Branching off this subject from the original thread.
On Wed, Apr 15, 2020 at 9:56 AM Andrew Purtell <andrew.purt...@gmail.com>
wrote:
Quick Start and Production are exclusive configurations.
Yes absolutely.
Quick Start, as you say, should have as few steps to up and running as
possible.
Production requires a real distributed filesystem for persistence and
that
means HDFS and that means, whatever the provisioning and deployment and
process management (Ambari or k8s or...) choices are going to be, they
will
not be a Quick Start.
We’ve always had this problem. The Quick Start simply can’t produce a
system capable of durability because prerequisites for durability are not
quick to set up.
Is this exclusively due to the implementation of `LocalFileSystem` or are
there other issues at play? I've seen there's also `RawLocalFileSystem` but
I haven't investigated their relationship, it's capabilities, or if we
might profit from its use for the Quick Start experience.
Specifically about /tmp... I agree that’s not a good default. Time and
again I’ve heard people complain that the tmp cleaner has removed their
test data. It shouldn’t be surprising but is and that is real feedback on
mismatch of user expectation to what we are providing in that
configuration. Addressing this aspect of the Quick Start experience would
be a simple change: make the default a new directory in $HOME, perhaps
“.hbase” .
I propose changing the default value of `hbase.tmp.dir` as shipped in the
default hbase-site.xml to be simply `tmp`, as I documented in my change on
HBASE-24106. That way it's not hidden somewhere and it's self-contained to
this unpacking of the source/binary distribution. I.e., there's no need to
worry about upgrading the data stored there when a user experiments with a
new version.
On Apr 15, 2020, at 9:40 AM, Nick Dimiduk <ndimi...@apache.org> wrote:
Hi folks,
I'd like to bring up the topic of the experience of new users as it
pertains to use of the `LocalFileSystem` and its associated (lack of)
data
durability guarantees. By default, an unconfigured HBase runs with its
root
directory on a `file:///` path. This patch is picked up as an instance
of
`LocalFileSystem`. Hadoop has long offered this class, but it has never
supported `hsync` or `hflush` stream characteristics. Thus, when HBase
runs
on this configuration, it is unable to ensure that WAL writes are
durable,
and thus will ACK a write without this assurance. This is the case,
even
when running in a fully durable WAL mode.
This impacts a new user, someone kicking the tires on HBase following
our
Getting Started docs. On Hadoop 2.8 and before, an unconfigured HBase
will
WARN and cary on. Hadoop 2.10+, HBase will refuse to start. The book
describes a process of disabling enforcement of stream capability
enforcement as a first step. This is a mandatory configuration for
running
HBase directly out of our binary distribution.
HBASE-24086 restores the behavior on Hadoop 2.10+ to that of running on
2.8: log a warning and cary on. The critique of this approach is that
it's
far too subtle, too quiet for a system operating in a state known to
not
provide data durability.
I have two assumptions/concerns around the state of things, which
prompted
my solution on HBASE-24086 and the associated doc update on
HBASE-24106.
1. No one should be running a production system on `LocalFileSystem`.
The initial implementation checked both for `LocalFileSystem` and
`hbase.cluster.distributed`. When running on the former and the latter
is
false, we assume the user is running a non-production deployment and
carry
on with the warning. When the latter is true, we assume the user
intended a
production deployment and the process terminates due to stream
capability
enforcement. Subsequent code review resulted in skipping the
`hbase.cluster.distributed` check and simply warning, as was done on
2.8
and earlier.
(As I understand it, we've long used the `hbase.cluster.distributed`
configuration to decide if the user intends this runtime to be a
production
deployment or not.)
Is this a faulty assumption? Is there a use-case we support where we
condone running production deployment on the non-durable
`LocalFileSystem`?
2. The Quick Start experience should require no configuration at all.
Our stack is difficult enough to run in a fully durable production
environment. We should make it a priority to ensure it's as easy as
possible to try out HBase. Forcing a user to make decisions about data
durability before they even launch the web ui is a terrible experience,
in
my opinion, and should be a non-starter for us as a project.
(In my opinion, the need to configure either `hbase.rootdir` or
`hbase.tmp.dir` away from `/tmp` is equally bad for a Getting Started
experience. It is a second, more subtle question of data durability
that
we
should avoid out of the box. But I'm happy to leave that for another
thread.)
Thank you for your time,
Nick
