[
https://issues.apache.org/jira/browse/HBASE-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038858#comment-13038858
]
Joey Echeverria commented on HBASE-1316:
----------------------------------------
bq. Do you think this patch will be generally useful Joey? If so, maybe once
its up working in hbase, it can be contrib'd back to zk?
Which part of the patch? I think it might be useful to get zk to publish native
artifacts, but I'm not sure if a full Java API that uses the native libraries
makes sense in general. It might make sense as a kind of general purpose
watchdog facility, probably in zk contrib.
bq. Tell me more about this? Are you thinking we need to build the native libs
in-line with a build each time?
Right now that's how my patch works. We could also build the native libraries
as a separate maven module that HBase could depend on, that way the assembly
could include all versions that we've built.
bq. Do you think this feature can be optionally enabled? If we fail to load the
required native lib, do we default to old-school session handling? Or, its on
always but we only use new-style if we find the native libs?
It's sort-of optionally available as is. I put my changes in the pom which
builds the native library into a profile which is activated base don the OS and
CPU (currently only activated for x86-64 Linux). We could turn off the
automatic activation entirely and require that you turn on the profile using
mvn -Pnative.
At runtime, it's completely optional. If you can't load the native library, it
falls back to using pure Java methods. It means that you wouldn't get the fast
path recovery that the native code is supposed to enable, but it prints a
warning to that effect in the logs.
bq. How does this timeout relate to the zk session timeout?
This configures the session timeout for the native zk session. Since you'll
want to tune the two separately, I put it in as a separate config. From the
previous project that I worked on, we found that 5-10 seconds was reasonable.
You would notice failures quickly but you're unlikely to get triggered by
temporary network glitches.
One thing I'm undecided about is how the region server would want to deal with
getting a session expiration notice for the native session. In my last project,
we had the native code call exit() to fail as fast as possible. We didn't want
our master process assigning a shard to a new server while the old one still
thought they had control of it. In the current patch, I'm sending an expired
session event to the Java zk handle. It would probably be better to expire the
Java session entirely. This can be done pretty easily by just connecting to zk
with the same session id and password and then closing the session. This way
anyone watching ephemeral nodes created by the Java session would get equally
fast failure notifications.
I'm busy this week, so I probably won't get back to this until next weekend.
> ZooKeeper: use native threads to avoid GC stalls (JNI integration)
> ------------------------------------------------------------------
>
> Key: HBASE-1316
> URL: https://issues.apache.org/jira/browse/HBASE-1316
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: Berk D. Demir
> Attachments: HBASE-1316-1.patch, zk_wrapper.tar.gz,
> zookeeper-native-Linux-amd64-64.tgz, zookeeper-native-headers.tgz
>
>
> From Joey Echeverria up on hbase-users@:
> We've used zookeeper in a write-heavy project we've been working on and
> experienced issues similar to what you described. After several days of
> debugging, we discovered that our issue was garbage collection. There was no
> way to guarantee we wouldn't have long pauses especially since our
> environment was the worst case for garbage collection, millions of tiny,
> short lived objects. I suspect HBase sees similar work loads frequently, if
> it's not constantly. With anything shorter than a 30 second session time out,
> we got session expiration events extremely frequently. We needed to use 60
> seconds for any real confidence that an ephemeral node disappearing meant
> something was unavailable.
> We really wanted quick recovery so we ended up writing a light-weight wrapper
> around the C API and used swig to auto-generate a JNI interface. It's not
> perfect, but since we switched to this method we've never seen a session
> expiration event and ephemeral nodes only disappear when there are network
> issues or a machine/process goes down.
> I don't know if it's worth doing the same kind of thing for HBase as it adds
> some "unnecessary" native code, but it's a solution that I found works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
