[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768468#comment-15768468 ] Sangjin Lee commented on HBASE-17018: - Your suggestion is interesting [~enis]. Thanks for the idea. In addition to what Joep mentioned above, I do worry about the capacity requirement a dual-writing system would have. It would essentially double the hdfs requirement for the storage, and at large scale it would add up to a meaningful amount. Also, how would a reader work in the case where the data made it into hdfs but not into hbase (e.g. hbase cluster was down for a while for an upgrade)? Would the reader still query hbase only and return no data if hbase is missing the data? If we want to address that situation, we're putting back the unspooling (migrating missing data from the backup location to hbase). I'm just trying to round out the idea... Thanks! > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, HBASE-17018.master.003.patch, > HBASE-17018.master.004.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. > https://reviews.apache.org/r/54882/ > For design of SpoolingBufferedMutatorImpl see > https://docs.google.com/document/d/1GTSk1Hd887gGJduUr8ZJ2m-VKrIXDUv9K3dr4u2YGls/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765011#comment-15765011
]
Sangjin Lee commented on HBASE-17018:
-
{quote}
Indeed if I were to wrap the put calls, it would alleviate the out of order
submissions between puts and flushes. It would mean I have to change the
flushCount in the SpoolingBufferedMutatorSubmission from a final field set in
the constructor to a volatile (or otherwise synchnonized) setter. Were you
thinking I'd add a new interface around the BlockingQueue, or simply have a
local method and hope the current code and later maintainers keep the
discipline to always call a local method to enqueue? Alternatively the entire
construction of the submission could be pushed into a separate class that keeps
the flushCount and does object creation and queue submission under a single
lock (perhaps simply a synchronized method). I can see if the coordinator could
do this work so we can avoid yet another class.
{quote}
I suppose there are a couple of options each of which has pros and cons. First
was extending {{LinkedBlockingQueue}} within this class to add the logic of
handling the flush count strongly. Something like:
{code}
final BlockingQueue inbound =
new LinkedBlockingQueue<>() {
public synchronized void put(SpoolingBufferedMutatorSubmission s)
throws InterruptedException {
// handle and set the flush count on the submission based on the type
super.put(s);
}
};
{code}
That way, any new code within this class that would put items in the inbound
queue would benefit from the same treatment. One drawback is that {{put()}} is
not the only way to add items to the queue (there is {{add()}}), and to be
complete we'd need to either override {{add()}} too or ensure no one calls it
(by overriding it to throw an exception or something). This overridden class
can be totally private to the SBMI.
I also thought of simply doing this as a private method within SBMI so we don't
need to play with extending the LinkedBlockingQueue. But the downside of that
is we need to ensure all new code that puts items in that inbound queue goes
through that method.
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018.master.004.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
> https://reviews.apache.org/r/54882/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762788#comment-15762788
]
Sangjin Lee commented on HBASE-17018:
-
I am supportive of the design (obviously) as I consulted with Joep and provided
feedback. I like the overall approach and the proposed WIP patch.
I went over the patch at a high level and I do have several comments and
questions.
(1)
What's not entirely clear to me from the patch is exactly how the state
transition will occur, especially how it will transition out of BAD into
TRANSITIONING or GOOD. Maybe that is still TODO? Did I read that right? We
talked offline about attempting flushing periodically if it is in a bad state
to probe the state of the HBase cluster. One idea we could use is to use the
ExceptionListener/RetriesExhaustedWithDetailsException/mayHaveClusterIssues.
Also, we want to think about under what condition we transition from GOOD to
BAD. I still think the exception listener has a lot of value as it can tell us
(more) about the cluster status. We should see if we can utilize the exception
listener in determining whether to transition. We should also tune the timeout
value to the same timeout value we're willing to wait for these operations so
that in most cases we would detect the timeout in the form of an exception
coming from the underlying {{BufferedMutatorImpl}} with the proper exception
listener.
(2)
In {{SpoolingBufferedMutatorProcessor.call()}}, I'm not quite sure how useful
the two-tiered timeout setup is. It appears that the overall timeout is
something like 100 seconds, but we divide it with finer-grained 1-second timed
waits and keep looping until we exhaust the overall timeout. Is it truly
necessary? Do we gain value by having the two-tiered mechanism? Since this is
all done in the same single thread, it's not like the thread will do anything
other than looping right back onto {{future.get()}} on the same submission.
IMO, this seems to be adding more complexity than is needed with not much
payoff. Why not simply have a single timeout with the desired overall timeout?
That would make this much simpler without losing any flexibility.
(3)
Somewhat related to the above, I'd like to see the submission state used more
explicitly in that method. Currently a bad state and a subsequent behavior
difference are implied by {{timeout == 0}}. Instead, it might be great if we
explicitly use the state from the coordinator to do different things (e.g.
{{coordinator.getState()}} instead of {{coordinator.getTimeout()}}). We can add
things like the timeout value into the state enum so the state becomes more
useful.
(4)
Regarding the out-of-order mutate calls with respect to flushes, one idea might
be to extend the {{BlockingQueue}} (specifically override {{put()}}) so that
each {{put()}} call can handle the flush count synchronously and internally as
part of the call. Then we may be able to eliminate the need for handling
out-of-order mutate calls and thus simplify further.
> Spooling BufferedMutator
>
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
>Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
[ https://issues.apache.org/jira/browse/HBASE-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281974#comment-15281974 ] Sangjin Lee commented on HBASE-13706: - I suspect that could work. cc [~gtCarrera9] > CoprocessorClassLoader should not exempt Hive classes > - > > Key: HBASE-13706 > URL: https://issues.apache.org/jira/browse/HBASE-13706 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0 > > Attachments: HBASE-13706-0.98.patch, HBASE-13706-branch-1.patch, > HBASE-13706-master-v2.patch, HBASE-13706-master-v2.patch, HBASE-13706.patch > > > CoprocessorClassLoader is used to load classes from the coprocessor jar. > Certain classes are exempt from being loaded by this ClassLoader, which means > they will be ignored in the coprocessor jar, but loaded from parent classpath > instead. > One problem is that we categorically exempt "org.apache.hadoop". > But it happens that Hive packages start with "org.apache.hadoop". > There is no reason to exclude hive classes from theCoprocessorClassLoader. > HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
[ https://issues.apache.org/jira/browse/HBASE-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281050#comment-15281050 ] Sangjin Lee commented on HBASE-13706: - Yes, that's what I heard too. I guess the question is how soon Phoenix 4.8 will be released as that may become a blocker for us to merge into the hadoop trunk... > CoprocessorClassLoader should not exempt Hive classes > - > > Key: HBASE-13706 > URL: https://issues.apache.org/jira/browse/HBASE-13706 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0 > > Attachments: HBASE-13706-0.98.patch, HBASE-13706-branch-1.patch, > HBASE-13706-master-v2.patch, HBASE-13706-master-v2.patch, HBASE-13706.patch > > > CoprocessorClassLoader is used to load classes from the coprocessor jar. > Certain classes are exempt from being loaded by this ClassLoader, which means > they will be ignored in the coprocessor jar, but loaded from parent classpath > instead. > One problem is that we categorically exempt "org.apache.hadoop". > But it happens that Hive packages start with "org.apache.hadoop". > There is no reason to exclude hive classes from theCoprocessorClassLoader. > HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
[ https://issues.apache.org/jira/browse/HBASE-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281033#comment-15281033 ] Sangjin Lee commented on HBASE-13706: - I thought Nick was not comfortable with backporting HBASE-15686, but I didn't read that he was uncomfortable with backporting this JIRA. The YARN timeline service feature also has a dependency on Phoenix, and it appears Phoenix is not yet ready to pick up 1.2.x? [~enis]? > CoprocessorClassLoader should not exempt Hive classes > - > > Key: HBASE-13706 > URL: https://issues.apache.org/jira/browse/HBASE-13706 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0 > > Attachments: HBASE-13706-0.98.patch, HBASE-13706-branch-1.patch, > HBASE-13706-master-v2.patch, HBASE-13706-master-v2.patch, HBASE-13706.patch > > > CoprocessorClassLoader is used to load classes from the coprocessor jar. > Certain classes are exempt from being loaded by this ClassLoader, which means > they will be ignored in the coprocessor jar, but loaded from parent classpath > instead. > One problem is that we categorically exempt "org.apache.hadoop". > But it happens that Hive packages start with "org.apache.hadoop". > There is no reason to exclude hive classes from theCoprocessorClassLoader. > HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13706) CoprocessorClassLoader should not exempt Hive classes
[ https://issues.apache.org/jira/browse/HBASE-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280870#comment-15280870 ] Sangjin Lee commented on HBASE-13706: - I asked in another JIRA, but is there appetite in porting this to the 1.1.x line? Thanks! > CoprocessorClassLoader should not exempt Hive classes > - > > Key: HBASE-13706 > URL: https://issues.apache.org/jira/browse/HBASE-13706 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.1.0, 0.98.12 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0 > > Attachments: HBASE-13706-0.98.patch, HBASE-13706-branch-1.patch, > HBASE-13706-master-v2.patch, HBASE-13706-master-v2.patch, HBASE-13706.patch > > > CoprocessorClassLoader is used to load classes from the coprocessor jar. > Certain classes are exempt from being loaded by this ClassLoader, which means > they will be ignored in the coprocessor jar, but loaded from parent classpath > instead. > One problem is that we categorically exempt "org.apache.hadoop". > But it happens that Hive packages start with "org.apache.hadoop". > There is no reason to exclude hive classes from theCoprocessorClassLoader. > HBase does not even include Hive jars. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15812) HttpServer fails to come up against hadoop trunk
[
https://issues.apache.org/jira/browse/HBASE-15812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279176#comment-15279176
]
Sangjin Lee commented on HBASE-15812:
-
LGTM. It would be great if this could be ported to 1.x lines. Thanks [~tedyu]!
> HttpServer fails to come up against hadoop trunk
>
>
> Key: HBASE-15812
> URL: https://issues.apache.org/jira/browse/HBASE-15812
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
> Attachments: 15812.v1.txt
>
>
> If you run HBase HttpServer against the hadoop trunk, it fails.
> {noformat}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/metrics/MetricsServlet
> at
> org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at
> org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:104)
> at
> org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.(InfoServer.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 26 more
> {noformat}
> The hadoop trunk removed {{MetricsServlet}} (HADOOP-12504).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15812) HttpServer fails to come up against hadoop trunk
Sangjin Lee created HBASE-15812:
---
Summary: HttpServer fails to come up against hadoop trunk
Key: HBASE-15812
URL: https://issues.apache.org/jira/browse/HBASE-15812
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Sangjin Lee
If you run HBase HttpServer against the hadoop trunk, it fails.
{noformat}
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/metrics/MetricsServlet
at
org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
at
org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:500)
at org.apache.hadoop.hbase.http.HttpServer.(HttpServer.java:104)
at
org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
at org.apache.hadoop.hbase.http.InfoServer.(InfoServer.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:550)
at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:333)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
... 26 more
{noformat}
The hadoop trunk removed {{MetricsServlet}} (HADOOP-12504).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258984#comment-15258984
]
Sangjin Lee commented on HBASE-15686:
-
The meaning of "excluded.classes" is these classes will be excluded from
exemption so that this is bit of a double negative. These classes will actually
be loaded by the coprocessor classloader. Could that be a source of confusion?
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
>Assignee: Ted Yu
> Attachments: 15686.v2.txt, 15686.v3.txt, 15686.wip
>
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258350#comment-15258350
]
Sangjin Lee commented on HBASE-15686:
-
My apologies for mixing discussions on HBASE-13706 and this JIRA itself.
I'd like to re-word this JIRA to discuss supporting an override mechanism for
the exempt classes, and move the discussion for backporting HBASE-13706 to that
JIRA. I believe Nick's and Enis's comments apply to HBASE-13706. Implementing
the override mechanism can be done in a later version.
Does that sound reasonable?
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
>Assignee: Ted Yu
> Attachments: 15686.v2.txt, 15686.wip
>
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253381#comment-15253381
]
Sangjin Lee commented on HBASE-15686:
-
Would it be a possibility to backport HBASE-13706 to the 1.0.x releases? I'm
not completely familiar with the currently supported release lines, but it
would be nice if we can get this in a 1.0.x line. Thanks!
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
> Attachments: 15686.wip
>
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253167#comment-15253167
]
Sangjin Lee commented on HBASE-15686:
-
[~jerryhe], yes, I see that the 2.0 list is pretty small. That said, if my
experience is any guide, getting this "system/exempt" classes list right is
trickier than it appears. This is the current default list in hadoop:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/resources/org.apache.hadoop.application-classloader.properties
Even getting the JDK packages right is not so easy as they are not explicitly
defined in terms of packages. But the penalty for not getting it right is
pretty severe, as classes just won't get loaded. So over time, I found myself
repeatedly modifying and improving the list. I suspect a similar thing may
happen on your end too.
This JIRA is another example: HADOOP-11831. {{org.w3c.dom}} is a JDK package,
and as such should be in the system classes. However, it turns out a
third-party library (xerces) has a class called
{{org.w3c.dom.ElementTraversal}} that the JDK does not have. The moment that
class is used under these classloaders, it won't work. So users' customizing
this list proves to be a handy way to address those interesting edge cases.
Most custom classloader solutions (OSGi, servlet engines, ...) have a way to
modify/customize the system classes/packages list for that reason. Just my 2
cents.
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
> Attachments: 15686.wip
>
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253102#comment-15253102
]
Sangjin Lee commented on HBASE-15686:
-
Either that (if that can be handled cleanly) or even an overloaded method...
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253070#comment-15253070
]
Sangjin Lee commented on HBASE-15686:
-
Thanks for the pointer [~jerryhe]. It looks like the earliest release that has
the fix for HBASE-13706 is 1.2.0? Is there a possibility of getting this into
the 1.0.x line?
Also, although HBASE-13706 should address our specific issue, I think it would
still be a good feature enhancement to allow configuring the exempt classes.
Otherwise, any modification to the exempt list would require a HBase release.
The changes can be defined through configuration or even as an additional
argument to {{addCoprocessor()}} or something to that effect...
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
[
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252853#comment-15252853
]
Sangjin Lee commented on HBASE-15686:
-
In case of hadoop's {{ApplicationClassLoader}}, the main way to define/override
it is via configuration (although there are other ways).
It allows "-" as a way to exclude certain packages or classes from the exempt
list. {{ApplicationClassLoader}}'s constructor takes the exempt classes
("system classes") as an argument. Thus, I suppose any way you can pass in the
modified exempt classes would work, including programmatically.
> unable to dynamically load a table coprocessor if it belongs in
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> cannot be loaded
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
> hadoop classes as exempt from loading from the coprocessor jar. Since our
> coprocessor sits in the coprocessor jar, and yet the loading of this class is
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt
> classes so that they can be loaded via table coprocessor classloader. See
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop
Sangjin Lee created HBASE-15686:
---
Summary: unable to dynamically load a table coprocessor if it
belongs in org.apache.hadoop
Key: HBASE-15686
URL: https://issues.apache.org/jira/browse/HBASE-15686
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 1.0.1
Reporter: Sangjin Lee
As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table
coprocessor (YARN-4062). However, we're finding that the coprocessor cannot be
loaded dynamically. A relevant snippet for the exception:
{noformat}
java.io.IOException: Class
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
cannot be loaded
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
at
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Class
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
cannot be loaded
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
at
org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
at
org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
... 8 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
... 10 more
{noformat}
We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all
hadoop classes as exempt from loading from the coprocessor jar. Since our
coprocessor sits in the coprocessor jar, and yet the loading of this class is
delegated to the parent which does not have this jar, the classloading fails.
What would be nice is the ability to exclude certain classes from the exempt
classes so that they can be loaded via table coprocessor classloader. See
hadoop's {{ApplicationClassLoader}} for a similar feature.
Is there any other way to load this coprocessor at the table scope?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
[
https://issues.apache.org/jira/browse/HBASE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15195735#comment-15195735
]
Sangjin Lee commented on HBASE-15436:
-
Thanks [~anoop.hbase]. That sounds plausible.
This does represent a pretty critical issue then, no? If a region server is in
a state where a socket timeout is thrown in this manner, flush will be stuck
for a LONG time. In a high throughput situation, this would imply a pretty
severe consequence and induce huge instability on the client.
For the background, we are working on using HBase for the timeline service v.2
(see YARN-4736) and node managers will be HBase clients. If a region server or
region servers are in an unhealthy state, this issue would cause a pretty big
cascading effect on the client cluster, correct?
Does this behavior exist in all other later releases?
> BufferedMutatorImpl.flush() appears to get stuck
>
>
> Key: HBASE-15436
> URL: https://issues.apache.org/jira/browse/HBASE-15436
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.2
>Reporter: Sangjin Lee
> Attachments: hbaseException.log, threaddump.log
>
>
> We noticed an instance where the thread that was executing a flush
> ({{BufferedMutatorImpl.flush()}}) got stuck when the (local one-node) cluster
> shut down and was unable to get out of that stuck state.
> The setup is a single node HBase cluster, and apparently the cluster went
> away when the client was executing flush. The flush eventually logged a
> failure after 30+ minutes of retrying. That is understandable.
> What is unexpected is that thread is stuck in this state (i.e. in the
> {{flush()}} call). I would have expected the {{flush()}} call to return after
> the complete failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
[
https://issues.apache.org/jira/browse/HBASE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191553#comment-15191553
]
Sangjin Lee commented on HBASE-15436:
-
Thanks [~anoop.hbase] for your comments. To answer your question,
bq. So after seeing this log how long u wait?
I believe the user tried to shut it down about 30 minutes after this failure:
{noformat}
Fri Feb 26 00:39:03 IST 2016, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=68065: row
'timelineservice.entity,naga!yarn_cluster!flow_1456425026132_1!���!M�����!YARN_CONTAINER!container_1456425026132_0001_01_01,99'
on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
hostname=localhost,16201,1456365764939, seqNum=0
...
2016-02-26 01:09:19,799 ERROR
org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15:
SIGTERM
{noformat}
Also, I'm not too sure it is the case that flush is still going through the
mutations. This is the stack trace of the thread that was in the {{flush()}}
call (taken *after* this exception was seen):
{noformat}
"pool-14-thread-1" prio=10 tid=0x7f4215268000 nid=0x46e6 waiting on
condition [0x7f41fe75d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xeeb5a010> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at
org.apache.hadoop.hbase.util.BoundedCompletionService.take(BoundedCompletionService.java:75)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:190)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1200)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1109)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
- locked <0xc246f268> (a
org.apache.hadoop.hbase.client.BufferedMutatorImpl)
at
org.apache.hadoop.yarn.server.timelineservice.storage.common.BufferedMutatorDelegator.flush(BufferedMutatorDelegator.java:66)
at
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.flush(HBaseTimelineWriterImpl.java:457)
at
org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager$WriterFlushTask.run(TimelineCollectorManager.java:230)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The stack trace strongly indicates that it is waiting for more tasks to be
completed and *is idle*. I wasn't the one who observed this, and don't have any
more thread dumps around that time.
> BufferedMutatorImpl.flush() appears to get stuck
>
>
> Key: HBASE-15436
> URL: https://issues.apache.org/jira/browse/HBASE-15436
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.2
>Reporter: Sangjin Lee
> Attachments: hbaseException.log, threaddump.log
>
>
> We noticed an instance where the thread that was executing a flush
> ({{BufferedMutatorImpl.flush()}}) got stuck
[jira] [Commented] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
[
https://issues.apache.org/jira/browse/HBASE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188261#comment-15188261
]
Sangjin Lee commented on HBASE-15436:
-
See YARN-4736 for more details.
> BufferedMutatorImpl.flush() appears to get stuck
>
>
> Key: HBASE-15436
> URL: https://issues.apache.org/jira/browse/HBASE-15436
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.2
>Reporter: Sangjin Lee
> Attachments: hbaseException.log, threaddump.log
>
>
> We noticed an instance where the thread that was executing a flush
> ({{BufferedMutatorImpl.flush()}}) got stuck when the (local one-node) cluster
> shut down and was unable to get out of that stuck state.
> The setup is a single node HBase cluster, and apparently the cluster went
> away when the client was executing flush. The flush eventually logged a
> failure after 30+ minutes of retrying. That is understandable.
> What is unexpected is that thread is stuck in this state (i.e. in the
> {{flush()}} call). I would have expected the {{flush()}} call to return after
> the complete failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
[
https://issues.apache.org/jira/browse/HBASE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HBASE-15436:
Attachment: threaddump.log
hbaseException.log
The hbaseException.log file shows the exception and the failure during
{{flush()}}. The threaddump.log file shows the full thread stack trace dump
after the shutdown mechanism was unable to shut down the thread that was stuck
in the {{flush()}} call.
> BufferedMutatorImpl.flush() appears to get stuck
>
>
> Key: HBASE-15436
> URL: https://issues.apache.org/jira/browse/HBASE-15436
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.2
>Reporter: Sangjin Lee
> Attachments: hbaseException.log, threaddump.log
>
>
> We noticed an instance where the thread that was executing a flush
> ({{BufferedMutatorImpl.flush()}}) got stuck when the (local one-node) cluster
> shut down and was unable to get out of that stuck state.
> The setup is a single node HBase cluster, and apparently the cluster went
> away when the client was executing flush. The flush eventually logged a
> failure after 30+ minutes of retrying. That is understandable.
> What is unexpected is that thread is stuck in this state (i.e. in the
> {{flush()}} call). I would have expected the {{flush()}} call to return after
> the complete failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
[
https://issues.apache.org/jira/browse/HBASE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HBASE-15436:
Description:
We noticed an instance where the thread that was executing a flush
({{BufferedMutatorImpl.flush()}}) got stuck when the (local one-node) cluster
shut down and was unable to get out of that stuck state.
The setup is a single node HBase cluster, and apparently the cluster went away
when the client was executing flush. The flush eventually logged a failure
after 30+ minutes of retrying. That is understandable.
What is unexpected is that thread is stuck in this state (i.e. in the
{{flush()}} call). I would have expected the {{flush()}} call to return after
the complete failure.
was:
We noticed an instance where the thread that was executing a flush
({{BufferedMutatorImpl.flush()}} got stuck when the (local one-node) cluster
shut down and was unable to get out of that stuck state.
The setup is a single node HBase cluster, and apparently the cluster went away
when the client was executing flush. The flush eventually logged a failure
after 30+ minutes of retrying. That is understandable.
What is unexpected is that thread is stuck in this state (i.e. in the
{{flush()}} call). I would have expected the {{flush()}} call to return after
the complete failure.
> BufferedMutatorImpl.flush() appears to get stuck
>
>
> Key: HBASE-15436
> URL: https://issues.apache.org/jira/browse/HBASE-15436
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.2
>Reporter: Sangjin Lee
>
> We noticed an instance where the thread that was executing a flush
> ({{BufferedMutatorImpl.flush()}}) got stuck when the (local one-node) cluster
> shut down and was unable to get out of that stuck state.
> The setup is a single node HBase cluster, and apparently the cluster went
> away when the client was executing flush. The flush eventually logged a
> failure after 30+ minutes of retrying. That is understandable.
> What is unexpected is that thread is stuck in this state (i.e. in the
> {{flush()}} call). I would have expected the {{flush()}} call to return after
> the complete failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15436) BufferedMutatorImpl.flush() appears to get stuck
Sangjin Lee created HBASE-15436:
---
Summary: BufferedMutatorImpl.flush() appears to get stuck
Key: HBASE-15436
URL: https://issues.apache.org/jira/browse/HBASE-15436
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.0.2
Reporter: Sangjin Lee
We noticed an instance where the thread that was executing a flush
({{BufferedMutatorImpl.flush()}} got stuck when the (local one-node) cluster
shut down and was unable to get out of that stuck state.
The setup is a single node HBase cluster, and apparently the cluster went away
when the client was executing flush. The flush eventually logged a failure
after 30+ minutes of retrying. That is understandable.
What is unexpected is that thread is stuck in this state (i.e. in the
{{flush()}} call). I would have expected the {{flush()}} call to return after
the complete failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
