[
https://issues.apache.org/jira/browse/HDFS-12583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190720#comment-16190720
]
Yiqun Lin commented on HDFS-12583:
----------------------------------
Thanks for the review and comment, [~vagarychen]!
bq. Could you please elaborate a bit on how you got the error in the
description?
The error in description is just copied from previous failure test in
HDFS-12307. Actually, we can just throw a IOException like following way to
reproduce this:
{code}
private XceiverClientSpi getClient(Pipeline pipeline)
throws IOException {
String containerName = pipeline.getContainerName();
try {
return clientCache.get(containerName,
new Callable<XceiverClientSpi>() {
@Override
public XceiverClientSpi call() throws Exception {
throw new IOException("Throw exception when getting
XceiverClient.");
}
});
} catch (Exception e) {
throw new IOException("Exception getting XceiverClient.", e);
}
}
{code}
Having tested in my local, the stack info can be printed as following:
{noformat}
2017-10-04 10:48:54,647 [Thread-206] ERROR handlers.KeyProcessTemplate
(KeyProcessTemplate.java:handleCall(100)) ozone
9aaa47d4-f6be-494c-b38f-f5c000997fadvolume/894199ed-b2c6-4ec8-be99-d768dbe58bc1bucket/test-key0
hdfs 5e5f7f56-601f-4f43-b95f-98bc140ee686 - IOException. ex : {}
java.io.IOException: Exception getting XceiverClient.
at
org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:158)
at
org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:127)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
at
org.apache.hadoop.ozone.web.storage.DistributedStorageHandler.newKeyWriter(DistributedStorageHandler.java:397)
at
org.apache.hadoop.ozone.web.handlers.KeyHandler$2.doProcess(KeyHandler.java:174)
at
org.apache.hadoop.ozone.web.handlers.KeyProcessTemplate.handleCall(KeyProcessTemplate.java:91)
at
org.apache.hadoop.ozone.web.handlers.KeyHandler.putKey(KeyHandler.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
org.apache.hadoop.ozone.web.netty.ObjectStoreJerseyContainer$RequestRunner.run(ObjectStoreJerseyContainer.java:232)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Throw
exception when getting XceiverClient.
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:132)
at
com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2381)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2351)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:149)
... 23 more
Caused by: java.io.IOException: Throw exception when getting XceiverClient.
at
org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:153)
at
org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:1)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
... 28 more
{noformat}
Attach the updated patch to address comments.
> Ozone: Fix swallow exceptions which makes hard to debug failures
> ----------------------------------------------------------------
>
> Key: HDFS-12583
> URL: https://issues.apache.org/jira/browse/HDFS-12583
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Affects Versions: HDFS-7240
> Reporter: Yiqun Lin
> Assignee: Yiqun Lin
> Attachments: HDFS-12583-HDFS-7240.001.patch,
> HDFS-12583-HDFS-7240.002.patch
>
>
> There are some places that swallow exceptions that makes client hard to debug
> the failure. For example, if we get xceiver client from xceiver client
> manager error, client only gets the error info like this:
> {noformat}
> org.apache.hadoop.ozone.web.exceptions.OzoneException: Exception getting
> XceiverClient.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:119)
> at
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createUsingDefault(StdValueInstantiator.java:243)
> {noformat}
> The error exception stack is missing. We should print the error log as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
