[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679209#comment-15679209
]
Steve Loughran commented on HADOOP-13824:
-
patchs looks good, except I'd go
{code}
displayError(cmd, ex.toString());
{code}
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HADOOP-13824.001.patch
>
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679191#comment-15679191 ] Steve Loughran commented on HADOOP-13449: - I can talk morning Palo Alto time from 09:00 to 11:00. It's good to open the calls to all. I have a webex conf #, or we can use google+ hangouts (which I prefer, though they will block anyone from China trying to dial in) > S3Guard: Implement DynamoDBMetadataStore. > - > > Key: HADOOP-13449 > URL: https://issues.apache.org/jira/browse/HADOOP-13449 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Mingliang Liu > Attachments: HADOOP-13449-HADOOP-13345.000.patch, > HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch, > HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch, > HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch, > HADOOP-13449-HADOOP-13345.007.patch > > > Provide an implementation of the metadata store backed by DynamoDB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679171#comment-15679171
]
Steve Loughran commented on HADOOP-13811:
-
Looking at the stack trace of mine, it's a thread interrupt -> stream abort ->
parser failure.
Assuming it is a network problem surfacing as errors in the XML parser (i.e.
the input stream being closed prematurely), this *may* go away on a retry. But
if a thread has been interrupted, it should really stay interrupted, rather
than trigger a retry. That is: error handler to walk the exception tree and
look for an {{AbortedException}} at the bottom, and upconvert to an
InterruptedIOException().
I think that's separate from Luke's cause though.
Luke, are you in a position to build hadoop & Spark yourself? Or see if you can
replicate this on the HDP sandbox, which contains much of the S3a phase II
work, including the ignoring of exceptions in fake directory cleanup
http://hortonworks.com/products/sandbox/
Otherwise, I can build up a snapshot of spark trunk with Hadoop 2.8-snapshot;
I'll share this with you and we can see if we can identify what's happening, or
at least ignore it better when it's non-critical
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679156#comment-15679156
]
Steve Loughran commented on HADOOP-13811:
-
I've seen this too. I was doing other things on the network at the time; I
suspect this may jus be a network failure manifesting as an XML parser error.
{code}
2016-11-18 19:38:28,167 [JobGenerator] WARN dstream.FileInputDStream
(Logging.scala:logWarning(87)) - Error finding new files
org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on
spark-cloud/S3AStreamingSuite/streaming/streaming/:
com.amazonaws.AmazonClientException: Failed to sanitize XML document destined
for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler:
Failed to sanitize XML document destined for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:116)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1584)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1339)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1315)
at org.apache.hadoop.fs.Globber.listStatus(Globber.java:76)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:234)
at org.apache.hadoop.fs.Globber.glob(Globber.java:148)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1770)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:1975)
at
org.apache.spark.streaming.dstream.FileInputDStream.findNewFiles(FileInputDStream.scala:205)
at
org.apache.spark.streaming.dstream.FileInputDStream.compute(FileInputDStream.scala:149)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:336)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:334)
at scala.Option.orElse(Option.scala:289)
at
org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:331)
at
org.apache.spark.streaming.dstream.MappedDStream.compute(MappedDStream.scala:36)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:336)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:334)
at scala.Option.orElse(Option.scala:289)
at
org.apache.spark.streaming.dstream.DStream.getOrCompute(DStream.scala:331)
at
org.apache.spark.streaming.dstream.FilteredDStream.compute(FilteredDStream.scala:36)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
at
org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:336)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:334)
