[
https://issues.apache.org/jira/browse/HADOOP-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810267#comment-16810267
]
Steve Loughran commented on HADOOP-16228:
-----------------------------------------
This looks like a bug in the AWS SDK, something in its XML parsing of the LIST
response
* Is this a genuine AWS S3 endpoint, or someone else's s3-compatible one?
* is there something "odd" about the files in your directory tree (i.e. unusual
characters in them)
* if you install the aws sdk, does an {{aws s3 ls}} command work?
* Does this still happen on the Hadoop 3.2 release?
* Is it only one specific bucket/path?
Given it's an AWS-side problem, we can't fix it, but since it's getting time to
update the AWS SDK on trunk, maybe you could help us see if this problem goes
away from you then.
otherwise, without that update, I'm afraid you'll have to turn v2 listing off,
at least, unless we can find some other root cause.
One trick would be actually grab the HTTP level logs, see:
https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html#Enabling_low-level_logging
I don't want those logs as they'll inevitably contain sensitive filenames or
something, but if you can see anything obvious in the response (like you don't
get XML back), that would help track this down
> Throwing OOM exception for ListStatus (v2) when using S3A
> ----------------------------------------------------------
>
> Key: HADOOP-16228
> URL: https://issues.apache.org/jira/browse/HADOOP-16228
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Ranith Sardar
> Priority: Major
>
> {code:java}
> @InterfaceStability.Unstable
> public static final String LIST_VERSION = "fs.s3a.list.version";
> @InterfaceStability.Unstable
> public static final int DEFAULT_LIST_VERSION = 2;
> {code}
> If the files in the bucket are more than 1k, it will throw OOM error for V2
> version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
