[
https://issues.apache.org/jira/browse/HADOOP-13139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135370#comment-16135370
]
Jason Lowe commented on HADOOP-13139:
-------------------------------------
Had a user that ran into this on one of our clusters that upgraded to 2.8.
They were running a pre-2.8 version of the S3AFileSystem code with their job
and it failed like this:
{noformat}
java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307)
at
java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1230)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:280)
at
com.yahoo.prism.UseLocalKeyS3AFileSystem.initializeFileSystem(UseLocalKeyS3AFileSystem.java:68)
at
com.yahoo.prism.UseLocalKeyS3AFileSystem.initialize(UseLocalKeyS3AFileSystem.java:113)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2670)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:95)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2704)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2686)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:374)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
[...]
{noformat}
The problem is that core-default in 2.8 removed fs.s3a.threads.core but changed
the existing fs.s3a.threads.max to 10. The old pre-2.8 S3AFileSystem code had
code defaults of 15 and 256, respectively. So when a 2.8 job client (in this
case an Oozie server) submits the job, picking up the 2.8 core-default settings
for fs.s3a.threads.max for job.xml but the job itself runs with the older
S3AFileSystem code the job fails because it tries to initialize a threadpool
with core threads=15 and max threads=10.
Not sure if this is considered simply an invalid setup, but I suspect this
won't be the first case of someone submitting a job with a 2.8 or later client
(e.g.: via an Oozie server upgraded independently of a user's job code) and
failing because the user hasn't upgraded to the 2.8 or later S3AFileSystem code
yet.
If we had added a deprecated core-default value for fs.s3a.threads.core then
the older code would have gotten consistent values for core and max threads.
As it is now, it gets half of the new default settings, and those aren't
compatible with the older, other half of the defaults. Thoughts on whether
this is worth doing in a followup JIRA?
> Branch-2: S3a to use thread pool that blocks clients
> ----------------------------------------------------
>
> Key: HADOOP-13139
> URL: https://issues.apache.org/jira/browse/HADOOP-13139
> Project: Hadoop Common
> Issue Type: Task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Pieter Reuse
> Assignee: Pieter Reuse
> Fix For: 2.8.0
>
> Attachments: HADOOP-13139-001.patch, HADOOP-13139-branch-2.001.patch,
> HADOOP-13139-branch-2.002.patch, HADOOP-13139-branch-2-003.patch,
> HADOOP-13139-branch-2-004.patch, HADOOP-13139-branch-2-005.patch,
> HADOOP-13139-branch-2-006.patch
>
>
> HADOOP-11684 is accepted into trunk, but was not applied to branch-2. I will
> attach a patch applicable to branch-2.
> It should be noted in CHANGES-2.8.0.txt that the config parameter
> 'fs.s3a.threads.core' has been been removed and the behavior of the
> ThreadPool for s3a has been changed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
