[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972796#comment-15972796
]
Steve Loughran commented on HADOOP-14138:
-----------------------------------------
The problem with the current mechanism is that it forces a transitive classload
of all dependencies, which is O(filesystem)*O(dependencies); doesn't handle
failure well.
What I'm trying to understand here is "why do you want to ignore
core-default.xml, given that sacrifices the ability to allow installations to
override it at the core-site layer? It seems you've made that policy decision,
and now suffering the consequences. And while the service discovery of S3A was
the first place to break, you are still left with the problem of "how to pick
up credentials".
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
> Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
