[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975477#comment-15975477
]
Siddharth Seth commented on HADOOP-14138:
-----------------------------------------
Not reading core-default was a bug, and is fixed. core-site.xml is independent
of what is in core-default.xml, if the code defaults were used.
See: HDFS-4820, HADOOP-7956. While searching for these bugs, there's several
others which relate to removing deprecated entries, fixing default values etc
from *-default files (This was within the HDFS project itself).
My concern is that an explicit requirement on the *-default files is getting
introduced, while the files should be removed IMO.
I get the problem with the serviceloaders, and the additional time introduced.
Thoughts there was an alternate plan to reduce this cost, while retaining
service loaders.
Could you please elaborate on "How to pick up credentials" - and why that needs
to be part of core-default? (And not a conf.get(PARAMETER, DEFAULT)).
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
> Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
