[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977816#comment-15977816
]
Aaron Fabbri commented on HADOOP-14138:
---------------------------------------
[[email protected]]'s original proposal here seems like the right approach
for the short term goal (serious perf regression) to me, especially given the
points about the core-default config being required for things like FileContext
anyways.. Changing core-default.xml behavior is a big, incompatible change, so
I'm glad we're not tackling that up front.
I personally think the xml default approach works well in practice. Having all
the defaults in one spot makes things easy to find, as well as isolating
vendor-specific defaults in an easy to change place (versus having to keep
commit changes throughout the code). To me, having to change defaults is
pretty common (we frequently have to tweak core-default settings for a shipping
product), and being able to do that in a default config is very low-friction
compared to code changes.
For the problem of keeping code defaults and core-default.xml up to date, some
good ideas here. Another idea is this: Let humans keep them in sync, and
enforce with a pre-commit check script.
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
> Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
