[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981108#comment-15981108
]
Steve Loughran commented on HADOOP-14138:
-----------------------------------------
Actually, doing it in code would work with some @Annotations: have 1+ class
which defines the constants, have something else which scans for annotated
classes, enumerates all constants declared their and generates the XML and MD
from it
{code}
@DefaultValues
public class CoreDefaults {
@Attribute
public static final Attribute FS_DEFAULT_NAME = new
Attribute("fs.default.name", "file://", "default filesystem", "URI"}
}
{code}
You could then programatically get at {{FS_DEFAULT_NAME.key}},
{{FS_DEFAULT_NAME.value}}, {{FS_DEFAULT_NAME.type}},
{{FS_DEFAULT_NAME.description}}, use that for generating things, or just
hooking up the IDEs.
And you could transit from existing declarations by having the existing code
reference the specific fields.
This would work for core, hdfs. MR. Where it would be somewhat complicated is
in things like S3A., where the constants are currently set in the hadoop-aws
module. All the S3A constants would have to be initially defined in
hadoop-common, so that they would go into core-default.
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
> Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
