[ https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976768#comment-15976768 ]
Steve Loughran commented on HADOOP-14138: ----------------------------------------- those JIRAs are so old they are implicitly dead. right now things are, well, confused, as we have both code and XML defaults. The XML ones win. Benefit: easy place to see what is there, one single spot to track down the declarations without having access to the source price: * the defaults in the source files need to be kept in sync, when they aren't they diverge and people loading in the configs without the defaults get undefined outcomes * sometimes those defaults are actually broken (S3AOutputStream temp dir setup prior to HADOOP-13560), and nobody notices * if someone has mistyped an entry (HADOOP-12735) then its not valid and nobody knows why. Personally, I like the XML file for its one-stop shop. But I don't like the maintenance/sync problem Thinking about it, maybe the way to go is an IDL mech which generates the XML, the java constant names and the default value strings. That way: one single thing to work off. Either XML -> XSLT -> .java constants file, or see if we can't use protoc, avroc or even, dare I say it, CORBA and the {{idlc}} compiler, to go from a .idl declaration to java code. Though none of the protoc/avroc/idlj workflows would generate the core-default.xml file. Our own XML file format, adding in name of generated constant, maybe desired value type if not String, could be used via <xslt> to generate everything. It could be done, though it'd need XSL skills > Remove S3A ref from META-INF service discovery, rely on existing core-default > entry > ----------------------------------------------------------------------------------- > > Key: HADOOP-14138 > URL: https://issues.apache.org/jira/browse/HADOOP-14138 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.9.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Critical > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3 > > Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch > > > As discussed in HADOOP-14132, the shaded AWS library is killing performance > starting all hadoop operations, due to classloading on FS service discovery. > This is despite the fact that there is an entry for fs.s3a.impl in > core-default.xml, *we don't need service discovery here* > Proposed: > # cut the entry from > {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}} > # when HADOOP-14132 is in, move to that, including declaring an XML file > exclusively for s3a entries > I want this one in first as its a major performance regression, and one we > coula actually backport to 2.7.x, just to improve load time slightly there too -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org