Steve Loughran updated HADOOP-14132:
    Parent: HADOOP-15220  (was: HADOOP-14831)

> Filesystem discovery to stop loading implementation classes
> -----------------------------------------------------------
>                 Key: HADOOP-14132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14132
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
>    Affects Versions: 2.7.3
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
> Integration testing of Hadoop with the HADOOP-14040 has shown up that the 
> move to a shaded AWS JAR is slowing all hadoop client code down.
> I believe this is due to how we use service discovery to identify FS 
> implementations: the implementation classes themselves are instantiated.
> This has known problems today with classloading, but clearly impacts 
> performance too, especially with complex transitive dependencies unique to 
> the loaded class.
> Proposed: have lightweight service declaration classes which implement an 
> interface declaring
> # schema
> # classname of FileSystem impl
> # classname of AbstractFS impl
> # homepage (for third party code, support, etc)
> These are what we register and scan in the FS to look for services.
> This will leave the question about what to do for existing filesystems? I 
> think we'll need to retain the old code for external ones, while moving the 
> hadoop modules to the new ones

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to