[
https://issues.apache.org/jira/browse/HDFS-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755046#comment-16755046
]
Steve Loughran commented on HDFS-13934:
---------------------------------------
More thoughts based on HADOOP-15229 work
* as with the new open() command; use the builder API; support custom options
for apps which really know what they are doing/command line options for tools
* move to async upload calls; its up to the FS implementation to schedule the
work as it chooses.
* the final complete() command to include any FS checksum at the end. Because
if you are storing the checksums for incremental distcp, an atomic complete +
checksum is better than complete() + getchecksum(). Faster and free of race
conditions.
> Multipart uploaders to be created through API call to FileSystem/FileContext,
> not service loader
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-13934
> URL: https://issues.apache.org/jira/browse/HDFS-13934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, fs/s3, hdfs
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Priority: Major
>
> the Multipart Uploaders are created via service loaders. This is troublesome
> # HADOOP-12636, HADOOP-13323, HADOOP-13625 highlight how the load process
> forces the transient loading of dependencies. If a dependent class cannot be
> loaded (e.g aws-sdk is not on the classpath), that service won't load.
> Without error handling round the load process, this stops any uploader from
> loading. Even with that error handling, the performance hit of that load,
> especially with reshaded dependencies, hurts performance (HADOOP-13138).
> # it makes wrapping the the load with any filter impossible, stops transitive
> binding through viewFS, mocking, etc.
> # It complicates security in a kerberized world. If you have an FS instance
> of user A, then you should be able to create an MPU instance with that user's
> permissions. currently, if a service were to try to create one, you'd be
> looking at doAs() games around the service loading, and a more complex bind
> process.
> Proposed
> # remove the service loader mech entirely
> # add to FS & FC as createMultipartUploader(path) call, which will create one
> bound to the current FS, with its permissions, DTs, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
