steveloughran opened a new pull request #1679: HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. URL: https://github.com/apache/hadoop/pull/1679 This is my proposal for an API for multipart uploads through the FS, rather than via service loading. It builds on the work done in createFile() and openFile() for builders and futures. * a builder API for configuring the uploader, including specifying the permissions of uploaded files. * MultipartUploader is now interface; org.apache.hadoop.fs.impl.AbstractMultipartUploader is the base class for implementations to extend. * all the operations in the interface now return FutureCompletable<>; implementations can perform the operations synchronously or asynchronously. * there is a new common path capability for the hasPathCapabilities() probe. FileSystemMultipartUploader and its builder moved to fs.impl; I needed a class org.apache.hadoop.fs.InternalOperations to get at rename/3 from the new package. It really is time to make the good rename method public. S3AMultipartUploader moved to fs.s3a.impl, alongside its builder. The uploader does not get a reference to the base FS; it gets a StoreContext and WriteOperationHelper and has to work with them. All operations and now async and executed in the executor offered by the StoreContext. Tests are all fixed up to cope with the changed API. The builders can be created from the file system; we can also extend file context and the filter classes... Though I would not pass the operation through checksum FS as the uploads will not be checksummed. One thing I am unsure about is viewFS: even though we can do with the remapping when the builder is created, the actual uploads need the full path in the store. That is unless, this and the WiP copy operation both take a path remapper in the builder which is then used to map from View FS paths to ones in their store. Suggestions here are welcome. No changes into the specification; yet. Let's get feedback first. Change-Id: Ib526fe6db8e9282c634181be231d4124ea6c78fc
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
