[
https://issues.apache.org/jira/browse/HDDS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822092#comment-16822092
]
Steve Loughran commented on HDDS-1301:
--------------------------------------
All the work in mapreduce for the s3a committer can be applied to Ozone too:
you can define a new implementation of PathOutputCommitProtocol, and then
declare the factory class for it {{PathOutputCommitterFactory}} , and when an
MR client (or suitably configured Spark app) looks for an output committer for
a FileOutputFormat, you can serve up your committer instead of the legacy
"let's commit by rename" algorithm.
Happy to talk about how to do this/point you at the s3a impl. Key point: for
your store, no matter what the applications think, rename is optional
> Optimize recursive ozone filesystem apis
> ----------------------------------------
>
> Key: HDDS-1301
> URL: https://issues.apache.org/jira/browse/HDDS-1301
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Lokesh Jain
> Assignee: Lokesh Jain
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1301.001.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This Jira aims to optimise recursive apis in ozone file system. These are the
> apis which have a recursive flag which requires an operation to be performed
> on all the children of the directory. The Jira would add support for
> recursive apis in Ozone manager in order to reduce the number of rpc calls to
> Ozone Manager. Also currently these operations are not atomic. This Jira
> would make all the operations in ozone filesystem atomic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
