[
https://issues.apache.org/jira/browse/HUDI-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397683#comment-17397683
]
ASF GitHub Bot commented on HUDI-1292:
--------------------------------------
prashantwason commented on pull request #3427:
URL: https://github.com/apache/hudi/pull/3427#issuecomment-897222618
To summarize:
1. Async [clean, compact, cluster] - these are in the same process but run
parallel in thread
- We can take a lock in syncMetadata()
- We can sync only as part of commit
- We can depend on transaction support
2. Parallel [clean, compact, cluster] - these are in different processes
- We can use the proposed setting to disable sync in all but one pipeline
- We can depend on transaction support
Only transaction support fixed both the cases but this may not come by this
weekend ([I have a patch for you to
review](https://github.com/apache/hudi/commit/f417390aed56c05dbaedeac6ecf47294996bb591)
though). In the short run, we need to:
1. Enable the setting in this PR
2. Handle the async operations as @vinothchandar is proposing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> [Umbrella] RFC-15 : File Listing and Query Planning Optimizations
> ------------------------------------------------------------------
>
> Key: HUDI-1292
> URL: https://issues.apache.org/jira/browse/HUDI-1292
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration, Writer Core
> Affects Versions: 0.9.0
> Reporter: Vinoth Chandar
> Assignee: Prashant Wason
> Priority: Major
> Labels: hudi-umbrellas, pull-request-available
> Fix For: 0.10.0
>
>
> This is the umbrella ticket that tracks the overall implementation of RFC-15
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
