[
https://issues.apache.org/jira/browse/HUDI-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397682#comment-17397682
]
ASF GitHub Bot commented on HUDI-1292:
--------------------------------------
prashantwason commented on a change in pull request #3427:
URL: https://github.com/apache/hudi/pull/3427#discussion_r687252668
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java
##########
@@ -44,6 +44,13 @@
.sinceVersion("0.7.0")
.withDocumentation("Enable the internal metadata table which serves
table metadata like level file listings");
+ // Enable syncing the Metadata Table
+ public static final ConfigProperty<Boolean> METADATA_SYNC_ENABLE_PROP =
ConfigProperty
Review comment:
@leesf the current problem is that metadata table is not safe when
written from multiple pipelines together. So we need to keep it enabled in all
pipelines but only have one pipeline write to it.
Two alternate suggestion:
1. Change the name of this config :
The writing-to-metadata-table is being called the "sync" here. We can change
it to read-only mode?
hoodie.metadata.readonly=true?
2. Introduce metadata table mode instead of enable disable:
hoodie.metadata.mode=[disabled,readwrite,readonly]. readwrite and
readonly imply enabled=true and users only provide the single config mode.
I also do not want another config to confuse users. The other option is to
enable multi-writer mode for metadata table. I have that implemented as part of
another PR (metadata table next version):
https://github.com/apache/hudi/pull/3426/commits/f417390aed56c05dbaedeac6ecf47294996bb591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> [Umbrella] RFC-15 : File Listing and Query Planning Optimizations
> ------------------------------------------------------------------
>
> Key: HUDI-1292
> URL: https://issues.apache.org/jira/browse/HUDI-1292
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration, Writer Core
> Affects Versions: 0.9.0
> Reporter: Vinoth Chandar
> Assignee: Prashant Wason
> Priority: Major
> Labels: hudi-umbrellas, pull-request-available
> Fix For: 0.10.0
>
>
> This is the umbrella ticket that tracks the overall implementation of RFC-15
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
