xushiyan commented on code in PR #8929:
URL: https://github.com/apache/hudi/pull/8929#discussion_r1302429672
##########
hudi-aws/src/main/java/org/apache/hudi/aws/sync/AWSGlueCatalogSyncClient.java:
##########
@@ -91,16 +92,23 @@ public class AWSGlueCatalogSyncClient extends
HoodieSyncClient {
private static final Logger LOG =
LoggerFactory.getLogger(AWSGlueCatalogSyncClient.class);
private static final int MAX_PARTITIONS_PER_REQUEST = 100;
private static final long BATCH_REQUEST_SLEEP_MILLIS = 1000L;
+ /**
+ * athena v2/v3 table property
+ * see https://docs.aws.amazon.com/athena/latest/ug/querying-hudi.html
+ */
+ private static final String ENABLE_MDT_LISTING =
"hudi.metadata-listing-enabled";
private final AWSGlue awsGlue;
private final String databaseName;
private final Boolean skipTableArchive;
+ private final String enableMetadataTable;
public AWSGlueCatalogSyncClient(HiveSyncConfig config) {
super(config);
this.awsGlue = AWSGlueClientBuilder.standard().build();
this.databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
this.skipTableArchive =
config.getBooleanOrDefault(GlueCatalogSyncClientConfig.GLUE_SKIP_TABLE_ARCHIVE);
+ this.enableMetadataTable =
Boolean.toString(config.getBoolean(GLUE_METADATA_FILE_LISTING)).toUpperCase();
Review Comment:
right `hoodie.meta.sync.metadata_file_listing` is a separate config. my main
idea is to infer it from the master flag `hoodie.metadata.enable`, just like
for all readers. in `BaseHoodieTableFileIndex`, we check the master flag with a
dedicated default value `DEFAULT_METADATA_ENABLE_FOR_READERS`, which all
readers should follow. There are too many configs already we should avoid
unnecessary extra configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
