danny0405 commented on code in PR #20072:
URL: https://github.com/apache/hudi/pull/20072#discussion_r4118478576
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -394,6 +399,56 @@ public void preWrite(String instantTime,
WriteOperationType writeOperationType,
runPreWriteValidators(instantTime, writeOperationType, metaClient,
recordsOpt);
}
+ /**
+ * Returns the table to write a bucket of records to the given instant, with
the write schema validated.
+ *
+ * <p>The buckets of an instant share its table config, so only the first
bucket of the instant loads it. Every bucket
+ * reloads the timeline: it moves while the instant is written (the previous
instant or a table service can complete
+ * meanwhile), and the file system view of the write handles, served by the
timeline server, rejects a client timeline
+ * older than its own. The write schema is validated, and the internal
schema resolved, again only when a commit has
+ * completed since the last validation, since only a commit can change the
table schema.
+ */
+ @VisibleForTesting
+ HoodieTable<T, List<HoodieRecord<T>>, List<HoodieKey>, List<WriteStatus>>
initTableForInstant(
+ WriteOperationType operationType, String instantTime) {
+ if (instantTableState != null && instantTableState.isFor(operationType,
instantTime)) {
+ HoodieTableMetaClient metaClient = instantTableState.getMetaClient();
+ metaClient.reloadActiveTimeline();
+ String latestCommitCompletionTime =
getLatestCommitCompletionTime(metaClient);
Review Comment:
this is a perf regression, before the patch, there is no per-bucket timeline
loading(file listing) which is a huge pressure to the filesystem.
The `validateWriteSchema` by default does nothing.
Can we keep it simple, not a fan of such cache/reuse pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
