danny0405 commented on code in PR #19304:
URL: https://github.com/apache/hudi/pull/19304#discussion_r4043583657
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,6 +1546,38 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ /**
+ * Resolves the record key encoding of a single-field {@code
ComplexKeyGenerator} table onto the write config,
+ * so that every key generator built from it keys records the way the
table's data is keyed. A table that does
+ * not carry {@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} yet gets it
deduced from its data and backfilled
+ * under the transaction lock, or the write fails when the encoding cannot
be determined and
+ * {@code hoodie.write.complex.keygen.validation.enable} is on.
+ * Public because the streamer keys its records before {@link #initTable}
runs and has to call this itself.
+ */
+ public void resolveComplexKeygenEncoding(HoodieTableMetaClient metaClient) {
+ if
(!KeyGenUtils.isComplexKeyGenEncodingTracked(metaClient.getTableConfig())) {
+ return;
+ }
+ if (!metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent()) {
+ executeUsingTxnManager(Option.empty(), () ->
backfillComplexKeygenEncoding(metaClient));
+ }
+ config.setValue(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING,
metaClient.getTableConfig().getComplexKeyGenEncoding().get().name());
+ }
+
+ private void backfillComplexKeygenEncoding(HoodieTableMetaClient metaClient)
{
Review Comment:
Write config can be updated freely during execution while table config is
kind of immutable. The conf injection like this is hard to maintain in general.
The new option should only be applied to the `ComplexAvroKeyGenerator` which
already has a `TypedProperties` in its constructor, is it feasible we set up
the props before instantiating the `ComplexAvroKeyGenerator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
