hudi-agent commented on code in PR #19304:
URL: https://github.com/apache/hudi/pull/19304#discussion_r3749031167
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,6 +1547,49 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ private void handleComplexKeygenEncoding(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if
(!tableConfig.getTableVersion().lesserThan(org.apache.hudi.common.table.HoodieTableVersion.NINE))
{
+ return; // table version 9+ uses fixed encoding; no deduction needed
+ }
+ if
(!KeyGenUtils.isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig)) {
+ return;
+ }
+ if (config.autoDeduceComplexKeygenEncoding()) {
+ if (!tableConfig.populateMetaFields()) {
+ LOG.warn("Skipping complex key encoding auto-deduction for table {}
because meta fields are "
+ + "disabled (virtual keys); relying on the configured {}.",
metaClient.getBasePath(),
+ HoodieWriteConfig.COMPLEX_KEYGEN_NEW_ENCODING.key());
+ return;
+ }
+ org.apache.hudi.common.util.Option<Boolean> cachedEncoding =
KeyGenUtils.readComplexKeyEncodingFromAuxFile(
+ metaClient.getStorage(), metaClient.getBasePath());
+ if (cachedEncoding.isPresent()) {
+ config.setValue(HoodieWriteConfig.COMPLEX_KEYGEN_NEW_ENCODING,
String.valueOf(cachedEncoding.get()));
+ LOG.info("Using cached complex key encoding from aux file: {}",
cachedEncoding.get());
+ } else {
+ String recordKeyField = tableConfig.getRecordKeyFields().get()[0];
+ org.apache.hudi.common.util.Option<Boolean> deducedEncoding =
+ KeyGenUtils.deduceComplexKeyEncodingFromData(metaClient,
recordKeyField);
+ if (deducedEncoding.isPresent()) {
+
KeyGenUtils.writeComplexKeyEncodingToAuxFile(metaClient.getStorage(),
metaClient.getBasePath(), deducedEncoding.get());
+ config.setValue(HoodieWriteConfig.COMPLEX_KEYGEN_NEW_ENCODING,
String.valueOf(deducedEncoding.get()));
Review Comment:
🤖 The deduced encoding is pinned onto the write client's `config`, and the
Spark upsert/insert path picks it up because it builds its key generator from
`client.getConfig()` (same object, read lazily in `HoodieCreateRecordUtils`).
But the record-level DELETE path in `HoodieSparkSqlWriter` builds its key
generator from the separate `hoodieConfig` (from `parameters`), which this
deduction never mutates. Combined with the validation guard now being bypassed
by default, could a delete on an upgraded single-field ComplexKeyGenerator
table generate keys with the default (field:value) encoding and silently fail
to match the stored bare-value keys? @yihua
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,6 +1547,49 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ private void handleComplexKeygenEncoding(HoodieTableMetaClient metaClient) {
Review Comment:
🤖 nit: could you add `import` statements for `HoodieTableVersion` and
`org.apache.hudi.common.util.Option` at the top of the file rather than
inlining the fully-qualified names here (and at line 1562)? The FQN noise makes
the guard conditions a bit harder to scan at a glance.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
