rangareddy commented on issue #17168:
URL: https://github.com/apache/hudi/issues/17168#issuecomment-5366103879
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-9738).
**Findings: confirmed, and the parsing code is itself the strongest
evidence.**
**Write side does not escape.**
`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/BuiltinKeyGenerator.java:224`
and `:275` append `DEFAULT_RECORD_KEY_PARTS_SEPARATOR` directly around raw
field values, so a value containing the separator goes in verbatim - which is
how you get `customer_name:Bob, Johnson`.
**Read side compensates with a heuristic.**
`KeyGenUtils.extractRecordKeysByFields`
(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:139`)
starts with:
```java
// if there is no ',' and ':', then it's a key value
if (!recordKey.contains(DEFAULT_RECORD_KEY_PARTS_SEPARATOR) ||
!recordKey.contains(DEFAULT_COLUMN_VALUE_SEPARATOR)) {
return new String[] {recordKey};
}
```
and then does index arithmetic with repeated
`lastIndexOf(DEFAULT_RECORD_KEY_PARTS_SEPARATOR, keyValueSep2)` calls at
`:166`, `:181`, `:204` and `:208` - scanning backwards to guess where one field
ends and the next begins.
That backwards-scanning logic exists *only* because the separator is
unescaped in the value. It is a heuristic, which means whether a key
round-trips correctly depends on the data rather than on the format. Your
caution is well placed: absence of an observed correctness bug is not the same
as correctness here, and a value containing `:` as well as `,` is the case most
likely to break it.
Worth noting that changing the encoding is an on-disk compatibility change
for existing record-index entries, so it likely needs to ride an index-version
bump rather than being fixed in place.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
