wombatu-kun opened a new pull request, #20314: URL: https://github.com/apache/hudi/pull/20314
### Describe the issue this Pull Request addresses Closes #20313. On a Flink bucket-index table with one primary key and two or more partition fields, an equality filter on the primary key returns no rows: the complex key generator stores the key as `uuid:id1` (`FIELD_PREFIXED`) and the bucket writers hash that, while `PrimaryKeyPruners.getBucketIdFunc` hashed the bare literal `id1`. ### Summary and Changelog - `HoodieTableSource` reads the record key encoding recorded in the table config and hashes `<field>:<value>` when both the table and the job's key generator prefix. When the table records `FIELD_PREFIXED` but the job would not prefix, the stored keys can take either form (Spark prefixes, Flink with an explicit complex key generator and one partition field does not), so the pruning is skipped, as it is for a table without a recorded encoding (no data files are read at planning time) and for a streaming read before the table exists. - `RowDataKeyGen#isFieldPrefixedSingleRecordKey` exposes the writer's prefix decision to the pruner. - `PrimaryKeyPruners#getBucketIdFunc` hashes the prefixed key through `BucketIdentifier.getBucketId(recordKey, indexKeyFields, numBuckets)`, as the writers do. - `KeyGenUtils.extractRecordKeysByFields` gets the missing `keyValueSep2 > 0` guard, so a skipped trailing value holding `:` (e.g. `'a,b:c:d'`) no longer loops forever; only inputs that never returned change. - Tests: `ITTestHoodieDataSource` reproduces both table shapes with Flink SQL on COW and MOR, `TestHoodieTableSource` checks the pruning bucket against the writer's key generator and the skipped cases, and `TestKeyGenUtils` covers the parser. ### Impact Primary key equality queries on these tables return their rows again. A `FIELD_PREFIXED` table read by a job that would not prefix, such as one Spark wrote with one partition field, now reads every bucket and finds its rows. Other key generators, compound keys and `VALUE_ONLY` tables keep their bucket ids. No storage format, API or configuration change. ### Risk Level Low. The bucket changes only when the table and the job both prefix, and then follows the writers' hashing; a table without a recorded encoding is read without bucket pruning until its next write or upgrade records it. Verified: the multi-partition IT fails on master and the single-partition IT on this PR's first revision `ee336b06e3e1`, both with `Expected: is "[+I[id1]]" but: was "[]"`, and each new unit-test assertion fails with its part of the fix removed. Merges cleanly with #20292. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
