reswqa opened a new pull request, #10549: URL: https://github.com/apache/paimon/pull/10549
### Purpose Fix lookup merge buffer serialization when reading with a projected schema. `LookupMergeFunction.Factory` currently creates the candidate buffer using the full `valueType`, even when the input rows follow a projected `readType`. Once the number of candidates for a key exceeds `lookup.merge-records-threshold`, the buffer switches to binary serialization. The mismatch between the serializer schema and the actual row layout can cause read failures when fields are pruned or reordered, including nested projections. This PR uses `readType` to derive the buffer schema when it is provided, preserving the projected field order and nested structure. Nullable fields from the internal `valueType` are retained by matching field IDs. This preserves the delete-record handling introduced in #9978, where omitted values may be null even if the corresponding fields are declared `NOT NULL` in the read schema. The original `readType` is still passed unchanged to the wrapped merge function. Calls without a `readType` continue to use `valueType`. No public API or table file-format changes. ### Tests Added regression coverage for: - Pruned and reordered fields during lookup merge buffering. - Delete records containing null values for fields declared `NOT NULL` in the read schema. - Nested field projections. - Projected batch reads across two data files with lookup changelog enabled and `lookup.merge-records-threshold` set to 1. Validated the following test classes with JDK 8: - `LookupMergeFunctionTest` - `LookupMergeFunctionUnitTest` - `LookupChangelogMergeFunctionWrapperTest` - `LookupMergeTreeCompactRewriterTest` - `PrimaryKeySimpleTableTest` all tests passed, with no failures, errors, or skips. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
