JingsongLi commented on PR #10415: URL: https://github.com/apache/paimon/pull/10415#issuecomment-6081149945
[P1] Include the physical row schema in the merged sidecar mapping cache key The physical-schema fix resolves the earlier reported cases, but `createUnionReader` still keys non-nested `formatReaderMappings` only by schema ID, format and requested field names. Different `writeCols` layouts can request the same fields while requiring different positional row decoders. I reproduced this with real writes to one data-evolution table with row sidecars enabled: - Row IDs 0–99: files physically containing `[f0, f1]` and `[f2]`. - Row IDs 100–199: files physically containing `[f1]` and `[f2]`. - Both `f2` files have a bitmap index. Query projection is `[f1, f2]`, with `f2 = 'b050'` and no explicit row ranges. The query returns two rows. The first `f1` is correctly `a050`; the second should be `second-a050`, but becomes a corrupt string beginning `nd-a050` and containing strings from subsequent rows (`second-a051` ... `second-a058`). Both groups use the same `FormatKey(schemaId, row, [f1])`, so the second sidecar is decoded with the first group's `[f0,f1]` schema. Please include the physical data schema/order in the cache key whenever the target format is the row sidecar, as the existing nested cache key already does. Reusing only projected names is unsafe for positional decoding. Validation: all 58 existing tests in `DataEvolutionSplitReadTest` and `DataEvolutionFileIndexTest` pass with standard JDK 8 Maven checks. The additional real-table multi-group case fails on this head. It passes when a temporary cache-key change includes `dataSchema.fields()` for row sidecars, and also passes when the selection-ratio setting forces Parquet. All temporary source changes were restored. Independent review confirmed this layout matches supported partial-column writing, and found no further issue in row-ID/sequence/DV position handling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
