JingsongLi commented on code in PR #10415:
URL: https://github.com/apache/paimon/pull/10415#discussion_r4206534476
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java:
##########
@@ -767,6 +755,21 @@ private FileRecordReader<InternalRow> createFileReader(
}
}
+ FileReadTarget readTarget =
+ readTarget(file, dataFilePathFactory, rowRanges,
fileIndexResult);
+ String formatIdentifier = readTarget.formatIdentifier;
+ FormatReaderMapping formatReaderMapping =
+ singleFileReaderMappings.computeIfAbsent(
+ new SingleFileKey(
+ schemaId,
+ formatIdentifier,
+ file.writeCols(),
+ readRowType,
+ nestedFieldEnabled),
+ key ->
+ formatBuilder(readRowType, fileFilters,
nestedFieldEnabled)
+ .build(formatIdentifier, schema,
dataSchema));
Review Comment:
[P1] Use the physical schema when switching to a row sidecar
The new bitmap-driven selection can choose a row sidecar for an ordinary
filtered read with `rowRanges == null`, but this mapping still uses the full
table schema, and `formatBuilder` adds the virtual `_ROW_ID` and
`_SEQUENCE_NUMBER` fields. The sidecar is written using the actual
`writeSchema`. Unlike Parquet, `RowBlockReader` decodes bytes using the
supplied field order and field count before applying the projection, so this
schema mismatch silently changes the values.
I reproduced two cases with row sidecars enabled and a selective bitmap
index:
- Write aligned `[f0, f1]` and `[f2]` files, index `f2`, and query only `f2`
with `f2 = 'b050'`. Scan pruning leaves the file that physically contains only
`f2`, but it is decoded using the full schema; `b050` becomes an empty string.
- Write seven INT columns and filter the indexed column for `50`. The writer
uses a one-byte null header, while the reader adds two virtual fields and
expects a two-byte header; `50` becomes `838860800`.
With residual filtering enabled, both queries return no rows instead of the
matching row. These ordinary filtered reads use Parquet and pass on the parent
commit.
Please build the row decoder's schema from
`dataFileSchema(file.writeCols())` and exclude the virtual tracking fields from
its physical decoding schema. Tracking values should continue to be assigned
from the manifest by `DataFileRecordReader`; applying only the `writeCols`
projection will not fix the seven-column case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
