suxiaogang223 opened a new pull request, #65253:
URL: https://github.com/apache/doris/pull/65253
### What problem does this PR solve?
Iceberg ORC tables written by Doris can produce invalid data file metrics
when the table contains partition columns and complex columns. A minimal
reproduction is an ORC table partitioned by an `int` column with a
`map<boolean, boolean>` data column, followed by querying `SELECT * FROM
table$data_files`. Iceberg fails while materializing readable metrics because
the committed bounds can be associated with the wrong field id.
Root cause: the ORC metrics collector used Iceberg top-level field order to
match ORC file statistics. Partition columns are not written into ORC data
files, so subsequent Iceberg fields can be matched with ORC statistics from the
wrong column. In the reproduced case, a boolean bound from a map child column
was written as the partition `int` field bound, which later caused Iceberg to
throw `BufferUnderflowException` when decoding the bound as an int.
This PR fixes ORC metrics collection by using the actual ORC writer schema
column id and the stored Iceberg field id attribute. It also encodes Iceberg
`int` bounds as 4-byte values instead of always using 8-byte integer bounds.
### Release note
None
### Check List (For Author)
- Test: Manual test / Static check
- Manual test: On Fedora before the fix, reproduced the ORC/ZLIB
`map<boolean, boolean>` partitioned table failure with `SELECT * FROM
table$data_files` returning `JNI_ERROR BufferUnderflowException`.
- Manual test: On Fedora before the fix, verified the Parquet/ZSTD
counterpart with the same schema succeeds.
- Static check: `git diff --check`.
- Not run: BE rebuild and regression test after the fix because
rebuilding Doris on the development machine is expensive and was deferred.
- Behavior changed: Yes. Iceberg ORC write metrics now use the actual ORC
file schema field ids and encode int bounds with the Iceberg int byte width.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
