Gabriel39 opened a new pull request, #65401:
URL: https://github.com/apache/doris/pull/65401
## Proposed changes
Relax Iceberg position delete Parquet reads so Doris can consume delete
files where `file_path` or `pos` are declared optional in Parquet metadata but
contain no actual null values.
The reader now materializes position delete columns as nullable, validates
that each batch has no nulls, then strips the nullable wrapper before building
delete ranges. Actual null values in `file_path` or `pos` still fail with a
corruption error.
This covers the legacy Iceberg Parquet reader, the shared delete-file helper
used by Iceberg writes, and the format_v2 position delete collector.
## Problem
Some writers, including DuckLake/DuckDB-produced data seen in the field, can
produce Iceberg position delete files whose required Iceberg fields are encoded
as optional Parquet fields. Doris previously rejected these files before
checking whether the data actually contained nulls.
## Testing
- `git diff --check`
- `JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
./run-be-ut.sh --run --filter='IcebergDeleteFileReaderHelperTest.*'`
- Blocked locally during CMake configuration because this checkout is
missing ANN/vector-search prerequisites (`contrib/openblas`, `contrib/faiss`,
OpenMP).
- A prior run without an external `DORIS_THIRDPARTY` also blocked before
compiling due missing `thirdparty/installed/bin/protoc` and Snappy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
