suxiaogang223 opened a new pull request, #70112:
URL: https://github.com/apache/doris/pull/70112
### What problem does this PR solve?
Issue Number: N/A
Related PR: None
Problem Summary:
Iceberg V3 `unknown` fields represent NULL-only values. Doris currently
exposes them as `UNSUPPORTED`, so users cannot project those columns or read
the complete table with `SELECT *`.
This PR adds read-only support:
- Preserve the logical `unknown` type for schema display and serialization,
while using the existing SQL NULL execution type.
- Emit NULL carriers in scan schemas and synthesize NULL before matching
physical fields or partition values. Retain parent null maps, collection
offsets, and map keys for nested values.
- Keep predicates on unknown columns out of physical-statistics pushdown.
Compare current unknown equality-delete keys as typed NULLs, while preserving
physical-type resolution for dropped historical keys whose retained schema only
records unknown.
- Reject data mutations against a target schema containing unknown,
including omitted/nested columns, metadata-only DELETE, and empty-table
rewrites. Keep unknown column creation unsupported.
- Add frontend/backend unit coverage and runtime-generated Parquet/ORC
regression fixtures for absent/present physical fields, nested values, metadata
display, NULL predicates and aggregates, schema promotion, historical
snapshots, and write rejection.
Other unsupported Iceberg types retain their existing behavior. No
Thrift/Protobuf definitions or storage-file formats change. The new unknown
scan semantics require the accompanying BE changes; mixed-version execution has
not been validated.
### Release note
Support reading Iceberg V3 unknown columns as NULL, including nested
columns. Tables containing unknown columns remain read-only.
### Check List (For Author)
- Test
- [ ] Regression test — `test_iceberg_unknown_read` added, not executed.
Its `.out` must be generated with the regression script and reviewed before
this PR is ready to merge.
- [ ] Unit Test — 11 new FE test methods and 4 new BE test methods
added, not executed.
- [ ] Manual test
- [ ] No need to test or manual test.
Static validation completed:
- Checkstyle for `fe-connector-iceberg` and `fe-core`: 0 violations.
- Repository clang-format 16 formatting and check-only validation passed.
- Connector import gate, BE build/header hygiene, and `git diff --check`
passed.
No Doris build, test-fixture compilation, FE/BE unit test, regression
test, or cluster/client validation has been run. Compilation and execution were
intentionally deferred for review. clang-tidy remains pending because it
requires a BE compilation database.
- Behavior changed:
- [ ] No.
- [x] Yes. Unknown columns become queryable as NULL; data mutations
against tables containing unknown columns are explicitly rejected.
- Does this need documentation?
- [ ] No.
- [x] Yes. Iceberg type-support documentation update is pending.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
