eldenmoon opened a new pull request, #70476: URL: https://github.com/apache/doris/pull/70476
### Background Doris supports VARIANT columns in Parquet OUTFILE and EXPORT. Parquet defines an unshredded VARIANT logical type with binary metadata and value fields. [Parquet Variant encoding specification](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Doris exports VARIANT as Parquet text, which loses the logical type. On import of a native Parquet VARIANT, a direct load can also change a root string such as `"true"` to a boolean or a root boolean to an integer. The Parquet converter now declares native VARIANT, and direct loads use the file's native type when the source slot has a single consumer. ### Details - **Frequency:** Affects Parquet export of VARIANT columns and direct loads of native Parquet VARIANT into a VARIANT destination. - **Root cause:** Parquet export used Arrow's UTF-8 mapping. The load plan used a VARCHAR source slot followed by a JSON parse for the VARIANT destination. - **Why existing tests missed it:** Text exports could be read as strings, and string-based comparisons could miss scalar type changes. - **Fix:** The Parquet converter owns the extension schema and uses the existing VARIANT SerDe. The load reader binds eligible source slots using each file's schema, including when a job reads both native and UTF-8 Parquet files. - **Other paths and branches:** A source slot shared by custom load expressions uses the planned text conversion. This PR targets master. - **Behavior change:** Parquet OUTFILE and EXPORT write VARIANT columns with the Parquet VARIANT annotation. An explicit `CAST(... AS STRING)` selects text output. ### Testing - Final code: 46 focused ASAN tests covering the Parquet writer and Arrow VARIANT paths passed. Clang-format 16, clang-tidy, and BE build hygiene checks passed. - Before the schema ownership refactor, `test_outfile_parquet_variant` failed on the master baseline because the exported field was TEXT and passed with the fix. Manual checks covered Stream Load root scalar types, EXPORT round-trip, and DuckDB recognizing the exported VARIANT logical type. - Local RELEASE comparison used master `9fb3e3a2a13f` and implementation `40c136a4e6d3` before the schema ownership refactor. One million rows contained 2,000 sparse paths, 20 sparse keys per row, and one materialized key. Segment metadata confirmed the sparse paths. Both sides used Snappy and the same loopback S3 test service. Each side ran one warm-up and five measured rounds. Each round verified all one million rows after S3 TVF reload. | Median | Master | Patched | Change | |---|---:|---:|---:| | Parquet output | 176.56 MB | 93.80 MB | -46.9% | | Export elapsed time | 1.199 s | 1.151 s | -4.0% | | Export BE CPU | 12.28 core-s | 17.76 core-s | +44.6% | | S3 TVF reload elapsed time | 1.920 s | 1.958 s | +2.0% | | S3 TVF reload BE CPU | 16.22 core-s | 14.52 core-s | -10.5% | The benchmark measured one export or reload at a time. Concurrent throughput was not measured. ### Release note Parquet exports now write VARIANT columns using the standard Parquet VARIANT logical type. Direct loads of native Parquet VARIANT preserve root scalar types when the source slot has a single consumer. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes. Update the Parquet export and import documentation to describe native VARIANT and explicit text export. - Pre-submission status: Partially verified. The final schema ownership refactor passed focused ASAN tests. The end-to-end run and RELEASE benchmark used the implementation before that refactor. ### AI Disclosure - Tool: OpenAI Codex - Generated: Implementation, tests, benchmark harness, and PR description - Human review: Pending -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
