eldenmoon opened a new pull request, #67675: URL: https://github.com/apache/doris/pull/67675
### What problem does this PR solve? Issue Number: [CIR-20720](http://39.106.86.136:8090/browse/CIR-20720) Problem Summary: Variant values already have canonical equality and hashing, but equality joins and ordering were not connected consistently through FE and BE. This PR: - adds a shared canonical comparator whose zero result is consistent with canonical equality; - enables Variant equality coercion, hash joins, and runtime filters; - implements `ColumnVariantV2` permutation and sort-column paths, including nullable multi-key sorting; - allows Variant keys in Sort, TopN, and window ordering in FE; - adds mixed-type, numeric-promotion, NULL, missing-path, array, object, Group By, Order By, and Join coverage; - adds a resumable `variant_p2` runner for correctness, spill stability, and native-versus-CAST performance tests. The large-data validation used the real `variant_p2` GitHub Events data with 44,273,863 rows and the `actor.login` and `actor.id` JSON keys. Every native Variant query matched the corresponding CAST query result hash. Forced-spill runs also matched for three native and three CAST executions per operation. The following 8-core Release measurements are medians of seven measured runs after two warmups. They were collected on a busy shared host, so they describe the current implementation but are not isolated performance claims. | Key / operation | Native Variant | CAST baseline | CAST faster by | | --- | ---: | ---: | ---: | | `actor.login` Group By | 1981.995 ms | 1118.949 ms | 1.77x | | `actor.login` Order By | 1820.382 ms | 156.941 ms | 11.60x | | `actor.login` broadcast Join | 1780.577 ms | 752.931 ms | 2.36x | | `actor.login` shuffle Join | 2009.484 ms | 1014.958 ms | 1.98x | | `actor.id` Group By | 1284.557 ms | 456.478 ms | 2.81x | | `actor.id` Order By | 1797.801 ms | 100.322 ms | 17.92x | | `actor.id` broadcast Join | 1578.926 ms | 389.745 ms | 4.05x | | `actor.id` shuffle Join | 1235.754 ms | 405.663 ms | 3.05x | ### Release note Support equality joins and ordering for Variant values. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - `./run-be-ut.sh --run --filter=ColumnVariantV2Test.*OrderingWorks:VariantV2ExecutionTest.CanonicalComparison:VariantV2ExecutionTest.TypedEquality:VariantCanonicalTest.ComparisonPreservesEqualityAndNumericOrder:VariantCanonicalTest.ContainerComparisonIsLexicographicAndTransitive` (7 passed) - `./run-fe-ut.sh --run org.apache.doris.nereids.rules.analysis.VariantEqualityContextTest` (5 passed) - `variant_p0/test_variant_ordering_comparison_error` passed - `variant_p0/test_variant_relational_corners` passed - `variant_p0/test_variant_join` passed - Release BE build passed - `variant_p2` 44,273,863-row correctness, forced-spill stability, and 8-core benchmark completed - `build-support/clang-format.sh`, `build-support/check-format.sh`, `build-support/check-build-hygiene.sh`, and `git diff --check` passed - Behavior changed: Yes. Variant values can participate in equality joins and ordering operations. - Does this need documentation: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
