This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from 584d73e3f95 change to rc06
new 25c71e386be [minor](Cooldown) Log the old rowset id when uploading
rowset to S3 #30339
new 6f8c133a371 [chore] Remove unused test_show_create_catalog.out (#30290)
new 09ae37dcc51 [pipelineX](localexchange) Adjust local exchange plan rule
(#30393)
new 144204fecce [Refactor](Rf) refactor the code of runtime filter (#30268)
new a954bab81af [fix](function) fix error result in time_to_sec and
timediff (#30248)
new 4bdc2cde2a6 [chore](Nereids) enable nereids if update from doris 1.x
(#30369)
new 9aaa6ba3515 [Fix](Variant) fix variant lost null info after
`cast_column` (#30153)
new 9aa3cdfa1b1 Change auto analyze max table width default value to 100.
(#30395)
new 18b5b72cd91 [improve](stack_trace) avoid print stack trace in group
commit (#30399)
new 1c449d32049 [Improvement](priv) move check priv out of analyze (#30403)
new 85431671953 [Nereids](Variant) Implement variant type and support new
sub column access method (#30348)
new 4bd97b50945 [docs](update) Update 2.0.4 release (#30416)
new 5e4674ab669 [fix](partial update) mishandling of exceptions in the
publish phase may result in data loss (#30366)
new 2befa75b9cc [Rf](exec) Support build exactly not cal by ndv (#30398)
new f25af15842f [Fix](Nereids) Fix lost predicate when query has a filter
at the right input of the outer join (#30374)
new 713798d5497 [feature](nereids)support mark join (#30133)
new fe082e374fc [doc](fix) K8s doc modify configmap code (#30420)
new 904182685b4 [debug](move-memtable) enable brpc debug log in regression
pipelines (#30389)
new 22322a38648 [refactor](join) remove unused RowRefListType from join
(#30442)
new f1a135619c8 [fix](fe) Fix `BackendHbResponse` serialization
compatibility problem (#30433)
new 1fdb3231542 [pipeline](load) p0 load failed in pipeline stream load
(#30438)
new cea9cf07a49 [docs](fix) Fix 2.0.4 Release note for master #30434
new 04237f60e05 [feature](Nereids): eager aggreagate support mix agg
function (#30400)
new 5d2b011a8df [fix](ES catalog) throw exception when ES meta track
failed (#30409)
new 46cadc9856c [minor](Prefetch) log slow prefetch io operation #30415
new d51cbf00f40 [minor fix](use_fix_replica) handle set use_fix_replica <
0 (#30358)
new d1918093727 [fix](pipeline) Fix non-prepared execute of UnionOperator
(#30355)
new f571ffe57fd (fix)[group commit] Row count is incorrect when enable
pipeline load (#30447)
new bedad15f039 [enhancement](scanner) add a lower bound for bytes in
scanner queue (#29624)
new e576412a56c fix iceberg table get split fail when with date type
conjuct (#30162)
new 823c469c5c9 [fix](rowsetreader) determine merge iterator considering
segment num (#29269)
The 31 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 6 +-
be/src/common/config.cpp | 3 +-
be/src/common/config.h | 1 +
be/src/common/logging.h | 40 +-
be/src/exec/exec_node.h | 11 +
be/src/exprs/bloom_filter_func.h | 17 +-
be/src/exprs/json_functions.cpp | 2 +-
be/src/exprs/runtime_filter.cpp | 19 +-
be/src/io/fs/buffered_reader.cpp | 9 +
be/src/io/fs/buffered_reader.h | 3 +-
be/src/olap/rowset/beta_rowset_reader.h | 11 +-
be/src/olap/tablet.cpp | 47 +-
be/src/olap/tablet.h | 6 +-
be/src/olap/txn_manager.cpp | 63 +-
be/src/olap/txn_manager.h | 40 +-
be/src/olap/wal/wal_dirs_info.cpp | 16 +-
be/src/olap/wal/wal_table.cpp | 6 +-
be/src/pipeline/exec/aggregation_sink_operator.cpp | 5 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 88 +-
be/src/pipeline/exec/hashjoin_build_sink.h | 14 +
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 71 +-
be/src/pipeline/exec/hashjoin_probe_operator.h | 18 +
be/src/pipeline/exec/join_build_sink_operator.cpp | 12 +-
be/src/pipeline/exec/join_probe_operator.cpp | 13 +-
be/src/pipeline/exec/join_probe_operator.h | 2 +
be/src/pipeline/exec/union_source_operator.cpp | 4 +-
be/src/pipeline/pipeline_fragment_context.cpp | 1 -
be/src/pipeline/pipeline_x/dependency.h | 1 +
.../pipeline_x/pipeline_x_fragment_context.cpp | 1 -
be/src/runtime/fragment_mgr.cpp | 6 +-
be/src/runtime/group_commit_mgr.cpp | 27 +-
be/src/runtime/plan_fragment_executor.cpp | 1 -
be/src/runtime/runtime_filter_mgr.cpp | 182 +--
be/src/runtime/runtime_filter_mgr.h | 46 +-
.../runtime/stream_load/stream_load_executor.cpp | 2 +-
be/src/vec/common/hash_table/hash_map.h | 4 +-
be/src/vec/common/hash_table/hash_map_context.h | 12 +-
.../common/hash_table/hash_map_context_creator.h | 37 +-
be/src/vec/common/hash_table/join_hash_table.h | 145 +-
be/src/vec/common/schema_util.cpp | 23 +-
be/src/vec/data_types/serde/data_type_serde.cpp | 2 +-
.../exec/join/null_aware_left_semi_join_impl.cpp | 26 +
be/src/vec/exec/join/process_hash_table_probe.h | 6 +-
.../vec/exec/join/process_hash_table_probe_impl.h | 191 ++-
be/src/vec/exec/join/vhash_join_node.cpp | 143 +-
be/src/vec/exec/join/vhash_join_node.h | 83 +-
be/src/vec/exec/join/vjoin_node_base.cpp | 34 +-
be/src/vec/exec/join/vjoin_node_base.h | 6 +-
be/src/vec/exec/scan/scanner_context.cpp | 10 +
be/src/vec/exec/scan/scanner_scheduler.cpp | 8 +-
be/src/vec/exec/vunion_node.cpp | 3 +-
be/src/vec/exec/vunion_node.h | 4 +-
be/src/vec/exprs/vcast_expr.cpp | 12 +-
be/src/vec/exprs/vexpr_context.cpp | 44 +
be/src/vec/exprs/vexpr_context.h | 4 +
be/src/vec/runtime/shared_hash_table_controller.h | 1 +
docs/en/docs/install/k8s-deploy/network.md | 4 +-
.../docs/install/k8s-deploy/persistent-volume.md | 12 +-
docs/en/docs/query-acceleration/statistics.md | 2 +-
docs/en/docs/releasenotes/release-2.0.4.md | 67 +
docs/sidebars.json | 1 +
docs/zh-CN/docs/install/k8s-deploy/network.md | 4 +-
.../docs/install/k8s-deploy/persistent-volume.md | 11 +-
docs/zh-CN/docs/query-acceleration/statistics.md | 2 +-
docs/zh-CN/docs/releasenotes/release-2.0.4.md | 60 +
.../doris/analysis/AlterColumnStatsStmt.java | 18 +-
.../apache/doris/analysis/AlterTableStatsStmt.java | 17 +-
.../doris/analysis/CreateMaterializedViewStmt.java | 9 +-
.../org/apache/doris/analysis/DateLiteral.java | 4 +
.../org/apache/doris/analysis/JoinOperator.java | 4 +-
.../java/org/apache/doris/analysis/SlotRef.java | 11 +-
.../org/apache/doris/analysis/StatementBase.java | 3 +
.../main/java/org/apache/doris/catalog/Env.java | 4 +
.../java/org/apache/doris/catalog/EsTable.java | 7 +-
.../external/elasticsearch/EsMetaStateTracker.java | 12 +-
.../doris/external/iceberg/util/IcebergUtils.java | 6 +-
.../org/apache/doris/nereids/StatementContext.java | 73 +
.../glue/translator/ExpressionTranslator.java | 22 +-
.../glue/translator/PhysicalPlanTranslator.java | 85 +-
.../glue/translator/PlanTranslatorContext.java | 5 +
.../doris/nereids/jobs/executor/Analyzer.java | 3 +
.../doris/nereids/jobs/executor/Rewriter.java | 10 +-
.../post/PushDownFilterThroughProject.java | 4 +
.../doris/nereids/processor/post/Validator.java | 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 11 +-
.../nereids/rules/analysis/BindSlotWithPaths.java | 87 ++
.../nereids/rules/analysis/CheckAfterRewrite.java | 5 +-
.../rules/analysis/CollectJoinConstraint.java | 2 +-
.../doris/nereids/rules/analysis/SlotBinder.java | 8 +-
.../nereids/rules/analysis/SubqueryToApply.java | 60 +-
.../mv/AbstractMaterializedViewRule.java | 66 +-
.../rules/expression/ExpressionOptimization.java | 4 +-
.../rules/expression/ExpressionRewrite.java | 47 +-
.../nereids/rules/expression/check/CheckCast.java | 4 +
.../rules/expression/rules/ElementAtToSlot.java | 89 ++
.../rules/expression/rules/FunctionBinder.java | 33 +-
.../TrySimplifyPredicateWithMarkJoinSlot.java | 113 ++
.../implementation/LogicalJoinToHashJoin.java | 1 +
.../LogicalJoinToNestedLoopJoin.java | 1 +
.../rules/rewrite/AdjustConjunctsReturnType.java | 5 +-
.../nereids/rules/rewrite/AdjustNullable.java | 16 +-
.../nereids/rules/rewrite/CheckDataTypes.java | 8 +-
.../rules/rewrite/CheckMatchExpression.java | 6 +-
.../rules/rewrite/ConvertInnerOrCrossJoin.java | 6 +-
.../rules/rewrite/DeferMaterializeTopNResult.java | 3 +-
.../rules/rewrite/EliminateDedupJoinCondition.java | 7 +-
.../rules/rewrite/EliminateJoinCondition.java | 8 +-
.../nereids/rules/rewrite/EliminateMarkJoin.java | 59 +
.../rewrite/EliminateNullAwareLeftAntiJoin.java | 5 +-
.../nereids/rules/rewrite/EliminateSemiJoin.java | 2 +
.../rules/rewrite/ExtractFilterFromCrossJoin.java | 4 +-
.../rules/rewrite/FindHashConditionForJoin.java | 1 +
.../doris/nereids/rules/rewrite/InApplyToJoin.java | 66 +-
.../doris/nereids/rules/rewrite/OrExpansion.java | 1 +
...oin.java => PushDownAggThroughJoinOneSide.java} | 50 +-
.../rules/rewrite/PushDownAliasThroughJoin.java | 3 +-
.../rewrite/PushDownCountThroughJoinOneSide.java | 216 ---
.../PushDownExpressionsInHashCondition.java | 13 +
.../rules/rewrite/PushDownFilterThroughJoin.java | 1 +
.../rewrite/PushDownFilterThroughProject.java | 2 +
.../rules/rewrite/PushDownJoinOtherCondition.java | 3 +-
.../rewrite/PushDownSumThroughJoinOneSide.java | 98 --
.../rewrite/UnCorrelatedApplyAggregateFilter.java | 2 +-
.../rules/rewrite/UnCorrelatedApplyFilter.java | 2 +-
.../rewrite/UnCorrelatedApplyProjectFilter.java | 2 +-
.../apache/doris/nereids/stats/JoinEstimation.java | 12 +-
.../trees/copier/LogicalPlanDeepCopier.java | 8 +-
.../doris/nereids/trees/expressions/Alias.java | 10 +-
.../trees/expressions/ArrayItemReference.java | 2 +-
.../nereids/trees/expressions/SlotReference.java | 77 +-
.../expressions/functions/scalar/ElementAt.java | 6 +-
.../scalar/PushDownToProjectionFunction.java | 31 +
.../expressions/functions/scalar/TimeToSec.java | 4 +-
.../doris/nereids/trees/plans/algebra/Join.java | 23 +-
.../doris/nereids/trees/plans/algebra/Project.java | 20 +
.../trees/plans/commands/UpdateCommand.java | 3 +-
.../nereids/trees/plans/logical/LogicalApply.java | 33 +-
.../plans/logical/LogicalCatalogRelation.java | 2 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 270 ++--
.../trees/plans/logical/LogicalOlapScan.java | 42 +-
.../trees/plans/logical/LogicalTVFRelation.java | 2 +-
.../nereids/trees/plans/logical/UsingJoin.java | 26 +-
.../trees/plans/physical/AbstractPhysicalJoin.java | 61 +-
.../plans/physical/PhysicalCatalogRelation.java | 2 +-
.../trees/plans/physical/PhysicalHashJoin.java | 59 +-
.../plans/physical/PhysicalNestedLoopJoin.java | 73 +-
.../trees/plans/physical/PhysicalTVFRelation.java | 2 +-
.../org/apache/doris/nereids/types/DataType.java | 6 +
.../apache/doris/nereids/types/VariantType.java | 83 ++
.../apache/doris/nereids/util/ExpressionUtils.java | 141 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 17 +-
.../doris/nereids/util/TypeCoercionUtils.java | 11 +
.../java/org/apache/doris/planner/EsScanNode.java | 2 +-
.../org/apache/doris/planner/HashJoinNode.java | 95 ++
.../apache/doris/planner/NestedLoopJoinNode.java | 17 +
.../org/apache/doris/planner/OlapScanNode.java | 16 +-
.../java/org/apache/doris/planner/ScanNode.java | 5 +-
.../apache/doris/planner/StreamLoadPlanner.java | 3 +
.../main/java/org/apache/doris/qe/Coordinator.java | 18 +-
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 1 +
.../main/java/org/apache/doris/qe/VariableMgr.java | 20 +
.../java/org/apache/doris/rewrite/FEFunctions.java | 6 +-
.../doris/statistics/StatisticConstants.java | 2 +-
.../rules/expression/PredicatesSplitterTest.java | 2 +-
.../rules/rewrite/ExistsApplyToJoinTest.java | 8 +-
.../PushDownCountThroughJoinOneSideTest.java | 139 --
.../rewrite/PushDownMinMaxSumThroughJoinTest.java | 357 +++++
.../rewrite/PushDownMinMaxThroughJoinTest.java | 183 ---
.../rewrite/PushDownSumThroughJoinOneSideTest.java | 135 --
.../nereids/trees/plans/PlanToStringTest.java | 2 +-
.../org/apache/doris/rewrite/FEFunctionsTest.java | 19 -
gensrc/script/doris_builtins_functions.py | 1 +
gensrc/thrift/PlanNodes.thrift | 6 +-
.../data/correctness/test_time_function.out | 9 +-
.../concurrency_update1.csv} | 0
.../concurrency_update2.csv | 0
.../fault_injection_p0/concurrency_update3.csv | 21 +
..._partial_update_publish_conflict_with_error.out | 47 +
.../nereids_p0/show/test_show_create_catalog.out | 4 -
.../nereids_p0/subquery/subquery_unnesting.out | 1494 ++++++++++++++++++++
.../subquery/test_subquery_in_project.out | 120 ++
.../push_down_count_through_join_one_side.out | 6 +-
.../eager_aggregate/push_down_max_through_join.out | 6 +-
.../eager_aggregate/push_down_min_through_join.out | 6 +-
.../push_down_sum_through_join_one_side.out | 6 +-
.../mv/join/left_outer/outer_join.out | 8 +
.../shape/query10.out | 85 +-
.../shape/query35.out | 53 +-
.../noStatsRfPrune/query10.out | 49 +-
.../noStatsRfPrune/query35.out | 51 +-
.../no_stats_shape/query10.out | 49 +-
.../no_stats_shape/query35.out | 51 +-
.../rf_prune/query10.out | 83 +-
.../rf_prune/query35.out | 53 +-
.../nereids_tpcds_shape_sf100_p0/shape/query10.out | 83 +-
.../nereids_tpcds_shape_sf100_p0/shape/query35.out | 53 +-
.../query_p0/show/test_show_create_catalog.out | 4 -
regression-test/data/variant_p0/load.out | 4 +-
.../variant_p0/schema_change/schema_change.out | 24 +
.../data/variant_p0/variant_hirachinal.out | 21 +
regression-test/pipeline/external/conf/be.conf | 3 +
regression-test/pipeline/p0/conf/be.conf | 3 +
regression-test/pipeline/p1/conf/be.conf | 5 +-
.../suites/correctness/test_time_function.groovy | 9 +
...rtial_update_publish_conflict_with_error.groovy | 171 +++
.../nereids_p0/subquery/subquery_unnesting.groovy | 132 ++
.../subquery/test_subquery_in_project.groovy | 55 +
.../nereids_rules_p0/eager_aggregate/basic.groovy | 2 +-
.../eager_aggregate/basic_one_side.groovy | 4 +-
.../push_down_count_through_join_one_side.groovy | 2 +-
.../push_down_max_through_join.groovy | 2 +-
.../push_down_min_through_join.groovy | 2 +-
.../push_down_sum_through_join_one_side.groovy | 2 +-
.../mv/join/left_outer/outer_join.groovy | 24 +-
.../tpch_sf100_unique_p2/load_one_step/load.groovy | 7 +-
.../suites/variant_p0/column_name.groovy | 8 +-
.../suites/variant_p0/complexjson.groovy | 9 +-
.../suites/variant_p0/insert_into_select.groovy | 2 +-
regression-test/suites/variant_p0/load.groovy | 4 +-
regression-test/suites/variant_p0/multi_var.groovy | 6 +-
.../variant_p0/schema_change/schema_change.groovy | 7 +
regression-test/suites/variant_p0/sql/gh_data.sql | 22 +-
.../suites/variant_p0/variant_hirachinal.groovy | 49 +
.../suites/variant_p0/with_index/load.groovy | 10 +-
.../suites/variant_p0/with_index/var_index.groovy | 4 +-
226 files changed, 5912 insertions(+), 2261 deletions(-)
create mode 100644 be/src/vec/exec/join/null_aware_left_semi_join_impl.cpp
create mode 100644 docs/en/docs/releasenotes/release-2.0.4.md
create mode 100644 docs/zh-CN/docs/releasenotes/release-2.0.4.md
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotWithPaths.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/ElementAtToSlot.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/TrySimplifyPredicateWithMarkJoinSlot.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateMarkJoin.java
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/{PushDownMinMaxThroughJoin.java
=> PushDownAggThroughJoinOneSide.java} (81%)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownCountThroughJoinOneSide.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownSumThroughJoinOneSide.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/PushDownToProjectionFunction.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/types/VariantType.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownCountThroughJoinOneSideTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownMinMaxSumThroughJoinTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownMinMaxThroughJoinTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownSumThroughJoinOneSideTest.java
copy
regression-test/data/{unique_with_mow_p0/partial_update/concurrency_update3.csv
=> fault_injection_p0/concurrency_update1.csv} (100%)
copy regression-test/data/{unique_with_mow_p0/partial_update =>
fault_injection_p0}/concurrency_update2.csv (100%)
create mode 100644
regression-test/data/fault_injection_p0/concurrency_update3.csv
create mode 100644
regression-test/data/fault_injection_p0/test_partial_update_publish_conflict_with_error.out
delete mode 100644
regression-test/data/nereids_p0/show/test_show_create_catalog.out
create mode 100644
regression-test/data/nereids_p0/subquery/subquery_unnesting.out
delete mode 100644
regression-test/data/query_p0/show/test_show_create_catalog.out
create mode 100644 regression-test/data/variant_p0/variant_hirachinal.out
create mode 100644
regression-test/suites/fault_injection_p0/test_partial_update_publish_conflict_with_error.groovy
create mode 100644
regression-test/suites/nereids_p0/subquery/subquery_unnesting.groovy
create mode 100644 regression-test/suites/variant_p0/variant_hirachinal.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
