This is an automated email from the ASF dual-hosted git repository.
924060929 pushed a change to branch fe_local_shuffle_rebase3
in repository https://gitbox.apache.org/repos/asf/doris.git
omit f056ebbed42 [fix](local shuffle) remove serial-without-LE validation
omit 7e15d5b3b91 [fix](local shuffle) serial validation throw to warn;
null-safe force_passthrough
omit 7c898c90050 [opt](local shuffle) single topological sweep for pipeline
num_tasks
omit 089459fbaae [fix](local shuffle) address review comments on PR #63366
omit 95af0ae33d6 [fix](build) macOS: python UDF boost/process via compat
shim; clang-format + license header
omit 716aa608da5 [fix](local shuffle) convert remaining ExchangeType refs
to TLocalPartitionType after rebase
omit 136cb9a6a89 [fix](local shuffle) fall back to LOCAL hash for serial
source in PARTITIONED join/intersect
omit 130a93715b1 [test](local shuffle) add DORIS-26103 regression: UNION
ALL + PartitionTopN + INTERSECT
omit 10ea6e474a7 [test](local shuffle) regression tests for
DORIS-26100/26101 global hash require
omit 071c0ce9fcd [fix](local shuffle) Intersect/Except:
requireGlobalExecutionHash instead of autoRequireHash
omit e3354670586 [fix](local shuffle) PARTITIONED hash join:
requireGlobalExecutionHash instead of requireHash
omit 5fb7c076540 [test](local shuffle) update
LocalShuffleNodeCoverageTest.testRepeatNode for noRequire fix
omit 700cd874539 [test](local shuffle) cover RepeatNode noRequire +
return-child-distribution behaviors
omit 92ce57133c9 [fix](local shuffle) RepeatNode: stop forwarding hash
require below repeat
omit efecd3511a1 [fix](local shuffle) Route to NereidsCoordinator based on
distributedPlans
omit 0f21876ae1e [doc](local shuffle) DANGER ZONE on sender_count to
prevent repeated mistakes
omit 62086e63aa2 [revert](local shuffle) Roll back the sender_count
std::max patch
omit f12e23c1c2f [fix](local shuffle) Mirror BE partition_exprs and
sender_count for FE-planned LE
omit 2ee76320cb4 [test](local shuffle) Switch order-sensitive checks to
order_qt_*
omit 4f81055ee6f [fix](local shuffle) Skip LocalExchange under
RecursiveCteNode and mark it serial (DORIS-25865)
omit f349f5d8492 [fix](local shuffle) Restore LoadStatistic fileNum
tracking under FE local-shuffle planner
omit 0ee9dfebba3 [test](local shuffle) Drop stats-sensitive shape
assertions from test_multilevel_join_agg_local_shuffle
omit f885cb1071e [test](local shuffle) Rewrite LocalExchangePlannerTest
with PlanShape DSL
omit a0be959eac5 [refactor](local shuffle) Move local exchange planning
from BE to FE
add 291dd42160a [fix](fe) Reject TopN in correlated scalar subquery
(#64251)
add 3a2d9d55f1e [fix](nereids) clamp the merged limit of MERGE_TOP_N by
the parent offset (#64306)
add 399d9d8f2bb [refactor](be) Remove redundant spill state (#64083)
add e8c06f265a2 [fix](be) Escape JSONB path member control characters
(#63517)
add c9e9f6c9451 [fix](ann) Clamp nprobe/efSearch to prevent FAISS errors
and recall loss (#64226)
add 7f5b6f510d1 [bug](cloud restore) rewrite table properties and
partition info in cloud restore (#63696)
add 9d3c35bc9b5 [refactor](be) Derive get_storage_field_type from
primitive type (#64341)
add 1450d6b8c60 [fix](test) fix flaky case in variables_persist test_mtmv
(#64365)
add 01f3d7c8f08 [fix](regression) Wait for stable colocate groups in plan
tests (#64361)
add 54a0c9f46e2 [improvement](filecache) limit file cache LRU replay
queues (#64381)
add fd807abdd49 [fix](iceberg) Cleanup when drop DB (#64269)
add ca20f0d2276 [fix](fe) Update fe-core compiler release level from 8 to
17 (#63843)
add 7a6ceedd1f5 [fix](BE) SIGSEGV in MergeSorterState when writing large
sorted Iceberg tables (#64126)
add 10704d7b3fc [refactor](runtime-filter) remove bitmap runtime filter
(#64314)
add f6491435313 [fix](test) fix flaky regression test agg_strategy (#64379)
add 0c2acc10a64 [feat](routine_load) Support SHOW ROUTINE LOAD TASK FOR
<db>.<job_name> (#64151)
add 8fffa028266 [fix](be) Return NaN for avg_weighted when sum of weights
is zero (#64333)
add 081aced387c [fix](regression) Address colocate stability review
comments (#64397)
add 6450603c922 [chore](ms) Change Default config for ms rate limit
(#64398)
add f25caf2adaf [fix](cloud) Fix reading packed inverted index file on
file cache miss (#64383)
add 92f800ac443 [improvement](cgroup) inactive_file should be treated as
available memory to avoid query be cancelled (#64347)
add 4ecdab582db [improvement](fe) Set default profile level to 2 (#64378)
add 0cf0be29977 [fix](regression) fix p2 case failure (#64420)
add 2493edcd2e8 [refactor](local shuffle) Move local exchange planning
from BE to FE
add 41e2492989f [test](local shuffle) Rewrite LocalExchangePlannerTest
with PlanShape DSL
add 428eeb977ee [test](local shuffle) Drop stats-sensitive shape
assertions from test_multilevel_join_agg_local_shuffle
add ee114aea7cd [fix](local shuffle) Restore LoadStatistic fileNum
tracking under FE local-shuffle planner
add b4ad8863c99 [fix](local shuffle) Skip LocalExchange under
RecursiveCteNode and mark it serial (DORIS-25865)
add 3cf5d30598f [test](local shuffle) Switch order-sensitive checks to
order_qt_*
add db4bf139300 [fix](local shuffle) Mirror BE partition_exprs and
sender_count for FE-planned LE
add 7c9f4961369 [revert](local shuffle) Roll back the sender_count
std::max patch
add 680b10b2682 [doc](local shuffle) DANGER ZONE on sender_count to
prevent repeated mistakes
add caacdb944a4 [fix](local shuffle) Route to NereidsCoordinator based on
distributedPlans
add eac6dcd0045 [fix](local shuffle) RepeatNode: stop forwarding hash
require below repeat
add dde70be7a23 [test](local shuffle) cover RepeatNode noRequire +
return-child-distribution behaviors
add 23701ff83e0 [test](local shuffle) update
LocalShuffleNodeCoverageTest.testRepeatNode for noRequire fix
add 3aafb3c17d4 [fix](local shuffle) PARTITIONED hash join:
requireGlobalExecutionHash instead of requireHash
add c2a86e952a2 [fix](local shuffle) Intersect/Except:
requireGlobalExecutionHash instead of autoRequireHash
add 9330c3ad714 [test](local shuffle) regression tests for
DORIS-26100/26101 global hash require
add f274f123fef [test](local shuffle) add DORIS-26103 regression: UNION
ALL + PartitionTopN + INTERSECT
add 083be352f85 [fix](local shuffle) fall back to LOCAL hash for serial
source in PARTITIONED join/intersect
add 133762c5c88 [fix](local shuffle) convert remaining ExchangeType refs
to TLocalPartitionType after rebase
add e778fee303a [fix](build) macOS: python UDF boost/process via compat
shim; clang-format + license header
add d9b6ef638b6 [fix](local shuffle) address review comments on PR #63366
add fdd3c400381 [opt](local shuffle) single topological sweep for pipeline
num_tasks
add a900f9b6c80 [fix](local shuffle) serial validation throw to warn;
null-safe force_passthrough
add 9f33ce47e1b [fix](local shuffle) remove serial-without-LE validation
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f056ebbed42)
\
N -- N -- N refs/heads/fe_local_shuffle_rebase3 (9f33ce47e1b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/common/cgroup_memory_ctl.cpp | 34 ++-
be/src/common/config.cpp | 4 +-
be/src/common/config.h | 2 +
be/src/core/data_type/data_type.cpp | 5 +
be/src/core/data_type/data_type.h | 4 +-
be/src/core/data_type/data_type_agg_state.h | 4 -
be/src/core/data_type/data_type_array.h | 4 -
be/src/core/data_type/data_type_bitmap.h | 4 -
be/src/core/data_type/data_type_date.h | 3 -
.../core/data_type/data_type_date_or_datetime_v2.h | 6 -
be/src/core/data_type/data_type_date_time.h | 4 -
be/src/core/data_type/data_type_hll.h | 4 -
be/src/core/data_type/data_type_ipv4.h | 4 -
be/src/core/data_type/data_type_ipv6.h | 3 -
be/src/core/data_type/data_type_jsonb.h | 3 -
be/src/core/data_type/data_type_map.h | 3 -
be/src/core/data_type/data_type_number_base.h | 53 -----
be/src/core/data_type/data_type_quantilestate.h | 3 -
be/src/core/data_type/data_type_string.h | 4 -
be/src/core/data_type/data_type_struct.h | 3 -
be/src/core/data_type/data_type_variant.h | 3 -
be/src/exec/operator/file_scan_operator.h | 3 -
be/src/exec/operator/mock_scan_operator.h | 4 -
be/src/exec/operator/multi_cast_data_streamer.cpp | 9 +-
.../operator/nested_loop_join_build_operator.cpp | 9 -
.../operator/nested_loop_join_build_operator.h | 1 -
.../operator/nested_loop_join_probe_operator.cpp | 116 +++++------
.../operator/nested_loop_join_probe_operator.h | 1 -
be/src/exec/operator/olap_scan_operator.h | 4 -
be/src/exec/operator/operator.h | 7 -
.../partitioned_aggregation_sink_operator.cpp | 4 +-
be/src/exec/operator/scan_operator.cpp | 34 ---
be/src/exec/operator/scan_operator.h | 7 -
.../operator/spill_iceberg_table_sink_operator.cpp | 4 +-
.../exec/operator/spill_sort_source_operator.cpp | 4 -
be/src/exec/operator/spill_sort_source_operator.h | 4 +-
be/src/exec/operator/spill_utils.h | 17 --
be/src/exec/runtime_filter/runtime_filter.cpp | 20 --
.../runtime_filter/runtime_filter_consumer.cpp | 21 --
.../exec/runtime_filter/runtime_filter_consumer.h | 6 +-
.../runtime_filter/runtime_filter_definitions.h | 6 -
.../exec/runtime_filter/runtime_filter_wrapper.cpp | 34 ---
.../exec/runtime_filter/runtime_filter_wrapper.h | 2 -
be/src/exec/runtime_filter/utils.cpp | 6 -
.../sink/writer/iceberg/viceberg_sort_writer.cpp | 4 +-
be/src/exec/sort/sorter.cpp | 6 +-
be/src/exec/spill/spill_file_writer.cpp | 5 +-
be/src/exec/spill/spill_file_writer.h | 3 -
.../aggregate/aggregate_function_avg_weighted.h | 8 +-
be/src/exprs/bitmapfilter_predicate.h | 133 ------------
be/src/exprs/create_predicate_function.h | 47 -----
be/src/exprs/vbitmap_predicate.cpp | 132 ------------
be/src/exprs/vbitmap_predicate.h | 83 --------
be/src/exprs/vexpr.h | 8 -
be/src/io/cache/block_file_cache.cpp | 111 ++++++++--
be/src/io/cache/block_file_cache.h | 11 +-
be/src/io/cache/lru_queue_recorder.cpp | 68 +++++-
be/src/io/cache/lru_queue_recorder.h | 11 +-
be/src/runtime/runtime_profile_counter_names.h | 1 -
be/src/storage/index/ann/faiss_ann_index.cpp | 15 +-
be/src/storage/predicate/bitmap_filter_predicate.h | 121 -----------
be/src/storage/predicate/column_predicate.h | 7 +-
be/src/storage/predicate/predicate_creator.cpp | 25 ---
be/src/storage/predicate/predicate_creator.h | 4 -
be/src/storage/segment/segment.cpp | 7 +-
be/src/storage/segment/segment.h | 3 +
be/src/storage/segment/segment_iterator.cpp | 24 +--
be/src/storage/segment/segment_iterator.h | 3 -
be/src/storage/tablet/tablet_reader.cpp | 1 -
be/src/storage/tablet/tablet_reader.h | 1 -
be/src/util/cgroup_util.cpp | 7 +
be/src/util/jsonb_document.h | 105 +++++++++-
be/src/util/mem_info.cpp | 16 +-
be/test/core/jsonb/jsonb_document_test.cpp | 113 ++++++++++
.../operator/spill_sort_sink_operator_test.cpp | 29 ---
.../operator/spillable_operator_test_helper.cpp | 3 +-
.../pipeline/multi_cast_data_streamer_test.cpp | 4 -
.../runtime_filter_consumer_test.cpp | 46 -----
.../runtime_filter/runtime_filter_wrapper_test.cpp | 219 +++-----------------
be/test/exec/runtime_filter/utils_test.cpp | 5 -
be/test/exec/sort/merge_sorter_state.cpp | 84 ++++++++
be/test/exprs/bitmapfilter_predicate_test.cpp | 129 ------------
be/test/exprs/bloom_filter_func_test.cpp | 104 ++--------
.../io/cache/block_file_cache_test_lru_dump.cpp | 73 ++++++-
.../io/cache/block_file_cache_test_meta_store.cpp | 28 ++-
be/test/io/cache/cache_lru_dumper_test.cpp | 51 ++++-
.../storage/index/ann/faiss_vector_index_test.cpp | 112 +++++++++-
be/test/vec/spill/spill_file_test.cpp | 1 -
be/test/vec/spill/spill_repartitioner_test.cpp | 1 -
cloud/src/common/config.h | 2 +-
.../meta_service_rate_limit_helper.cpp | 2 +-
fe/fe-core/pom.xml | 2 +-
.../java/org/apache/doris/backup/RestoreJob.java | 26 ++-
.../org/apache/doris/catalog/PartitionInfo.java | 31 ++-
.../org/apache/doris/catalog/TableProperty.java | 12 ++
.../cloud/common/util/CloudPropertyAnalyzer.java | 1 +
.../datasource/iceberg/IcebergMetadataOps.java | 113 +++++++++-
.../glue/translator/PhysicalPlanTranslator.java | 11 -
.../glue/translator/RuntimeFilterTranslator.java | 31 +--
.../doris/nereids/parser/LogicalPlanBuilder.java | 18 +-
.../generator/PatternDescribableProcessor.java | 2 +-
.../processor/post/RuntimeFilterGenerator.java | 64 +-----
.../post/RuntimeFilterPushDownVisitor.java | 25 +--
.../nereids/rules/analysis/SubExprAnalyzer.java | 12 +-
.../rules/exploration/join/JoinCommute.java | 25 ---
.../rules/rewrite/InferSetOperatorDistinct.java | 15 --
.../doris/nereids/rules/rewrite/MergeTopNs.java | 11 +-
...ProjectOtherJoinConditionForNestedLoopJoin.java | 5 -
.../plans/commands/ShowRoutineLoadTaskCommand.java | 19 +-
.../plans/physical/PhysicalNestedLoopJoin.java | 20 --
.../trees/plans/physical/RuntimeFilter.java | 50 +----
.../org/apache/doris/nereids/util/JoinUtils.java | 36 ----
.../apache/doris/planner/NestedLoopJoinNode.java | 20 --
.../org/apache/doris/planner/RuntimeFilter.java | 10 -
.../apache/doris/qe/RuntimeFilterTypeHelper.java | 27 ++-
.../java/org/apache/doris/qe/SessionVariable.java | 34 ++-
.../main/java/org/apache/doris/qe/VariableMgr.java | 6 +-
.../org/apache/doris/catalog/OlapTableTest.java | 120 +++++++++++
.../datasource/iceberg/IcebergMetadataOpTest.java | 229 +++++++++++++++++++++
.../rules/analysis/AnalyzeSubQueryTest.java | 21 ++
.../nereids/rules/rewrite/MergeTopNsTest.java | 20 +-
.../doris/qe/RuntimeFilterTypeHelperTest.java | 54 +++++
.../org/apache/doris/qe/SessionVariablesTest.java | 29 +++
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 3 +-
gensrc/thrift/Exprs.thrift | 4 +-
gensrc/thrift/PlanNodes.thrift | 8 +-
pytest/qe/palo2/src/test_query_bitmap_filter.py | 4 -
.../ann_search_params_clamp.out} | 6 +-
.../test_merge_topn_offset.out} | 8 +-
.../support_type/avg_weighted/avg_weighted.out | 3 +
.../join/test_runtime_filter_decimal256.out | 41 ----
.../sql_functions/json_functions/json_search.out | 3 +
.../query_p2/big_join_build/big_join_build.out | 5 +-
.../org/apache/doris/regression/suite/Suite.groovy | 15 ++
.../ann_index_p0/ann_search_params_clamp.groovy | 122 +++++++++++
.../test_backup_restore_colocate.groovy | 31 +--
.../test_packed_file_inverted_index_query.groovy | 137 ++++++++++++
.../test_colocate_join_of_column_order.groovy | 2 +
.../correctness_p0/test_merge_topn_offset.groovy | 50 +++++
.../write/test_iceberg_write_sorted_order.groovy | 109 ++++++++++
.../routine_load/test_show_routine_load.groovy | 16 +-
.../agg_strategy/agg_strategy.groovy | 11 +-
.../nereids_syntax_p0/sub_query_correlated.groovy | 9 +-
.../support_type/avg_weighted/avg_weighted.groovy | 14 ++
.../query_p0/join/colocate_join_with_rollup.groovy | 1 +
.../suites/query_p0/join/test_bitmap_filter.groovy | 1 -
.../join/test_bitmap_filter_nereids.groovy | 5 +-
.../join/test_runtime_filter_decimal256.groovy | 50 +----
.../json_functions/json_search.groovy | 10 +
.../subquery/correlated_scalar_subquery.groovy | 16 +-
.../query_p0/variables_persist/test_mtmv.groovy | 4 +-
.../query_p2/big_join_build/big_join_build.groovy | 6 +-
.../lateral_view/load_from_big_lateral_view.groovy | 2 +-
...st_show_create_table_with_storage_policy.groovy | 6 +-
154 files changed, 2250 insertions(+), 2007 deletions(-)
delete mode 100644 be/src/exprs/bitmapfilter_predicate.h
delete mode 100644 be/src/exprs/vbitmap_predicate.cpp
delete mode 100644 be/src/exprs/vbitmap_predicate.h
delete mode 100644 be/src/storage/predicate/bitmap_filter_predicate.h
delete mode 100644 be/test/exprs/bitmapfilter_predicate_test.cpp
copy
regression-test/data/{cloud_p0/multi_cluster/test_apsaradb_internal_stage.out
=> ann_index_p0/ann_search_params_clamp.out} (60%)
copy regression-test/data/{load/insert/test_publish_one_succ.out =>
correctness_p0/test_merge_topn_offset.out} (56%)
create mode 100644
regression-test/suites/ann_index_p0/ann_search_params_clamp.groovy
create mode 100644
regression-test/suites/cloud_p0/packed_file/test_packed_file_inverted_index_query.groovy
create mode 100644
regression-test/suites/correctness_p0/test_merge_topn_offset.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_sorted_order.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
