This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch auto-pick-52315-branch-3.1
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 1dd6c93f74e [fix](external) check duplicate column names for external
table schema (#52315)
add 24f3c3a10c3 branch-3.1: [fix](serde)fix orc remalloc memory leak
#52676 (#52673)
add d96c5fdcbc6 branch-3.1 [feature](iceberg catalog) support iceberg view
query #51376 (#52806)
add a92ada57b35 branch-3.1: [enhancement](cloud) Persist LRU information
for file cache #49456 (#52811)
add a236d1f04e1 branch-3.1: [Opt](cloud-mow) Do fast retry when commit
compaction job for mow tablet #52476 (#52843)
add 950ccccefda branch-3.1: [chore](cloud) Add async wrap for
bthread_fork_join with promise-future #52816 (#52845)
add 29d5d993c9b branch-3.1: [fix](mow) fix potential mem leak for
DeleteBitmap::get_agg #52718,#52756 (#52860)
add 7f216fc71cd branch-3.1: [fix](move-memtable) gracefully shutdown flush
token in TabletStream::pre_close #52719 (#52778)
add c2ae4ab1f9a branch-3.1: [fix](ci) exclude case with known issue of
memory leak #52784 (#52804)
add 16d330dc067 branch-3.1: [Fix](regression) update flink connector
version for test case #48292 (#52856)
add bdcf9978abf [fix](array) fix build when using gcc (#52759)
add e7430774d91 branch-3.1: [opt](ci) check stop grace and detected memory
leak|undefined-behavior #52725 (#52741)
add 11ad47fc1d4 branhc-3.1: [fix](case) let lazy_materialize_topn work
well (#52890)
add 1cdc43a1881 branch-3.1: [fix](bloomfilter) Correct bloom_filter_fpp
usage in BE #47384 (#52893)
add 64fef996197 branch-3.1: [feature](inverted index) add index searcher
after writing inverted index #43992 (#52877)
add d854139aa27 branch-3.1: [fix](inverted index) catch IO exception to
avoid coredump in inverted index string reader #51844 (#52867)
add f639783347a branch-3.1: [Exec](alias) support stddev alias to std
#52819 (#52866)
add 5a68db78bed branch-3.1: [improve](hive row count)Log stack info when
fetch hive row count fail. #52795 (#52840)
add 9020fdd141f [fix](inverted index) fix custom analyzer validation check
(#52826)
add fde839a848d branch-3.1: [env](compile) add compiler check and fix some
clang-tidy suggestions for vec/exec/format #45881 (#52832)
add 04b4ad300a3 branch-3.1: [fix](job) remove can not transform RUNNING to
NEED_SCHEDULE limit #52887 (#52910)
add f19fd9303f9 [fix](cloud) fix some multi-az failover bug and add
regression cases #52450 (#52685)
add 05d8b2508be [fix](external) check duplicate column names for external
table schema (#52315)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1dd6c93f74e)
\
N -- N -- N refs/heads/auto-pick-52315-branch-3.1 (05d8b2508be)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/agent/heartbeat_server.cpp | 7 +
.../{cloud_delete_task.h => cloud_cluster_info.h} | 18 +-
be/src/cloud/cloud_meta_mgr.cpp | 32 +-
be/src/cloud/cloud_meta_mgr.h | 9 +
be/src/cloud/cloud_tablet_mgr.cpp | 8 +
be/src/cloud/config.cpp | 4 +
be/src/cloud/config.h | 4 +
be/src/clucene | 2 +-
be/src/common/config.cpp | 9 +-
be/src/common/config.h | 11 +-
be/src/http/action/shrink_mem_action.cpp | 2 +
be/src/io/cache/block_file_cache.cpp | 241 ++++++++---
be/src/io/cache/block_file_cache.h | 130 ++----
be/src/io/cache/cache_lru_dumper.cpp | 465 +++++++++++++++++++++
be/src/io/cache/cache_lru_dumper.h | 83 ++++
be/src/io/cache/file_block.cpp | 8 +-
be/src/io/cache/file_cache_common.cpp | 57 +++
be/src/io/cache/file_cache_common.h | 99 +++++
be/src/io/cache/fs_file_cache_storage.cpp | 11 +-
be/src/io/cache/lru_queue_recorder.cpp | 132 ++++++
be/src/io/cache/lru_queue_recorder.h | 83 ++++
be/src/olap/push_handler.h | 4 +-
.../segment_v2/bloom_filter_index_writer.cpp | 12 +
be/src/olap/rowset/segment_v2/column_writer.cpp | 3 +-
be/src/olap/rowset/segment_v2/column_writer.h | 2 +
.../segment_v2/inverted_index_file_reader.cpp | 1 +
.../rowset/segment_v2/inverted_index_file_reader.h | 4 +-
.../segment_v2/inverted_index_file_writer.cpp | 61 +++
.../rowset/segment_v2/inverted_index_file_writer.h | 5 +
.../rowset/segment_v2/inverted_index_reader.cpp | 89 ++--
.../olap/rowset/segment_v2/inverted_index_reader.h | 5 +-
be/src/olap/rowset/segment_v2/segment_writer.cpp | 3 +
.../rowset/segment_v2/vertical_segment_writer.cpp | 4 +
be/src/olap/tablet_meta.cpp | 4 +-
be/src/runtime/exec_env.h | 3 +
be/src/runtime/exec_env_init.cpp | 8 +-
be/src/runtime/load_stream.cpp | 2 +
be/src/util/string_parser.hpp | 3 +-
.../aggregate_function_stddev.cpp | 1 +
be/src/vec/columns/column_nullable.h | 2 +-
be/src/vec/core/block.h | 6 +-
be/src/vec/data_types/data_type_ipv4.h | 8 +-
.../vec/data_types/serde/data_type_array_serde.cpp | 46 +-
.../vec/data_types/serde/data_type_array_serde.h | 19 +-
.../data_types/serde/data_type_bitmap_serde.cpp | 40 +-
.../vec/data_types/serde/data_type_bitmap_serde.h | 8 +-
.../data_types/serde/data_type_date64_serde.cpp | 84 ++--
.../vec/data_types/serde/data_type_date64_serde.h | 18 +-
.../serde/data_type_datetimev2_serde.cpp | 18 +-
.../data_types/serde/data_type_datetimev2_serde.h | 14 +-
.../data_types/serde/data_type_datev2_serde.cpp | 15 +-
.../vec/data_types/serde/data_type_datev2_serde.h | 14 +-
.../data_types/serde/data_type_decimal_serde.cpp | 18 +-
.../vec/data_types/serde/data_type_decimal_serde.h | 14 +-
.../vec/data_types/serde/data_type_hll_serde.cpp | 40 +-
be/src/vec/data_types/serde/data_type_hll_serde.h | 8 +-
.../vec/data_types/serde/data_type_ipv4_serde.cpp | 8 +-
be/src/vec/data_types/serde/data_type_ipv4_serde.h | 6 +-
.../vec/data_types/serde/data_type_ipv6_serde.cpp | 57 ++-
be/src/vec/data_types/serde/data_type_ipv6_serde.h | 6 +-
.../vec/data_types/serde/data_type_jsonb_serde.cpp | 57 ++-
.../vec/data_types/serde/data_type_jsonb_serde.h | 4 +-
.../vec/data_types/serde/data_type_map_serde.cpp | 36 +-
be/src/vec/data_types/serde/data_type_map_serde.h | 10 +-
.../vec/data_types/serde/data_type_nothing_serde.h | 8 +-
.../data_types/serde/data_type_nullable_serde.cpp | 16 +-
.../data_types/serde/data_type_nullable_serde.h | 14 +-
.../data_types/serde/data_type_number_serde.cpp | 62 ++-
.../vec/data_types/serde/data_type_number_serde.h | 14 +-
.../data_types/serde/data_type_object_serde.cpp | 57 ++-
.../vec/data_types/serde/data_type_object_serde.h | 8 +-
.../serde/data_type_quantilestate_serde.h | 42 +-
be/src/vec/data_types/serde/data_type_serde.h | 52 +--
.../vec/data_types/serde/data_type_string_serde.h | 16 +-
.../data_types/serde/data_type_struct_serde.cpp | 30 +-
.../vec/data_types/serde/data_type_struct_serde.h | 14 +-
be/src/vec/data_types/serde/data_type_time_serde.h | 5 +-
.../exec/format/arrow/arrow_pip_input_stream.cpp | 18 +-
.../vec/exec/format/arrow/arrow_pip_input_stream.h | 14 +-
.../vec/exec/format/arrow/arrow_stream_reader.cpp | 16 +-
be/src/vec/exec/format/arrow/arrow_stream_reader.h | 6 +-
be/src/vec/exec/format/avro/avro_jni_reader.cpp | 10 +-
be/src/vec/exec/format/avro/avro_jni_reader.h | 10 +-
be/src/vec/exec/format/column_type_convert.cpp | 4 +-
be/src/vec/exec/format/column_type_convert.h | 43 +-
be/src/vec/exec/format/csv/csv_reader.cpp | 11 +-
be/src/vec/exec/format/csv/csv_reader.h | 15 +-
.../file_reader/new_plain_binary_line_reader.cpp | 6 +-
.../file_reader/new_plain_binary_line_reader.h | 7 +-
.../file_reader/new_plain_text_line_reader.cpp | 16 +-
.../file_reader/new_plain_text_line_reader.h | 24 +-
be/src/vec/exec/format/format_common.h | 4 +-
be/src/vec/exec/format/generic_reader.h | 10 +-
be/src/vec/exec/format/jni_reader.cpp | 4 +-
be/src/vec/exec/format/jni_reader.h | 5 +-
be/src/vec/exec/format/json/new_json_reader.cpp | 42 +-
be/src/vec/exec/format/json/new_json_reader.h | 21 +-
be/src/vec/exec/format/orc/orc_memory_pool.h | 2 +
be/src/vec/exec/format/orc/vorc_reader.cpp | 202 +++++----
be/src/vec/exec/format/orc/vorc_reader.h | 81 ++--
be/src/vec/exec/format/parquet/schema_desc.h | 2 +-
.../exec/format/parquet/vparquet_group_reader.cpp | 2 +-
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 1 +
be/src/vec/exec/format/table/equality_delete.cpp | 13 +-
be/src/vec/exec/format/table/equality_delete.h | 4 +-
be/src/vec/exec/format/table/hudi_jni_reader.cpp | 6 +-
be/src/vec/exec/format/table/hudi_jni_reader.h | 3 +-
.../format/table/iceberg/arrow_schema_util.cpp | 10 +-
.../exec/format/table/iceberg/arrow_schema_util.h | 10 +-
.../exec/format/table/iceberg/partition_spec.cpp | 9 +-
.../vec/exec/format/table/iceberg/partition_spec.h | 10 +-
.../format/table/iceberg/partition_spec_parser.cpp | 10 +-
.../format/table/iceberg/partition_spec_parser.h | 8 +-
be/src/vec/exec/format/table/iceberg/schema.cpp | 8 +-
be/src/vec/exec/format/table/iceberg/schema.h | 8 +-
.../exec/format/table/iceberg/schema_parser.cpp | 13 +-
.../vec/exec/format/table/iceberg/schema_parser.h | 8 +-
be/src/vec/exec/format/table/iceberg/struct_like.h | 10 +-
be/src/vec/exec/format/table/iceberg/types.cpp | 13 +-
be/src/vec/exec/format/table/iceberg/types.h | 79 ++--
.../table/iceberg/unbound_partition_spec.cpp | 13 +-
.../format/table/iceberg/unbound_partition_spec.h | 8 +-
be/src/vec/exec/format/table/iceberg_reader.cpp | 46 +-
be/src/vec/exec/format/table/iceberg_reader.h | 6 +-
.../vec/exec/format/table/lakesoul_jni_reader.cpp | 6 +-
be/src/vec/exec/format/table/lakesoul_jni_reader.h | 2 +
.../exec/format/table/max_compute_jni_reader.cpp | 6 +-
.../vec/exec/format/table/max_compute_jni_reader.h | 7 +-
be/src/vec/exec/format/table/paimon_jni_reader.cpp | 10 +-
be/src/vec/exec/format/table/paimon_jni_reader.h | 4 +-
be/src/vec/exec/format/table/paimon_reader.cpp | 2 +
be/src/vec/exec/format/table/paimon_reader.h | 2 +
be/src/vec/exec/format/table/table_format_reader.h | 5 +-
.../format/table/transactional_hive_common.cpp | 2 +
.../exec/format/table/transactional_hive_common.h | 2 +
.../format/table/transactional_hive_reader.cpp | 34 +-
.../exec/format/table/transactional_hive_reader.h | 14 +-
.../format/table/trino_connector_jni_reader.cpp | 4 +-
.../exec/format/table/trino_connector_jni_reader.h | 7 +-
be/src/vec/exec/format/wal/wal_reader.cpp | 12 +-
be/src/vec/exec/format/wal/wal_reader.h | 9 +-
be/src/vec/exec/scan/vfile_scanner.cpp | 4 +-
.../functions/array/function_array_aggregation.cpp | 47 +--
be/test/cloud/cloud_compaction_test.cpp | 5 +
be/test/cloud/cloud_meta_mgr_test.cpp | 89 ++++
be/test/io/cache/block_file_cache_test.cpp | 232 +++-------
be/test/io/cache/block_file_cache_test_common.h | 134 ++++++
.../io/cache/block_file_cache_test_lru_dump.cpp | 404 ++++++++++++++++++
be/test/io/cache/cache_lru_dumper_test.cpp | 149 +++++++
be/test/io/cache/lru_queue_test.cpp | 117 ++++++
.../segment_v2/inverted_index_file_writer_test.cpp | 374 ++++++++++++++++-
.../segment_v2/inverted_index_reader_test.cpp | 167 +++++++-
be/test/vec/data_types/common_data_type_test.h | 1 -
.../serde/data_type_serde_object_test.cpp | 2 +-
cloud/src/meta-service/meta_service_job.cpp | 141 ++++---
.../create_preinstalled_scripts/iceberg/run14.sql | 87 ++++
.../main/java/org/apache/doris/common/Config.java | 3 +
.../apache/doris/analysis/FunctionCallExpr.java | 3 +-
.../java/org/apache/doris/analysis/IndexDef.java | 3 +-
.../apache/doris/analysis/InvertedIndexUtil.java | 9 +-
.../doris/catalog/BuiltinAggregateFunctions.java | 2 +-
.../main/java/org/apache/doris/catalog/Env.java | 6 +-
.../java/org/apache/doris/catalog/FunctionSet.java | 8 +
.../main/java/org/apache/doris/catalog/Index.java | 3 +-
.../doris/cloud/system/CloudSystemInfoService.java | 43 +-
.../apache/doris/datasource/ExternalCatalog.java | 11 +
.../doris/datasource/ExternalRowCountCache.java | 10 +-
.../doris/datasource/hive/HMSExternalTable.java | 4 +-
.../datasource/iceberg/IcebergExternalCatalog.java | 12 +-
.../datasource/iceberg/IcebergExternalTable.java | 108 ++++-
.../datasource/iceberg/IcebergMetadataCache.java | 41 ++
.../datasource/iceberg/IcebergMetadataOps.java | 106 ++++-
.../doris/datasource/iceberg/IcebergUtils.java | 81 ++--
.../iceberg/source/IcebergApiSource.java | 11 +
.../datasource/operations/ExternalMetadataOps.java | 33 ++
.../doris/load/routineload/RoutineLoadJob.java | 5 -
.../doris/nereids/rules/analysis/BindRelation.java | 43 +-
.../trees/plans/commands/info/IndexDefinition.java | 5 +-
.../org/apache/doris/planner/IcebergTableSink.java | 3 +
.../java/org/apache/doris/qe/ConnectContext.java | 4 +
.../java/org/apache/doris/qe/ShowExecutor.java | 8 +
.../main/java/org/apache/doris/system/Backend.java | 5 +
.../java/org/apache/doris/system/HeartbeatMgr.java | 4 +
.../cloud/system/CloudSystemInfoServiceTest.java | 33 ++
.../nereids/rules/analysis/GenerateFunction.java | 1 +
.../proto/file_cache.proto | 39 +-
gensrc/thrift/HeartbeatService.thrift | 5 +
...ead_write.out => default_vcg_auto_failover.out} | Bin
...out => failover_standby_disable_compaction.out} | Bin 129 -> 171 bytes
...vcluster.out => standby_disable_compaction.out} | Bin 129 -> 171 bytes
...ad_write.out => use_default_vcg_read_write.out} | Bin 299 -> 197 bytes
...ut => use_vcg_read_write_unhealthy_node_50.out} | Bin
...se_vcg_read_write.out => vcg_auto_failover.out} | Bin
...e.out => vcg_auto_failover_manual_failback.out} | Bin 299 -> 367 bytes
.../iceberg/test_iceberg_view_query_p0.out | Bin 0 -> 1033 bytes
.../inverted_index_p0/test_inverted_index_v3.out | Bin 223 -> 304 bytes
.../test_aggregate_all_functions.out | Bin 2765 -> 2803 bytes
regression-test/framework/pom.xml | 2 +-
.../regression/action/StreamLoadAction.groovy | 4 +
.../org/apache/doris/regression/suite/Suite.groovy | 94 +++--
.../doris/regression/suite/SuiteCluster.groovy | 3 +
regression-test/pipeline/cloud_p0/clean.sh | 3 +-
regression-test/pipeline/cloud_p0/run.sh | 2 +-
regression-test/pipeline/common/doris-utils.sh | 67 ++-
.../nonConcurrent/conf/regression-conf.groovy | 1 +
regression-test/pipeline/p0/conf/be.conf | 1 +
regression-test/pipeline/vault_p0/clean.sh | 4 +-
regression-test/pipeline/vault_p0/run.sh | 2 +-
.../bloom_filter_p0/test_bloom_filter.groovy | 63 ++-
.../virtual_compute_group}/ddl/nation.sql | 0
.../virtual_compute_group}/ddl/nation_delete.sql | 0
.../virtual_compute_group}/ddl/nation_load.sql | 0
.../default_vcg_auto_failover.groovy | 105 +++--
.../failover_standby_disable_compaction.groovy | 289 +++++++++++++
.../load_trigger_failover.groovy | 164 ++++++++
.../standby_disable_compaction.groovy | 276 ++++++++++++
.../use_default_vcg_read_write.groovy | 152 ++++---
.../use_vcg_read_write.groovy | 136 +++---
...oovy => use_vcg_read_write_routine_load.groovy} | 348 +++++++--------
...te.groovy => use_vcg_read_write_s3_load.groovy} | 320 +++++---------
...=> use_vcg_read_write_unhealthy_node_50.groovy} | 153 ++++---
.../virtual_compute_group/vcg_auto_failover.groovy | 88 ++--
...vy => vcg_auto_failover_manual_failback.groovy} | 286 ++++++++-----
...ocker_action.groovy => test_lru_persist.groovy} | 74 ++--
.../iceberg/test_iceberg_view_query_p0.groovy | 203 +++++++++
.../test_inverted_index_v3.groovy | 110 +++++
.../lazy_materialize_topn.groovy | 1 -
.../test_aggregate_all_functions.groovy | 1 +
228 files changed, 6746 insertions(+), 2333 deletions(-)
copy be/src/cloud/{cloud_delete_task.h => cloud_cluster_info.h} (75%)
create mode 100644 be/src/io/cache/cache_lru_dumper.cpp
create mode 100644 be/src/io/cache/cache_lru_dumper.h
create mode 100644 be/src/io/cache/lru_queue_recorder.cpp
create mode 100644 be/src/io/cache/lru_queue_recorder.h
create mode 100644 be/test/cloud/cloud_meta_mgr_test.cpp
create mode 100644 be/test/io/cache/block_file_cache_test_common.h
create mode 100644 be/test/io/cache/block_file_cache_test_lru_dump.cpp
create mode 100644 be/test/io/cache/cache_lru_dumper_test.cpp
create mode 100644 be/test/io/cache/lru_queue_test.cpp
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg/run14.sql
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/receiver/AbstractReceiver.java
=> gensrc/proto/file_cache.proto (56%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.out
=> default_vcg_auto_failover.out} (100%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{vcluster.out
=> failover_standby_disable_compaction.out} (71%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{vcluster.out
=> standby_disable_compaction.out} (71%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.out
=> use_default_vcg_read_write.out} (65%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.out
=> use_vcg_read_write_unhealthy_node_50.out} (100%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.out
=> vcg_auto_failover.out} (100%)
copy
regression-test/data/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.out
=> vcg_auto_failover_manual_failback.out} (81%)
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_view_query_p0.out
copy regression-test/suites/{tpch_sf100_p2 =>
cloud_p0/multi_cluster/virtual_compute_group}/ddl/nation.sql (100%)
copy regression-test/suites/{nereids_rules_p0/mv/tpch =>
cloud_p0/multi_cluster/virtual_compute_group}/ddl/nation_delete.sql (100%)
copy regression-test/suites/{tpch_sf100_p2 =>
cloud_p0/multi_cluster/virtual_compute_group}/ddl/nation_load.sql (100%)
create mode 100644
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/failover_standby_disable_compaction.groovy
create mode 100644
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/load_trigger_failover.groovy
create mode 100644
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/standby_disable_compaction.groovy
copy
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/{use_default_vcg_read_write.groovy
=> use_vcg_read_write_routine_load.groovy} (57%)
copy
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/{use_default_vcg_read_write.groovy
=> use_vcg_read_write_s3_load.groovy} (57%)
copy
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/{vcg_auto_failover.groovy
=> use_vcg_read_write_unhealthy_node_50.groovy} (77%)
copy
regression-test/suites/cloud_p0/multi_cluster/virtual_compute_group/{use_vcg_read_write.groovy
=> vcg_auto_failover_manual_failback.groovy} (67%)
copy regression-test/suites/demo_p0/{docker_action.groovy =>
test_lru_persist.groovy} (64%)
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_view_query_p0.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
