This is an automated email from the ASF dual-hosted git repository.
HappenLee pushed a change to branch 4.1_performance
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 213d7221504 (14_1)PushDownJoinOnAssertNumRows support hashJoin (#66384)
omit 7ea77dc124a pick eager-agg change (#66374)
add 818c08e39b0 [improvement](cloud) Use rendezvous hashing for colocate
tablet placement (#64638) (#66066)
add 5ea5dd73e13 [fix](replica) Resolve conflicting default replica
properties (#65836) (#66189)
add 9f5480e45a2 branch-4.1: [fix](cloud) Skip unnecessary update tablet to
ms when alter table property #65983 (#66256)
add 3c53ec749fc branch-4.1: [fix](docker)(case) Restore Kerberos/Paimon
external env and fix flaky file cache cases (#66314)
add 640a61b7f6d [test](paimon) Add P0 write correctness and snapshot
reference coverage (#66325)
add af64a59827c [fix](paimon) Backport JNI reader guardrails to branch-4.1
(#66297)
add 35d7084d565 branch-4.1: [fix](auto partition) restrict dummy load
locations to current compute group #66083 (#66352)
add afe6e647e4b branch-4.1: [fix](fe) Allow retention count ALTER in cloud
mode #66197 (#66278)
add 206f2f663f8 [refactor](cloud) Rename cloud compute group metadata
class (#65817) (#66175)
add 98f95869fcd branch-4.1: [fix](be) Avoid scan executor shutdown use
after free #65220 (#66217)
add 3ef0f866976 branch-4.1: [fix](cloud) Skip BE rowset promotion for
incomplete lazy commits (#66253)
add 9a77e41d8f1 branch-4.1: [Fix](user-property) Preserve legacy user
properties during desrialization #65781 (#66098)
add dda9300565b branch-4.1: [fix](cast) Handle DST fallback in timestamp
cast monotonicity #65903 (#66186)
add af7b9936265 branch-4.1: [fix](cloud) Prevent old clients from decoding
unknown MetaServiceCode as OK (#64148) (#66353)
add a700f04fee8 branch-4.1: [fix](mv) Fix scan materialization pattern
context check #65414 (#66086)
add 61720ba4927 [branch-4.1][pick] Backport Paimon and Parquet scan
improvements (#66359)
add b531a5795d3 branch-4.1: [fix](arrow) correct the Flight SQL GetTables
schema and the TIMESTAMPTZ arrow reader #66344 (#66355)
add a9641f1770f [enhancement](scan) Optimize Parquet V2 direct predicate
filtering (#66360)
add bd234fc068d [opt](parquet) accelerate fixed binary decimal decoding
(#66379)
add a73b2106fd3 [feat](cloud) cross compute group peer read routing with
peer cache I/O optimization (#63818) (#66167)
add 46bd26c7f6f [opt](parquet) Fuse fragmented nullable selection planning
(#66397)
add b8a683bfba9 [feature](variant) Support reading Iceberg Variant from
Parquet (#66302)
add 69f4d9a9d78 branch-4.1:[feature](eager-agg) Bilateral push-down for
eager aggregation #63690 (#66341)
add e0115cac0cd [fix](catalog) safely publish Hadoop properties (#66392)
add 9b094acb7f8 branch-4.1: [improvement](fe) Evaluate protobuf 4 with
Arrow 19 (#61943) (#63720)
add 28d350eceb2 branch-4.1:[enhance](eager-agg) adjust aggregation push
down strategy #66117 (#66415)
add 1237a436994 branch-4.1: [fix](reverted index) Validate nested variant
MATCH predicates #66207 (#66394)
add 36b84b7541c branch-4.1: [fix](be) Key the index lookup map by each
surviving index in remove_index #66316 (#66393)
add 92d5ba0510f branch-4.1: [chore](session-variable) remove the unused
session variable plan_nereids_dump (#66387)
add f0d8f2b0bee branch-4.1: [fix](job scheduler) Record task finish time
on execution failure #66232 (#66271)
add b4b2f795576 [fix](variant) Preserve nested Variant append atomicity
(#66421)
add 8df195315e1 [opt](paimon) support read paimon variant type using jni
reader (#66334)
add e5c6b601955 branch-4.1: [fix](fe) Preserve effective Compute Group for
audit logs #66281 (#66373)
add 0a0dd232a25 [fix](test) handle Polaris one-shot init outside compose
wait (#66268)
add 7f81c9847f6 branch-4.1: [fix](compaction) Fix sparse vertical
compaction destination offset #66306 (#66429)
new 99fb09e61ed pick eager-agg change (#66374)
new 180ae24c883 (14_1)PushDownJoinOnAssertNumRows support hashJoin (#66384)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (213d7221504)
\
N -- N -- N refs/heads/4.1_performance (180ae24c883)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/benchmark/benchmark_bit_pack.hpp | 10 +-
be/benchmark/benchmark_column_array_view.hpp | 2 +-
.../benchmark_column_array_view_distance.hpp | 2 +-
be/benchmark/benchmark_fastunion.hpp | 2 +-
be/benchmark/benchmark_hll_merge.hpp | 2 +-
be/benchmark/benchmark_main.cpp | 18 +-
be/benchmark/binary_cast_benchmark.hpp | 2 +-
be/benchmark/parquet/AGENTS.md | 75 +-
be/benchmark/parquet/README.md | 79 +-
.../parquet/benchmark_file_scanner_expr.hpp | 118 +
be/benchmark/parquet/benchmark_parquet_kernels.hpp | 470 ++-
.../parquet/benchmark_parquet_selection.hpp | 172 +
be/benchmark/parquet/parquet_benchmark_scenarios.h | 96 +-
be/src/agent/heartbeat_server.cpp | 12 +-
be/src/cloud/cloud_backend_service.cpp | 26 +-
be/src/cloud/cloud_backend_service.h | 3 +-
be/src/cloud/cloud_cluster_info.h | 3 +
be/src/cloud/cloud_internal_service.cpp | 587 +++-
be/src/cloud/cloud_internal_service.h | 14 +
be/src/cloud/cloud_meta_mgr.cpp | 28 +-
be/src/cloud/cloud_meta_mgr.h | 4 +
be/src/cloud/cloud_snapshot_loader.cpp | 2 +
be/src/cloud/cloud_storage_engine.cpp | 8 +-
be/src/cloud/cloud_storage_engine.h | 7 +-
be/src/cloud/cloud_warm_up_manager.cpp | 525 ++-
be/src/cloud/cloud_warm_up_manager.h | 78 +-
be/src/cloud/config.cpp | 34 +-
be/src/cloud/config.h | 24 +-
be/src/common/config.cpp | 6 +
be/src/common/config.h | 8 +
be/src/common/metrics/doris_metrics.h | 4 +
be/src/core/block/block.cpp | 5 +-
.../core/column/variant_v2/column_variant_v2.cpp | 168 +-
be/src/core/column/variant_v2/column_variant_v2.h | 63 +-
.../variant_v2/column_variant_v2_read_view.cpp | 3 +
.../data_type_serde/data_type_decimal_serde.cpp | 75 +
.../data_type_timestamptz_serde.cpp | 88 +
.../data_type_serde/data_type_timestamptz_serde.h | 5 +
be/src/exec/operator/file_scan_operator.cpp | 43 +
be/src/exec/operator/olap_scan_operator.cpp | 1 +
be/src/exec/operator/scan_operator.h | 12 +
be/src/exec/scan/access_path_parser.cpp | 95 +-
be/src/exec/scan/access_path_parser.h | 6 +
be/src/exec/scan/file_scanner_v2.cpp | 49 +-
be/src/exec/scan/file_scanner_v2.h | 3 +
be/src/exec/scan/olap_scanner.cpp | 4 +-
be/src/exec/scan/olap_scanner.h | 2 +
be/src/exec/scan/parallel_scanner_builder.cpp | 50 +-
be/src/exec/scan/parallel_scanner_builder.h | 5 +-
be/src/exec/scan/scanner.cpp | 9 +
be/src/exec/scan/scanner.h | 4 +
be/src/exec/scan/scanner_context.cpp | 14 +-
be/src/exec/scan/scanner_context.h | 1 +
be/src/exec/scan/scanner_scheduler.h | 8 +-
be/src/exec/scan/split_source_connector.h | 17 +
be/src/exec/sink/viceberg_merge_sink.cpp | 26 +-
be/src/exec/sink/viceberg_merge_sink.h | 1 +
.../exprs/function/function_variant_element_v2.cpp | 45 +-
be/src/exprs/vdirect_in_predicate.h | 71 +-
be/src/format/jni/jni_data_bridge.cpp | 80 +
be/src/format/jni/jni_data_bridge.h | 9 +
be/src/format/table/paimon_jni_reader.cpp | 8 +
be/src/format_v2/column_data.h | 24 +
be/src/format_v2/column_mapper.cpp | 380 ++-
be/src/format_v2/column_mapper.h | 26 +-
be/src/format_v2/column_mapper_nested.cpp | 116 -
be/src/format_v2/column_mapper_nested.h | 2 -
be/src/format_v2/file_reader.cpp | 8 +
be/src/format_v2/file_reader.h | 85 +-
be/src/format_v2/jni/jni_table_reader.cpp | 34 +
be/src/format_v2/jni/jni_table_reader.h | 2 +
be/src/format_v2/jni/paimon_jni_reader.h | 1 +
be/src/format_v2/parquet/native_schema_desc.cpp | 434 +++
be/src/format_v2/parquet/native_schema_desc.h | 4 +
be/src/format_v2/parquet/native_schema_node.cpp | 3 +-
be/src/format_v2/parquet/parquet_column_schema.cpp | 63 +-
be/src/format_v2/parquet/parquet_column_schema.h | 5 +
be/src/format_v2/parquet/parquet_profile.cpp | 23 +
be/src/format_v2/parquet/parquet_profile.h | 15 +
be/src/format_v2/parquet/parquet_reader.cpp | 142 +-
be/src/format_v2/parquet/parquet_reader.h | 13 +
be/src/format_v2/parquet/parquet_scan.cpp | 122 +-
be/src/format_v2/parquet/parquet_scan.h | 9 +-
be/src/format_v2/parquet/parquet_statistics.cpp | 464 ++-
be/src/format_v2/parquet/parquet_statistics.h | 2 +-
be/src/format_v2/parquet/parquet_type.h | 2 +
.../parquet/reader/count_column_reader.cpp | 11 +
.../parquet/reader/native/column_chunk_reader.cpp | 59 +
.../parquet/reader/native/column_chunk_reader.h | 8 +
.../parquet/reader/native/column_reader.cpp | 82 +-
.../parquet/reader/native/column_reader.h | 1 +
be/src/format_v2/parquet/reader/native/common.cpp | 212 ++
be/src/format_v2/parquet/reader/native/common.h | 19 +
.../parquet/reader/native_column_reader.cpp | 211 +-
.../parquet/reader/native_column_reader.h | 8 +-
.../parquet/reader/variant_column_reader.cpp | 1169 +++++++
.../parquet/reader/variant_column_reader.h | 67 +
be/src/format_v2/parquet/selection_vector.h | 105 +-
be/src/format_v2/schema_projection.cpp | 5 +
be/src/format_v2/table/hudi_reader.cpp | 12 +
be/src/format_v2/table/hudi_reader.h | 1 +
be/src/format_v2/table/iceberg_reader.cpp | 91 +
be/src/format_v2/table/iceberg_reader.h | 4 +
be/src/format_v2/table/paimon_reader.cpp | 12 +
be/src/format_v2/table/paimon_reader.h | 1 +
be/src/format_v2/table_reader.cpp | 110 +
be/src/format_v2/table_reader.h | 55 +-
be/src/io/cache/block_file_cache.cpp | 14 +-
be/src/io/cache/block_file_cache_downloader.cpp | 33 +-
be/src/io/cache/block_file_cache_profile.cpp | 43 +
be/src/io/cache/block_file_cache_profile.h | 12 +
be/src/io/cache/cached_remote_file_reader.cpp | 1392 +++++---
be/src/io/cache/cached_remote_file_reader.h | 222 +-
be/src/io/cache/file_block.cpp | 82 +-
be/src/io/cache/file_block.h | 8 +
be/src/io/cache/file_cache_common.h | 6 +-
be/src/io/cache/file_cache_storage.h | 21 +
be/src/io/cache/fs_file_cache_storage.cpp | 226 +-
be/src/io/cache/fs_file_cache_storage.h | 34 +-
be/src/io/cache/mem_file_cache_storage.cpp | 105 +-
be/src/io/cache/mem_file_cache_storage.h | 23 +-
be/src/io/cache/peer_file_cache_reader.cpp | 218 +-
be/src/io/cache/peer_file_cache_reader.h | 38 +-
be/src/io/cache/shard_mem_cache.cpp | 195 ++
.../shard_mem_cache.h} | 41 +-
be/src/io/file_factory.cpp | 1 +
be/src/io/fs/file_reader.cpp | 23 +
be/src/io/fs/file_reader.h | 17 +
be/src/io/fs/local_file_reader.cpp | 46 +
be/src/io/fs/local_file_reader.h | 2 +
be/src/io/fs/local_file_writer.cpp | 43 +
be/src/io/fs/local_file_writer.h | 3 +
be/src/io/fs/remote_file_system.cpp | 9 +-
be/src/io/io_common.h | 34 +
be/src/runtime/exec_env.h | 2 +
be/src/runtime/exec_env_init.cpp | 6 +
be/src/runtime/query_context.h | 1 -
be/src/service/http/action/peer_cache_action.cpp | 267 ++
be/src/service/http/action/peer_cache_action.h | 74 +
be/src/service/http_service.cpp | 7 +
be/src/service/internal_service.cpp | 30 +
be/src/service/internal_service.h | 1 +
be/src/storage/iterator/vertical_block_reader.cpp | 4 +-
be/src/storage/rowset/beta_rowset.cpp | 2 +
be/src/storage/rowset/beta_rowset_writer.cpp | 2 +
be/src/storage/rowset/rowset_meta.cpp | 6 +-
be/src/storage/segment/column_reader.h | 4 +-
be/src/storage/tablet/tablet_schema.cpp | 4 +-
be/src/util/bthread_utils.h | 62 +
be/test/cloud/cloud_meta_mgr_test.cpp | 23 +
be/test/cloud/cloud_warm_up_manager_peer_test.cpp | 527 +++
be/test/core/column/column_variant_v2_test.cpp | 27 +
.../data_type_serde_parquet_test.cpp | 84 +
.../data_type_serde_timestamptz_test.cpp | 85 +
be/test/exec/scan/access_path_parser_test.cpp | 87 +
be/test/exec/scan/file_scanner_v2_test.cpp | 96 +
be/test/exec/scan/scanner_late_arrival_rf_test.cpp | 49 +-
be/test/exec/sink/viceberg_merge_sink_test.cpp | 38 +
be/test/exprs/expr_zonemap_filter_test.cpp | 44 +-
be/test/format/jni/jni_data_bridge_test.cpp | 89 +
be/test/format_v2/column_mapper_test.cpp | 904 ++---
be/test/format_v2/jni/jni_table_reader_test.cpp | 63 +
be/test/format_v2/jni/paimon_jni_reader_test.cpp | 16 +
.../parquet/parquet_benchmark_scenarios_test.cpp | 66 +-
.../parquet/parquet_reader_control_test.cpp | 204 ++
be/test/format_v2/parquet/parquet_reader_test.cpp | 822 +++++
be/test/format_v2/parquet/parquet_scan_test.cpp | 130 +-
be/test/format_v2/parquet/parquet_schema_test.cpp | 397 +++
.../format_v2/parquet/parquet_statistics_test.cpp | 507 ++-
.../parquet/variant_column_reader_test.cpp | 1821 ++++++++++
be/test/format_v2/table/hudi_reader_test.cpp | 45 +
be/test/format_v2/table/iceberg_reader_test.cpp | 39 +
be/test/format_v2/table/paimon_reader_test.cpp | 44 +
be/test/format_v2/table_reader_request_test.cpp | 26 +
be/test/format_v2/table_reader_test.cpp | 172 +-
be/test/io/cache/block_file_cache_test.cpp | 1302 +++++++-
.../cache/cached_remote_file_reader_peer_test.cpp | 3493 ++++++++++++++++++++
be/test/io/fs/local_file_system_test.cpp | 137 +
.../compaction/vertical_compaction_test.cpp | 112 +-
be/test/storage/tablet/tablet_schema_test.cpp | 56 +
cloud/src/meta-service/meta_service.h | 7 +
cloud/src/meta-service/meta_service_helper.h | 40 +-
cloud/src/meta-service/meta_service_txn.cpp | 6 +
cloud/test/meta_service_helper_test.cpp | 250 ++
cloud/test/meta_service_http_test.cpp | 9 +-
cloud/test/meta_service_test.cpp | 8 +
cloud/test/txn_lazy_commit_test.cpp | 111 +
conf/be.conf | 2 +-
conf/fe.conf | 2 +-
docker/thirdparties/docker-compose/hive/README.md | 1 +
.../thirdparties/docker-compose/hive/README_ZH.md | 1 +
.../docker-compose/hive/scripts/hive-module-lib.sh | 26 +-
.../hive/scripts/init-hive-baseline.sh | 8 +-
.../create_preinstalled_scripts/paimon/run13.sql | 17 +
.../kerberos/entrypoint-hive-master.sh | 11 +-
docker/thirdparties/run-thirdparties-docker.sh | 6 +-
.../apache/doris/common/jni/vec/ColumnType.java | 55 +-
.../apache/doris/common/jni/vec/ColumnValue.java | 8 +
.../apache/doris/common/jni/vec/VectorColumn.java | 31 +
.../doris/common/jni/vec/VectorColumnVariant.java | 214 ++
.../apache/doris/common/jni/JniScannerTest.java | 12 +
.../common/jni/vec/VectorColumnVariantTest.java | 128 +
.../org/apache/doris/paimon/PaimonColumnValue.java | 146 +-
.../org/apache/doris/paimon/PaimonJniScanner.java | 419 ++-
.../org/apache/doris/paimon/PaimonTypeUtils.java | 6 +
.../apache/doris/paimon/PaimonColumnValueTest.java | 251 +-
.../apache/doris/paimon/PaimonJniScannerTest.java | 534 ++-
.../rolemapping/UnifiedRoleMappingCelEngine.java | 2 +-
.../java/org/apache/doris/catalog/VariantType.java | 6 +
.../main/java/org/apache/doris/common/Config.java | 6 +
.../apache/doris/alter/SchemaChangeHandler.java | 26 +-
.../org/apache/doris/analysis/TableScanParams.java | 10 +
.../apache/doris/catalog/ColocateTableIndex.java | 3 +
.../doris/catalog/InternalSchemaInitializer.java | 8 +-
.../org/apache/doris/catalog/TableProperty.java | 28 +-
.../cloud/alter/CloudSchemaChangeHandler.java | 15 +-
.../doris/cloud/catalog/BalanceTypeEnum.java | 3 +-
.../cloud/catalog/CloudColocatePlacement.java | 64 +
...omputeGroup.java => CloudComputeGroupMeta.java} | 28 +-
.../cloud/catalog/CloudInstanceStatusChecker.java | 30 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 76 +-
.../doris/cloud/catalog/CloudTabletRebalancer.java | 16 +-
.../apache/doris/cloud/rpc/MetaServiceClient.java | 70 +-
.../doris/cloud/system/CloudSystemInfoService.java | 187 +-
.../transaction/CloudGlobalTransactionMgr.java | 6 +
.../org/apache/doris/datasource/CatalogMgr.java | 25 +-
.../apache/doris/datasource/CatalogProperty.java | 20 +-
.../apache/doris/datasource/ExternalCatalog.java | 18 +-
.../doris/datasource/iceberg/IcebergUtils.java | 33 +-
.../datasource/iceberg/source/IcebergScanNode.java | 40 +
.../PaimonLatestSnapshotProjectionLoader.java | 64 +-
.../paimon/PaimonPartitionInfoLoader.java | 2 +
.../apache/doris/datasource/mvcc/MvccTable.java | 25 +
.../doris/datasource/mvcc/MvccTableInfo.java | 25 +-
.../org/apache/doris/datasource/mvcc/MvccUtil.java | 22 +-
.../datasource/paimon/PaimonExternalCatalog.java | 34 +-
.../datasource/paimon/PaimonExternalMetaCache.java | 22 +
.../datasource/paimon/PaimonExternalTable.java | 85 +-
.../datasource/paimon/PaimonReaderOptions.java | 315 ++
.../doris/datasource/paimon/PaimonScanParams.java | 98 +-
.../datasource/paimon/PaimonSysExternalTable.java | 144 +-
.../datasource/paimon/PaimonTableDecorators.java | 41 +
.../apache/doris/datasource/paimon/PaimonUtil.java | 5 +-
.../doris/datasource/paimon/PaimonUtils.java | 19 +
.../datasource/paimon/source/PaimonScanNode.java | 77 +-
.../datasource/paimon/source/PaimonSource.java | 34 +-
.../metastore/AbstractPaimonProperties.java | 84 +-
.../org/apache/doris/job/task/AbstractTask.java | 1 +
.../mysql/privilege/CommonUserProperties.java | 4 +-
.../org/apache/doris/nereids/StatementContext.java | 138 +-
.../glue/translator/PhysicalPlanTranslator.java | 1 +
.../doris/nereids/rules/analysis/BindRelation.java | 13 +-
.../doris/nereids/rules/analysis/BindSink.java | 3 +
...terializedViewAggregateOnNoneAggregateRule.java | 4 +-
...IcebergMergeSinkToPhysicalIcebergMergeSink.java | 1 +
.../rewrite/AccessPathExpressionCollector.java | 9 +
.../rules/rewrite/AccessPathPlanCollector.java | 7 +
.../rules/rewrite/CheckMatchExpression.java | 26 +-
.../nereids/rules/rewrite/NestedColumnPruning.java | 6 +
.../nereids/rules/rewrite/SlotTypeReplacer.java | 5 +
.../doris/nereids/trees/expressions/Cast.java | 50 +-
.../plans/commands/AlterComputeGroupCommand.java | 4 +-
.../trees/plans/commands/ExecuteCommand.java | 41 +-
.../trees/plans/commands/IcebergMergeCommand.java | 2 +
.../trees/plans/commands/IcebergUpdateCommand.java | 1 +
.../trees/plans/commands/ShowClustersCommand.java | 8 +-
.../trees/plans/commands/WarmUpClusterCommand.java | 6 +-
.../plans/commands/merge/MergeIntoCommand.java | 8 +
.../trees/plans/logical/LogicalFileScan.java | 3 +-
.../plans/logical/LogicalIcebergMergeSink.java | 36 +-
.../plans/physical/PhysicalIcebergMergeSink.java | 26 +-
.../org/apache/doris/persist/gson/GsonUtils.java | 6 +-
.../org/apache/doris/planner/IcebergMergeSink.java | 18 +-
.../org/apache/doris/planner/OlapTableSink.java | 12 +-
.../java/org/apache/doris/qe/AuditLogHelper.java | 12 +-
.../java/org/apache/doris/qe/ConnectContext.java | 11 +
.../java/org/apache/doris/qe/SessionVariable.java | 35 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 6 +
.../apache/doris/service/FrontendServiceImpl.java | 16 +
.../service/arrowflight/FlightSqlSchemaHelper.java | 103 +-
.../java/org/apache/doris/system/HeartbeatMgr.java | 4 +
.../org/apache/doris/system/SystemInfoService.java | 4 +
.../tablefunction/PaimonTableValuedFunction.java | 28 +-
.../doris/alter/InternalSchemaAlterTest.java | 5 +
.../apache/doris/catalog/TablePropertyTest.java | 113 +
.../cloud/WarmUpClusterOnTablesParseTest.java | 20 +-
.../cloud/catalog/CloudColocatePlacementTest.java | 232 ++
...oupTest.java => CloudComputeGroupMetaTest.java} | 135 +-
.../catalog/CloudInstanceStatusCheckerTest.java | 23 +-
.../doris/cloud/catalog/CloudReplicaTest.java | 346 ++
.../doris/cloud/rpc/MetaServiceProxyTest.java | 170 +-
.../cloud/system/CloudSystemInfoServiceTest.java | 104 +-
.../transaction/CloudGlobalTransactionMgrTest.java | 77 +
.../apache/doris/datasource/CatalogMgrTest.java | 191 ++
.../doris/datasource/CatalogPropertyTest.java | 137 +
.../doris/datasource/iceberg/IcebergUtilsTest.java | 29 +
.../iceberg/source/IcebergScanNodeTest.java | 114 +-
.../paimon/PaimonExternalMetaCacheTest.java | 273 +-
.../datasource/paimon/PaimonExternalTableTest.java | 505 ++-
.../datasource/paimon/PaimonReaderOptionsTest.java | 279 ++
.../datasource/paimon/PaimonScanParamsTest.java | 92 +-
.../doris/datasource/paimon/PaimonUtilTest.java | 10 +
.../paimon/source/PaimonScanNodeTest.java | 108 +
.../datasource/paimon/source/PaimonSourceTest.java | 33 +
.../metastore/AbstractPaimonPropertiesTest.java | 116 +-
.../doris/job/base/AbstractJobStatusTest.java | 23 +
.../doris/mysql/privilege/CloudAuthTest.java | 8 +-
.../mysql/privilege/CommonUserPropertiesTest.java | 86 +
.../nereids/StatementContextMvccSnapshotTest.java | 277 ++
.../apache/doris/nereids/StatementContextTest.java | 36 +-
.../rules/rewrite/CheckMatchExpressionTest.java | 14 +
.../rules/rewrite/PruneNestedColumnTest.java | 19 +
.../rules/rewrite/VariantPruningLogicTest.java | 19 +
.../trees/expressions/CastMonotonicityTest.java | 126 +
.../commands/AlterComputeGroupCommandTest.java | 4 +-
.../trees/plans/commands/ExecuteCommandTest.java | 114 +
.../plans/commands/IcebergUpdateCommandTest.java | 27 +
.../gson/GsonProtobufCompatibilityTest.java} | 35 +-
.../apache/doris/planner/IcebergMergeSinkTest.java | 12 +
.../apache/doris/planner/OlapTableSinkTest.java | 64 +
.../org/apache/doris/qe/AuditLogHelperTest.java | 9 +
.../FlightSqlSchemaHelperArrowTypeTest.java | 209 ++
.../doris/statistics/AnalysisManagerTest.java | 107 +
.../PaimonTableValuedFunctionTest.java | 87 +
fe/pom.xml | 10 +-
gensrc/proto/cloud.proto | 28 +-
gensrc/proto/internal_service.proto | 13 +
gensrc/thrift/BackendService.thrift | 1 +
gensrc/thrift/DataSinks.thrift | 2 +
gensrc/thrift/HeartbeatService.thrift | 1 +
gensrc/thrift/PaloInternalService.thrift | 6 +-
gensrc/thrift/Types.thrift | 1 +
.../test_cloud_colocate_consistent_hash_join.out | 5 +
.../test_timestamptz_rf_partition_prune_dst.out | 13 +
.../iceberg/iceberg_variant_shredded.parquet | Bin 0 -> 34528 bytes
...f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json | 74 +
.../b7958052-e154-425f-8850-f0011d0272c5-m0.avro | Bin 0 -> 7778 bytes
...823-1-b7958052-e154-425f-8850-f0011d0272c5.avro | Bin 0 -> 4758 bytes
.../iceberg/test_iceberg_varbinary.out | 1 +
.../iceberg/test_iceberg_variant_read.out | 178 +
.../paimon/test_paimon_catalog_variant.out | 26 +
.../paimon/test_paimon_jni_reader_guardrails.out | 44 +
regression-test/pipeline/p0/conf/be.conf | 2 +-
regression-test/pipeline/p0/conf/fe.conf | 2 +-
...test_audit_log_hint_compute_group_docker.groovy | 64 +
.../cloud_p0/balance/test_balance_warm_up.groovy | 25 +-
.../test_balance_warm_up_use_peer_cache.groovy | 85 +-
...e_warm_up_with_compaction_use_peer_cache.groovy | 46 +-
.../balance/test_cross_cg_bvar_metrics.groovy | 280 ++
.../test_cross_cg_candidate_management.groovy | 377 +++
.../balance/test_cross_cg_inverted_index.groovy | 195 ++
.../test_cross_cg_peer_read_colocate.groovy | 309 ++
.../balance/test_cross_cg_peer_read_core.groovy | 351 ++
.../balance/test_cross_cg_server_fill.groovy | 319 ++
...test_cross_cg_server_fill_dynamic_config.groovy | 305 ++
.../test_cross_cg_server_fill_three_cg.groovy | 272 ++
.../balance/test_peer_cache_http_api.groovy | 311 ++
.../test_peer_fetch_concurrent_same_block.groovy | 230 ++
.../balance/test_peer_fetch_pool_isolation.groovy | 244 ++
.../balance/test_peer_read_async_warmup.groovy | 44 +-
.../read_from_peer/test_read_from_peer.groovy | 188 +-
...test_cloud_colocate_consistent_hash_join.groovy | 67 +
...st_cloud_alter_partition_retention_count.groovy | 61 +
.../test_timestamptz_rf_partition_prune_dst.groovy | 90 +
.../cache/test_file_cache_query_limit.groovy | 21 +-
.../cache/test_file_cache_statistics.groovy | 65 +-
.../iceberg/test_iceberg_varbinary.groovy | 20 +-
.../iceberg/test_iceberg_variant_read.groovy | 1049 ++++++
.../paimon/test_paimon_catalog_variant.groovy | 91 +
.../paimon/test_paimon_jdbc_catalog.groovy | 211 +-
.../test_paimon_jni_reader_guardrails.groovy | 231 ++
.../external_table_p2/es/test_es_query.groovy | 7 +-
...paimon_write_key_dynamic_memory_negative.groovy | 160 +
.../test_paimon_write_snapshot_refs.groovy | 192 ++
.../test_paimon_write_source_models.groovy | 294 ++
.../test_paimon_write_thread_lifecycle.groovy | 154 +
.../auto_partition/test_auto_new_recycle.groovy | 22 +
.../search/test_disable_root_variant_match.groovy | 22 +-
.../suites/variant_p0/variant_with_rowstore.groovy | 7 +-
run-be-ut.sh | 1 +
run-regression-test.sh | 7 +-
381 files changed, 35533 insertions(+), 2943 deletions(-)
create mode 100644 be/benchmark/parquet/benchmark_file_scanner_expr.hpp
create mode 100644 be/benchmark/parquet/benchmark_parquet_selection.hpp
create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.cpp
create mode 100644 be/src/format_v2/parquet/reader/variant_column_reader.h
create mode 100644 be/src/io/cache/shard_mem_cache.cpp
copy be/src/io/{fs/local_file_writer.h => cache/shard_mem_cache.h} (52%)
create mode 100644 be/src/service/http/action/peer_cache_action.cpp
create mode 100644 be/src/service/http/action/peer_cache_action.h
create mode 100644 be/src/util/bthread_utils.h
create mode 100644 be/test/cloud/cloud_warm_up_manager_peer_test.cpp
create mode 100644 be/test/format/jni/jni_data_bridge_test.cpp
create mode 100644 be/test/format_v2/parquet/variant_column_reader_test.cpp
create mode 100644 be/test/io/cache/cached_remote_file_reader_peer_test.cpp
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
create mode 100644
fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/VectorColumnVariant.java
create mode 100644
fe/be-java-extensions/java-common/src/test/java/org/apache/doris/common/jni/vec/VectorColumnVariantTest.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudColocatePlacement.java
rename
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/{ComputeGroup.java =>
CloudComputeGroupMeta.java} (89%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonReaderOptions.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonTableDecorators.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/catalog/TablePropertyTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudColocatePlacementTest.java
rename
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/{ComputeGroupTest.java
=> CloudComputeGroupMetaTest.java} (61%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudReplicaTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogPropertyTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonReaderOptionsTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/CommonUserPropertiesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextMvccSnapshotTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/CastMonotonicityTest.java
copy fe/fe-core/src/{main/java/org/apache/doris/datasource/mvcc/MvccTable.java
=> test/java/org/apache/doris/persist/gson/GsonProtobufCompatibilityTest.java}
(51%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/service/arrowflight/FlightSqlSchemaHelperArrowTypeTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/tablefunction/PaimonTableValuedFunctionTest.java
create mode 100644
regression-test/data/cloud_p0/multi_cluster/test_cloud_colocate_consistent_hash_join.out
create mode 100644
regression-test/data/datatype_p0/timestamptz/test_timestamptz_rf_partition_prune_dst.out
create mode 100644
regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded.parquet
create mode 100644
regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/00002-5d3f3ae6-7100-4eb0-a42e-e52ddc62d9e3.metadata.json
create mode 100644
regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/b7958052-e154-425f-8850-f0011d0272c5-m0.avro
create mode 100644
regression-test/data/external_table_p0/iceberg/iceberg_variant_shredded_table/snap-5420489606554005823-1-b7958052-e154-425f-8850-f0011d0272c5.avro
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_variant_read.out
create mode 100644
regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
create mode 100644
regression-test/data/external_table_p0/paimon/test_paimon_jni_reader_guardrails.out
create mode 100644
regression-test/suites/audit/test_audit_log_hint_compute_group_docker.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_bvar_metrics.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_candidate_management.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_inverted_index.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_peer_read_colocate.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_peer_read_core.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_server_fill.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_server_fill_dynamic_config.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_cross_cg_server_fill_three_cg.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_peer_cache_http_api.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_peer_fetch_concurrent_same_block.groovy
create mode 100644
regression-test/suites/cloud_p0/balance/test_peer_fetch_pool_isolation.groovy
create mode 100644
regression-test/suites/cloud_p0/multi_cluster/test_cloud_colocate_consistent_hash_join.groovy
create mode 100644
regression-test/suites/cloud_p0/partition/test_cloud_alter_partition_retention_count.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_rf_partition_prune_dst.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_variant_read.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_jni_reader_guardrails.groovy
create mode 100644
regression-test/suites/paimon_write/test_paimon_write_key_dynamic_memory_negative.groovy
create mode 100644
regression-test/suites/paimon_write/test_paimon_write_snapshot_refs.groovy
create mode 100644
regression-test/suites/paimon_write/test_paimon_write_source_models.groovy
create mode 100644
regression-test/suites/paimon_write/test_paimon_write_thread_lifecycle.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
