This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch mor_value_predicate_pushdown_control
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 0bfacc2b38c Merge branch 'master' into
mor_value_predicate_pushdown_control
omit dbfbb499d23 [feature](scan) Add read_mor_as_dup_tables session
variable to read MOR tables without merge
omit 8079739554d [feature](scan) Implement BE-side value predicate pushdown
for MOR tables
omit a06783e5ffc [feature](scan) Add session variable to control value
predicate pushdown for MOR tables
add 6fac5f248e9 [chore](planner) clear code under analysis package (#60844)
add 0c5a7fcc2ae [fix](cloud) Fix tablets permanently invisible to
compaction scheduler due to race condition in `CloudTabletMgr::get_tablet`
(#60832)
add 567661f0c06 [feature](maxcompute)support maxcompute ram_role_arn and
ecs_ram_role authenticate. (#60649)
add 151401506a2 [refine](function)Some functions are not using the default
const handling logic. (#60810)
add e0578fe620c [fix](iceberg) Invalidate related-table cache on external
table refresh (#60867)
add 8a076456db3 [test](search) Add regression test for wildcard query on
variant subcolumns with multi-index (#60834)
add a4091ad5c65 [fix](search) fix MATCH_ALL_DOCS losing occur attribute in
multi-field expansion (#60873)
add c981585f367 [fix](case)fix some external regression test. (#60744)
add 056c8d0fc34 [feature](storage) Add OzoneProperties to support Apache
Ozone (#60809)
add 8cb5a4210f7 [fix](variant) Disable strict mode for variant internal
cast to fix INSERT INTO SELECT returning all NULLs (#60881)
add 37253328bea [feat](http) Add sync and export cloud meta API (#60739)
add 82de8f605f8 [feature](docker)add docker base slim image (#60458)
add 8d9d26c253b [feat](catalog) Support include_table_list property and
refactor lower_case_database/table_names for external catalog (#60580)
add 5434070b634 [fix](cloud) change cloud FE getVersion rpc metric to
getTableVersion and getPartitionVersion (#60745)
add b64445effbf [feat](fe) add recovery_journal_id to truncate bdbje logs
(#60738)
add 69262cb5902 [fix](cache) Fix null pointer dereference in prefetch
dryrun fallback path (#60870)
add ee37bd66691 [improve](cloud) reduce fe call get_version to
meta_service (#60467)
add 79d57ec5e6c [feat](Variant) Support NestedGroup for variant columns
with search DSL NESTED clause (#60847)
add 61a3d3b2a87 [opt](s3) Skip S3 listing for deterministic file paths
using HEAD requests (#60414)
add 3e8112219ed [fix](ci) allow stream load cost more time (#60909)
add 6828cb125ea [fix](web) FE startup fail due to websocket startup
(#60369)
add 6d4e218a2e3 [fix](search) inject MATCH_ALL_DOCS for multi-MUST_NOT
queries in lucene mode (#60891)
add a81c501905f [opt](build) enable maven cache and check maven version
(#60819)
add 04c59ea291d [fix](recycler) Fix metrics statistics issues in recycler
workflows (#60697)
add be53e027072 [Enhancement](recycler) Add fault injection config for
s3/azure delete rate limit (#60882)
add 1e20b8a87df [fix](insert-overwrite) skip deleted partitions before
replace to avoid DdlException (#60914)
add e95a17ef809 [Enhancement](udf) clear cache when droping function
(#60630)
add e66e76fabe9 [feat](common) Add DORIS_CHECK macro for precondition
assertions (#60935)
add 8364e8a899c [fix](cloud) fix active-priority rebalancer starvation
(#60835)
add 9d2f65e288b [feat](iceberg) support aliyun dlf iceberg rest catalog
(#60796)
add 9ec8abbfd35 [fix](paimon) add rest external catalog for gson
comaptiblity (#60916)
add 38132964998 [fix](nereids)Fix decompose repeat nest rewrite need to
derive stats after construct cteProducer (#60811)
add 0eaac8c104f [fix](inverted index) implement phrase-level BM25 scoring
with phrase frequency (#60331)
add 69c5e3908ee [fix](pipeline) Fix COW violation in block column
operations when filtering and clearing (PART I) (#60936)
add c53d9cc3998 [fix](regression) stabilize parquet_meta tvf assertions
across s3 buckets (#60938)
add 17a81a419a4 [check](column)const is only allowed at the top level.
(#60578)
add d437efea333 [chore](thirdparty)(paimon-cpp) reuse Doris Arrow stack
and isolate external headers (#60946)
add 2dbb0c734fc [fix](priv) skip catalog priv check if using customized
access controller (#60945)
add cc9df57cee7 [fix](be) Default to the Default chain when S3 role_arn is
set without provider_type (#60822)
add ae18f177d2f [feat](agent) Add AGENTS.md and code review skill
documentation (#60963)
add fe4934f8d54 [feat](CI) Add OpenCode code review workflow with
AGENTS.md and skills (#60967)
add 016fd06046d [fix](csv reader) fix incorrect column parsing when using
enclose for CSV files with UTF-8 BOM (#60864)
add 5715a8518aa [fix](jdbc) Fix PG array null detection and SQL Server
date format pushdown (#60954)
add 463718b8fb8 [opt](maxcompute) Optimize split generation for LIMIT
queries with partition equality predicates (#60895)
add bd2210ae598 [Enhancement](CI) Optimize review skills and and code
owner for them (#60969)
add c6ee9cf25ea [fix](compile) auto-regenerate hadoop-deps/lib when
missing due to Ma… (#60964)
add 6f6768c7c51 [opt](in list) remove redundant condition in camp_field
(#60912)
add f0fba98184c [bug](udf) udf should cache classloader in static load
(#60709)
add 9186b6213f5 [opt](query cache) Limit instance count for non-scan
fragments when query cache is enabled (#60878)
add adca4d6c51f [fix](stream-load) Fix LZ4 compressed stream load
decompress made no progress error (#60852)
add 2f01d606c6f [chore](analysis) clean fe analysis package code (#60961)
add 163e02b5ca3 [Bug](join) fix left_semi_direct_return_opt get wrong
result with local shuffle (#60952)
add 55b99a7eda9 [fix](point-query) Fix point query ignoring session
timezone for functions like from_unixtime (#60913)
add 41231c73585 [Feature](iceberg) Support schema change for complex types
in Iceberg external tables (#60169)
add 85d9e5a1e47 [fix](metrics) Fix wrong scanner metrics (#60966)
add 59e5f6bbf7f [Opt](cloud) Support packed file for delete bitmap storage
(#60411)
add 7b417605e27 [fix](test) make hive compress split assertion BE-aware
(#60947)
add 5e5179e584e [feature](scan) Add session variable to control value
predicate pushdown for MOR tables
add 4a0c30ea7f0 [feature](scan) Implement BE-side value predicate pushdown
for MOR tables
add 89d3715b3aa [feature](scan) Add read_mor_as_dup_tables session
variable to read MOR tables without merge
add 1c6db7775dd [fix](scan) Refactor table list matching to use
TableNameInfo-style component parsing
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (0bfacc2b38c)
\
N -- N -- N refs/heads/mor_value_predicate_pushdown_control
(1c6db7775dd)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.claude/skills/code-review/SKILL.md | 841 +++++++++++++++++++++
.github/CODEOWNERS | 2 +
.github/workflows/opencode-review.yml | 115 +++
AGENTS.md | 50 ++
be/src/agent/task_worker_pool.cpp | 14 +-
be/src/cloud/cloud_meta_mgr.cpp | 56 +-
be/src/cloud/cloud_tablet_mgr.cpp | 50 +-
be/src/cloud/delete_bitmap_file_reader.cpp | 40 +-
be/src/cloud/delete_bitmap_file_reader.h | 13 +
be/src/cloud/delete_bitmap_file_writer.cpp | 64 +-
be/src/cloud/delete_bitmap_file_writer.h | 17 +-
be/src/common/config.cpp | 4 +
be/src/common/config.h | 3 +
be/src/common/status.h | 8 +
be/src/io/cache/cached_remote_file_reader.cpp | 29 +-
be/src/io/fs/packed_file_manager.cpp | 4 +-
be/src/io/fs/packed_file_manager.h | 1 +
be/src/olap/in_list_predicate.h | 26 +-
be/src/olap/rowset/segment_creator.cpp | 18 +-
be/src/olap/rowset/segment_v2/column_reader.cpp | 2 +
.../olap/rowset/segment_v2/column_reader_cache.cpp | 1 -
be/src/olap/rowset/segment_v2/column_writer.h | 7 +
.../olap/rowset/segment_v2/index_file_writer.cpp | 2 +-
.../inverted_index/query/phrase_query.cpp | 52 +-
.../segment_v2/inverted_index/query/phrase_query.h | 3 +-
.../query/phrase_query/exact_phrase_matcher.cpp | 9 +
.../query/phrase_query/exact_phrase_matcher.h | 1 +
.../phrase_query/ordered_sloppy_phrase_matcher.cpp | 13 +
.../phrase_query/ordered_sloppy_phrase_matcher.h | 2 +
.../query/phrase_query/sloppy_phrase_matcher.cpp | 13 +
.../query/phrase_query/sloppy_phrase_matcher.h | 2 +
.../inverted_index/util/docid_set_iterator.h | 7 +
.../segment_v2/inverted_index/util/mock_iterator.h | 2 +
.../inverted_index/util/union_term_iterator.h | 6 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 23 +-
.../variant/hierarchical_data_iterator.cpp | 17 +-
.../rowset/segment_v2/variant/nested_group_path.h | 68 ++
.../segment_v2/variant/nested_group_provider.cpp | 123 +++
.../segment_v2/variant/nested_group_provider.h | 228 ++++++
.../segment_v2/variant/variant_column_reader.cpp | 482 +++++++++---
.../segment_v2/variant/variant_column_reader.h | 99 ++-
.../variant/variant_column_writer_impl.cpp | 131 +++-
.../variant/variant_column_writer_impl.h | 13 +-
.../variant/variant_external_meta_reader.cpp | 1 -
.../rowset/segment_v2/variant/variant_statistics.h | 71 +-
.../exec/nested_loop_join_probe_operator.h | 11 +-
be/src/runtime/descriptors.cpp | 9 +
be/src/runtime/descriptors.h | 10 +-
be/src/runtime/exec_env.h | 4 +
be/src/runtime/runtime_state.h | 5 +
be/src/runtime/user_function_cache.cpp | 51 +-
be/src/runtime/user_function_cache.h | 3 +
be/src/runtime_filter/runtime_filter_producer.cpp | 13 +-
be/src/runtime_filter/runtime_filter_producer.h | 18 +-
.../runtime_filter_producer_helper.cpp | 2 +-
be/src/runtime_filter/runtime_filter_wrapper.cpp | 3 +-
be/src/service/point_query_executor.cpp | 4 +
be/src/udf/python/python_server.cpp | 73 ++
be/src/udf/python/python_server.h | 3 +
be/src/udf/python/python_server.py | 164 +++-
be/src/util/s3_util.cpp | 3 +-
be/src/vec/columns/column.cpp | 11 +
be/src/vec/columns/column.h | 5 +
be/src/vec/columns/column_array.cpp | 4 +-
be/src/vec/columns/column_map.cpp | 1 +
be/src/vec/columns/column_nullable.cpp | 4 +-
be/src/vec/columns/column_struct.cpp | 1 +
be/src/vec/columns/column_variant.cpp | 4 +-
be/src/vec/columns/column_variant.h | 1 +
be/src/vec/common/variant_util.cpp | 60 +-
be/src/vec/common/variant_util.h | 1 +
be/src/vec/core/block.cpp | 15 +-
be/src/vec/exec/format/csv/csv_reader.cpp | 18 +-
be/src/vec/exec/format/csv/csv_reader.h | 2 +
.../file_reader/new_plain_text_line_reader.cpp | 9 +-
.../file_reader/new_plain_text_line_reader.h | 9 +
.../exec/format/table/max_compute_jni_reader.cpp | 34 +-
be/src/vec/exec/scan/olap_scanner.cpp | 30 +-
be/src/vec/exprs/vexpr.cpp | 9 +
be/src/vec/exprs/vexpr.h | 6 +-
be/src/vec/exprs/vexpr_context.cpp | 15 +-
be/src/vec/exprs/vexpr_context.h | 54 +-
be/src/vec/exprs/vsearch.cpp | 104 ++-
be/src/vec/functions/binary_arithmetic.h | 2 +
be/src/vec/functions/cast/cast_to_variant.h | 8 +
be/src/vec/functions/divide.cpp | 2 +
be/src/vec/functions/function_bit.cpp | 2 +
be/src/vec/functions/function_bit_shift.cpp | 2 +
be/src/vec/functions/function_search.cpp | 352 ++++++++-
be/src/vec/functions/function_search.h | 18 +
be/src/vec/functions/functions_comparison.h | 6 -
be/src/vec/functions/int_div.cpp | 2 +
be/src/vec/functions/modulo.cpp | 2 +
be/src/vec/functions/multiply.cpp | 2 +
be/src/vec/json/json_parser.cpp | 30 +-
be/src/vec/json/path_in_data.cpp | 24 +
be/src/vec/json/path_in_data.h | 11 +
be/src/vec/olap/olap_data_convertor.cpp | 4 +
be/src/vec/runtime/vparquet_transformer.cpp | 26 +-
.../sink/writer/maxcompute/vmc_table_writer.cpp | 4 +-
be/test/cloud/cloud_tablet_mgr_test.cpp | 181 +++++
be/test/io/s3_client_factory_test.cpp | 243 +++++-
.../segment_v2/external_col_meta_util_test.cpp | 1 -
.../query/phrase_query/phrase_freq_test.cpp | 559 ++++++++++++++
.../segment_v2/nested_group_path_filter_test.cpp | 159 ++++
.../rowset/segment_v2/nested_group_path_test.cpp | 113 +++
.../segment_v2/nested_group_provider_test.cpp | 237 ++++++
be/test/vec/columns/column_array_test.cpp | 73 +-
.../column_check_const_only_in_top_level_test.cpp | 311 ++++++++
be/test/vec/exprs/try_cast_expr_test.cpp | 1 +
.../exprs/vexpr_evalute_inverted_index_test.cpp | 12 +-
be/test/vec/exprs/vsearch_expr_test.cpp | 46 +-
.../function/cast/function_variant_cast_test.cpp | 148 ++++
.../vec/function/function_search_nested_test.cpp | 382 ++++++++++
be/test/vec/function/function_search_test.cpp | 8 +
be/test/vec/jsonb/json_parser_test.cpp | 24 +-
bin/start_fe.sh | 12 +-
build.sh | 51 +-
cloud/src/common/bvars.cpp | 5 +
cloud/src/common/bvars.h | 3 +
cloud/src/common/config.h | 4 +
cloud/src/meta-service/meta_service.h | 14 +-
cloud/src/meta-service/meta_service_partition.cpp | 141 +++-
cloud/src/meta-service/meta_service_txn.cpp | 240 +++++-
cloud/src/recycler/azure_obj_client.cpp | 6 +
cloud/src/recycler/recycler.cpp | 286 ++++++-
cloud/src/recycler/recycler.h | 19 +-
cloud/src/recycler/s3_obj_client.cpp | 6 +
cloud/test/meta_service_test.cpp | 54 +-
cloud/test/meta_service_versioned_read_test.cpp | 39 +
cloud/test/txn_lazy_commit_test.cpp | 38 +
docker/runtime/base-image/Dockerfile_base_slim | 64 ++
.../postgresql/init/02-create-table.sql | 7 +-
.../docker-compose/postgresql/init/04-insert.sql | 5 +
.../sqlserver/init/03-create-table.sql | 7 +
.../docker-compose/sqlserver/init/04-insert.sql | 9 +
.../apache/doris/common/jni/utils/ExpiringMap.java | 19 +-
.../doris/common/jni/utils/UdfClassCache.java | 24 +-
.../java/org/apache/doris/udf/BaseExecutor.java | 27 +-
.../java/org/apache/doris/udf/UdafExecutor.java | 7 +-
.../java/org/apache/doris/udf/UdfExecutor.java | 10 +-
.../doris/maxcompute/MaxComputeJniScanner.java | 12 +-
.../doris/maxcompute/MaxComputeJniWriter.java | 12 +-
fe/fe-common/pom.xml | 18 +
.../main/java/org/apache/doris/common/Config.java | 57 ++
.../org/apache/doris/common/FormatOptions.java | 0
.../doris/common/maxcompute}/MCProperties.java | 19 +-
.../apache/doris/common/maxcompute/MCUtils.java | 78 ++
.../org/apache/doris/nereids/search/SearchLexer.g4 | 21 +-
.../apache/doris/nereids/search/SearchParser.g4 | 7 +-
.../src/main/java/org/apache/doris/DorisFE.java | 12 +
.../org/apache/doris/analysis/AggregateInfo.java | 10 +-
.../java/org/apache/doris/analysis/BrokerDesc.java | 18 -
.../apache/doris/analysis/ChannelDescription.java | 41 -
.../org/apache/doris/analysis/CopyFromParam.java | 261 +------
.../org/apache/doris/analysis/DataDescription.java | 371 ---------
.../java/org/apache/doris/analysis/DbName.java | 6 +-
.../apache/doris/analysis/DefaultValueExprDef.java | 3 -
.../org/apache/doris/analysis/DescriptorTable.java | 18 +-
.../apache/doris/analysis/DistributionDesc.java | 8 -
.../org/apache/doris/analysis/EncryptKeyName.java | 4 -
.../apache/doris/analysis/FunctionCallExpr.java | 72 +-
.../org/apache/doris/analysis/FunctionName.java | 28 -
.../org/apache/doris/analysis/GroupingInfo.java | 5 +-
.../apache/doris/analysis/ImportColumnDesc.java | 12 -
.../apache/doris/analysis/ImportColumnsStmt.java | 5 -
.../apache/doris/analysis/ImportDeleteOnStmt.java | 5 -
.../apache/doris/analysis/ImportSequenceStmt.java | 5 -
.../org/apache/doris/analysis/ImportWhereStmt.java | 5 -
.../org/apache/doris/analysis/InsertTarget.java | 39 -
.../java/org/apache/doris/analysis/KeysDesc.java | 108 +--
.../org/apache/doris/analysis/LargeIntLiteral.java | 2 -
.../org/apache/doris/analysis/LimitElement.java | 31 +-
.../apache/doris/analysis/ListPartitionDesc.java | 14 -
.../java/org/apache/doris/analysis/LockTable.java | 65 --
.../org/apache/doris/analysis/OrderByElement.java | 48 +-
.../org/apache/doris/analysis/OutFileClause.java | 3 -
.../org/apache/doris/analysis/PartitionDesc.java | 121 ---
.../apache/doris/analysis/RangePartitionDesc.java | 16 -
.../org/apache/doris/analysis/RedirectStatus.java | 12 +-
.../org/apache/doris/analysis/ResourceDesc.java | 11 -
.../org/apache/doris/analysis/SearchPredicate.java | 8 +-
.../java/org/apache/doris/analysis/Separator.java | 2 +-
.../java/org/apache/doris/analysis/SetVar.java | 8 -
.../java/org/apache/doris/analysis/SortInfo.java | 1 +
.../org/apache/doris/analysis/StageAndPattern.java | 21 +-
.../org/apache/doris/analysis/StatementBase.java | 61 --
.../org/apache/doris/analysis/TupleDescriptor.java | 25 -
.../java/org/apache/doris/catalog/ColumnType.java | 65 +-
.../main/java/org/apache/doris/catalog/Env.java | 16 +
.../org/apache/doris/catalog/JdbcResource.java | 16 +-
.../java/org/apache/doris/catalog/OlapTable.java | 76 +-
.../org/apache/doris/catalog/RefreshManager.java | 6 +
.../java/org/apache/doris/catalog/Resource.java | 4 -
.../doris/clone/DynamicPartitionScheduler.java | 2 +-
.../org/apache/doris/cloud/catalog/CloudEnv.java | 11 +-
.../cloud/catalog/CloudFEVersionSynchronizer.java | 220 ++++++
.../apache/doris/cloud/catalog/CloudPartition.java | 52 +-
.../cloud/catalog/CloudSyncVersionDaemon.java | 203 +++++
.../doris/cloud/catalog/CloudTabletRebalancer.java | 72 +-
.../cloud/datasource/CloudInternalCatalog.java | 52 +-
.../doris/cloud/persist/CloudMetaSyncPoint.java | 69 ++
.../apache/doris/cloud/rpc/MetaServiceClient.java | 5 +
.../apache/doris/cloud/rpc/MetaServiceProxy.java | 12 +-
.../transaction/CloudGlobalTransactionMgr.java | 122 ++-
.../java/org/apache/doris/common/ClientPool.java | 3 +
.../java/org/apache/doris/common/FeConstants.java | 1 +
.../doris/common/proc/PartitionsProcDir.java | 74 +-
.../org/apache/doris/common/util/LocationPath.java | 4 +
.../org/apache/doris/common/util/PrintableMap.java | 2 +-
.../java/org/apache/doris/common/util/S3Util.java | 232 ++++++
.../org/apache/doris/datasource/CatalogIf.java | 11 +
.../apache/doris/datasource/CatalogProperty.java | 25 +-
.../apache/doris/datasource/ExternalCatalog.java | 161 +++-
.../apache/doris/datasource/ExternalDatabase.java | 15 +-
.../apache/doris/datasource/InternalCatalog.java | 13 +-
.../AbstractVendedCredentialsProvider.java | 2 +-
.../credentials/VendedCredentialsFactory.java | 2 +-
.../doris/RemoteDorisExternalCatalog.java | 3 +-
.../doris/datasource/es/EsExternalCatalog.java | 3 +-
.../doris/datasource/hive/HMSExternalCatalog.java | 8 +-
.../datasource/iceberg/IcebergExternalCatalog.java | 7 +-
.../datasource/iceberg/IcebergMetadataOps.java | 223 +++++-
.../iceberg/IcebergVendedCredentialsProvider.java | 15 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 5 +-
.../datasource/jdbc/client/JdbcClientConfig.java | 5 +-
.../jdbc/client/JdbcPostgreSQLClient.java | 6 +-
.../doris/datasource/jdbc/source/JdbcScanNode.java | 19 +
.../lakesoul/LakeSoulExternalCatalog.java | 2 +-
.../maxcompute/MaxComputeExternalCatalog.java | 36 +-
.../maxcompute/MaxComputeExternalTable.java | 8 +-
.../maxcompute/source/MaxComputeScanNode.java | 146 +++-
.../datasource/paimon/PaimonExternalCatalog.java | 3 +-
.../paimon/PaimonRestExternalCatalog.java} | 14 +-
.../paimon/PaimonVendedCredentialsProvider.java | 2 +-
.../datasource/property/ConnectorProperty.java | 2 +
.../metastore/AbstractIcebergProperties.java | 99 ++-
.../IcebergFileSystemMetaStoreProperties.java | 32 +-
.../metastore/IcebergGlueMetaStoreProperties.java | 16 +-
.../metastore/IcebergHMSMetaStoreProperties.java | 42 +-
.../metastore/IcebergJdbcMetaStoreProperties.java | 55 +-
.../property/metastore/IcebergRestProperties.java | 78 +-
.../IcebergS3TablesMetaStoreProperties.java | 10 -
.../property/metastore/MetastoreProperties.java | 2 +
.../TrinoConnectorPropertiesFactory.java} | 21 +-
.../storage/AbstractS3CompatibleProperties.java | 37 +
.../datasource/property/storage/COSProperties.java | 1 +
.../property/storage/MinioProperties.java | 4 +
.../datasource/property/storage/OBSProperties.java | 1 +
.../datasource/property/storage/OSSProperties.java | 6 +-
.../property/storage/OzoneProperties.java | 153 ++++
.../datasource/property/storage/S3Properties.java | 1 +
.../property/storage/StorageProperties.java | 116 ++-
.../doris/datasource/test/TestExternalCatalog.java | 3 +-
.../TrinoConnectorExternalCatalog.java | 3 +-
.../org/apache/doris/fs/obj/AzureObjStorage.java | 99 ++-
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 108 +++
.../config/WebServerFactoryCustomizerConfig.java | 42 +-
.../doris/httpv2/rest/TableQueryPlanAction.java | 2 +-
.../httpv2/rest/manager/MetaBackupAction.java | 329 ++++++++
.../doris/insertoverwrite/InsertOverwriteUtil.java | 13 +-
.../org/apache/doris/journal/JournalEntity.java | 6 +
.../apache/doris/journal/bdbje/BDBEnvironment.java | 92 +++
.../apache/doris/journal/bdbje/BDBJEJournal.java | 75 ++
.../mysql/privilege/AccessControllerManager.java | 28 +-
.../java/org/apache/doris/nereids/CTEContext.java | 15 +-
.../org/apache/doris/nereids/StatementContext.java | 22 +-
.../doris/nereids/glue/LogicalPlanAdapter.java | 6 -
.../doris/nereids/parser/LogicalPlanBuilder.java | 10 +-
.../nereids/rules/analysis/ExpressionAnalyzer.java | 129 +++-
.../rewrite/AccessPathExpressionCollector.java | 28 +-
.../rules/rewrite}/AccessPathInfo.java | 5 +-
.../rules/rewrite/AccessPathPlanCollector.java | 35 +-
.../rewrite/DecomposeRepeatWithPreAggregation.java | 4 +-
.../nereids/rules/rewrite/NestedColumnPruning.java | 186 +++--
.../rules/rewrite/RewriteSearchToSlots.java | 1 -
.../nereids/rules/rewrite/SlotTypeReplacer.java | 5 +-
.../rules/rewrite/VariantSubPathPruning.java | 9 +-
.../expressions/functions/generator/Explode.java | 5 +
.../functions/generator/ExplodeOuter.java | 5 +
.../expressions/functions/scalar/ElementAt.java | 2 +
.../functions/scalar/SearchDslParser.java | 185 ++++-
.../trees/plans/commands/BackupCommand.java | 3 +-
.../trees/plans/commands/DropFunctionCommand.java | 35 +-
.../trees/plans/commands/RestoreCommand.java | 3 +-
.../trees/plans/commands/ShowTableCommand.java | 3 +-
.../trees/plans/commands/UpdateCommand.java | 10 +-
.../trees/plans/commands/info/CopyIntoInfo.java | 29 +-
.../trees/plans/commands/info/LockTableInfo.java | 31 +-
.../worker/job/UnassignedShuffleJob.java | 19 +-
.../trees/plans/logical/LogicalSubQueryAlias.java | 9 +-
.../apache/doris/nereids/types/VariantType.java | 10 +
.../apache/doris/nereids/util/AggregateUtils.java | 1 +
.../java/org/apache/doris/persist/EditLog.java | 10 +
.../org/apache/doris/persist/OperationType.java | 1 +
.../org/apache/doris/persist/gson/GsonUtils.java | 2 +
.../apache/doris/planner/MaxComputeTableSink.java | 3 +-
.../org/apache/doris/qe/PointQueryExecutor.java | 6 +
.../java/org/apache/doris/qe/SessionVariable.java | 81 +-
.../org/apache/doris/qe/cache/PartitionRange.java | 391 ----------
.../apache/doris/service/FrontendServiceImpl.java | 12 +
.../doris/statistics/util/StatisticsUtil.java | 2 +-
.../org/apache/doris/task/CleanUDFCacheTask.java | 5 +-
.../java/org/apache/doris/catalog/ColumnTest.java | 9 +-
.../org/apache/doris/catalog/OlapTableTest.java | 132 ++++
.../cloud/catalog/CloudTabletRebalancerTest.java | 72 +-
.../org/apache/doris/common/util/S3UtilTest.java | 208 +++++
.../doris/datasource/ExternalCatalogTest.java | 139 ++++
.../doris/datasource/IncludeTableListTest.java | 424 +++++++++++
.../AbstractVendedCredentialsProviderTest.java | 6 +-
.../iceberg/IcebergMetadataOpsValidationTest.java | 180 +++++
.../doris/datasource/iceberg/IcebergUtilsTest.java | 5 +
.../datasource/jdbc/JdbcExternalCatalogTest.java | 7 +-
.../datasource/jdbc/source/JdbcScanNodeTest.java | 133 ++++
...aseNameComparedLowercaseMetaCacheFalseTest.java | 134 ++++
...baseNameComparedLowercaseMetaCacheTrueTest.java | 134 ++++
...abaseNameStoredLowercaseMetaCacheFalseTest.java | 138 ++++
...tabaseNameStoredLowercaseMetaCacheTrueTest.java | 138 ++++
.../lowercase/ShowAndSelectLowercaseTest.java | 599 +++++++++++++++
.../maxcompute/source/MaxComputeScanNodeTest.java | 438 +++++++++++
.../metastore/IcebergDlfRestCatalogTest.java | 124 +++
.../IcebergJdbcMetaStorePropertiesTest.java | 5 +-
.../metastore/IcebergRestPropertiesTest.java | 128 +++-
.../IcebergUnityCatalogRestCatalogTest.java | 1 -
.../property/storage/COSPropertiesTest.java | 8 +-
.../property/storage/OBSPropertyTest.java | 7 +-
.../property/storage/OSSPropertiesTest.java | 5 +-
.../property/storage/OzonePropertiesTest.java | 180 +++++
.../property/storage/S3PropertiesTest.java | 4 +-
.../property/storage/StoragePropertiesTest.java | 421 +++++++++++
.../doris/journal/bdbje/BDBEnvironmentTest.java | 88 ++-
.../doris/journal/bdbje/BDBJEJournalTest.java | 87 +++
.../privilege/AccessControllerManagerTest.java | 174 +++++
.../nereids/rules/analysis/ReadMorAsDupTest.java | 2 +-
.../DecomposeRepeatWithPreAggregationTest.java | 16 +-
.../rules/rewrite/PruneNestedColumnTest.java | 176 +++++
.../rules/rewrite/VariantPruningLogicTest.java | 243 ++++++
.../functions/scalar/SearchDslParserTest.java | 185 +++++
.../worker/job/UnassignedShuffleJobTest.java | 487 ++++++++++++
fe/pom.xml | 25 +
fs_brokers/cdc_client/build.sh | 2 +-
gensrc/proto/cloud.proto | 18 +-
gensrc/proto/internal_service.proto | 2 +
gensrc/proto/segment_v2.proto | 19 +
gensrc/thrift/AgentService.thrift | 1 +
gensrc/thrift/DataSinks.thrift | 1 +
gensrc/thrift/Descriptors.thrift | 5 +-
gensrc/thrift/Exprs.thrift | 5 +-
gensrc/thrift/FrontendService.thrift | 15 +
.../cdc_client/build.sh => hooks/setup_worktree.sh | 23 +-
regression-test/conf/regression-conf.groovy | 2 +
.../hive/test_hive_case_sensibility.out | 2 +
.../iceberg/iceberg_schema_change_ddl.out | 28 +-
.../test_iceberg_hadoop_case_sensibility.out | 5 +
.../iceberg/test_iceberg_hms_case_sensibility.out | 2 +
.../iceberg/test_iceberg_rest_case_sensibility.out | 5 +
.../jdbc/test_pg_jdbc_catalog.out | 33 +-
.../jdbc/test_sqlserver_jdbc_catalog.out | 26 +-
.../tvf/test_parquet_meta_tvf.out | 44 +-
.../iceberg/test_iceberg_dlf_rest_catalog.out | 17 +
.../data/load_p0/stream_load/enclose_with_bom.csv | 2 +
.../test_csv_with_enclose_and_escape.out | 2 +
.../nereids_p0/cte/test_cte_multicast_complex.out | 145 ++++
.../decompose_repeat/decompose_repeat.out | 17 +
.../point_query_p0/test_point_query_timezone.out | 15 +
.../data/pythonudaf_p0/test_pythonudaf_drop.out | 10 +
.../data/pythonudf_p0/test_pythonudf_drop.out | 10 +
.../data/pythonudtf_p0/test_pythonudtf_drop.out | 13 +
.../join/test_cte_exists/test_cte_exists.out | 44 ++
...est_search_variant_wildcard_custom_analyzer.out | 45 ++
.../test_packed_delete_bitmap.out | 63 ++
.../variant_p0/test_variant_cast_strict_mode.out | 19 +
.../regression/action/StreamLoadAction.groovy | 6 +-
.../hive/test_hive_case_sensibility.groovy | 2 +-
.../hive/test_hive_compress_type.groovy | 33 +-
.../hive/test_orc_lazy_mat_profile.groovy | 2 +-
.../iceberg/iceberg_schema_change_ddl.groovy | 3 +-
.../test_iceberg_hadoop_case_sensibility.groovy | 4 +-
.../test_iceberg_hms_case_sensibility.groovy | 2 +-
.../test_iceberg_rest_case_sensibility.groovy | 4 +-
...test_iceberg_schema_change_complex_types.groovy | 232 ++++++
.../jdbc/test_pg_jdbc_catalog.groovy | 10 +-
.../jdbc/test_sqlserver_jdbc_catalog.groovy | 24 +
...ta_with_lower_table_conf_show_and_select.groovy | 5 +-
.../tvf/test_parquet_meta_tvf.groovy | 45 +-
.../hive/test_hive_translation_insert_only.groovy | 7 +-
.../hive/test_orc_merge_io_input_streams.groovy | 17 +-
.../hive/test_parquet_complex_cross_page.groovy | 9 +-
.../iceberg/test_iceberg_dlf_rest_catalog.groovy | 45 ++
.../write/test_mc_limit_split_optimization.groovy | 266 +++++++
.../maxcompute/write/test_mc_write_ctas.groovy | 2 +-
.../maxcompute/write/test_mc_write_insert.groovy | 2 +-
.../write/test_mc_write_large_data.groovy | 2 +-
.../write/test_mc_write_partitions.groovy | 2 +-
.../write/test_mc_write_static_partitions.groovy | 2 +-
.../maxcompute/write/test_mc_write_types.groovy | 2 +-
.../test_csv_with_enclose_and_escape.groovy | 10 +
.../test_json_lz4_decompress_progress.groovy | 84 ++
.../cte/test_cte_multicast_complex.groovy | 284 +++++++
.../decompose_repeat/decompose_repeat.groovy | 13 +-
.../test_point_query_timezone.groovy | 71 ++
.../pythonudaf_p0/test_pythonudaf_drop.groovy | 96 +++
.../udaf_scripts/python_udaf_drop_a/drop_udaf.py | 38 +-
.../python_udaf_drop_a/python_udaf_drop_test.zip | Bin 0 -> 850 bytes
.../udaf_scripts/python_udaf_drop_b/drop_udaf.py | 37 +-
.../python_udaf_drop_b/python_udaf_drop_test.zip | Bin 0 -> 817 bytes
.../suites/pythonudf_p0/test_pythonudf_drop.groovy | 96 +++
.../udf_scripts/python_udf_drop_a/drop_udf.py | 22 +-
.../python_udf_drop_a/python_udf_drop_test.zip | Bin 0 -> 2143 bytes
.../udf_scripts/python_udf_drop_b/drop_udf.py | 22 +-
.../python_udf_drop_b/python_udf_drop_test.zip | Bin 0 -> 2143 bytes
.../pythonudtf_p0/test_pythonudtf_drop.groovy | 130 ++++
.../udtf_scripts/python_udtf_drop_a/drop_udtf.py | 20 +-
.../python_udtf_drop_a/python_udtf_drop_test.zip | Bin 0 -> 662 bytes
.../udtf_scripts/python_udtf_drop_b/drop_udtf.py | 21 +-
.../python_udtf_drop_b/python_udtf_drop_test.zip | Bin 0 -> 664 bytes
.../join/test_cte_exists/test_cte_exists.groovy | 78 ++
..._search_variant_wildcard_custom_analyzer.groovy | 191 +++++
.../cloud_delete_bitmap/test_cu_compaction.groovy | 6 +-
.../cloud_delete_bitmap/test_load.groovy | 6 +-
.../test_packed_delete_bitmap.groovy | 178 +++++
regression-test/suites/variant_p0/nested2.groovy | 284 +++----
.../test_variant_cast_strict_mode.groovy | 131 ++++
thirdparty/build-thirdparty.sh | 38 +-
thirdparty/download-thirdparty.sh | 10 +
thirdparty/paimon-cpp-cache.cmake | 47 +-
.../patches/apache-arrow-17.0.0-paimon.patch | 224 ++++++
.../paimon-cpp-buildutils-static-deps.patch | 160 +++-
428 files changed, 19367 insertions(+), 3714 deletions(-)
create mode 100644 .claude/skills/code-review/SKILL.md
create mode 100644 .github/workflows/opencode-review.yml
create mode 100644 AGENTS.md
mode change 100644 => 100755 be/src/olap/rowset/segment_v2/column_writer.h
create mode 100644 be/src/olap/rowset/segment_v2/variant/nested_group_path.h
create mode 100644
be/src/olap/rowset/segment_v2/variant/nested_group_provider.cpp
create mode 100644
be/src/olap/rowset/segment_v2/variant/nested_group_provider.h
mode change 100644 => 100755 be/src/vec/olap/olap_data_convertor.cpp
create mode 100644 be/test/cloud/cloud_tablet_mgr_test.cpp
create mode 100644
be/test/olap/rowset/segment_v2/inverted_index/query/phrase_query/phrase_freq_test.cpp
create mode 100644
be/test/olap/rowset/segment_v2/nested_group_path_filter_test.cpp
create mode 100644 be/test/olap/rowset/segment_v2/nested_group_path_test.cpp
create mode 100644
be/test/olap/rowset/segment_v2/nested_group_provider_test.cpp
create mode 100644
be/test/vec/columns/column_check_const_only_in_top_level_test.cpp
create mode 100644 be/test/vec/function/function_search_nested_test.cpp
create mode 100644 docker/runtime/base-image/Dockerfile_base_slim
rename fe/{fe-core =>
fe-common}/src/main/java/org/apache/doris/common/FormatOptions.java (100%)
rename
fe/{fe-core/src/main/java/org/apache/doris/datasource/property/constants =>
fe-common/src/main/java/org/apache/doris/common/maxcompute}/MCProperties.java
(90%)
create mode 100644
fe/fe-common/src/main/java/org/apache/doris/common/maxcompute/MCUtils.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/ChannelDescription.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/InsertTarget.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/LockTable.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudFEVersionSynchronizer.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudSyncVersionDaemon.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/persist/CloudMetaSyncPoint.java
rename fe/fe-core/src/main/java/org/apache/doris/{analysis/AlterUserType.java
=> datasource/paimon/PaimonRestExternalCatalog.java} (70%)
copy
fe/fe-core/src/main/java/org/apache/doris/{analysis/ImportDeleteOnStmt.java =>
datasource/property/metastore/TrinoConnectorPropertiesFactory.java} (66%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/OzoneProperties.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/MetaBackupAction.java
rename fe/fe-core/src/main/java/org/apache/doris/{analysis =>
nereids/rules/rewrite}/AccessPathInfo.java (92%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/IncludeTableListTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalDatabaseNameComparedLowercaseMetaCacheFalseTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalDatabaseNameComparedLowercaseMetaCacheTrueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalDatabaseNameStoredLowercaseMetaCacheFalseTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalDatabaseNameStoredLowercaseMetaCacheTrueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ShowAndSelectLowercaseTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/maxcompute/source/MaxComputeScanNodeTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/IcebergDlfRestCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/OzonePropertiesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/StoragePropertiesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AccessControllerManagerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/VariantPruningLogicTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/distribute/worker/job/UnassignedShuffleJobTest.java
copy fs_brokers/cdc_client/build.sh => hooks/setup_worktree.sh (66%)
create mode 100644
regression-test/data/external_table_p2/iceberg/test_iceberg_dlf_rest_catalog.out
create mode 100644
regression-test/data/load_p0/stream_load/enclose_with_bom.csv
create mode 100644
regression-test/data/nereids_p0/cte/test_cte_multicast_complex.out
create mode 100644
regression-test/data/point_query_p0/test_point_query_timezone.out
create mode 100644 regression-test/data/pythonudaf_p0/test_pythonudaf_drop.out
create mode 100644 regression-test/data/pythonudf_p0/test_pythonudf_drop.out
create mode 100644 regression-test/data/pythonudtf_p0/test_pythonudtf_drop.out
create mode 100644
regression-test/data/query_p0/join/test_cte_exists/test_cte_exists.out
create mode 100644
regression-test/data/search/test_search_variant_wildcard_custom_analyzer.out
create mode 100644
regression-test/data/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.out
create mode 100644
regression-test/data/variant_p0/test_variant_cast_strict_mode.out
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_change_complex_types.groovy
create mode 100644
regression-test/suites/external_table_p2/iceberg/test_iceberg_dlf_rest_catalog.groovy
create mode 100644
regression-test/suites/external_table_p2/maxcompute/write/test_mc_limit_split_optimization.groovy
create mode 100644
regression-test/suites/load_p0/stream_load/test_json_lz4_decompress_progress.groovy
create mode 100644
regression-test/suites/nereids_p0/cte/test_cte_multicast_complex.groovy
create mode 100644
regression-test/suites/point_query_p0/test_point_query_timezone.groovy
create mode 100644
regression-test/suites/pythonudaf_p0/test_pythonudaf_drop.groovy
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudaf_p0/udaf_scripts/python_udaf_drop_a/drop_udaf.py
(55%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudaf_p0/udaf_scripts/python_udaf_drop_a/python_udaf_drop_test.zip
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudaf_p0/udaf_scripts/python_udaf_drop_b/drop_udaf.py
(58%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudaf_p0/udaf_scripts/python_udaf_drop_b/python_udaf_drop_test.zip
create mode 100644
regression-test/suites/pythonudf_p0/test_pythonudf_drop.groovy
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudf_p0/udf_scripts/python_udf_drop_a/drop_udf.py
(64%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudf_p0/udf_scripts/python_udf_drop_a/python_udf_drop_test.zip
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudf_p0/udf_scripts/python_udf_drop_b/drop_udf.py
(64%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudf_p0/udf_scripts/python_udf_drop_b/python_udf_drop_test.zip
create mode 100644
regression-test/suites/pythonudtf_p0/test_pythonudtf_drop.groovy
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudtf_p0/udtf_scripts/python_udtf_drop_a/drop_udtf.py
(64%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudtf_p0/udtf_scripts/python_udtf_drop_a/python_udtf_drop_test.zip
copy fs_brokers/cdc_client/build.sh =>
regression-test/suites/pythonudtf_p0/udtf_scripts/python_udtf_drop_b/drop_udtf.py
(64%)
mode change 100755 => 100644
create mode 100644
regression-test/suites/pythonudtf_p0/udtf_scripts/python_udtf_drop_b/python_udtf_drop_test.zip
create mode 100644
regression-test/suites/query_p0/join/test_cte_exists/test_cte_exists.groovy
create mode 100644
regression-test/suites/search/test_search_variant_wildcard_custom_analyzer.groovy
create mode 100644
regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.groovy
create mode 100644
regression-test/suites/variant_p0/test_variant_cast_strict_mode.groovy
create mode 100644 thirdparty/patches/apache-arrow-17.0.0-paimon.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
