This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch auto-pick-57898-branch-4.0
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 5d239d0c24e [Feature] doris cross-cluster query (#57898)
add 01c5026f286 branch-4.0: [test](case) fix the hdfs case uri error
#58556 (#58575)
add 1d838332481 branch-4.0: [fix](hive) Fix StackOverflowError in insert
overwrite on S3-compatible storage #58504 (#58534)
add 6a27e215623 [feat](function) Add function of deduplicate_map (#58403)
(#58537)
add d02422d45a9 branch-4.0: [chore](dep)Upgrade Shade #58071 (#58581)
add daa1b6fd096 branch-4.0: [fix](rf)Fix ColumnPredicate rf may not reset
judge #58466 (#58558)
add fb4fc9e7f69 branch-4.0: Fix typo in Nereids EXPLAIN output ('planed' →
'planned') #57337 (#58560)
add cafa0d8fe30 branch-4.0: [fix](regression) Rewrite case and add some
info #58453 (#58587)
add ede3490bb3b branch-4.0: [fix](metacache) Fix query hang caused by
Caffeine cache deadlock in external table queries #57856 (#58066)
add c718494e0bf branch-4.0: [opt](load)mysql load use http utils to create
client #58450 (#58517)
add 8f7dcac75b0 branch-4.0: [fix](mtls)brpc SetFailed handle error_code=0
pick brpc #2560 #58394 (#58519)
add 7ce93865ade branch-4.0: [fix](paimon)Support user-defined S3 config
prefixes and unify to HDFS S3A protocol #57116 (#58544)
add 9b250553e44 branch-4.0: [fix](s3)fix: unify AWS connection parameter
between FE and BE #58514 (#58543)
add ad76d9deeff branch-4.0: [opt](scheduler) Improve Graceful Shutdown
Behavior for BE and FE, and Optimize Query Retry During BE Shutdown #56601
#58019 (#58526)
add 5c8a6f23e57 branch-4.0: [improvement](jdbc) Improve JdbcDataSource
cleanup thread reliability and observability #58484 (#58536)
add a6e83c974a6 branch-4.0: [feature](iceberg) Support Partition Evolution
DDL for Iceberg Tables (#57972) (#58538)
add 42c88d166d5 branch-4.0: [fix](nereids)fix
pushdownLimit/topnThroughJoin dead loop #58305 (#58564)
add b7a789a86d3 branch-4.0: [opt](coordinator) All Command can use
NereidsCoordinator #58360 (#58561)
add 31557ff76c2 branch-4.0: [opt](nereids) support random local shuffle
for union #58042 (#58555)
add 9546f5c7cf8 branch-4.0: [feature](nereids) Support dereference
expression #57532 (#58546)
add 4338163767e branch-4.0: [opt](hive) Speed up Hive insert on partition
tables using cache #58166 (#58540)
add 0c1283a2d29 branch-4.0: [chore](dep)upgrade netty and hadoop #57226
(#57609)
add 091d904f831 branch-4.0: [fix](nereids) fix copy into load orc/parquet
format fail (#57362) (#58552)
add 0a0ec5ba91d branch-4.0: [fix](nereids) Fix logicalRepeat java type
convert err when start fe in ide #58510 (#58549)
add 40b30b0998a branch-4.0: [fix](test) Make the result of the
test_insert_limit test case more clearer #58288 (#58520)
add 45081ca2cd9 branch-4.0: [improve](glue)Enforce HTTPS protocol for Glue
endpoint #58306 (#58367)
add e31e1f5dc3b branch-4.0: [opt](tde) Fix rotate framework #58447 (#58531)
add 41f12fcedec branch-4.0: [feat](test)Add Azure Blob Test #57268 (#57457)
add f62c3bd7f44 branch-4.0: [fix](paimon) Align incremental query behavior
with Spark Paimon for single snapshot scenario #58239 (#58254)
add 4d547dc0d72 [Feature] doris cross-cluster query (#57898)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5d239d0c24e)
\
N -- N -- N refs/heads/auto-pick-57898-branch-4.0 (4d547dc0d72)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/src/common/config.cpp | 1 +
be/src/common/config.h | 6 +
be/src/http/action/health_action.cpp | 27 +-
be/src/olap/column_predicate.h | 10 +-
be/src/olap/comparison_predicate.h | 1 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 23 +-
be/src/pipeline/exec/exchange_sink_operator.h | 1 +
be/src/runtime/exec_env.cpp | 7 +
be/src/runtime/exec_env.h | 3 +
be/src/runtime/snapshot_loader.cpp | 2 +-
be/src/service/doris_main.cpp | 3 +
be/src/util/s3_util.cpp | 2 +-
be/src/vec/columns/column_map.cpp | 4 +-
be/src/vec/exec/format/orc/vorc_reader.cpp | 2 +-
.../exec/format/parquet/vparquet_column_reader.cpp | 1 -
be/src/vec/exec/jni_connector.cpp | 2 +-
be/src/vec/functions/function_map.cpp | 27 ++
be/test/http/http_client_test.cpp | 4 +-
be/test/vec/exec/orc/orc_reader_fill_data_test.cpp | 12 +-
be/test/vec/function/function_map_test.cpp | 83 ++++++
build.sh | 26 +-
.../java/org/apache/doris/jdbc/JdbcDataSource.java | 30 +-
fe/be-java-extensions/preload-extensions/pom.xml | 8 +
fe/fe-core/pom.xml | 61 +---
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 12 +
.../src/main/java/org/apache/doris/DorisFE.java | 52 +++-
.../main/java/org/apache/doris/alter/Alter.java | 42 +++
.../java/org/apache/doris/alter/AlterOpType.java | 4 +
.../doris/analysis/AddPartitionFieldClause.java | 108 ++++++++
.../apache/doris/analysis/AlterTableClause.java | 14 +
.../doris/analysis/DropPartitionFieldClause.java | 113 ++++++++
.../analysis/ReplacePartitionFieldClause.java | 145 ++++++++++
.../org/apache/doris/backup/BackupHandler.java | 1 +
.../org/apache/doris/catalog/AzureResource.java | 5 +-
.../doris/catalog/BuiltinScalarFunctions.java | 2 +
.../org/apache/doris/catalog/RefreshManager.java | 12 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 8 +-
.../java/org/apache/doris/common/CacheFactory.java | 22 +-
.../java/org/apache/doris/common/FeConstants.java | 26 --
.../doris/common/profile/SummaryProfile.java | 17 ++
.../org/apache/doris/common/util/LocationPath.java | 9 +
.../apache/doris/datasource/ExternalObjectLog.java | 10 +
.../doris/datasource/ExternalSchemaCache.java | 2 +-
.../doris/datasource/hive/HMSTransaction.java | 98 +++++--
.../doris/datasource/hive/HiveMetaStoreCache.java | 63 ++++-
.../hudi/source/HudiCachedFsViewProcessor.java | 2 +-
.../hudi/source/HudiCachedMetaClientProcessor.java | 1 -
.../hudi/source/HudiCachedPartitionProcessor.java | 2 +-
.../datasource/iceberg/IcebergExternalCatalog.java | 52 ++++
.../datasource/iceberg/IcebergMetadataCache.java | 8 +-
.../datasource/iceberg/IcebergMetadataOps.java | 139 ++++++++++
.../doris/datasource/metacache/MetaCache.java | 6 +-
.../datasource/paimon/PaimonMetadataCache.java | 2 +-
.../datasource/paimon/source/PaimonScanNode.java | 12 +-
.../doris/datasource/property/ParamRules.java | 23 ++
.../metastore/AWSGlueMetaStoreBaseProperties.java | 4 +-
.../metastore/AbstractPaimonProperties.java | 49 +++-
.../IcebergFileSystemMetaStoreProperties.java | 4 +-
.../metastore/IcebergHMSMetaStoreProperties.java | 3 -
.../property/metastore/IcebergRestProperties.java | 9 +-
.../PaimonAliyunDLFMetaStoreProperties.java | 6 +-
.../PaimonFileSystemMetaStoreProperties.java | 7 +-
.../metastore/PaimonHMSMetaStoreProperties.java | 6 +-
.../metastore/PaimonRestMetaStoreProperties.java | 2 +-
.../storage/AbstractS3CompatibleProperties.java | 55 ++--
.../property/storage/AzureProperties.java | 123 ++++++++-
.../property/storage/AzurePropertyUtils.java | 36 ++-
.../property/storage/BrokerProperties.java | 8 +
.../property/storage/HdfsCompatibleProperties.java | 7 +
.../property/storage/LocalProperties.java | 6 +
.../property/storage/StorageProperties.java | 62 ++++-
.../property/storage/exception/AzureAuthType.java} | 18 +-
.../apache/doris/encryption/KeyManagerStore.java | 8 +-
.../org/apache/doris/fs/obj/AzureObjStorage.java | 11 +
.../java/org/apache/doris/fs/obj/S3ObjStorage.java | 7 +-
.../java/org/apache/doris/httpv2/HttpServer.java | 24 +-
.../doris/httpv2/entity/ResponseEntityBuilder.java | 5 +
.../org/apache/doris/httpv2/rest/HealthAction.java | 6 +
.../doris/httpv2/rest/RestApiStatusCode.java | 3 +-
.../doris/httpv2/rest/manager/HttpUtils.java | 4 +
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 3 +-
.../org/apache/doris/load/StreamLoadHandler.java | 2 +-
.../apache/doris/load/loadv2/MysqlLoadManager.java | 3 +-
.../org/apache/doris/nereids/NereidsPlanner.java | 2 +-
.../glue/translator/PhysicalPlanTranslator.java | 46 ++++
.../doris/nereids/parser/LogicalPlanBuilder.java | 108 +++++++-
.../nereids/rules/analysis/BindExpression.java | 22 +-
.../nereids/rules/analysis/ExpressionAnalyzer.java | 156 ++++++++++-
.../rewrite/PushDownLimitDistinctThroughJoin.java | 14 +-
.../rewrite/PushDownTopNDistinctThroughJoin.java | 11 +-
.../rules/rewrite/PushDownTopNThroughJoin.java | 17 +-
.../trees/copier/LogicalPlanDeepCopier.java | 2 +-
.../trees/expressions/DereferenceExpression.java | 41 +++
.../functions/scalar/DeduplicateMap.java | 70 +++++
.../expressions/visitor/ExpressionVisitor.java | 5 +
.../trees/plans/commands/AlterTableCommand.java | 8 +-
.../plans/commands/info/AddPartitionFieldOp.java | 113 ++++++++
.../trees/plans/commands/info/CopyIntoInfo.java | 5 +-
.../plans/commands/info/DropPartitionFieldOp.java | 126 +++++++++
.../commands/info/ReplacePartitionFieldOp.java | 157 +++++++++++
.../commands/insert/AbstractInsertExecutor.java | 6 +-
.../insert/BaseExternalTableInsertExecutor.java | 23 +-
.../plans/commands/insert/HiveInsertExecutor.java | 57 ++++
.../plans/commands/insert/OlapInsertExecutor.java | 6 +-
.../worker/job/UnassignedJobBuilder.java | 6 +-
.../worker/job/UnassignedLocalShuffleUnionJob.java | 99 +++++++
.../org/apache/doris/nereids/types/StructType.java | 4 +
.../org/apache/doris/persist/KeyOperationInfo.java | 4 +-
.../org/apache/doris/planner/DataPartition.java | 3 +-
.../org/apache/doris/planner/ExchangeNode.java | 13 +-
.../org/apache/doris/planner/HiveTableSink.java | 49 +++-
.../java/org/apache/doris/planner/UnionNode.java | 10 +
.../java/org/apache/doris/qe/SessionVariable.java | 11 -
.../java/org/apache/doris/qe/StmtExecutor.java | 21 +-
.../main/java/org/apache/doris/system/Backend.java | 52 +++-
.../org/apache/doris/system/SystemInfoService.java | 40 +--
.../org/apache/doris/common/CacheFactoryTest.java | 6 +-
.../apache/doris/common/util/LocationPathTest.java | 17 ++
.../metacache/MetaCacheDeadlockTest.java | 114 ++++++++
.../paimon/source/PaimonScanNodeTest.java | 52 ++--
.../doris/datasource/property/ParamRulesTest.java | 41 +++
.../AWSGlueMetaStoreBasePropertiesTest.java | 19 +-
.../metastore/AbstractPaimonPropertiesTest.java | 89 ++++++
.../PaimonRestMetaStorePropertiesTest.java | 14 +-
.../property/storage/AzurePropertiesTest.java | 47 +++-
.../property/storage/AzurePropertyUtilsTest.java | 16 ++
.../property/storage/OSSPropertiesTest.java | 10 +
.../nereids/distribute/LocalShuffleUnionTest.java | 141 ++++++++++
.../nereids/rules/analysis/TestDereference.java | 86 ++++++
.../plans/commands/AlterTableCommandTest.java | 104 ++++++-
.../commands/info/AddPartitionFieldOpTest.java | 183 ++++++++++++
.../commands/info/DropPartitionFieldOpTest.java | 82 ++++++
.../commands/info/ReplacePartitionFieldOpTest.java | 156 +++++++++++
.../apache/doris/planner/HiveTableSinkTest.java | 9 +
.../doris/utframe/DemoMultiBackendsTest.java | 3 +-
fe/pom.xml | 83 +++++-
gensrc/thrift/Partitions.thrift | 3 +
.../one_level_nestedtypes_with_s3data.out | 20 +-
...t => one_level_nestedtypes_with_s3data.out.txt} | 20 +-
.../test_iceberg_partition_evolution_ddl.out | 119 ++++++++
...est_iceberg_partition_evolution_query_write.out | 142 ++++++++++
.../external_table_p0/paimon/paimon_incr_read.out | 14 +
.../paimon/test_paimon_catalog.out | 24 +-
.../data/load_p0/tvf/test_tvf_lz4_compress.out | 10 +
.../test_iceberg_mtmv_with_partition_evolution.out | 13 +
.../data/query_p1/test_insert_limit.out | 94 +++----
regression-test/framework/pom.xml | 4 +-
.../query_retry/test_retry_no_scan_node.groovy | 80 ++++++
.../update_use_nereids_coordinator.groovy | 32 ++-
.../one_level_nestedtypes_with_s3data.groovy | 148 +++++++---
.../hive/test_hive_partitions.groovy | 97 +++++++
.../test_iceberg_partition_evolution_ddl.groovy | 306 +++++++++++++++++++++
..._iceberg_partition_evolution_query_write.groovy | 245 +++++++++++++++++
.../paimon/paimon_incr_read.groovy | 13 +-
.../paimon/test_paimon_catalog_varbinary.groovy | 8 +-
.../azure_non_catalog_all_test.groovy | 206 ++++++++++++++
.../backup_restore_azure.groovy | 63 +++--
.../hive_on_hms_and_dlf.groovy | 116 +++++++-
.../iceberg_and_hive_on_glue.groovy | 179 ++++++++++++
.../load_p0/tvf/test_tvf_lz4_compress.groovy | 53 ++--
...st_iceberg_mtmv_with_partition_evolution.groovy | 170 ++++++++++++
.../nereids_p0/stats/col_stats/column_stats.groovy | 18 +-
.../mv/pre_rewrite/limit/query_with_limit.groovy | 19 ++
.../suites/query_p0/test_dereference.groovy | 69 +++++
.../suites/query_p1/test_insert_limit.groovy | 25 +-
...SetFailed-to-handle-error_code-0-properly.patch | 303 ++++++++++++++++++++
167 files changed, 6366 insertions(+), 667 deletions(-)
create mode 100644 be/test/vec/function/function_map_test.cpp
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/AddPartitionFieldClause.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/DropPartitionFieldClause.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/ReplacePartitionFieldClause.java
copy
fe/fe-core/src/main/java/org/apache/doris/{httpv2/rest/RestApiStatusCode.java
=> datasource/property/storage/exception/AzureAuthType.java} (73%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/DereferenceExpression.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DeduplicateMap.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/AddPartitionFieldOp.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropPartitionFieldOp.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ReplacePartitionFieldOp.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/distribute/worker/job/UnassignedLocalShuffleUnionJob.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/MetaCacheDeadlockTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/AbstractPaimonPropertiesTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/distribute/LocalShuffleUnionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/TestDereference.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/AddPartitionFieldOpTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/DropPartitionFieldOpTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ReplacePartitionFieldOpTest.java
copy
regression-test/data/datatype_p0/nested_types/base_cases/{one_level_nestedtypes_with_s3data.out
=> one_level_nestedtypes_with_s3data.out.txt} (84%)
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_ddl.out
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_partition_evolution_query_write.out
create mode 100644
regression-test/data/mtmv_p0/test_iceberg_mtmv_with_partition_evolution.out
create mode 100644
regression-test/suites/cloud_p0/query_retry/test_retry_no_scan_node.groovy
copy
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/RestApiStatusCode.java =>
regression-test/suites/coordinator_p0/update_use_nereids_coordinator.groovy
(58%)
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_ddl.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_partition_evolution_query_write.groovy
create mode 100644
regression-test/suites/external_table_p2/refactor_catalog_param/azure_non_catalog_all_test.groovy
rename regression-test/suites/{external_table_p0/refactor_storage_param =>
external_table_p2/refactor_catalog_param}/backup_restore_azure.groovy (69%)
create mode 100644
regression-test/suites/mtmv_p0/test_iceberg_mtmv_with_partition_evolution.groovy
create mode 100644 regression-test/suites/query_p0/test_dereference.groovy
create mode 100644
thirdparty/patches/brpc-2560-Refactor-Socket-SetFailed-to-handle-error_code-0-properly.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
