This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
from 0a9ea42b154 branch-4.2: [feature](variant) Support relational
operations for Variant (#68882)
new 59bcdeb5d28 [fix](regression) Restore missing ES query results on
branch-4.1 (#68852)
new e903b027cac [fix](hudi) Align preloaded Parquet format classes on
branch-4.1 (#68856)
new 0e98481c14b branch-4.1: [fix](auth) Add missing privilege checks for
several Nereids commands- #66218 (#68857)
new 0425d79bfda [opt](lance) add prefilter option for filter execution
control (#68813)
new add815d2622 branch-4.1: [fix](parquet) Restore column ownership after
interrupted reads #66780 (#68840)
new f26a25f5788 [fix](expr) Fix DATETIMEV2 array range tail omission
(#68855)
new 8d4660e9bb3 [fix](type) Preserve VARBINARY length in serialized blocks
on branch-4.1 (#68859)
new 7856e3bf67f [fix](regression) Backport external and Kerberos fixes to
branch-4.1 (#68150, #66433) (#68869)
new b5331698c28 [fix](regression) Keep HDFS array export out of ordinary
P0 runs (#68838)
new 8f20c3a5853 [fix](regression) stabilize external timestamp
expectations on 4.1 (#68864)
new da63b55148f [fix](regression) Stabilize Paimon Rust IN filter checks
on branch-4.1 (#68899)
new 201450c7503 [fix](regression) Stabilize information schema timezone
checks on branch-4.1 (#68920)
new 10dd002771c branch-4.1: [improvement](streaming) Improve streaming job
error messages #68241 (#68867)
new 8bf18b20599 branch-4.1: [test](regression) Remove unsupported
non-cloud auto-partition idempotence case #68070 (#68850)
new 1a30544d6e3 branch-4.1: [fix](fe) Skip insert after losing concurrent
CTAS table creation #68849 (#68881)
new f4c950fa084 [fix](paimon) Default new tables to portable ORC LTZ
encoding (#68924)
new f3f0ef37305 branch-4.1: [fix](fe) Fix MySQL packet truncation over TLS
#68703 (#68925)
new ed5ada37e6a [branch-4.1](fix) Correct external binary partitions and
timestamp handling (#68873)
new 368955561f6 branch-4.1: [fix](pipeline) Make PipelineTask::set_running
an unconditional exchange #67032 (#68929)
new e0f0406527e [fix](cloud) Bound abort task submissions for non-normal
clusters (#68066) (#68947)
new e2f91209976 branch-4.1: [feature](fe) Improve MySQL 9 client and
Connector/J compatibility (#68824) (#68944)
new 352e3a98e5c [fix](iceberg) Backport nested partition and default fixes
to branch-4.1 (#66825)
new 0e7ab62c013 [improvement](fe) Share CloudReplica route identifiers
(#68783) (#68950)
new ceecc8bee6d [fix](regression) branch-4.1: Reuse pinned Kerberos
dependencies without host build networking (#68954)
new b30e0c95c58 [fix](cloud) Fix SHOW TABLETS visible version (#68467)
(#68949)
new a946eefa6c2 branch-4.1: [improvement](streaming) Support one-time S3
streaming ingestion #68007 (#68943)
new 57ae1dc7361 [fix](file cache) keep disk resource limit mode hysteresis
across checks (#67313) (#68945)
The 27 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/core/data_type/data_type_factory.cpp | 2 +-
be/src/core/data_type/data_type_varbinary.cpp | 7 +
be/src/core/data_type/data_type_varbinary.h | 2 +
be/src/exec/pipeline/pipeline_task.h | 12 +-
.../sink/writer/iceberg/viceberg_table_writer.cpp | 151 +-
.../sink/writer/iceberg/viceberg_table_writer.h | 8 +
.../exprs/function/array/function_array_range.cpp | 24 +-
be/src/format/parquet/vparquet_column_reader.cpp | 22 +-
be/src/format/table/iceberg/schema.cpp | 29 +-
be/src/format/table/iceberg/schema.h | 4 +
.../transformer/iceberg_partition_function.cpp | 76 +-
.../transformer/iceberg_partition_function.h | 4 +
be/src/format_v2/lance/lance_reader_helper.cpp | 8 +
be/src/format_v2/lance/lance_reader_helper.h | 198 +
.../iceberg_position_delete_sys_table_reader.cpp | 10 +
be/src/format_v2/table/iceberg_reader.cpp | 37 +-
be/src/format_v2/table/iceberg_reader.h | 4 +
be/src/format_v2/table/lance_reader.cpp | 282 +-
be/src/format_v2/table/lance_reader.h | 13 +-
be/src/io/cache/block_file_cache.cpp | 29 +-
be/src/io/cache/block_file_cache.h | 4 +-
.../core/data_type/data_type_varbinary_test.cpp | 80 +
be/test/core/value/merge_partitioner_test.cpp | 91 +
be/test/exec/pipeline/pipeline_task_test.cpp | 58 +
.../writer/iceberg/viceberg_table_writer_test.cpp | 91 +-
.../parquet/parquet_scalar_column_reader_test.cpp | 170 +
be/test/format/table/iceberg/schema_test.cpp | 16 +
...eberg_position_delete_sys_table_reader_test.cpp | 29 +
be/test/format_v2/table/iceberg_reader_test.cpp | 33 +
be/test/format_v2/table/lance_reader_test.cpp | 77 +-
be/test/io/cache/block_file_cache_test.cpp | 108 +-
.../docker-compose/kerberos/Dockerfile | 11 +-
.../kerberos/entrypoint-hive-master.sh | 26 +
.../docker-compose/kerberos/hadoop-hive.env.tpl | 4 +
.../docker-compose/kerberos/kerberos.yaml.tpl | 7 +-
.../kerberos/sql/create_paimon_hive_table.hql | 20 +
docker/thirdparties/run-thirdparties-docker.sh | 47 +-
docs/lance-ann-profile.md | 17 +-
docs/lance-prefilter-profile.md | 24 +-
.../apache/doris/common/jni/vec/VectorColumn.java | 9 +
.../jni/vec/VectorColumnTimestampTzTest.java | 86 +
fe/be-java-extensions/jdbc-scanner/pom.xml | 6 +
.../org/apache/doris/jdbc/MySQLJdbcExecutor.java | 9 +-
.../org/apache/doris/jdbc/OracleJdbcExecutor.java | 27 +-
.../org/apache/doris/jdbc/TrinoJdbcExecutor.java | 32 +-
.../doris/jdbc/JdbcTimestampBackportTest.java | 70 +
.../apache/doris/jdbc/JdbcTimestampBindTest.java | 6 +-
.../doris/jdbc/JdbcTimestampSemanticsTest.java | 4 +-
.../jdbc/JdbcZonedTimestampIntegrationTest.java | 52 +-
.../doris/jdbc/MySqlTimestampIntegrationTest.java | 124 +-
.../doris/jdbc/OracleSessionTimeZoneTest.java | 73 +
.../doris/maxcompute/MaxComputeJniWriter.java | 9 +
.../doris/maxcompute/MaxComputeJniWriterTest.java | 40 +
fe/be-java-extensions/preload-extensions/pom.xml | 6 +-
.../main/java/org/apache/doris/common/Config.java | 12 +
.../java/org/apache/doris/common/ConfigBase.java | 28 +-
.../java/org/apache/doris/common/ConfigTest.java | 68 +
.../apache/doris/analysis/VarBinaryLiteral.java | 30 +-
.../java/org/apache/doris/catalog/JdbcTable.java | 13 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 23 +-
.../cloud/datasource/CloudInternalCatalog.java | 5 +
.../apache/doris/common/proc/TabletsProcDir.java | 6 +-
.../apache/doris/common/profile/AggCounter.java | 5 +
.../apache/doris/datasource/InternalCatalog.java | 11 +
.../datasource/iceberg/IcebergMetadataOps.java | 16 +-
.../datasource/iceberg/IcebergTransaction.java | 24 +-
.../iceberg/IcebergWriteSchemaContext.java | 7 +-
.../doris/datasource/jdbc/client/JdbcClient.java | 26 +
.../doris/datasource/jdbc/source/JdbcScanNode.java | 195 +-
.../datasource/lance/source/LanceScanNode.java | 8 +-
.../doris/datasource/paimon/PaimonMetadataOps.java | 7 +
.../apache/doris/datasource/paimon/PaimonUtil.java | 7 +-
.../doris/httpv2/rest/StreamingJobAction.java | 2 +-
.../streaming/DataSourceConfigValidator.java | 24 +-
.../insert/streaming/StreamingInsertJob.java | 86 +-
.../insert/streaming/StreamingJobProperties.java | 35 +-
.../streaming/StreamingJobSchedulerTask.java | 7 +-
.../insert/streaming/StreamingMultiTblTask.java | 7 +-
.../job/offset/SourceOffsetProviderFactory.java | 7 +-
.../job/offset/jdbc/JdbcSourceOffsetProvider.java | 20 +-
.../org/apache/doris/job/offset/s3/S3Offset.java | 2 +
.../job/offset/s3/S3SourceOffsetProvider.java | 30 +-
.../job/scheduler/StreamingTaskScheduler.java | 6 +-
.../apache/doris/job/util/StreamingJobUtils.java | 11 +-
.../doris/mysql/CachingSha2PasswordExchange.java | 236 +
.../org/apache/doris/mysql/MysqlCapability.java | 10 +-
.../java/org/apache/doris/mysql/MysqlChannel.java | 6 +-
.../password/NativePasswordResolver.java | 45 +-
.../org/apache/doris/nereids/StatementContext.java | 11 +
.../glue/translator/PhysicalPlanTranslator.java | 3 +-
.../nereids/properties/DistributionSpecMerge.java | 17 +-
.../nereids/trees/expressions/literal/Literal.java | 36 +
.../expressions/literal/VarBinaryLiteral.java | 30 +-
.../trees/plans/commands/AddConstraintCommand.java | 17 +
.../commands/AdminRotateTdeRootKeyCommand.java | 15 +
.../commands/AdminSetEncryptionRootKeyCommand.java | 9 +
.../trees/plans/commands/AlterJobCommand.java | 21 +-
.../plans/commands/CancelWarmUpJobCommand.java | 9 +
.../plans/commands/CreateDictionaryCommand.java | 38 +-
.../commands/DropCatalogRecycleBinCommand.java | 15 +
.../plans/commands/DropConstraintCommand.java | 41 +-
.../plans/commands/DropDictionaryCommand.java | 12 +-
.../trees/plans/commands/DropStageCommand.java | 15 +
.../commands/ShowTabletsFromTableCommand.java | 40 +-
.../trees/plans/commands/WarmUpClusterCommand.java | 42 +-
.../trees/plans/commands/info/CreateJobInfo.java | 3 +-
.../commands/insert/InsertIntoTableCommand.java | 8 +
.../plans/physical/PhysicalIcebergMergeSink.java | 92 +-
.../org/apache/doris/planner/DataPartition.java | 10 +
.../java/org/apache/doris/qe/GlobalVariable.java | 6 +
.../org/apache/doris/qe/MysqlConnectProcessor.java | 40 +-
.../java/org/apache/doris/system/HeartbeatMgr.java | 11 +-
.../CdcStreamTableValuedFunction.java | 13 +-
.../FullTextSearchTableValuedFunction.java | 2 +-
.../LanceExternalSearchTableValuedFunction.java | 16 +
.../VectorSearchTableValuedFunction.java | 13 +-
.../VarBinaryLiteralOrderingTest.java} | 22 +-
.../org/apache/doris/catalog/CreateTableTest.java | 94 +
.../cloud/WarmUpClusterOnTablesParseTest.java | 6 +
.../doris/cloud/catalog/CloudReplicaTest.java | 80 +
.../cloud/datasource/CloudInternalCatalogTest.java | 72 +
.../common/proc/CloudProcVersionDisplayTest.java | 16 +
.../datasource/iceberg/IcebergTransactionTest.java | 14 +
.../iceberg/IcebergWriteSchemaContextTest.java | 17 +
.../jdbc/source/JdbcQueryWrappingTest.java | 306 +
.../jdbc/source/JdbcTimestampProjectionTest.java | 7 +-
.../paimon/PaimonBinaryPartitionTest.java} | 24 +-
.../datasource/paimon/PaimonMetadataOpsTest.java | 80 +
.../StreamingInsertJobCheckDataQualityTest.java | 6 +
.../StreamingInsertJobOffsetPersistenceTest.java | 105 +-
.../StreamingInsertJobStatusTransitionTest.java} | 22 +-
.../streaming/StreamingJobPropertiesTest.java | 46 +-
.../JdbcSourceOffsetProviderErrorHandlingTest.java | 18 +-
.../job/offset/s3/S3SourceOffsetProviderTest.java | 151 +
.../job/scheduler/StreamingTaskSchedulerTest.java | 70 +
.../apache/doris/mysql/MysqlCapabilityTest.java | 20 +-
.../doris/mysql/MysqlHandshakePacketTest.java | 9 +-
.../org/apache/doris/mysql/MysqlProtoTest.java | 41 +
.../password/NativePasswordResolverTest.java | 233 +
.../nereids/parser/VarBinaryLiteralParserTest.java | 4 +-
.../trees/expressions/VarBinaryLiteralTest.java | 5 +-
.../expressions/literal/TimeV2LiteralTest.java | 110 +
.../commands/ConstraintAuthorizationTest.java | 169 +
.../plans/commands/ShowConfigCommandTest.java | 7 +-
.../plans/commands/insert/InsertUtilsTest.java | 47 +
.../physical/PhysicalIcebergMergeSinkTest.java | 135 +
.../java/org/apache/doris/qe/VariableMgrTest.java | 12 +
.../org/apache/doris/system/HeartbeatMgrTest.java | 64 +
.../doris/cdcclient/sink/DorisBatchStreamLoad.java | 11 +
.../cdcclient/sink/DorisBatchStreamLoadTest.java | 84 +
gensrc/proto/data.proto | 2 +
gensrc/thrift/Partitions.thrift | 2 +
gensrc/thrift/PlanNodes.thrift | 8 +-
.../write/test_iceberg_write_complex_evolution.out | 12 +-
.../lance/test_lance_full_text_search.out | 9 +
.../lance/test_lance_vector_search_two_phase.out | 7 +-
.../data/external_table_p2/es/test_es_query.out | 206 +
.../hudi/test_hudi_partition_prune.out | 8 +-
.../external_table_p2/hudi/test_hudi_timestamp.out | 36 +-
.../jdbc/test_oceanbase_jdbc_catalog.out | 6 +-
.../jdbc/test_sqlserver_jdbc_catalog.out | 18 +-
.../maxcompute/test_max_compute_timestamp.out | 128 +-
.../test_array_range_datetime_tail.out | 29 +
.../array_functions/test_array_functions.out | 14 +-
.../auth_call/test_ddl_constraint_auth.groovy | 126 +
...auth.groovy => test_ddl_dictionary_auth.groovy} | 91 +-
...et_auth.groovy => test_recycle_bin_auth.groovy} | 71 +-
.../suites/export/test_array_export.groovy | 3 +-
.../iceberg/test_iceberg_deletion_vector.groovy | 42 +-
.../test_iceberg_write_binary_partitions.groovy | 21 +
.../test_iceberg_write_complex_evolution.groovy | 26 +-
.../test_information_schema_timezone.groovy | 436 +-
.../jdbc/test_mysql_jdbc_catalog.groovy | 5 +-
.../kerberos/test_single_hive_kerberos.groovy | 1 +
.../lance/test_lance_full_text_search.groovy | 28 +-
.../lance/test_lance_multivector_coverage.groovy | 3 +-
.../lance/test_lance_multivector_search.groovy | 3 +-
.../lance/test_lance_prepared_vector_search.groovy | 2 +-
.../lance/test_lance_vector_search.groovy | 11 +-
.../test_lance_vector_search_two_phase.groovy | 46 +-
.../test_paimon_rust_reader_eq_for_null.groovy | 3 +
.../write/test_paimon_orc_ltz_encoding.groovy | 125 +
.../refactor_storage_param/hdfs_all_test.groovy | 2 +
.../external_table_p0/tvf/test_http_tvf.groovy | 12 +-
.../tvf/test_s3_tvf_number_range.groovy | 32 +-
.../external_table_p2/es/test_es_query.groovy | 1 +
.../hudi/test_hudi_partition_prune.groovy | 1 +
.../jdbc/test_oceanbase_jdbc_catalog.groovy | 1 +
.../jdbc/test_sqlserver_jdbc_catalog.groovy | 1 +
.../maxcompute/test_max_compute_timestamp.groovy | 1 +
.../paimon/test_paimon_hms_catalog.groovy | 4 +-
.../hive_on_hms_and_dlf.groovy | 4 +-
.../iceberg_on_hms_and_filesystem_and_dlf.groovy | 4 +-
.../test_streaming_mysql_job_create_alter.groovy | 2 +-
.../cdc/test_streaming_mysql_job_dup.groovy | 2 +-
.../cdc/test_streaming_postgres_job_dup.groovy | 2 +-
.../test_streaming_insert_job_crud.groovy | 8 +-
.../test_streaming_insert_job_s3_once.groovy | 107 +
.../test_streaming_job_max_retry.groovy | 2 +
.../test_array_range_datetime_tail.groovy | 35 +
.../select_tablets/select_with_tablets.groovy | 3 -
.../test_auto_partition_idempotence.groovy | 96 -
thirdparty/patches/lance-c-foyer.patch | 6936 ++++++++++++++++++--
thirdparty/vars.sh | 10 +-
204 files changed, 13272 insertions(+), 1627 deletions(-)
create mode 100644 be/test/format/parquet/parquet_scalar_column_reader_test.cpp
create mode 100644
docker/thirdparties/docker-compose/kerberos/sql/create_paimon_hive_table.hql
create mode 100644
fe/be-java-extensions/java-common/src/test/java/org/apache/doris/common/jni/vec/VectorColumnTimestampTzTest.java
create mode 100644
fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcTimestampBackportTest.java
create mode 100644
fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/OracleSessionTimeZoneTest.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/mysql/CachingSha2PasswordExchange.java
copy fe/fe-core/src/test/java/org/apache/doris/{common/CheckedMathTest.java =>
analysis/VarBinaryLiteralOrderingTest.java} (62%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/cloud/datasource/CloudInternalCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/source/JdbcQueryWrappingTest.java
copy fe/fe-core/src/test/java/org/apache/doris/{common/CheckedMathTest.java =>
datasource/paimon/PaimonBinaryPartitionTest.java} (50%)
copy fe/fe-core/src/test/java/org/apache/doris/{common/CheckedMathTest.java =>
job/extensions/insert/streaming/StreamingInsertJobStatusTransitionTest.java}
(62%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/job/offset/s3/S3SourceOffsetProviderTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/job/scheduler/StreamingTaskSchedulerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/mysql/authenticate/password/NativePasswordResolverTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ConstraintAuthorizationTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSinkTest.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/sink/DorisBatchStreamLoadTest.java
create mode 100644
regression-test/data/nereids_function_p0/scalar_function/test_array_range_datetime_tail.out
create mode 100644
regression-test/suites/auth_call/test_ddl_constraint_auth.groovy
copy regression-test/suites/auth_call/{test_ddl_database_auth.groovy =>
test_ddl_dictionary_auth.groovy} (51%)
copy regression-test/suites/auth_call/{test_show_tablet_auth.groovy =>
test_recycle_bin_auth.groovy} (54%)
create mode 100644
regression-test/suites/external_table_p0/paimon/write/test_paimon_orc_ltz_encoding.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_s3_once.groovy
create mode 100644
regression-test/suites/nereids_function_p0/scalar_function/test_array_range_datetime_tail.groovy
delete mode 100644
regression-test/suites/partition_p0/auto_partition/test_auto_partition_idempotence.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
