This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch
optimize_s3_skip_list_for_deterministic_paths
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 4702df99f97 [fix](s3) Skip HEAD optimization for path-style URLs to
avoid false positives
discard 26026e37134 [opt](s3) Treat non-negated bracket patterns as
deterministic for HEAD request optimization
discard d495108da49 [opt](s3) Skip S3 listing for deterministic file paths
using HEAD requests
add 14c7b204dcb [fix](predicate) Avoid recomputing predicates (#60484)
add 705ca19d56c [fix](agg) Fix local shuffle before agg (#60494)
add 2b0b87be27e [fix](inverted index) Fix empty string MATCH on keyword
index returning wrong results (#60500)
add e33e2b05e4f [chore](ci)Delete sonar cloud (#60517)
add f39bb99174e [fix](build) fix duplicate symbol errors when compiling BE
UT with Clang (#60504)
add 019adf8431c [fix](nereids)
PhysicalDictionarySink.resetLogicalProperties() does not reset logical
properties properly (#60495)
add b633fbe899a [enhance](memtable) support dynamic modification of flush
thread pool size (#60423)
add e1a7abee9b4 [enhance](load) only set brokerLoadBatchSize when
enableMemtableOnSinkNode is true (#60301)
add d655b4bd76c [Improve](StreamingJob) add max_filter_ratio and strict
mode for mysql/pg streaming job (#60473)
add cbcbb04107a [fix](workload policy)query time based be policy not work
due to wrong finish_time_ (#60456)
add acc2a8fb213 [regression-test](variant) disable doc mode in
test_variant_count_on_index_fault_injection (#60518)
add 6031d3741db [fix](nereids)EliminateGroupByKeyByUniform should replace
exprId for alias (#60020)
add 5660f111cd6 [fix](translator) fix set operation compute wrong result
(#60525)
add 5275e0e386b [Feature](join) support LEFT_SEMI_DIRECT_RETURN_OPT
(#59832)
add e010d8ddbca [enhancement](filecache) add filesystem leak cleaner
(#59269)
add 00b4c7fd7a2 [Improve](StreamingJob) add more metrics to observe the
streaming job (#60493)
add 4624f23cb0b [fix](case) Fix some docker case run in ci (#60087)
add 539a1a4277a [fix](topn) fix AcceptNullPredicate adding back all null
rows instead of only those in original bitmap (#60537)
add f26b21d5ce4 [fix](test) fix p2 export cases (#60333)
add 0649ef929cd [opt](s3) Skip S3 listing for deterministic file paths
using HEAD requests
add bbee759af91 [opt](s3) Treat non-negated bracket patterns as
deterministic for HEAD request optimization
add 1e8a877cc6d [fix](s3) Skip HEAD optimization for path-style URLs to
avoid false positives
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4702df99f97)
\
N -- N -- N
refs/heads/optimize_s3_skip_list_for_deterministic_paths (1e8a877cc6d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/sonarcloud.yml | 158 -----
be/CMakeLists.txt | 4 +-
be/src/common/config.cpp | 31 +-
be/src/common/config.h | 11 +-
be/src/io/cache/cache_block_meta_store.cpp | 27 +
be/src/io/cache/cache_block_meta_store.h | 3 +
be/src/io/cache/fs_file_cache_storage.cpp | 771 +++++++++++++++++++--
be/src/io/cache/fs_file_cache_storage.h | 69 +-
be/src/olap/accept_null_predicate.h | 6 +-
be/src/olap/column_predicate.h | 32 +-
be/src/olap/comparison_predicate.h | 6 +
be/src/olap/in_list_predicate.h | 7 +
be/src/olap/memtable_flush_executor.cpp | 29 +-
be/src/olap/memtable_flush_executor.h | 3 +
be/src/olap/null_predicate.h | 1 +
.../inverted_index/analyzer/analyzer.cpp | 6 +-
.../rowset/segment_v2/inverted_index_reader.cpp | 7 +-
be/src/pipeline/dependency.h | 6 +
be/src/pipeline/exec/aggregation_sink_operator.h | 4 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 11 +
be/src/pipeline/exec/hashjoin_build_sink.h | 8 +
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 13 +-
.../pipeline/exec/join/process_hash_table_probe.h | 5 +
.../exec/join/process_hash_table_probe_impl.h | 20 +
be/src/pipeline/exec/olap_scan_operator.cpp | 23 +-
be/src/pipeline/exec/scan_operator.cpp | 3 +-
be/src/runtime/workload_group/workload_group.cpp | 29 +
be/src/runtime/workload_group/workload_group.h | 3 +
.../workload_group/workload_group_manager.cpp | 7 +
.../workload_group/workload_group_manager.h | 2 +
.../runtime/workload_management/task_controller.h | 11 +-
be/src/runtime_filter/runtime_filter_consumer.cpp | 7 +
be/src/runtime_filter/runtime_filter_consumer.h | 4 +-
be/src/runtime_filter/runtime_filter_producer.h | 11 +
.../runtime_filter_producer_helper.cpp | 12 +
.../runtime_filter_producer_helper.h | 2 +
be/src/runtime_filter/runtime_filter_selectivity.h | 36 +-
be/src/runtime_filter/runtime_filter_wrapper.h | 18 +
be/src/vec/exprs/vexpr_context.h | 7 +-
be/src/vec/functions/match.cpp | 13 +-
be/src/vec/sink/writer/vfile_result_writer.cpp | 6 +
.../io/cache/block_file_cache_test_meta_store.cpp | 178 +----
.../fs_file_cache_storage_leak_cleaner_test.cpp | 718 +++++++++++++++++++
be/test/olap/accept_null_predicate_test.cpp | 38 +
be/test/olap/memtable_flush_executor_test.cpp | 123 ++++
be/test/runtime/workload_sched_policy_test.cpp | 41 ++
.../runtime_filter_selectivity_test.cpp | 69 +-
.../apache/doris/job/cdc/DataSourceConfigKeys.java | 4 +
.../CommitOffsetRequest.java} | 25 +-
.../doris/job/cdc/request/FetchRecordRequest.java | 12 -
.../job/cdc/request/JobBaseRecordRequest.java | 4 -
.../doris/job/cdc/request/WriteRecordRequest.java | 13 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 2 +-
.../doris/httpv2/rest/StreamingJobAction.java | 17 +-
.../streaming/DataSourceConfigValidator.java | 16 +-
.../insert/streaming/StreamingInsertJob.java | 101 ++-
.../streaming/StreamingJobSchedulerTask.java | 1 +
.../insert/streaming/StreamingJobStatistic.java | 3 +
.../insert/streaming/StreamingMultiTblTask.java | 37 +-
.../apache/doris/job/util/StreamingJobUtils.java | 5 +-
.../apache/doris/load/loadv2/BrokerLoadJob.java | 17 +
.../apache/doris/load/loadv2/LoadLoadingTask.java | 4 +-
.../java/org/apache/doris/metric/MetricRepo.java | 86 +++
.../glue/translator/PhysicalPlanTranslator.java | 4 +-
.../rewrite/EliminateGroupByKeyByUniform.java | 2 +-
.../nereids/rules/rewrite/ExprIdRewriter.java | 59 +-
.../plans/physical/PhysicalDictionarySink.java | 2 +-
.../java/org/apache/doris/qe/SessionVariable.java | 6 +
.../doris/load/loadv2/LoadLoadingTaskTest.java | 57 ++
.../nereids/rules/rewrite/ExprIdRewriterTest.java | 278 ++++++++
.../cdcclient/service/PipelineCoordinator.java | 10 +-
.../doris/cdcclient/sink/DorisBatchStreamLoad.java | 31 +-
.../doris/cdcclient/sink/HttpPutBuilder.java | 9 +-
.../apache/doris/cdcclient/sink/LoadStatistic.java | 29 +-
gensrc/thrift/PaloInternalService.thrift | 3 +-
regression-test/data/export_p2/test_outfile_p2.out | 4 -
.../inverted_index_p0/test_empty_string_match.out | 26 +
...t.out => test_streaming_mysql_job_errormsg.out} | 6 +-
.../distribute/set_operation_colocate.out | 456 ++++++++++++
.../data/query_p0/join/test_left_join1.out | 6 +
.../cloud_p0/auth/test_set_default_cluster.groovy | 2 +-
...est_balance_use_compute_group_properties.groovy | 9 +-
.../test_balance_warm_up_task_abnormal.groovy | 13 +-
.../suites/cloud_p0/multi_cluster/test_tvf.groovy | 2 +-
.../test_clean_tablet_when_drop_force_table.groovy | 9 +-
.../export_p2/test_export_max_file_size.groovy | 107 ++-
.../test_outfile_orc_max_file_size.groovy | 95 +--
.../suites/export_p2/test_outfile_p2.groovy | 72 --
...t_variant_count_on_index_fault_injection.groovy | 1 +
.../test_empty_string_match.groovy | 78 +++
.../cdc/test_streaming_mysql_job.groovy | 4 +-
.../test_streaming_mysql_job_create_alter.groovy | 2 +-
.../cdc/test_streaming_mysql_job_errormsg.groovy | 45 +-
.../cdc/test_streaming_mysql_job_metrics.groovy | 180 +++++
.../cdc/test_streaming_mysql_job_restart_fe.groovy | 10 +-
.../cdc/test_streaming_postgres_job.groovy | 4 +-
.../cdc/test_streaming_postgres_job_split.groovy | 6 +-
.../streaming_job/test_streaming_insert_job.groovy | 6 +-
.../test_streaming_insert_job_offset.groovy | 12 +-
...st_streaming_job_alter_offset_restart_fe.groovy | 24 +-
.../test_streaming_job_restart_fe.groovy | 12 +-
.../test_routin_load_abnormal_job_monitor.groovy | 15 +-
.../distribute/set_operation_colocate.groovy | 72 ++
.../suites/query_p0/join/test_left_join1.groovy | 19 +
.../suites/query_profile/profile_size_limit.groovy | 45 +-
105 files changed, 3919 insertions(+), 859 deletions(-)
delete mode 100644 .github/workflows/sonarcloud.yml
create mode 100644 be/test/io/cache/fs_file_cache_storage_leak_cleaner_test.cpp
copy
fe/fe-common/src/main/java/org/apache/doris/job/cdc/{split/AbstractSourceSplit.java
=> request/CommitOffsetRequest.java} (74%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/load/loadv2/LoadLoadingTaskTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ExprIdRewriterTest.java
copy
fe/fe-core/src/main/java/org/apache/doris/job/disruptor/ExecuteTaskEvent.java
=>
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/sink/LoadStatistic.java
(63%)
delete mode 100644 regression-test/data/export_p2/test_outfile_p2.out
create mode 100644
regression-test/data/inverted_index_p0/test_empty_string_match.out
copy
regression-test/data/job_p0/streaming_job/cdc/{test_streaming_postgres_job_split.out
=> test_streaming_mysql_job_errormsg.out} (78%)
create mode 100644
regression-test/data/nereids_syntax_p0/distribute/set_operation_colocate.out
delete mode 100644 regression-test/suites/export_p2/test_outfile_p2.groovy
create mode 100644
regression-test/suites/inverted_index_p0/test_empty_string_match.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_metrics.groovy
create mode 100644
regression-test/suites/nereids_syntax_p0/distribute/set_operation_colocate.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
