This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a change to branch nested_column_prune
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 44302514c30 fix ut
discard f50cf6b3e42 [feature](reader) Optimize Complex Type Column Reading
with Column Pruning.
add 741a0126402 [bug](test) fix the different test case with same table
name (#58197)
add 61022afb04c [fix](window) allow constant expression evaluation in
LAG/LEAD offset parameter (#58200)
add 83cd8fc34a6 [opt](nereids) when we check if there is a target for
runtime filter, we should use ctx.probeSlot not ctx.probeExpr (#57425)
add ba6bd8b695d [chore](cloud) Expose download command in
tools/fdb/fdb_ctl.sh for convenience (#58257)
add 14a2c498dac [refactor](mysql output)Use to_string when outputting
plain text to MySQL. (#57824)
add 67d6fa91d4f [fix](jni)Add hadoop-mapreduce-client-core dependencies
(#58217)
add 68433a71258 [Fix](package)Revert be-extension package management
changes due to class loading issues (#58255)
add eb393e81f0b [chore](proc) Fix wrong keys type in show proc index
schema (#58201)
add 0598d2a09b5 [fix](test) fix regression sumRewrite (#58232)
add 85b4b2e1f5a [feat](doris compose) fix start be check ulimit fail and
add env variables (#58229)
add 05ee9e7238e [enhancement](parquet)support column predicate tree
min-max filter for parquet page index. (#57771)
add 7c176afb966 [fix](nereids)PushDownJoinOnAssertNumRows rule process
project alias bug (#58174)
add 8f3cece752d [refactor](scan) Remove col_name_to_value_range (#58283)
add bbdff177617 [refine](expr)Use the new execute function in the expr
context. (#58241)
add 0a7b587ca28 [fix](test) Make the result of the test_insert_limit test
case more clearer (#58288)
add 636978a9142 [fix](test) remove spark load test cases (#58269)
add 4557e472faf [fix](test) Add 'waitForSchemaChangeDone' after drop
column from table (#58205)
add e344c67f2c2 [fix](cases) Remove badcase
test_auto_partition_with_single_replica_insert (#58259)
add 460e78b5110 [Improvement](join) add direct mapping opt for join
(#57960)
add 353a10b892f [fix](test) fix regression add_project_for_unique_function
(#58278)
add 7c470ce70cf [fix](regression-case) Fix test_dml_stream_load_auth, use
sync function but not async (#58286)
add 9031e7cf822 [be](chore) remove useless Schema in push handler (#58300)
add bf080303f17 [fix](nereids) setOperator column prunning bug (#58237)
add 2d3b1b67c68 [fix](filecache) fix regression failure due to data
compression (#58246)
add 689377ac851 [fix](test) fix unstable check_before_quit cases (#58295)
add 5704db91e5b [imp](checker) add NPE check for mysql_fetch_row to
prevent coredump (#57853)
add 3373b56aa9b [fix](type) Unexpected throw exception due to unknown file
type (#57755)
add 6d54dd447e9 [fix](test) fix unstable query result (#58293)
add af6148900d7 [FIX](regression-test) Fix case test_s3_tvf_number_range,
use the right file path in HDFS (#58292)
add b3780780733 [fix](auth)Fixed the issue nereids not check for the
existence of permission items (#58282)
add 96c1b993ac0 [fix](blockrule)Fix concurrency issues in SqlBlockRuleMgr
after restart fe (#58296)
add 2c97c2e18c0 [fix](case)Fix temp table case (#58299)
add c8b130ad82a [fix](variable) support set ON and ALL to variable (#58287)
add a13241b76c7 [feature](reader) Optimize Complex Type Column Reading
with Column Pruning (#57204)
add b4db4c3cce5 [opt](function) better signature match for time type
(#58193)
add 1f0b95f4076 [fix](inverted index) fix multi-position phrase query
handling in MultiPhraseQuery (#57993)
add 8471d2cc623 [fix](deps) update hadoop libs to 3.4.2.2 to support jdk17
(#58307)
add 79729b03841 [refactor](scanner) remove scanner scheduler class since
it is already split into SimplifiedScheduler (#58310)
new 993e70869c9 fix
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (44302514c30)
\
N -- N -- N refs/heads/nested_column_prune (993e70869c9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/rowid_fetcher.cpp | 4 +-
be/src/io/file_factory.h | 5 +-
be/src/olap/block_column_predicate.cpp | 32 +
be/src/olap/block_column_predicate.h | 23 +
be/src/olap/column_predicate.h | 7 +
be/src/olap/comparison_predicate.h | 61 +-
be/src/olap/in_list_predicate.h | 60 +-
be/src/olap/null_predicate.h | 15 +
be/src/olap/push_handler.cpp | 21 +-
be/src/olap/push_handler.h | 5 +-
.../inverted_index/query/query_helper.cpp | 29 +
.../segment_v2/inverted_index/query/query_helper.h | 3 +
.../ISegmenter.h => inverted_index_common.cpp} | 31 +-
.../olap/rowset/segment_v2/inverted_index_common.h | 62 +-
.../rowset/segment_v2/inverted_index_common_impl.h | 61 +
be/src/olap/rowset/segment_v2/row_ranges.h | 2 +
be/src/olap/schema_change.cpp | 3 +-
be/src/pipeline/common/join_utils.h | 104 +-
be/src/pipeline/exec/file_scan_operator.cpp | 2 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 44 +-
be/src/pipeline/exec/hashjoin_build_sink.h | 6 +-
.../exec/join/process_hash_table_probe_impl.h | 33 +-
.../exec/partitioned_hash_join_probe_operator.cpp | 2 +-
be/src/runtime/exec_env.h | 3 -
be/src/runtime/exec_env_init.cpp | 5 -
be/src/runtime/query_context.h | 8 +-
be/src/runtime/workload_group/workload_group.cpp | 8 +-
be/src/runtime/workload_group/workload_group.h | 10 +-
be/src/service/internal_service.cpp | 29 +-
be/src/vec/common/hash_table/hash_map_context.h | 65 +
be/src/vec/common/hash_table/join_hash_table.h | 59 +-
.../data_types/serde/data_type_bitmap_serde.cpp | 15 +
.../vec/data_types/serde/data_type_bitmap_serde.h | 3 +
.../vec/data_types/serde/data_type_hll_serde.cpp | 15 +
be/src/vec/data_types/serde/data_type_hll_serde.h | 3 +
.../vec/data_types/serde/data_type_jsonb_serde.cpp | 6 +
.../data_types/serde/data_type_nullable_serde.cpp | 10 +
.../data_types/serde/data_type_nullable_serde.h | 2 +
.../data_types/serde/data_type_number_serde.cpp | 12 +-
.../serde/data_type_quantilestate_serde.cpp | 18 +
.../serde/data_type_quantilestate_serde.h | 3 +
be/src/vec/data_types/serde/data_type_serde.cpp | 6 +
be/src/vec/data_types/serde/data_type_serde.h | 5 +
be/src/vec/exec/format/avro/avro_jni_reader.cpp | 6 +-
be/src/vec/exec/format/avro/avro_jni_reader.h | 4 +-
be/src/vec/exec/format/jni_reader.cpp | 6 +-
be/src/vec/exec/format/jni_reader.h | 6 +-
be/src/vec/exec/format/orc/vorc_reader.cpp | 17 +-
be/src/vec/exec/format/orc/vorc_reader.h | 15 +-
be/src/vec/exec/format/parquet/parquet_common.h | 18 +-
be/src/vec/exec/format/parquet/parquet_predicate.h | 29 +-
.../exec/format/parquet/vparquet_column_reader.cpp | 49 +-
.../exec/format/parquet/vparquet_column_reader.h | 30 +-
.../exec/format/parquet/vparquet_group_reader.cpp | 51 +-
.../exec/format/parquet/vparquet_group_reader.h | 5 +-
.../exec/format/parquet/vparquet_page_index.cpp | 18 +-
.../vec/exec/format/parquet/vparquet_page_index.h | 7 +-
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 514 +-
be/src/vec/exec/format/parquet/vparquet_reader.h | 56 +-
be/src/vec/exec/format/table/hive_reader.cpp | 26 +-
be/src/vec/exec/format/table/hive_reader.h | 4 -
be/src/vec/exec/format/table/hudi_jni_reader.cpp | 6 +-
be/src/vec/exec/format/table/hudi_jni_reader.h | 4 +-
be/src/vec/exec/format/table/hudi_reader.cpp | 13 +-
be/src/vec/exec/format/table/hudi_reader.h | 9 +-
be/src/vec/exec/format/table/iceberg_reader.cpp | 41 +-
be/src/vec/exec/format/table/iceberg_reader.h | 12 +-
.../format/table/iceberg_sys_table_jni_reader.cpp | 5 +-
.../format/table/iceberg_sys_table_jni_reader.h | 3 +-
.../vec/exec/format/table/lakesoul_jni_reader.cpp | 5 +-
be/src/vec/exec/format/table/lakesoul_jni_reader.h | 3 +-
.../exec/format/table/max_compute_jni_reader.cpp | 6 +-
.../vec/exec/format/table/max_compute_jni_reader.h | 4 +-
be/src/vec/exec/format/table/paimon_jni_reader.cpp | 6 +-
be/src/vec/exec/format/table/paimon_jni_reader.h | 4 +-
be/src/vec/exec/format/table/paimon_reader.h | 15 +-
.../format/table/paimon_sys_table_jni_reader.cpp | 6 +-
.../format/table/paimon_sys_table_jni_reader.h | 4 +-
.../format/table/transactional_hive_reader.cpp | 16 +-
.../exec/format/table/transactional_hive_reader.h | 6 +-
.../format/table/trino_connector_jni_reader.cpp | 5 +-
.../exec/format/table/trino_connector_jni_reader.h | 3 +-
be/src/vec/exec/jni_connector.cpp | 25 +-
be/src/vec/exec/jni_connector.h | 6 +-
be/src/vec/exec/scan/file_scanner.cpp | 93 +-
be/src/vec/exec/scan/file_scanner.h | 2 -
be/src/vec/exec/scan/meta_scanner.cpp | 6 +-
be/src/vec/exec/scan/scanner_context.cpp | 5 +-
be/src/vec/exec/scan/scanner_context.h | 9 +-
be/src/vec/exec/scan/scanner_scheduler.cpp | 22 +-
be/src/vec/exec/scan/scanner_scheduler.h | 74 +-
be/src/vec/exprs/vexpr_context.cpp | 64 +-
be/src/vec/exprs/vexpr_context.h | 13 +-
be/src/vec/functions/function_search.cpp | 30 +-
be/src/vec/sink/vmysql_result_writer.cpp | 84 +-
be/src/vec/sink/writer/vfile_result_writer.cpp | 5 +-
.../rowset/segment_v2/index_reader_helper_test.cpp | 2 +-
be/test/olap/wal/wal_manager_test.cpp | 4 +-
be/test/scan/mock_scanner_scheduler.h | 30 -
be/test/scan/scanner_context_test.cpp | 12 +-
.../vec/exec/format/parquet/parquet_expr_test.cpp | 24 +-
.../vec/exec/format/parquet/parquet_read_lines.cpp | 7 +-
.../exec/format/parquet/parquet_reader_test.cpp | 75 +-
.../exec/format/table/hive/hive_reader_test.cpp | 11 +-
.../format/table/iceberg/iceberg_reader_test.cpp | 14 +-
be/test/vec/exec/orc/orc_read_lines.cpp | 4 +-
be/test/vec/exec/orc_reader_test.cpp | 4 +-
be/test/vec/exec/vfile_scanner_exception_test.cpp | 4 +-
be/test/vec/function/function_search_test.cpp | 85 +-
cloud/src/recycler/meta_checker.cpp | 49 +-
docker/runtime/doris-compose/cluster.py | 21 +-
docker/runtime/doris-compose/command.py | 10 +-
.../scripts/create_preinstalled_scripts/run82.hql | 14 +
.../decimals_1_10/decimals_1_10.parquet | Bin 0 -> 3874 bytes
fe/be-java-extensions/avro-scanner/pom.xml | 8 +-
fe/be-java-extensions/hadoop-deps/pom.xml | 4 +
fe/be-java-extensions/hadoop-hudi-scanner/pom.xml | 34 +-
.../iceberg-metadata-scanner/pom.xml | 10 +-
fe/be-java-extensions/java-udf/pom.xml | 3 +-
fe/be-java-extensions/jdbc-scanner/pom.xml | 3 +-
fe/be-java-extensions/max-compute-scanner/pom.xml | 8 +-
fe/be-java-extensions/paimon-scanner/pom.xml | 14 +-
fe/be-java-extensions/preload-extensions/pom.xml | 9 +-
.../trino-connector-scanner/pom.xml | 3 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 7 +-
.../apache/doris/blockrule/SqlBlockRuleMgr.java | 18 +-
.../apache/doris/common/proc/IndexInfoProcDir.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 13 +
.../post/RuntimeFilterPushDownVisitor.java | 2 +-
.../nereids/rules/analysis/CheckAfterRewrite.java | 12 +
.../doris/nereids/rules/rewrite/ColumnPruning.java | 12 +-
.../rules/rewrite/PushDownJoinOnAssertNumRows.java | 19 +-
.../nereids/rules/rewrite/PushDownProject.java | 33 +-
.../expressions/functions/SearchSignature.java | 46 +-
.../trees/expressions/functions/window/Lag.java | 2 +-
.../trees/expressions/functions/window/Lead.java | 4 +-
.../trees/expressions/literal/TimeV2Literal.java | 61 +
.../literal/format/DateTimeChecker.java | 12 +-
.../expressions/literal/format/TimeChecker.java | 71 +
.../plans/physical/PhysicalNestedLoopJoin.java | 8 +-
.../doris/nereids/util/TypeCoercionUtils.java | 7 +
.../doris/blockrule/SqlBlockRuleMgrTest.java} | 38 +-
.../doris/nereids/parser/NereidsParserTest.java | 6 +
.../rewrite/PushDownJoinOnAssertNumRowsTest.java | 3 +-
.../expressions/literal/DateTimeLiteralTest.java | 1 -
.../literal/format/TimeCheckerTest.java | 61 +
.../data/cloud_p0/cache/ttl/test_show_ttl.out | 4 +-
.../sql-functions/doc_date_functions_test.out | 6 +-
.../hive/test_hive_get_schema_from_table.out | 2 +-
.../hive/test_hive_page_index.out | 51888 +++++++++++++++++++
.../data/load_p0/spark_load/all_types1.txt | 11 -
.../data/load_p0/spark_load/all_types2.txt | 11 -
.../data/load_p0/spark_load/test_spark_load.out | 37 -
.../data/nereids_function_p0/scalar_function/S.out | 173 +-
.../data/nereids_rules_p0/sumRewrite.out | 58 +-
.../add_project_for_unique_function.out | 13 +-
.../lag_lead_signature.out} | 20 +-
...t_auto_partition_with_single_replica_insert.csv | 78 -
...t_auto_partition_with_single_replica_insert.out | 161 -
.../data/query_p1/test_insert_limit.out | 94 +-
.../tpcds_sf100/noStatsRfPrune/query2.out | 8 +-
.../tpcds_sf100/noStatsRfPrune/query38.out | 78 +-
.../tpcds_sf100/noStatsRfPrune/query87.out | 78 +-
.../tpcds_sf100/no_stats_shape/query2.out | 8 +-
.../tpcds_sf100/no_stats_shape/query38.out | 78 +-
.../tpcds_sf100/no_stats_shape/query87.out | 78 +-
.../shape_check/tpcds_sf100/rf_prune/query2.out | 4 +-
.../shape_check/tpcds_sf100/rf_prune/query38.out | 78 +-
.../shape_check/tpcds_sf100/rf_prune/query59.out | 4 +-
.../shape_check/tpcds_sf100/rf_prune/query87.out | 78 +-
.../data/shape_check/tpcds_sf100/shape/query2.out | 4 +-
.../data/shape_check/tpcds_sf100/shape/query38.out | 78 +-
.../data/shape_check/tpcds_sf100/shape/query59.out | 4 +-
.../data/shape_check/tpcds_sf100/shape/query87.out | 78 +-
.../data/shape_check/tpcds_sf1000/hint/query2.out | 4 +-
.../data/shape_check/tpcds_sf1000/hint/query38.out | 78 +-
.../data/shape_check/tpcds_sf1000/hint/query59.out | 4 +-
.../data/shape_check/tpcds_sf1000/hint/query87.out | 78 +-
.../data/shape_check/tpcds_sf1000/shape/query2.out | 4 +-
.../shape_check/tpcds_sf1000/shape/query38.out | 78 +-
.../shape_check/tpcds_sf1000/shape/query54.out | 30 +-
.../shape_check/tpcds_sf1000/shape/query59.out | 4 +-
.../shape_check/tpcds_sf1000/shape/query87.out | 78 +-
.../shape_check/tpcds_sf10t_orc/shape/query2.out | 8 +-
.../shape_check/tpcds_sf10t_orc/shape/query59.out | 6 +-
.../doris/regression/suite/SuiteCluster.groovy | 7 +
.../auth_call/test_dml_stream_load_auth.groovy | 21 +-
.../suites/auth_p0/test_grant_auth.groovy | 14 +
.../check_before_quit/check_before_quit.groovy | 12 +-
.../cloud_p0/cache/ttl/alter_ttl_max_int64.groovy | 2 +-
.../cloud_p0/cache/ttl/create_table_like.groovy | 2 +-
.../suites/cloud_p0/cache/ttl/test_ttl.groovy | 2 +-
.../cloud_p0/cache/ttl/test_ttl_lru_evict.groovy | 2 +-
.../cloud_p0/cache/ttl/test_ttl_preempt.groovy | 2 +-
.../suites/cloud_p0/recycler/check_meta.groovy | 37 +
.../suites/demo_p0/docker_action.groovy | 4 +-
.../hive/test_hive_page_index.groovy | 98 +
.../tvf/test_s3_tvf_number_range.groovy | 16 +-
.../jsonb_p0/test_jsonb_predict_is_null.groovy | 5 +
.../load_p0/spark_load/test_spark_load.groovy | 150 -
.../nereids_function_p0/scalar_function/S.groovy | 3 +
.../nereids_syntax_p0/lag_lead_signature.groovy | 31 +-
...uto_partition_with_single_replica_insert.groovy | 78 -
.../binary_functions/test_binary_function.groovy | 2 +-
.../test_binary_for_digest.groovy | 2 +-
.../suites/query_p1/test_insert_limit.groovy | 25 +-
.../shape_check/tpcds_sf1000/shape/query54.groovy | 2 +-
.../suites/temp_table_p0/test_temp_table.groovy | 6 +-
thirdparty/vars.sh | 8 +-
tools/fdb/fdb_ctl.sh | 5 +-
210 files changed, 54927 insertions(+), 2472 deletions(-)
copy
be/src/olap/rowset/segment_v2/{inverted_index/analyzer/ik/core/ISegmenter.h =>
inverted_index_common.cpp} (63%)
create mode 100644 be/src/olap/rowset/segment_v2/inverted_index_common_impl.h
delete mode 100644 be/test/scan/mock_scanner_scheduler.h
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run82.hql
create mode 100644
docker/thirdparties/docker-compose/hive/scripts/preinstalled_data/parquet_table/decimals_1_10/decimals_1_10.parquet
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/format/TimeChecker.java
copy fe/fe-core/src/{main/java/org/apache/doris/metric/AutoMappedMetric.java
=> test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java} (51%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/format/TimeCheckerTest.java
create mode 100644
regression-test/data/external_table_p0/hive/test_hive_page_index.out
delete mode 100644 regression-test/data/load_p0/spark_load/all_types1.txt
delete mode 100644 regression-test/data/load_p0/spark_load/all_types2.txt
delete mode 100644 regression-test/data/load_p0/spark_load/test_spark_load.out
copy regression-test/data/{correctness_p0/test_cast_null.out =>
nereids_syntax_p0/lag_lead_signature.out} (73%)
delete mode 100644
regression-test/data/partition_p0/auto_partition/test_auto_partition_with_single_replica_insert.csv
delete mode 100644
regression-test/data/partition_p0/auto_partition/test_auto_partition_with_single_replica_insert.out
create mode 100644
regression-test/suites/external_table_p0/hive/test_hive_page_index.groovy
delete mode 100644
regression-test/suites/load_p0/spark_load/test_spark_load.groovy
delete mode 100644
regression-test/suites/partition_p0/auto_partition/test_auto_partition_with_single_replica_insert.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
