This is an automated email from the ASF dual-hosted git repository.
gabotechs pushed a change to branch gabotechs/add-q-error-stats-benchmarks
in repository https://gitbox.apache.org/repos/asf/datafusion.git
discard f08df357fc Do not abort on parse errors
omit 6366a30fae Add some focused unit tests
omit b96a42cd26 Rollback unintended changes
omit 13ccf04b38 Avoid registering a duplicate table
omit 310edba3e6 Store reports even on failures
omit b516bec4e8 Use parser for splitting statements
omit 141de83d8d Add dfbench statistics command
add cd28203e3e perf: `array_agg()` performance improvements (#23716)
add 5f7feda66b fix: accept LargeUtf8 and Utf8View patterns in SIMILAR TO
planning (#23735)
add 1727b7d709 Fill in missing utf8view support in function type coercion
(#23916)
add 043d97fb2a perf: Optimize hashing, null-free fast path for
`percentile_cont`, `median` (#23954)
add 5941552e46 perf: null-free fast path for COUNT(DISTINCT) primitive
accumulator (#23956)
add 1955d5af75 chore: refactor `VarianceAccumulator`, add tests and
benchmark (#23977)
add 2fb472a217 test: add regression coverage and docs for NULL format
handling (#23669)
add 2f25454ad0 fix: preserve aggregate filter pushdown order (#22926)
add 2e3626e72e minor(test): cover partially ordered aggregate spilling
(#23947)
add 95226ac7d0 feat: centralizing higher-order list lambda evaluation
helpers (#23911)
add 0f7bdddd37 fix(common): preserve an exact zero through filter
selectivity estimation (#23936)
add 86da5a85a2 chore(ordered-partial-aggregate): move
`OrderedPartialAggregateStream` to generators for readability (#23951)
add 46510030a0 test: improve `round` sqllogictest coverage (#23973)
add 802d8d2dc1 test: improve `gcd` sqllogictest coverage (#23972)
add cc1326a609 fix: preserve dictionary-value nulls in scalar regex
operators (#23966)
add 0bcbe40024 test: improve `rpad` sqllogictest coverage (#23968)
add 64bc392b52 test: improve `lpad` sqllogictest coverage (#23969)
add eacdf7554a Add benchmarks for hashjoin candidate equality filtering
(#23980)
add 455a3add52 docs: Fixes incorrect type name in `UserDefinedLogicalNode`
comment (#23992)
add 44aae94865 perf: precompile formats in to_time (#23964)
add 6a0e771549 Optimize Spark hex null handling (#23688)
add 62cfc0ceae fix(datasource): avoid over-conservative transformation of
num_rows statistics in file scan config (#23670)
add 3aef16c740 refactor(proto): migrate scalar subquery serde (#23915)
add 541caab85a refactor: mark the ExecutionPlan proto dispatch traits as
non-public API (#24001)
add 833e501459 fix(proto): prevent duplicate partition statistics on
roundtrip (#23999)
add f398301748 Report peak MemoryPool reservation per query in benchmarks
(#23985)
add 30ae8bf340 refactor(proto): move PartitionedFile / FileGroup serde
into datafusion-datasource (#24006)
add 1ae6b872aa perf: Replace SipHash with foldhash in
`BoundedWindowAggExec` (#23984)
add 39d5064142 chore: refactor `MaterializingSortMergeJoinStream` into
generators and simplify code to be textbook like as possible (#23976)
add a589b4bbae refactor(proto): put Partitioning / sort-expression serde
on the types (#24003)
add 6636d6be25 fix: keep a CoalescePartitionsExec required by a
SinglePartition child (#23948)
add f8b9ed81f0 fix: TopK aggregation drops groups whose MIN/MAX value is
NULL (#23684)
add aa9b7bb63f feat: add GroupColumn support for Duration in multi-column
GROUP BY (#23783)
add dbcb5c0f72 refactor: move planning APIs to session crate (#23842)
add 2eaba476e7 bench: use seedable rng for reproducibility (#23653)
add 285559e44f test: Fix data_pagesize_limit extraction in parquet writer
props roundtrip test (#23664)
add 7547280712 bench: extend BoundedWindowAggExec many-partitions
benchmark (#24032)
add 219d256362 feat: add GroupColumn support for Float16 in multi-column
GROUP BY (#23785)
add 9f1c5c92b9 fix(sql): preserve source qualifiers in CTAS with explicit
schema (#23879)
add 5cf7bff9f8 refactor: move arrow integer hex dispatch to
datafusion-common (#23917)
add 062cbcbc6b test: add IN list slt coverage for temporal, Decimal128 and
Interval types (#23875)
add f2b483598b feat: Add support for `explode_outer` function for arrays
(#22100)
add 8c46388903 fix: reject nested arrays in array_distance (#23995)
add b4cde12a85 fix: Improve error message for metadata conflict in schema
(#23952)
add b9022565e2 fix: rows_to_array cleanup for expecting single field
(#24040)
add f3a895ffb2 minor: Add `slt` test for nullable window retract (#24025)
add 60d9b67ec9 WindowTopN dense_rank benchmark (#24050)
add eb8e38e1de minor(fix): correct to_date results for formatted pre-epoch
datetimes (#24049)
add 66c3840c02 minor(test): strengthen sort-merge join spilling coverage
(#23988)
add 212d6137f1 refactor(hash-aggr): Support spilling for single mode
aggregation (#23965)
add 21ad189782 test: improve `find_in_set` sqllogictest coverage (#23970)
add 37baf4293d perf: preserve dictionary encoding for `character_length`,
`initcap`, and `reverse` (#23930)
add 9051efdb87 feat: switch VirtualTable producer to use expressions field
instead of deprecated values (#23672)
add 62650eff81 feat: add GroupColumn support for Interval in multi-column
GROUP BY (#23786)
add 28c8afa751 IN LIST: isolate branchless filter implementation (#23907)
add f9dde71ec3 feat: drop generator on error to free memory faster (#23967)
add 882d90658a docs: Add more documentation about `PartialSortExec`
operator (#24048)
add 47ca49043a perf: skip re-slicing window partition batches with nothing
to prune (#24047)
add b2ac10f977 feat: eliminate LEFT/RIGHT JOINs with redundant sides
(#23566)
add f248f451bc fix: handle empty patterns in regexp_instr (#24054)
add 17694ac0d0 feat(parquet): multi-column lexicographic stats reorder for
TopK sort pushdown (#23888)
add a3f0f939a4 fix: prevent incorrect results when pushing filters through
anti joins (#24045)
add 3ef9a8c221 perf: track `BoundedWindowAggExec` Linear-mode watermark
once per stream (#24033)
add 26fbf4d015 bench: add nested-type (List/Struct/Map) cases to
first_value/last_value benchmark (#24075)
add db0c31bfe1 fix(ffi): preserve aggregate null-handling support (#23908)
add bf613a1b3f chore(deps): bump taiki-e/install-action from 2.85.2 to
2.85.6 (#24081)
add 597170e925 chore(deps): bump the codeql-actions group with 2 updates
(#24080)
add 9114e78953 bench: add ArrowBytesMap benchmarks (#24078)
add 179b32c9b6 perf: gather Linear-mode window input more efficiently
(#24034)
add 30eccb44e9 Docs: Update PR template to ask for user-visible rationale
(#24053)
add c8665ce95a docs: document all fields and methods of
`DFParquetMetadata` (#24037)
add bc8b1a7a93 chore(deps): bump cryptography from 48.0.1 to 50.0.0
(#24091)
add 9eb31bfd16 fix(physical-plan): preserve Exact(0) in FilterExec for
null_count, distinct_count and total_byte_size upon empty input (#24000)
add e948f17dbf fix: UnionExec now conforms each batch to the union's
declared schema (#23861)
add e8015e41a7 refactor: unify `ParquetFileReader` and
`CachedParquetFileReader` (#24036)
add f6d4f04dc4 chore: cleanup `OrderedPartialAggregateStream` more (#24012)
add d813358848 perf: use Vec in ArrowBytesMap (#24071)
add c1366b5549 chore: apply workspace lints to all crates (#24076)
add 31ffab1418 feat(functions-aggregate): support nested types (List,
Struct, Map) in first_value / last_value GroupsAccumulator (#23628)
add e7304eec73 fix: do not derive ordering for arithmetic that can
overflow (#23910)
add a0124a4f0f fix: preserve projection field metadata during physical
planning (#23981)
add 2dd1a14232 perf: preallocate RowsGroupColumn buffers in take_n (#24070)
add 426b351513 Add config-matrix tests in enforce_distribution.rs for
range-satisfaction settings (#23627)
add 9db5c5de5e feat: add Spark-compatible atan2 function (#23962)
add c4a539bd8e chore(deps-dev): bump fast-uri from 3.1.4 to 3.1.5 in
/datafusion/wasmtest/datafusion-wasm-app (#24092)
add 3e3a92de29 feat: Calculate non-distinct `sum` from column statistics
when available (#23863)
add 373fab7bd5 Add support for running sql benchmarks with command line
arguments (#23772)
add ce2f153ebf fix(proto): preserve empty projection when ser/de
MemoryScanExec (#24087)
add 5eba27f5bd feat: prune unread Parquet leaves when a nested column is
cast to a narrower type (#24090)
add aa38d3c6b4 feat(pruning): expose pruning predicate IN-list rewrite
size cap as a config option (#24074)
add f27e50c136 refactor join-key equality filtering (#23843)
add 70c26a0671 fix: Fix nullability of logical `InSubquery` expression
(#23429)
new 6b39df17a0 Add dfbench statistics command
new a2123e70e7 Use parser for splitting statements
new 4d6565e001 Store reports even on failures
new ffd58c4b50 Avoid registering a duplicate table
new e1dad5cbb8 Rollback unintended changes
new 4f1c721d52 Add some focused unit tests
new e2f79d3ce7 Do not abort on parse errors
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f08df357fc)
\
N -- N -- N refs/heads/gabotechs/add-q-error-stats-benchmarks
(e2f79d3ce7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/ISSUE_TEMPLATE/feature_request.yml | 2 +-
.github/pull_request_template.md | 11 +-
.github/workflows/audit.yml | 2 +-
.github/workflows/breaking_changes_detector.yml | 2 +-
.github/workflows/codeql.yml | 4 +-
.github/workflows/dependencies.yml | 2 +-
.github/workflows/dev.yml | 6 +-
.github/workflows/docs.yaml | 2 +-
.github/workflows/docs_pr.yaml | 2 +-
.github/workflows/rust.yml | 8 +-
Cargo.lock | 57 +-
Cargo.toml | 17 +-
benchmarks/Cargo.toml | 8 +-
benchmarks/README.md | 8 +
benchmarks/benches/sql.rs | 6 +-
benchmarks/queries/h2o/window.sql | 50 +
benchmarks/sql_benchmarks/README.md | 119 +-
.../sql_benchmarks/clickbench/clickbench.suite | 25 +
.../clickbench_extended/clickbench_extended.suite | 21 +
.../clickbench_sorted/clickbench_sorted.suite | 28 +
benchmarks/sql_benchmarks/h2o/h2o.suite | 33 +
.../sql_benchmarks/hj/benchmarks/q24.benchmark | 31 +
.../sql_benchmarks/hj/benchmarks/q25.benchmark | 33 +
benchmarks/sql_benchmarks/hj/hj.suite | 21 +
benchmarks/sql_benchmarks/imdb/imdb.suite | 26 +
benchmarks/sql_benchmarks/nlj/nlj.suite | 11 +
.../predicate_eval/predicate_eval.suite | 33 +-
.../push_down_topk/push_down_topk.suite | 21 +
benchmarks/sql_benchmarks/smj/smj.suite | 11 +
.../sql_benchmarks/sort_tpch/sort_tpch.suite | 28 +
benchmarks/sql_benchmarks/tpcds/tpcds.suite | 21 +
benchmarks/sql_benchmarks/tpch/tpch.suite | 47 +
.../sql_benchmarks/wide_schema/wide_schema.suite | 14 +
benchmarks/src/bin/benchmark_runner.rs | 1463 ++++++++++++++++++--
benchmarks/src/bin/external_aggr.rs | 16 +-
benchmarks/src/cancellation.rs | 16 +-
benchmarks/src/clickbench.rs | 3 +-
benchmarks/src/dict.rs | 1 +
benchmarks/src/h2o.rs | 3 +-
benchmarks/src/hj.rs | 47 +
benchmarks/src/imdb/run.rs | 13 +-
benchmarks/src/lib.rs | 1 +
benchmarks/src/nlj.rs | 1 +
benchmarks/src/smj.rs | 1 +
benchmarks/src/sort_pushdown.rs | 13 +-
benchmarks/src/sort_tpch.rs | 14 +-
benchmarks/src/sql_benchmark_runner.rs | 227 ++-
benchmarks/src/sql_benchmark_suite.rs | 849 ++++++++++++
benchmarks/src/tpcds/run.rs | 3 +-
benchmarks/src/tpch/run.rs | 3 +-
benchmarks/src/util/memory.rs | 30 +-
benchmarks/src/util/memory_pool.rs | 381 +++++
benchmarks/src/util/mod.rs | 2 +
benchmarks/src/util/options.rs | 5 +-
benchmarks/src/util/run.rs | 140 +-
.../examples/dataframe/cache_factory.rs | 5 +-
.../examples/relation_planner/table_sample.rs | 9 +-
datafusion/common/src/config.rs | 11 +
.../common/src/file_options/parquet_writer.rs | 7 +-
datafusion/common/src/lib.rs | 2 +-
datafusion/common/src/stats.rs | 44 +-
datafusion/common/src/unnest.rs | 93 +-
datafusion/common/src/utils/hex.rs | 40 +
datafusion/core/benches/map_query_sql.rs | 9 +-
.../core/benches/parquet_nested_schema_pruning.rs | 36 +-
datafusion/core/benches/parquet_query_sql.rs | 36 +-
datafusion/core/benches/parquet_struct_query.rs | 11 +-
datafusion/core/src/execution/context/mod.rs | 19 +-
datafusion/core/src/execution/session_state.rs | 25 +-
datafusion/core/src/physical_planner.rs | 516 ++++---
datafusion/core/tests/dataframe/mod.rs | 151 ++
.../core/tests/fuzz_cases/equivalence/ordering.rs | 33 +-
.../tests/fuzz_cases/equivalence/projection.rs | 32 +-
.../core/tests/fuzz_cases/equivalence/utils.rs | 19 +-
.../memory_limit/memory_limit_validation/mod.rs | 1 +
.../memory_limit_validation/smj_mem_validation.rs | 105 ++
.../memory_limit_validation/sort_mem_validation.rs | 48 +-
.../memory_limit/memory_limit_validation/utils.rs | 107 +-
.../tests/memory_limit/union_nullable_spill.rs | 13 +-
.../tests/parquet/dynamic_row_group_pruning.rs | 152 ++
.../physical_optimizer/aggregate_statistics.rs | 221 ++-
.../physical_optimizer/enforce_distribution.rs | 95 +-
.../physical_optimizer/ensure_requirements.rs | 145 +-
.../tests/physical_optimizer/filter_pushdown.rs | 227 ++-
.../core/tests/physical_optimizer/test_utils.rs | 23 +-
datafusion/core/tests/sql/mod.rs | 1 +
datafusion/core/tests/sql/union_nullable.rs | 204 +++
.../core/tests/user_defined/user_defined_plan.rs | 7 +-
datafusion/datasource-parquet/src/access_plan.rs | 345 ++++-
datafusion/datasource-parquet/src/bloom_filter.rs | 12 +-
datafusion/datasource-parquet/src/metadata.rs | 85 +-
datafusion/datasource-parquet/src/mod.rs | 1 +
.../src/nested_schema_pruning.rs | 775 +++++++++++
datafusion/datasource-parquet/src/opener/mod.rs | 26 +-
.../datasource-parquet/src/projection_read_plan.rs | 725 +++++++++-
datafusion/datasource-parquet/src/push_decoder.rs | 23 +-
datafusion/datasource-parquet/src/reader.rs | 156 +--
datafusion/datasource-parquet/src/sort.rs | 95 +-
datafusion/datasource-parquet/src/source.rs | 73 +
datafusion/datasource/Cargo.toml | 5 +
datafusion/datasource/src/file_scan_config/mod.rs | 45 +-
datafusion/datasource/src/mod.rs | 4 +
datafusion/datasource/src/proto.rs | 238 ++++
datafusion/execution/src/async_stream.rs | 69 +-
datafusion/expr/src/expr.rs | 61 +-
datafusion/expr/src/expr_fn.rs | 3 +-
datafusion/expr/src/expr_rewriter/mod.rs | 7 +-
datafusion/expr/src/expr_schema.rs | 88 +-
datafusion/expr/src/logical_plan/extension.rs | 2 +-
datafusion/expr/src/tree_node.rs | 6 +-
datafusion/expr/src/type_coercion/functions.rs | 8 +-
datafusion/expr/src/window_state.rs | 13 -
datafusion/ffi/src/session/mod.rs | 10 +
datafusion/ffi/src/tests/mod.rs | 9 +-
datafusion/ffi/src/tests/udf_udaf_udwf.rs | 7 +
datafusion/ffi/src/udaf/mod.rs | 28 +
datafusion/ffi/tests/ffi_udaf.rs | 16 +
.../src/aggregate/count_distinct/native.rs | 52 +-
datafusion/functions-aggregate/Cargo.toml | 5 +
.../functions-aggregate/benches/first_last.rs | 272 +++-
datafusion/functions-aggregate/benches/variance.rs | 83 ++
datafusion/functions-aggregate/src/array_agg.rs | 408 ++++--
datafusion/functions-aggregate/src/first_last.rs | 341 ++++-
.../functions-aggregate/src/first_last/state.rs | 459 +++++-
datafusion/functions-aggregate/src/median.rs | 88 +-
.../functions-aggregate/src/percentile_cont.rs | 80 +-
datafusion/functions-aggregate/src/sum.rs | 129 +-
datafusion/functions-aggregate/src/variance.rs | 104 +-
datafusion/functions-nested/benches/map.rs | 18 +-
datafusion/functions-nested/src/array_any_match.rs | 134 +-
datafusion/functions-nested/src/array_filter.rs | 115 +-
datafusion/functions-nested/src/array_first.rs | 128 +-
datafusion/functions-nested/src/distance.rs | 68 +-
datafusion/functions-nested/src/lambda_utils.rs | 269 +++-
datafusion/functions/Cargo.toml | 2 +-
datafusion/functions/benches/concat.rs | 18 +-
datafusion/functions/benches/concat_ws.rs | 10 +-
datafusion/functions/benches/date_bin.rs | 7 +-
.../functions/benches/dictionary_encoding.rs | 8 +-
datafusion/functions/benches/gcd.rs | 4 +-
datafusion/functions/benches/lcm.rs | 4 +-
datafusion/functions/benches/make_date.rs | 15 +-
datafusion/functions/benches/pad.rs | 10 +-
datafusion/functions/benches/regx.rs | 36 +-
datafusion/functions/benches/to_char.rs | 40 +-
datafusion/functions/benches/to_local_time.rs | 9 +-
datafusion/functions/benches/to_time.rs | 11 +-
datafusion/functions/src/datetime/to_date.rs | 44 +-
datafusion/functions/src/datetime/to_time.rs | 31 +-
datafusion/functions/src/datetime/to_timestamp.rs | 15 +-
datafusion/functions/src/datetime/to_unixtime.rs | 2 +-
datafusion/functions/src/regex/regexpinstr.rs | 90 +-
datafusion/functions/src/string/to_hex.rs | 46 +-
.../functions/src/unicode/character_length.rs | 31 +-
datafusion/functions/src/unicode/initcap.rs | 115 +-
datafusion/functions/src/unicode/reverse.rs | 35 +-
datafusion/optimizer/src/eliminate_join.rs | 126 +-
.../src/simplify_expressions/expr_simplifier.rs | 30 +
datafusion/physical-expr-common/Cargo.toml | 4 +
.../benches/arrow_bytes_map.rs | 82 ++
datafusion/physical-expr-common/src/binary_map.rs | 22 +-
datafusion/physical-expr-common/src/sort_expr.rs | 96 ++
.../physical-expr/src/equivalence/ordering.rs | 8 +-
.../src/equivalence/properties/dependency.rs | 8 +-
datafusion/physical-expr/src/expressions/binary.rs | 346 ++++-
.../src/expressions/binary/kernels.rs | 22 +-
.../physical-expr/src/expressions/in_list.rs | 1 +
.../{primitive_filter.rs => branchless_filter.rs} | 608 +-------
.../src/expressions/in_list/primitive_filter.rs | 483 +------
.../src/expressions/in_list/strategy.rs | 5 +-
datafusion/physical-expr/src/partitioning.rs | 390 ++++++
datafusion/physical-expr/src/projection.rs | 133 +-
datafusion/physical-expr/src/window/aggregate.rs | 7 +-
datafusion/physical-expr/src/window/mod.rs | 1 +
.../physical-expr/src/window/sliding_aggregate.rs | 7 +-
datafusion/physical-expr/src/window/standard.rs | 3 +-
datafusion/physical-expr/src/window/window_expr.rs | 60 +-
datafusion/physical-optimizer/Cargo.toml | 1 +
.../src/ensure_requirements/enforce_sorting/mod.rs | 51 +-
.../physical-optimizer/src/filter_pushdown.rs | 8 +
datafusion/physical-optimizer/src/optimizer.rs | 64 +-
.../physical-optimizer/src/topk_aggregation.rs | 30 +-
datafusion/physical-plan/Cargo.toml | 4 +
datafusion/physical-plan/benches/bounded_window.rs | 280 ++++
datafusion/physical-plan/benches/multi_group_by.rs | 272 +++-
.../src/aggregates/aggregate_hash_table/common.rs | 43 +
.../aggregates/aggregate_hash_table/final_table.rs | 3 +
.../aggregate_hash_table/partial_reduce_table.rs | 3 +
.../aggregate_hash_table/partial_table.rs | 2 +
.../aggregate_hash_table/single_table.rs | 2 +
.../aggregates/group_values/multi_group_by/mod.rs | 246 +++-
.../group_values/multi_group_by/row_backed.rs | 123 +-
.../src/aggregates/grouped_topk_stream.rs | 17 +-
datafusion/physical-plan/src/aggregates/mod.rs | 174 +--
.../src/aggregates/ordered_final_stream.rs | 7 +-
.../src/aggregates/ordered_partial_stream.rs | 380 ++---
.../physical-plan/src/aggregates/single_stream.rs | 675 +++++++--
.../src/aggregates/topk/hash_table.rs | 307 +++-
.../src/aggregates/topk/priority_map.rs | 292 +++-
datafusion/physical-plan/src/execution_plan.rs | 6 +
datafusion/physical-plan/src/filter.rs | 191 ++-
datafusion/physical-plan/src/filter_pushdown.rs | 3 +
datafusion/physical-plan/src/joins/cross_join.rs | 54 +-
.../physical-plan/src/joins/hash_join/exec.rs | 32 +-
.../src/joins/sort_merge_join/exec.rs | 10 +-
.../joins/sort_merge_join/materializing_stream.rs | 1193 ++++++++--------
.../src/joins/sort_merge_join/tests.rs | 141 ++
.../physical-plan/src/joins/symmetric_hash_join.rs | 66 +-
datafusion/physical-plan/src/joins/utils.rs | 469 ++++++-
datafusion/physical-plan/src/projection.rs | 70 +-
datafusion/physical-plan/src/proto.rs | 80 +-
datafusion/physical-plan/src/repartition/mod.rs | 133 +-
datafusion/physical-plan/src/scalar_subquery.rs | 62 +
datafusion/physical-plan/src/sorts/partial_sort.rs | 83 +-
datafusion/physical-plan/src/union.rs | 130 +-
datafusion/physical-plan/src/unnest.rs | 484 ++++++-
.../src/windows/bounded_window_agg_exec.rs | 247 +++-
datafusion/physical-plan/src/windows/proto.rs | 37 +-
datafusion/proto-common/Cargo.toml | 6 +
.../proto-common/proto/datafusion_common.proto | 2 +
datafusion/proto-common/src/from_proto/mod.rs | 5 +-
datafusion/proto-common/src/generated/mod.rs | 1 +
datafusion/proto-common/src/generated/pbjson.rs | 22 +
datafusion/proto-common/src/generated/prost.rs | 2 +
datafusion/proto-common/src/to_proto/mod.rs | 53 +-
datafusion/proto-models/Cargo.toml | 6 +
datafusion/proto-models/proto/datafusion.proto | 20 +-
.../src/generated/datafusion_proto_common.rs | 2 +
datafusion/proto-models/src/generated/mod.rs | 1 +
datafusion/proto-models/src/generated/pbjson.rs | 119 +-
datafusion/proto-models/src/generated/prost.rs | 54 +-
datafusion/proto/Cargo.toml | 8 +-
datafusion/proto/src/bytes/mod.rs | 1 +
datafusion/proto/src/convert.rs | 2 +-
datafusion/proto/src/logical_plan/file_formats.rs | 10 +-
datafusion/proto/src/logical_plan/from_proto.rs | 19 +-
datafusion/proto/src/logical_plan/mod.rs | 6 +-
datafusion/proto/src/logical_plan/to_proto.rs | 20 +-
datafusion/proto/src/physical_plan/from_proto.rs | 196 +--
datafusion/proto/src/physical_plan/mod.rs | 125 +-
datafusion/proto/src/physical_plan/to_proto.rs | 120 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 24 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 137 +-
datafusion/proto/tests/proto_integration.rs | 4 +
datafusion/pruning/src/lib.rs | 4 +-
datafusion/pruning/src/pruning_predicate.rs | 284 +++-
datafusion/session/src/lib.rs | 10 +
datafusion/session/src/physical_optimizer.rs | 84 ++
datafusion/session/src/planner.rs | 198 +++
datafusion/session/src/session.rs | 51 +
datafusion/spark/benches/hex.rs | 10 +
.../spark/src/function/math/{hypot.rs => atan2.rs} | 32 +-
datafusion/spark/src/function/math/hex.rs | 198 ++-
datafusion/spark/src/function/math/mod.rs | 4 +
datafusion/sql/src/expr/function.rs | 18 +-
datafusion/sql/src/expr/mod.rs | 4 -
datafusion/sql/src/select.rs | 56 +-
datafusion/sql/src/statement.rs | 10 +-
datafusion/sql/src/unparser/expr.rs | 1 +
datafusion/sql/src/unparser/plan.rs | 2 +-
datafusion/sqllogictest/src/test_context.rs | 24 +
.../test_files/aggregate_memory_spill.slt | 80 +-
.../sqllogictest/test_files/aggregates_topk.slt | 250 +++-
.../test_files/array/array_any_match.slt | 23 +
.../sqllogictest/test_files/array/array_filter.slt | 14 +
.../sqllogictest/test_files/array/array_length.slt | 49 +-
datafusion/sqllogictest/test_files/binary.slt | 18 +-
.../sqllogictest/test_files/datetime/dates.slt | 14 +
.../test_files/datetime/timestamps.slt | 28 +
datafusion/sqllogictest/test_files/ddl.slt | 14 +
.../test_files/dynamic_filter_pushdown_config.slt | 43 +-
.../sqllogictest/test_files/first_last_nested.slt | 80 ++
.../test_files/functional_dependencies.slt | 10 +-
datafusion/sqllogictest/test_files/functions.slt | 123 +-
datafusion/sqllogictest/test_files/group_by.slt | 97 +-
datafusion/sqllogictest/test_files/in_list.slt | 247 ++++
.../sqllogictest/test_files/information_schema.slt | 2 +
datafusion/sqllogictest/test_files/joins.slt | 613 ++++++++
datafusion/sqllogictest/test_files/math.slt | 151 ++
datafusion/sqllogictest/test_files/metadata.slt | 6 +
datafusion/sqllogictest/test_files/order.slt | 43 +-
.../test_files/ordered_aggregate_spill.slt | 52 +
.../test_files/parquet_nested_schema_pruning.slt | 565 ++++++++
.../test_files/push_down_filter_regression.slt | 38 +
.../sqllogictest/test_files/range_partitioning.slt | 174 +--
.../test_files/regexp/regexp_instr.slt | 12 +
.../test_files/sort_merge_join_spill.slt | 252 ++++
.../sqllogictest/test_files/spark/math/atan2.slt | 152 +-
.../test_files/string/string_literal.slt | 131 ++
.../test_files/string/string_query.slt.part | 80 +-
datafusion/sqllogictest/test_files/topk.slt | 5 +-
.../sqllogictest/test_files/type_coercion.slt | 46 +-
datafusion/sqllogictest/test_files/unnest.slt | 183 +++
datafusion/sqllogictest/test_files/window.slt | 17 +-
.../src/logical_plan/consumer/rel/read_rel.rs | 45 +-
.../src/logical_plan/producer/rel/read_rel.rs | 78 +-
.../test_plans/join_with_expression_key.json | 189 ++-
.../mixed_join_equal_and_indistinct.json | 296 +++-
.../mixed_join_equal_and_indistinct_left.json | 296 +++-
.../tests/testdata/test_plans/multiple_joins.json | 170 ++-
.../test_plans/non_nullable_lists.substrait.json | 36 +-
.../test_plans/scalar_fn_logb_expr.substrait.json | 51 +-
.../scalar_fn_to_between_expr.substrait.json | 69 +-
..._to_built_in_binary_expr_and_not.substrait.json | 99 +-
...r_fn_to_built_in_binary_expr_xor.substrait.json | 99 +-
.../select_count_from_select_1.substrait.json | 8 +-
.../wasmtest/datafusion-wasm-app/package-lock.json | 12 +-
docs/source/library-user-guide/upgrading/55.0.0.md | 223 ++-
docs/source/user-guide/configs.md | 1 +
docs/source/user-guide/sql/scalar_functions.md | 21 +-
docs/source/user-guide/sql/select.md | 23 +
uv.lock | 167 ++-
312 files changed, 23382 insertions(+), 5645 deletions(-)
create mode 100644 benchmarks/sql_benchmarks/clickbench/clickbench.suite
create mode 100644
benchmarks/sql_benchmarks/clickbench_extended/clickbench_extended.suite
create mode 100644
benchmarks/sql_benchmarks/clickbench_sorted/clickbench_sorted.suite
create mode 100644 benchmarks/sql_benchmarks/h2o/h2o.suite
create mode 100644 benchmarks/sql_benchmarks/hj/benchmarks/q24.benchmark
create mode 100644 benchmarks/sql_benchmarks/hj/benchmarks/q25.benchmark
create mode 100644 benchmarks/sql_benchmarks/hj/hj.suite
create mode 100644 benchmarks/sql_benchmarks/imdb/imdb.suite
create mode 100644 benchmarks/sql_benchmarks/nlj/nlj.suite
create mode 100644
benchmarks/sql_benchmarks/push_down_topk/push_down_topk.suite
create mode 100644 benchmarks/sql_benchmarks/smj/smj.suite
create mode 100644 benchmarks/sql_benchmarks/sort_tpch/sort_tpch.suite
create mode 100644 benchmarks/sql_benchmarks/tpcds/tpcds.suite
create mode 100644 benchmarks/sql_benchmarks/tpch/tpch.suite
create mode 100644 benchmarks/sql_benchmarks/wide_schema/wide_schema.suite
create mode 100644 benchmarks/src/sql_benchmark_suite.rs
create mode 100644 benchmarks/src/util/memory_pool.rs
create mode 100644
datafusion/core/tests/memory_limit/memory_limit_validation/smj_mem_validation.rs
create mode 100644 datafusion/core/tests/sql/union_nullable.rs
create mode 100644 datafusion/datasource-parquet/src/nested_schema_pruning.rs
create mode 100644 datafusion/datasource/src/proto.rs
create mode 100644 datafusion/functions-aggregate/benches/variance.rs
create mode 100644 datafusion/physical-expr-common/benches/arrow_bytes_map.rs
copy datafusion/physical-expr/src/expressions/in_list/{primitive_filter.rs =>
branchless_filter.rs} (50%)
create mode 100644 datafusion/physical-plan/benches/bounded_window.rs
create mode 100644 datafusion/session/src/physical_optimizer.rs
create mode 100644 datafusion/session/src/planner.rs
copy datafusion/spark/src/function/math/{hypot.rs => atan2.rs} (72%)
create mode 100644 datafusion/sqllogictest/test_files/first_last_nested.slt
create mode 100644
datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt
create mode 100644 datafusion/sqllogictest/test_files/sort_merge_join_spill.slt
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
