This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch dependabot/uv/all-uv-deps-a8cc4bc276
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit 98e155f50c chore(deps): bump pydata-sphinx-theme
add 2178f275ed perf: Optimize prefix-group processing in `PartialSortExec`
(#24979)
add 32662e7501 fix: report a non-nullable return field for named_struct()
and struct() (#25306)
add ec97d1c902 Push down topk through join (#21621)
add b376290aad fix: map pushed-down filter columns by position instead of
by name (#25259)
add 7917a9a65a bench: cover dictionary group keys with a shared values
array (#25198)
add cf5d573e14 feat: transfer parent and dynamic filters across
HashJoinExec equi-join keys (inner and semi joins) (#25255)
add fe28a2b37b fix: count each probe row once in HashJoinExec
probe_hit_rate and avg_fanout (#25272)
add a0631edb77 perf: reuse cached dictionary value hashes in
vectorized_append (#25185)
add cc06f1b1a0 ci: share workspace check artifacts with feature checks
(#25249)
add 22651d24cc test: bound merge fan-in in ordered aggregate spill tests
(#25252)
add 5a47d21c2b chore(deps): bump rstest from 0.26.1 to 0.27.0 (#25323)
add b88a8eae00 chore(deps): bump dirs from 6.0.0 to 7.0.0 (#25324)
add 210ea75f03 chore(deps): bump the all-other-cargo-deps group with 3
updates (#25322)
add 10c1007dc0 chore(deps): bump the codeql-actions group across 1
directory with 2 updates (#25319)
add e5dbcf4ad6 chore(deps): bump astral-sh/setup-uv from 10.0.1 to 10.1.0
(#25320)
add 8bd6629db6 chore(deps): bump taiki-e/install-action from 2.87.6 to
2.87.12 (#25321)
add 767d1422d9 fix: register the sort-merge join stream as a spillable
memory consumer (#25250)
add 2c1f74f10b fix: evaluate struct-returning UDFs once across repeated
field accesses (#23691)
add be5a96e76d fix: prune row groups when file statistics collapse the
predicate to a constant (#24770)
add 18cbb877ba fix: do not use the top-k aggregate stream for a DISTINCT
limit with no ordering direction (#25349)
add 62dc7ac37b Support INSERT OVERWRITE for MemTable (#24969)
add 5a4490502d dev: include ASF status checks in the local lint suite
(#25313)
add 86ba5e05fc docs(physical-optimizer): warn that wrapper rules must
forward schema_check (#25357)
add 606ae0f697 dev: include security auditing in the local lint suite
(#25359)
add ddad4a821a feat: DISTINCT handling for Aggregate UDFs (#25288)
add 2cb6b54085 fix(proto): preserve CSV sink writer options (#25058)
add b0b547136f refactor: exhaustively destructure dynamic filter and
scalar subquery proto hooks (#25178)
add 6ff484a96d fix: ignore constant aggregate completion boundaries
(#25019)
add ecb0fea9e3 feat: destructure proto hooks (#25204)
add 140c7c5a4c fix: Handle nulls correctly when extracting nested arrays
from nullable struct/map (#25122)
add e3163d46af feat: Support lazy per-file Parquet Arrow schema derivation
(#25343)
add 28c0dec0a4 fix: report regex compile failures as DataFusion errors,
consistently across the regexp family (#25352)
add 4e907557ad perf: Unify, optimize `map[k]` and `map_extract(key)`
(#25201)
add dcd385e3e1 perf: reuse one StatisticsContext across
ensure_distribution (#25098)
add 0342332e1d chore(docs): Fix the rendering of an expression in
docstring (#25379)
add a436660bd0 fix(docs): Fix code snippet in LogicalPlan docs (#25377)
add 6c27203c8e v55 upgrade guide: Remove unreleased-note (#25396)
add 0bef032a4a fix(physical-plan): CoalescePartitionsExec panic on
wasm32-unknown-unknown (#24890)
add 16ecdc9bdf docs: move misplaced upgrade notes to DataFusion 55 (#25331)
add 587e74c844 fix: preserve short-circuit evaluation in Boolean CASE
simplification (#25143)
add 768fc8a108 fix(proto): preserve Parquet source and sink state (#25057)
add 97f72625f4 perf: allow concurrent TrackConsumersPool updates (#24993)
add b300cea226 fix: preserve UPPER and LOWER input nullability (#25381)
add e5469e1570 chore(docs): Fix the documentation for
TableProvider::scan()'s `limit` argument (#25397)
add 790c463830 Add aggregate-internal submetrics (#25051)
add 902ee8a67e feat: push dynamic filters through nested loop joins
(#25030)
add 7deda30943 fix: avoid overflow and underflow in vector functions
(#25289)
add 0f8f3b9d8b perf: optimize `levenshtein` (#23543)
add 06e9fae2ef perf: embed projections in ASOF joins (#24873)
add 56361282fc docs: Add SKILL.md for designing benchmark (#25285)
add a2b809331f feat: simplify contradicting and redundant predicates on a
column (#25207)
add a7cac9afe8 fix: omit ordering fields from the state of
order-insensitive aggregates (#25402)
add c4f72bad34 bench: SQL benchmark suite for null-aware (NOT IN) joins
(#25386)
add 22f9a92178 perf(range): avoid eager ok_or in gen_range_timestamp
(#25308)
add 62f039f0d9 dev: include dependency checks in the local lint suite
(#25399)
add 1dea714201 chore: Update sqlparser to 0.63.0 (#25278)
add bf67e970ba Avoid normalizing hidden values in sliced list set
operations (#25300)
add 98b26c5f9e ci: skip queue-verified Rust jobs on pushes to main (#25365)
add 3ab72e61e3 fix: leave memory for aggregate spill replay (#25383)
add c4f5a9e0f2 fix: do not push filters on volatile group keys below
Aggregate (#25416)
add edc936f38b fix: `NOT IN (subquery)` with a constant value ignores
NULLs in the subquery (#25348)
add 3a647e49dd ci: skip queue-verified Dev and Dependencies jobs on pushes
to main (#25443)
add b9cea816dd fix: prevent unsafe integer interval propagation (#25234)
add 3b16a3d0ee fix: preserve MERGE target qualifier bindings (#24429)
add 0e292dcbfd fix: allow free RANGE window frames over Duration and
Interval ORDER BY types (#24916)
add 5a2000a2ff bench: add projection_subquery SQL benchmark suite (#25346)
add bacbf2aaf8 Release consumed sort input batches after partial output
(#25406)
add 64871d923c perf: Improve efficiency of `array_has_any` on sliced lists
(#25427)
add 879c8fbfa1 dev: include examples README checks in the local lint suite
(#25466)
add 4fb5e08267 chore(deps): bump soupsieve from 2.8.4 to 2.9 (#25444)
add cb14804859 fix(datetime): floor negative scalar timestamps in
date_trunc (#25430)
add 696dd18a8e bench: keep the expect_plan check out of the measured
region (#25441)
add efd36e7b66 fix: preserve date_trunc sorting for timezone-aware
timestamps (#25409)
add b631f2c7d9 chore: Update to arrow/parquet `60.0.0` and `object_store`
`0.14.2` (#25335)
add d967e68458 fix: accept all timestamp precisions in
`generate_series`/`range` (#25173)
add 6e4d9bae3c fix: Spark `map_from_arrays`, `map_from_entries` for sliced
lists (#25431)
add a522cd5bce perf: Reduce peak memory usage of NLJ by 2X using segmented
batch layout (#25371)
add d477a2b038 fix: shrink NLJ load outcome to satisfy Clippy (#25492)
add ccfe704806 fix: only propagate cast statistics through safe
conversions (#25227)
add de8803cd3f perf: Improve efficiency of `array_resize` for sliced
arrays (#25479)
add 24a7f13f51 chore: Add helpers for computing `OffsetBuffer` span,
length (#25481)
add 42f3888ca2 perf: Use agg `DistinctHandling` in join optimization
(#25385)
add 710c748db8 Match Parquet map key/value by position in nested schema
coercion (#25342)
add 925d7f8ffd perf: Avoid needless conversion of `Metadata` to `HashMap`
(#25510)
add 38a1865b7e fix: reject LargeList offsets that exceed i32 (#25516)
add b4a8c824b4 ci: move the large file check into a script shared with
rust_lint.sh (#25440)
add d20936c06f feat(pwmj): support left/right mark joins (#25467)
add 0e3c398ee1 fix: count Parquet decoding in the scan's elapsed_compute
(#25486)
add 43f567a976 fix: account for hash-join row-index storage (#25508)
add 5b7cc87031 chore: remove deprecated Parquet APIs past the API health
policy (#25202)
add 7f3e657343 perf: reuse column normalization context across expressions
(#25010)
add a9579bdfd5 fix: duplicate ordering state field names in partial
aggregates (#25196)
add 267b2c9898 perf: Share backing buffers in string view functions
(#25501)
add f7e2db3d92 minor: remove redundant state from the NestedLoopJoin
fallback coordinator (#25533)
add 04d683d6bd perf: preserve left functional dependencies for ASOF joins
(#24799)
add 4e603e15d9 fix: restrict reverse cast constraint propagation to safe
conversions (#25531)
add 9bffbea2a4 minor: move `create_group_accumulator` out of the legacy
aggregate stream (#25539)
add b8683f3d0a perf: Size builder capacity using visible array span, not
total size (#25502)
add 1cfc355796 minor: derive `group_column_supported_type` from
`make_group_column` (#25541)
add 6700d02160 test: bound merge fan-in in the final hash aggregate spill
test (#25424)
add 918ab2c998 fix: preserve sorts for non-monotonic temporal casts
(#25526)
add 99af44a5e6 dev: include generated config and function docs in the
local lint suite (#25525)
add 923d64fd54 fix: reject mixed-sign intervals in range functions (#25520)
add 993b3f3231 fix: preserve statistics for safe casts without extrema
(#25517)
add 1e09a2a3d7 perf: Avoid copying when materializing output in
OrderedPartialAggregateStream (#25312)
add 6574a8c3b2 fix(substrait): consume chained window functions whose
default names collide (#25181)
add b527fec4be refactor(hash-aggr): share one spill context between the
spilling aggregate streams (#25538)
add c607caf4be fix: sort-merge join filter columns follow the filter's own
column order (#25489)
add cee1dc7ace refactor: factor SortMergeJoinExec::execute into a reusable
sort_merge_join_stream (#25490)
add 708617a994 FilterExec : expose push_batch_with_filter on
LimitedBatchCoalescer (#25503)
add 31a4ca07fe test: characterize correlated `NOT IN` null-aware behaviour
(#25558)
add 0576a0b400 fix: compare ScalarValue maps using Arrow ordering (#25555)
add 4fe342d715 fix: Prevent SubqueryAlias to be pushed past FilterExec
when unparsing (#25547)
add 2424daeb18 fix: preserve computed names in derived SQL projections
(#25521)
add c55319e0b6 docs: Document the convention for internal public APIs
(#25495)
add a948ff6380 fix: Emit equality conditions for Substrait CASE base
expressions (#25191)
add 44fcc5d7f7 chore(deps): bump pydata-sphinx-theme
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (98e155f50c)
\
N -- N -- N refs/heads/dependabot/uv/all-uv-deps-a8cc4bc276
(44fcc5d7f7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.ai/skills/add-benchmarks/SKILL.md | 84 +
.github/workflows/audit.yml | 8 +-
.github/workflows/breaking_changes_detector.yml | 2 +-
.github/workflows/codeql.yml | 4 +-
.github/workflows/dependencies.yml | 18 +-
.github/workflows/dev.yml | 10 +-
.github/workflows/docs.yaml | 4 +-
.github/workflows/docs_pr.yaml | 4 +-
.github/workflows/large_files.yml | 20 +-
.github/workflows/rust.yml | 110 +-
Cargo.lock | 443 +++---
Cargo.toml | 28 +-
benchmarks/README.md | 56 +
benchmarks/bench.sh | 73 +
benchmarks/sql_benchmarks/README.md | 7 +-
.../array_agg_distinct/array_agg_distinct.suite | 11 +
.../array_agg_distinct/benchmarks/q01.benchmark | 11 +
.../null_aware_join/benchmarks/q01.benchmark | 26 +
.../null_aware_join/benchmarks/q02.benchmark | 27 +
.../null_aware_join/benchmarks/q03.benchmark | 26 +
.../null_aware_join/benchmarks/q04.benchmark | 34 +
.../null_aware_join/benchmarks/q05.benchmark | 35 +
.../null_aware_join/benchmarks/q06.benchmark | 36 +
.../null_aware_join/benchmarks/q07.benchmark | 36 +
.../null_aware_join/benchmarks/q08.benchmark | 50 +
.../null_aware_join/benchmarks/q09.benchmark | 45 +
.../null_aware_join/init/cleanup.sql | 4 +
.../sql_benchmarks/null_aware_join/init/load.sql | 55 +
.../null_aware_join/null_aware_join.suite | 31 +
.../projection_subquery/benchmarks/q01.benchmark | 3 +
.../projection_subquery/benchmarks/q02.benchmark | 3 +
.../projection_subquery/benchmarks/q03.benchmark | 3 +
.../projection_subquery/benchmarks/q04.benchmark | 3 +
.../projection_subquery/benchmarks/q05.benchmark | 3 +
.../projection_subquery/benchmarks/q06.benchmark | 3 +
.../projection_subquery/benchmarks/q07.benchmark | 3 +
.../projection_subquery/init/cleanup.sql | 2 +
.../projection_subquery/load/tables.sql | 25 +
.../projection_subquery.benchmark.template | 45 +
.../projection_subquery/projection_subquery.suite | 23 +
.../projection_subquery/queries/q01.sql | 7 +
.../projection_subquery/queries/q02.sql | 6 +
.../projection_subquery/queries/q03.sql | 6 +
.../projection_subquery/queries/q04.sql | 12 +
.../projection_subquery/queries/q05.sql | 6 +
.../projection_subquery/queries/q06.sql | 6 +
.../projection_subquery/queries/q07.sql | 7 +
.../projection_subquery/results/q01_in_bare.csv | 2 +
.../results/q02_in_coalesce.csv | 2 +
.../results/q03_in_correlated_eq.csv | 2 +
.../results/q04_in_two_columns.csv | 2 +
.../results/q05_not_in_bare.csv | 2 +
.../results/q06_exists_correlated.csv | 2 +
.../results/q07_in_correlated_residual.csv | 2 +
benchmarks/src/sql_benchmark.rs | 65 +-
ci/scripts/check_asf_yaml_status_checks.py | 164 +-
...workflows.sh => check_circular_dependencies.sh} | 16 +-
ci/scripts/check_examples_docs.sh | 142 +-
ci/scripts/check_generated_docs.sh | 152 ++
ci/scripts/check_large_files.sh | 138 ++
...n_workflows.sh => check_unused_dependencies.sh} | 17 +-
ci/scripts/{rust_toml_fmt.sh => security_audit.sh} | 64 +-
ci/scripts/utils/tool_versions.sh | 1 +
datafusion-cli/Cargo.toml | 2 +-
datafusion-cli/src/main.rs | 20 +-
.../examples/builtin_functions/regexp.rs | 3 +-
datafusion-testing | 2 +-
datafusion/catalog-listing/src/helpers.rs | 4 +-
datafusion/catalog-listing/src/table.rs | 4 +-
datafusion/catalog/src/memory/table.rs | 5 +-
datafusion/common/src/config.rs | 2 +-
datafusion/common/src/dfschema.rs | 23 +-
datafusion/common/src/hash_utils.rs | 24 +-
datafusion/common/src/heap_size.rs | 22 +-
datafusion/common/src/metadata.rs | 94 +-
datafusion/common/src/nested_struct.rs | 12 +-
datafusion/common/src/param_value.rs | 2 +-
datafusion/common/src/scalar/mod.rs | 61 +-
datafusion/common/src/utils/memory.rs | 2 +-
datafusion/common/src/utils/mod.rs | 58 +-
datafusion/core/src/datasource/file_format/csv.rs | 1 +
.../core/src/datasource/file_format/parquet.rs | 44 +-
datafusion/core/src/datasource/memory_test.rs | 130 +-
datafusion/core/src/optimizer_rule_reference.md | 13 +-
datafusion/core/src/physical_planner.rs | 35 +-
datafusion/core/tests/dataframe/mod.rs | 20 +-
datafusion/core/tests/fifo/mod.rs | 5 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 90 +-
.../memory_limit/memory_limit_validation/mod.rs | 1 +
.../memory_limit_validation/nlj_mem_validation.rs | 57 +
.../core/tests/parquet/string_in_list_pruning.rs | 10 +-
.../physical_optimizer/enforce_distribution.rs | 284 +++-
.../tests/physical_optimizer/filter_pushdown.rs | 1525 ++++++++++++++++--
.../tests/physical_optimizer/sanity_checker.rs | 21 +
datafusion/core/tests/sql/explain_analyze.rs | 58 +
datafusion/core/tests/sql/joins.rs | 115 +-
datafusion/core/tests/sql/path_partition.rs | 2 +
datafusion/core/tests/sql/sql_api.rs | 155 +-
datafusion/core/tests/sql/unparser.rs | 27 +-
.../tests/user_defined/user_defined_aggregates.rs | 14 +-
.../user_defined/user_defined_scalar_functions.rs | 8 +-
.../user_defined/user_defined_window_functions.rs | 8 +-
datafusion/datasource-arrow/src/file_format.rs | 6 +-
datafusion/datasource-arrow/src/source.rs | 7 +-
datafusion/datasource-csv/src/file_format.rs | 52 +-
datafusion/datasource-parquet/src/file_format.rs | 68 +-
datafusion/datasource-parquet/src/metadata.rs | 26 +-
datafusion/datasource-parquet/src/opener/mod.rs | 895 ++++++++++-
datafusion/datasource-parquet/src/page_filter.rs | 87 +-
datafusion/datasource-parquet/src/push_decoder.rs | 14 +-
datafusion/datasource-parquet/src/reader.rs | 28 +-
.../datasource-parquet/src/row_group_filter.rs | 11 +
.../datasource-parquet/src/schema_coercion.rs | 741 +++++++--
datafusion/datasource-parquet/src/sink.rs | 109 +-
datafusion/datasource-parquet/src/source.rs | 110 +-
.../src/statistics_order_tests.rs | 35 +-
datafusion/datasource/src/file_scan_config/mod.rs | 16 +-
datafusion/datasource/src/memory.rs | 24 +-
datafusion/execution/src/async_stream.rs | 3 +-
datafusion/execution/src/cache/mod.rs | 6 +-
datafusion/execution/src/memory_pool/pool.rs | 123 +-
datafusion/expr-common/src/accumulator.rs | 109 +-
datafusion/expr-common/src/groups_accumulator.rs | 10 +
datafusion/expr/Cargo.toml | 5 +
datafusion/expr/benches/normalize_columns.rs | 97 ++
datafusion/expr/src/expr.rs | 55 +-
datafusion/expr/src/expr_rewriter/mod.rs | 156 +-
datafusion/expr/src/lib.rs | 12 +-
datafusion/expr/src/logical_plan/builder.rs | 93 +-
datafusion/expr/src/logical_plan/dml.rs | 125 +-
datafusion/expr/src/logical_plan/invariants.rs | 9 +-
datafusion/expr/src/logical_plan/plan.rs | 19 +-
datafusion/expr/src/udaf.rs | 125 +-
datafusion/expr/src/utils.rs | 93 +-
datafusion/ffi/src/record_batch_stream.rs | 4 +-
datafusion/ffi/src/udaf/mod.rs | 11 +-
datafusion/functions-aggregate-common/Cargo.toml | 4 +
.../benches/groups_accumulator_adapter.rs | 62 +
.../src/aggregate/groups_accumulator.rs | 720 ++++++++-
.../src/aggregate/groups_accumulator/nulls.rs | 4 +-
.../functions-aggregate/benches/array_agg.rs | 6 +-
.../functions-aggregate/benches/nth_value.rs | 33 +-
.../functions-aggregate/benches/percentile_cont.rs | 43 +-
datafusion/functions-aggregate/src/any_value.rs | 5 +
.../functions-aggregate/src/approx_distinct.rs | 6 +
.../functions-aggregate/src/approx_median.rs | 6 +
.../src/approx_percentile_cont.rs | 8 +
.../src/approx_percentile_cont_with_weight.rs | 6 +
datafusion/functions-aggregate/src/array_agg.rs | 285 +++-
.../functions-aggregate/src/bit_and_or_xor.rs | 13 +
datafusion/functions-aggregate/src/bool_and_or.rs | 11 +
datafusion/functions-aggregate/src/correlation.rs | 8 +
datafusion/functions-aggregate/src/covariance.rs | 15 +
datafusion/functions-aggregate/src/first_last.rs | 20 +-
datafusion/functions-aggregate/src/grouping.rs | 10 +
datafusion/functions-aggregate/src/min_max.rs | 11 +
datafusion/functions-aggregate/src/nth_value.rs | 8 +
datafusion/functions-aggregate/src/regr.rs | 8 +
datafusion/functions-aggregate/src/stddev.rs | 12 +
datafusion/functions-aggregate/src/variance.rs | 6 +
datafusion/functions-nested/Cargo.toml | 4 +
datafusion/functions-nested/benches/array_has.rs | 61 +-
.../functions-nested/benches/array_resize.rs | 34 +
.../functions-nested/benches/array_set_ops.rs | 105 +-
datafusion/functions-nested/benches/map.rs | 98 +-
datafusion/functions-nested/benches/range.rs | 98 ++
datafusion/functions-nested/src/array_add.rs | 97 +-
datafusion/functions-nested/src/array_compact.rs | 20 +-
datafusion/functions-nested/src/array_has.rs | 110 +-
datafusion/functions-nested/src/array_normalize.rs | 43 +-
datafusion/functions-nested/src/array_scale.rs | 23 +-
datafusion/functions-nested/src/array_subtract.rs | 13 +
datafusion/functions-nested/src/concat.rs | 37 +-
datafusion/functions-nested/src/cosine_distance.rs | 57 +-
datafusion/functions-nested/src/distance.rs | 39 +-
datafusion/functions-nested/src/except.rs | 57 +-
datafusion/functions-nested/src/extract.rs | 26 +-
datafusion/functions-nested/src/map_extract.rs | 122 +-
datafusion/functions-nested/src/position.rs | 27 +-
datafusion/functions-nested/src/range.rs | 53 +-
datafusion/functions-nested/src/remove.rs | 17 +-
datafusion/functions-nested/src/replace.rs | 29 +-
datafusion/functions-nested/src/resize.rs | 100 +-
datafusion/functions-nested/src/reverse.rs | 3 +-
datafusion/functions-nested/src/set_ops.rs | 177 ++-
datafusion/functions-nested/src/sort.rs | 17 +-
datafusion/functions-nested/src/string.rs | 63 +-
datafusion/functions-nested/src/utils.rs | 140 +-
datafusion/functions-table/src/generate_series.rs | 140 +-
datafusion/functions/benches/misc/get_field.rs | 197 ++-
datafusion/functions/src/binaries.rs | 2 +-
datafusion/functions/src/core/arrow_field.rs | 22 +-
datafusion/functions/src/core/arrow_metadata.rs | 21 +-
datafusion/functions/src/core/getfield.rs | 352 +++--
datafusion/functions/src/core/named_struct.rs | 5 +-
datafusion/functions/src/core/overlay.rs | 18 +-
datafusion/functions/src/core/struct.rs | 18 +-
datafusion/functions/src/core/with_metadata.rs | 11 +-
datafusion/functions/src/datetime/date_trunc.rs | 94 +-
datafusion/functions/src/encoding/inner.rs | 20 +-
datafusion/functions/src/regex/mod.rs | 49 +-
datafusion/functions/src/regex/regexpcount.rs | 180 ++-
datafusion/functions/src/regex/regexpinstr.rs | 83 +-
datafusion/functions/src/regex/regexplike.rs | 80 +-
datafusion/functions/src/regex/regexpmatch.rs | 62 +-
datafusion/functions/src/regex/regexpreplace.rs | 46 +-
datafusion/functions/src/string/common.rs | 22 +-
datafusion/functions/src/string/levenshtein.rs | 196 ++-
datafusion/functions/src/string/lower.rs | 29 +-
datafusion/functions/src/string/split_part.rs | 2 +-
datafusion/functions/src/string/upper.rs | 29 +-
datafusion/functions/src/strings.rs | 39 +-
datafusion/functions/src/unicode/common.rs | 4 +-
datafusion/functions/src/unicode/initcap.rs | 36 +-
datafusion/functions/src/unicode/substr.rs | 2 +-
datafusion/functions/src/unicode/substrindex.rs | 29 +-
datafusion/functions/src/unicode/translate.rs | 55 +-
datafusion/functions/src/utils.rs | 589 ++++++-
.../optimizer/src/analyzer/function_rewrite.rs | 22 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 115 +-
.../src/decorrelate_predicate_subquery.rs | 119 +-
.../optimizer/src/eliminate_aggregate_distinct.rs | 533 +++++++
datafusion/optimizer/src/eliminate_join.rs | 598 ++++++-
.../optimizer/src/extract_leaf_expressions.rs | 170 +-
datafusion/optimizer/src/lib.rs | 1 +
datafusion/optimizer/src/optimizer.rs | 7 +-
.../optimizer/src/propagate_empty_relation.rs | 4 +-
datafusion/optimizer/src/push_down_filter.rs | 104 +-
datafusion/optimizer/src/push_down_limit.rs | 260 ++--
.../src/push_down_limit/topk_through_join.rs | 1635 ++++++++++++++++++++
datafusion/optimizer/src/rewrite_set_comparison.rs | 16 +-
.../src/simplify_expressions/expr_simplifier.rs | 236 +--
.../optimizer/src/simplify_expressions/regex.rs | 12 +-
.../src/simplify_expressions/simplify_exprs.rs | 19 +-
.../simplify_expressions/simplify_predicates.rs | 354 ++++-
.../src/simplify_expressions/udf_preimage.rs | 6 +-
.../optimizer/src/simplify_expressions/utils.rs | 17 +-
datafusion/optimizer/src/test/udfs.rs | 125 +-
datafusion/optimizer/src/unions_to_filter.rs | 54 +-
datafusion/optimizer/src/utils.rs | 89 +-
datafusion/physical-expr-common/Cargo.toml | 1 +
.../physical-expr-common/src/binary_view_map.rs | 5 +-
datafusion/physical-expr-common/src/lib.rs | 1 +
.../physical-expr-common/src/metrics/value.rs | 17 +
datafusion/physical-expr-common/src/regex.rs | 195 +++
datafusion/physical-expr-common/src/sort_expr.rs | 23 +-
datafusion/physical-expr-common/src/utils.rs | 2 +-
datafusion/physical-expr/benches/case_when.rs | 9 +-
datafusion/physical-expr/src/aggregate.rs | 140 +-
datafusion/physical-expr/src/expressions/binary.rs | 298 +++-
.../src/expressions/binary/kernels.rs | 42 +-
datafusion/physical-expr/src/expressions/cast.rs | 434 +++++-
.../src/expressions/dynamic_filters/mod.rs | 56 +-
.../physical-expr/src/expressions/try_cast.rs | 7 +-
datafusion/physical-expr/src/partitioning.rs | 86 +-
datafusion/physical-expr/src/planner.rs | 26 +-
datafusion/physical-expr/src/projection.rs | 324 +++-
datafusion/physical-expr/src/scalar_subquery.rs | 38 +-
.../ensure_requirements/enforce_distribution.rs | 38 +-
.../src/ensure_requirements/mod.rs | 37 +-
datafusion/physical-plan/Cargo.toml | 5 +
.../benches/dictionary_group_values.rs | 74 +-
datafusion/physical-plan/benches/partial_sort.rs | 133 ++
.../src/aggregates/aggregate_hash_table/common.rs | 62 +-
.../aggregate_hash_table/common_ordered.rs | 74 +-
.../src/aggregates/aggregate_hash_table/mod.rs | 11 +-
.../aggregate_hash_table/ordered_partial_table.rs | 28 +-
.../aggregate_hash_table/partial_table.rs | 1 +
.../src/aggregates/aggregate_stream.rs | 195 ++-
.../src/aggregates/group_values/metrics.rs | 219 ++-
.../src/aggregates/group_values/mod.rs | 2 +-
.../group_values/multi_group_by/bytes_view.rs | 6 +-
.../group_values/multi_group_by/dictionary.rs | 194 ++-
.../aggregates/group_values/multi_group_by/mod.rs | 105 +-
.../src/aggregates/grouped_hash_stream.rs | 203 ++-
.../physical-plan/src/aggregates/hash_stream.rs | 488 +++---
datafusion/physical-plan/src/aggregates/mod.rs | 660 +++++++-
.../src/aggregates/ordered_final_stream.rs | 471 ++++--
.../src/aggregates/ordered_partial_stream.rs | 396 +++--
.../src/aggregates/ordered_single_stream.rs | 208 +--
.../physical-plan/src/aggregates/single_stream.rs | 209 +--
datafusion/physical-plan/src/aggregates/spill.rs | 293 ++++
datafusion/physical-plan/src/coalesce/mod.rs | 148 +-
.../physical-plan/src/coalesce_partitions.rs | 22 +
datafusion/physical-plan/src/filter.rs | 132 +-
datafusion/physical-plan/src/filter_pushdown.rs | 150 +-
datafusion/physical-plan/src/joins/asof_join.rs | 119 ++
.../physical-plan/src/joins/hash_join/exec.rs | 447 +++++-
.../src/joins/hash_join/partitioned_hash_eval.rs | 8 +-
.../physical-plan/src/joins/hash_join/stream.rs | 41 +-
.../physical-plan/src/joins/logical_batch.rs | 750 +++++++++
datafusion/physical-plan/src/joins/mod.rs | 1 +
.../physical-plan/src/joins/nested_loop_join.rs | 408 +++--
.../src/joins/piecewise_merge_join/exec.rs | 173 ++-
.../joins/piecewise_merge_join/existence_join.rs | 131 +-
.../piecewise_merge_join/right_existence_join.rs | 395 ++++-
.../src/joins/piecewise_merge_join/utils.rs | 32 +-
.../src/joins/sort_merge_join/exec.rs | 203 ++-
.../src/joins/sort_merge_join/filter.rs | 50 +-
.../joins/sort_merge_join/materializing_stream.rs | 7 +-
.../src/joins/sort_merge_join/tests.rs | 484 +++++-
datafusion/physical-plan/src/joins/utils.rs | 8 +-
.../physical-plan/src/operator_statistics/mod.rs | 6 +-
datafusion/physical-plan/src/projection.rs | 75 +-
datafusion/physical-plan/src/repartition/mod.rs | 111 +-
datafusion/physical-plan/src/sorts/builder.rs | 210 ++-
.../physical-plan/src/sorts/multi_level_merge.rs | 541 ++++++-
datafusion/physical-plan/src/sorts/partial_sort.rs | 507 +++++-
.../physical-plan/src/sorts/streaming_merge.rs | 11 +
.../physical-plan/src/spill/spill_manager.rs | 33 +-
datafusion/physical-plan/src/union.rs | 4 +-
.../proto-common/proto/datafusion_common.proto | 14 +-
datafusion/proto-common/src/from_proto/mod.rs | 179 ++-
datafusion/proto-common/src/generated/pbjson.rs | 99 +-
datafusion/proto-common/src/generated/prost.rs | 25 +-
datafusion/proto-common/src/to_proto/mod.rs | 70 +-
datafusion/proto-models/proto/datafusion.proto | 24 +-
.../src/generated/datafusion_proto_common.rs | 25 +-
datafusion/proto-models/src/generated/pbjson.rs | 278 ++++
datafusion/proto-models/src/generated/prost.rs | 30 +-
datafusion/proto/src/logical_plan/from_proto.rs | 17 +-
datafusion/proto/src/logical_plan/to_proto.rs | 9 +-
datafusion/proto/tests/cases/plans/joins.rs | 92 +-
datafusion/proto/tests/cases/plans/mod.rs | 17 +
datafusion/proto/tests/cases/plans/sinks.rs | 188 ++-
datafusion/proto/tests/cases/plans/sources.rs | 98 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 52 +-
datafusion/proto/tests/cases/serialize.rs | 6 +-
datafusion/session/src/physical_optimizer.rs | 29 +
datafusion/session/src/session.rs | 4 +-
datafusion/session/src/table.rs | 10 +-
datafusion/spark/src/function/array/shuffle.rs | 27 +-
datafusion/spark/src/function/map/utils.rs | 163 +-
datafusion/spark/src/function/math/unhex.rs | 20 +-
datafusion/spark/src/function/string/quote.rs | 9 +-
datafusion/spark/src/function/url/url_decode.rs | 19 +-
datafusion/spark/src/function/url/url_encode.rs | 19 +-
datafusion/spark/src/function/utils.rs | 28 +
datafusion/sql/src/expr/function.rs | 5 +
datafusion/sql/src/expr/mod.rs | 36 +-
datafusion/sql/src/expr/order_by.rs | 13 +-
datafusion/sql/src/parser.rs | 41 +-
datafusion/sql/src/planner.rs | 2 +-
datafusion/sql/src/query.rs | 43 +-
datafusion/sql/src/statement.rs | 197 +--
datafusion/sql/src/unparser/ast.rs | 45 +-
datafusion/sql/src/unparser/dialect.rs | 1 -
datafusion/sql/src/unparser/expr.rs | 23 +-
datafusion/sql/src/unparser/plan.rs | 36 +-
datafusion/sql/tests/cases/params.rs | 14 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 57 +-
datafusion/sql/tests/common/mod.rs | 21 +-
datafusion/sql/tests/sql_integration.rs | 8 +-
datafusion/sqllogictest/src/test_context.rs | 19 +-
datafusion/sqllogictest/test_files/aggregate.slt | 33 +-
.../test_files/aggregate_memory_spill.slt | 10 +
.../test_files/aggregates_simplify.slt | 264 ++++
.../sqllogictest/test_files/array/array_length.slt | 17 +
.../sqllogictest/test_files/array/array_range.slt | 9 +
datafusion/sqllogictest/test_files/array_agg.slt | 7 +-
.../sqllogictest/test_files/array_normalize.slt | 15 +
datafusion/sqllogictest/test_files/arrow_field.slt | 6 +
datafusion/sqllogictest/test_files/asof_join.slt | 83 +-
datafusion/sqllogictest/test_files/case.slt | 63 +
datafusion/sqllogictest/test_files/cast.slt | 137 ++
.../sqllogictest/test_files/cosine_distance.slt | 27 +
.../test_files/datetime/arith_interval_double.slt | 12 +-
.../test_files/datetime/timestamps.slt | 48 +-
datafusion/sqllogictest/test_files/ddl.slt | 16 +
.../sqllogictest/test_files/dictionary_struct.slt | 32 +-
.../test_files/dynamic_filter_pushdown_config.slt | 272 ++++
.../test_files/eliminate_join_distinct.slt | 222 +++
datafusion/sqllogictest/test_files/explain.slt | 15 +
.../sqllogictest/test_files/explain_analyze.slt | 2 +-
.../test_files/filter_without_sort_exec.slt | 96 ++
datafusion/sqllogictest/test_files/functions.slt | 24 +
datafusion/sqllogictest/test_files/group_by.slt | 100 ++
.../sqllogictest/test_files/information_schema.slt | 4 +-
.../test_files/join_dynamic_filter_transfer.slt | 135 ++
datafusion/sqllogictest/test_files/joins.slt | 12 +-
datafusion/sqllogictest/test_files/map.slt | 46 +-
datafusion/sqllogictest/test_files/merge_into.slt | 235 ++-
.../test_files/null_aware_anti_join.slt | 363 ++++-
.../test_files/null_aware_mark_join.slt | 512 +++++-
datafusion/sqllogictest/test_files/order.slt | 10 +
.../test_files/ordered_aggregate_spill.slt | 16 +-
.../sqllogictest/test_files/parquet_statistics.slt | 87 ++
.../test_files/push_down_filter_parquet.slt | 44 +-
.../test_files/push_down_filter_regression.slt | 38 +-
.../test_files/push_down_topk_through_join.slt | 430 ++++-
datafusion/sqllogictest/test_files/pwmj.slt | 171 ++
.../test_files/regexp/regexp_count.slt | 35 +-
.../test_files/regexp/regexp_instr.slt | 32 +-
.../sqllogictest/test_files/regexp/regexp_like.slt | 95 +-
.../test_files/regexp/regexp_match.slt | 78 +-
.../test_files/regexp/regexp_replace.slt | 25 +
datafusion/sqllogictest/test_files/scalar.slt | 2 +-
.../test_files/simplify_predicates.slt | 113 +-
.../test_files/single_distinct_to_groupby.slt | 8 +-
.../test_files/spark/map/map_from_arrays.slt | 14 +
.../test_files/spark/map/map_from_entries.slt | 14 +
datafusion/sqllogictest/test_files/struct.slt | 67 +-
datafusion/sqllogictest/test_files/subquery.slt | 53 +-
.../sqllogictest/test_files/table_functions.slt | 187 +++
.../test_files/tpch/plans/q16.slt.part | 6 +-
datafusion/sqllogictest/test_files/union.slt | 4 +-
datafusion/sqllogictest/test_files/window.slt | 84 +-
.../src/logical_plan/consumer/rel/project_rel.rs | 60 +
.../substrait/src/logical_plan/consumer/utils.rs | 8 +-
.../src/logical_plan/producer/expr/if_then.rs | 124 +-
.../tests/cases/roundtrip_logical_plan.rs | 96 +-
datafusion/substrait/tests/cases/serialize.rs | 408 ++++-
datafusion/wasmtest/src/lib.rs | 22 +-
dev/rust_lint.sh | 26 +
dev/update_config_docs.sh | 34 +-
dev/update_function_docs.sh | 38 +-
docs/source/contributor-guide/api-health.md | 29 +
docs/source/contributor-guide/testing.md | 100 ++
.../library-user-guide/functions/adding-udfs.md | 19 +
docs/source/library-user-guide/upgrading/54.0.0.md | 43 -
docs/source/library-user-guide/upgrading/55.0.0.md | 45 +-
docs/source/library-user-guide/upgrading/56.0.0.md | 263 ++++
docs/source/user-guide/configs.md | 2 +-
docs/source/user-guide/metrics.md | 19 +
uv.lock | 6 +-
425 files changed, 31320 insertions(+), 5979 deletions(-)
create mode 100644 .ai/skills/add-benchmarks/SKILL.md
create mode 100644
benchmarks/sql_benchmarks/array_agg_distinct/array_agg_distinct.suite
create mode 100644
benchmarks/sql_benchmarks/array_agg_distinct/benchmarks/q01.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q01.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q02.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q03.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q04.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q05.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q06.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q07.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q08.benchmark
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/benchmarks/q09.benchmark
create mode 100644 benchmarks/sql_benchmarks/null_aware_join/init/cleanup.sql
create mode 100644 benchmarks/sql_benchmarks/null_aware_join/init/load.sql
create mode 100644
benchmarks/sql_benchmarks/null_aware_join/null_aware_join.suite
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q01.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q02.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q03.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q04.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q05.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q06.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/benchmarks/q07.benchmark
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/init/cleanup.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/load/tables.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/projection_subquery.benchmark.template
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/projection_subquery.suite
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q01.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q02.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q03.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q04.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q05.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q06.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/queries/q07.sql
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q01_in_bare.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q02_in_coalesce.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q03_in_correlated_eq.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q04_in_two_columns.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q05_not_in_bare.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q06_exists_correlated.csv
create mode 100644
benchmarks/sql_benchmarks/projection_subquery/results/q07_in_correlated_residual.csv
mode change 100644 => 100755 ci/scripts/check_asf_yaml_status_checks.py
copy ci/scripts/{check_no_cargo_install_in_workflows.sh =>
check_circular_dependencies.sh} (64%)
create mode 100755 ci/scripts/check_generated_docs.sh
create mode 100755 ci/scripts/check_large_files.sh
copy ci/scripts/{check_no_cargo_install_in_workflows.sh =>
check_unused_dependencies.sh} (61%)
copy ci/scripts/{rust_toml_fmt.sh => security_audit.sh} (51%)
create mode 100644
datafusion/core/tests/memory_limit/memory_limit_validation/nlj_mem_validation.rs
create mode 100644 datafusion/expr/benches/normalize_columns.rs
create mode 100644
datafusion/functions-aggregate-common/benches/groups_accumulator_adapter.rs
create mode 100644 datafusion/functions-nested/benches/range.rs
create mode 100644 datafusion/optimizer/src/eliminate_aggregate_distinct.rs
create mode 100644
datafusion/optimizer/src/push_down_limit/topk_through_join.rs
create mode 100644 datafusion/physical-expr-common/src/regex.rs
create mode 100644 datafusion/physical-plan/benches/partial_sort.rs
create mode 100644 datafusion/physical-plan/src/aggregates/spill.rs
create mode 100644 datafusion/physical-plan/src/joins/logical_batch.rs
create mode 100644
datafusion/sqllogictest/test_files/eliminate_join_distinct.slt
create mode 100644
datafusion/sqllogictest/test_files/join_dynamic_filter_transfer.slt
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
