This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/proto-c40a7389eb
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit 562ad7ee13 chore(deps): bump the proto group with 3 updates
add 899a762230 Populate partition column statistics for PartitionedFile
(#19284)
add 7c05b208e1 refactor: move metrics module to `datafusion-common` crate
(#19247)
add 9d4fe15895 Add library user guide for extending SQL syntax (#19265)
add b828fbfa54 chore(deps): bump taiki-e/install-action from 2.62.65 to
2.62.67 (#19295)
add 58377bf2e3 chore(deps): bump ctor from 0.6.1 to 0.6.3 (#19328)
add fc8824011b Refactor `power()` signature away from user defined (#18968)
add 933657e2c8 feat: Support sliding window queries for MedianAccumulator
by implementing `retract_batch` (#19278)
add 02c647ae1e chore: enforce `clippy::allow_attributes` for optimizer and
macros (#19310)
add 3a67d86c94 fix(concat): correct nullability inference (nullable only
if all arguments nullable) (#19189)
add 266a7d060e chore(deps): bump taiki-e/install-action from 2.62.67 to
2.63.3 (#19349)
add ea8087d77e chore(deps): bump clap from 4.5.50 to 4.5.53 (#19326)
add 199cf06d20 chore(deps): bump insta from 1.43.2 to 1.44.3 (#19327)
add 79cfe8e921 Add runtime config options for `list_files_cache_limit` and
`list_files_cache_ttl` (#19108)
add 89af694117 remove repartition exec from coalesce batches optimizer
(#19239)
add be0cf05f93 minor: cleanup unnecessary config in `decimal.slt` (#19352)
add c53a44891b Fix panic for `GROUPING SETS(())` and handle empty-grouping
aggregates (#19252)
add 775277ae3c Update datafusion-core crate to Rust 2024 edition (#19332)
add efd793ba3e Minor: clean up titles and links n extending operators and
optimizer pages (#19317)
add 1f26716280 Update 4 crates to rust 2024 edition (#19357)
add 50d20dd553 preserve Field metadata in first_value/last_value (#19335)
add 3a41cc6078 Establish the high level API for sort pushdown and the
optimizer rule and support reverse files and row groups (#19064)
add b806117bfa Fix flaky SpillPool channel test by synchronizing reader
and writer tasks (#19110)
add 2b67ba81c6 [minor] Upgrade rust version (#19363)
add 401345b6fd Minor: fix cargo fmt (#19368)
add 7900cd6e98 chore: enforce clippy::allow_attributes for proto, pruning,
session (#19350)
add 59dcc362ca Update remaining crates to rust 2024 edition (#19361)
add 2e95f2673f Minor: Make `ProjectionExpr::new` easier to use with
constants (#19343)
add fe11ad6b46 fix: Added nullable return from date_add(#19151) (#19229)
add 3879b24503 fix: spark sha1 nullability reporting (#19242)
add 1e4bd7565b Feat: DefaultListFilesCache prefix-aware for partition
pruning optimization (#19298)
add 14cd71eaba feat: add compression level configuration for JSON/CSV
writers (#18954)
add 4e7bba4909 Extend in_list benchmark coverage (#19376)
add d68b629dc6 Add hashing microbenchmark `with_hashes` (#19373)
add cb3fa1c2c7 fix: derive custom nullability for the spark `next_day`
(#19253)
add 0a0416dbfe Move `newlines_in_values` from `FileScanConfig` to
`CsvSource` (#19313)
add 2c9077a8bf [datafusion-cli] Implement average LIST duration for object
store profiling (#19127)
add 8120da40a9 chore(deps): bump taiki-e/install-action from 2.63.3 to
2.64.0 (#19382)
add ead8209803 update insta snapshots (#19381)
add d59ebac9f3 feat(spark): implement Spark `try_sum` function (#18569)
add c2747eb109 feat: Support log for Decimal32 and Decimal64 (#18999)
add cbf33d1ee9 Fix regression for negative-scale decimal128 in log (#19315)
add d493f3d441 Add Decimal support to Ceil and Floor (#18979)
add 8550010bd5 Fix input handling for encoding functions & various
refactors (#18754)
add 75d2473ba7 Remove SchemaAdapter (#19345)
add 887aa9f8c6 fix: preserve ListFilesCache TTL when not set in config
(#19401)
add 91cfb69903 feat(proto): Add protobuf serialization for HashExpr
(#19379)
add 2e3707e380 fix: projection for `CooperativeExec` and
`CoalesceBatchesExec` (#19400)
add b3d2cb622c Fix ORDER BY positional reference regression with aliased
aggregates (#19412)
add 5419ff5902 feat: hash partitioning satisfies subset (#19304)
add 8cc8c11de0 Optimize muti-column grouping with StringView/ByteView
(option 2) - 25% faster (#19413)
add 6fa9c1ad11 Optimize hashing for StringView and ByteView (15-70%
faster) (#19374)
add eb30c19b30 Implement disk spilling for all grouping ordering modes in
GroupedHashAggregateStream (#19287)
add 32e6fe887a feat: update FFI TableProvider and ExecutionPlan to use FFI
Session and TaskContext (#19281)
add d8e68a404a refactor: add ParquetOpenerBuilder to reduce test code
duplication (#19405)
add 4249e4ecd3 bench: add `range_and_generate_series` (#19428)
add 65a6bc4231 chore: use extend instead of manual loop in multi group by
(#19429)
add 2c3566ce85 doc: add example for cache factory (#19139)
add 1acaf7a9a2 chore(deps): bump taiki-e/install-action from 2.64.0 to
2.64.2 (#19399)
add 9fe9ec744b fix: spark crc32 custom nullability (#19271)
add 9a9e4dd5c9 Add recursive protection on planner's
`create_physical_expr` (#19299)
new d9992c7699 chore(deps): bump the proto group with 3 updates
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (562ad7ee13)
\
N -- N -- N refs/heads/dependabot/cargo/main/proto-c40a7389eb
(d9992c7699)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/audit.yml | 2 +-
.github/workflows/rust.yml | 4 +-
Cargo.lock | 36 +-
Cargo.toml | 6 +-
benchmarks/src/bin/external_aggr.rs | 4 +-
benchmarks/src/bin/mem_profile.rs | 27 +-
benchmarks/src/cancellation.rs | 10 +-
benchmarks/src/clickbench.rs | 6 +-
benchmarks/src/h2o.rs | 4 +-
benchmarks/src/hj.rs | 2 +-
benchmarks/src/imdb/convert.rs | 2 +-
benchmarks/src/imdb/run.rs | 8 +-
benchmarks/src/nlj.rs | 6 +-
benchmarks/src/smj.rs | 2 +-
benchmarks/src/sort_tpch.rs | 4 +-
benchmarks/src/tpcds/run.rs | 4 +-
benchmarks/src/tpch/run.rs | 8 +-
benchmarks/src/util/options.rs | 2 +-
benchmarks/src/util/run.rs | 2 +-
datafusion-cli/Cargo.toml | 2 +-
datafusion-cli/examples/cli-session-context.rs | 2 +-
datafusion-cli/src/catalog.rs | 50 +-
datafusion-cli/src/cli_context.rs | 2 +-
datafusion-cli/src/command.rs | 2 +-
datafusion-cli/src/exec.rs | 27 +-
datafusion-cli/src/functions.rs | 4 +-
datafusion-cli/src/helper.rs | 10 +-
datafusion-cli/src/highlighter.rs | 4 +-
datafusion-cli/src/main.rs | 12 +-
datafusion-cli/src/object_storage.rs | 51 +-
datafusion-cli/src/object_storage/instrumented.rs | 114 +-
datafusion-cli/src/print_format.rs | 66 +-
datafusion-cli/src/print_options.rs | 8 +-
datafusion-cli/tests/cli_integration.rs | 34 +-
...errides@explain_plan_environment_overrides.snap | 1 -
.../tests/snapshots/[email protected] | 2 +-
datafusion-cli/tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
datafusion-cli/tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/cli_quick_test@batch_size.snap | 2 +-
.../cli_quick_test@default_explain_plan.snap | 1 -
.../tests/snapshots/[email protected] | 2 +-
.../tests/snapshots/[email protected] | 2 +-
.../examples/builtin_functions/function_factory.rs | 2 +-
.../examples/builtin_functions/regexp.rs | 60 +-
.../examples/custom_data_source/csv_json_opener.rs | 2 +-
.../custom_data_source/custom_datasource.rs | 8 +-
.../custom_data_source/custom_file_casts.rs | 6 +-
.../custom_data_source/custom_file_format.rs | 6 +-
.../custom_data_source/default_column_values.rs | 21 +-
.../custom_data_source/file_stream_provider.rs | 6 +-
datafusion-examples/examples/data_io/catalog.rs | 4 +-
.../examples/data_io/json_shredding.rs | 79 +-
.../examples/data_io/parquet_advanced_index.rs | 14 +-
.../examples/data_io/parquet_embedded_index.rs | 30 +-
.../examples/data_io/parquet_encrypted.rs | 4 +-
.../examples/data_io/parquet_exec_visitor.rs | 23 +-
.../examples/data_io/parquet_index.rs | 12 +-
.../examples/data_io/remote_catalog.rs | 2 +-
.../examples/dataframe/cache_factory.rs | 233 ++
.../examples/dataframe/dataframe.rs | 6 +-
datafusion-examples/examples/dataframe/main.rs | 5 +
.../memory_pool_execution_plan.rs | 2 +-
.../examples/execution_monitoring/tracing.rs | 14 +-
.../external_dependency/dataframe_to_s3.rs | 2 +-
.../ffi/ffi_example_table_provider/src/lib.rs | 7 +-
.../examples/ffi/ffi_module_interface/Cargo.toml | 2 +-
.../examples/ffi/ffi_module_interface/src/lib.rs | 7 +-
.../examples/ffi/ffi_module_loader/Cargo.toml | 3 +-
.../examples/ffi/ffi_module_loader/src/main.rs | 13 +-
datafusion-examples/examples/flight/server.rs | 2 +-
datafusion-examples/examples/flight/sql_server.rs | 4 +-
.../examples/proto/composed_extension_codec.rs | 2 +-
.../examples/query_planning/analyzer_rule.rs | 4 +-
.../examples/query_planning/expr_api.rs | 29 +-
.../examples/query_planning/optimizer_rule.rs | 2 +-
.../examples/query_planning/plan_to_sql.rs | 2 +-
.../examples/relation_planner/match_recognize.rs | 2 +-
.../examples/relation_planner/pivot_unpivot.rs | 5 +-
.../examples/relation_planner/table_sample.rs | 31 +-
datafusion-examples/examples/sql_ops/analysis.rs | 2 +-
datafusion-examples/examples/sql_ops/frontend.rs | 2 +-
datafusion-examples/examples/sql_ops/query.rs | 10 +-
datafusion-examples/examples/udf/advanced_udaf.rs | 4 +-
datafusion-examples/examples/udf/advanced_udf.rs | 6 +-
datafusion-examples/examples/udf/async_udf.rs | 26 +-
datafusion-examples/examples/udf/simple_udtf.rs | 9 +-
datafusion/catalog-listing/Cargo.toml | 2 +-
datafusion/catalog-listing/src/config.rs | 86 +-
datafusion/catalog-listing/src/table.rs | 113 +-
datafusion/catalog/Cargo.toml | 2 +-
datafusion/common-runtime/Cargo.toml | 2 +-
datafusion/common/Cargo.toml | 7 +-
datafusion/common/benches/with_hashes.rs | 209 ++
datafusion/common/src/config.rs | 68 +
datafusion/common/src/dfschema.rs | 2 +-
datafusion/common/src/file_options/csv_writer.rs | 17 +
datafusion/common/src/file_options/json_writer.rs | 18 +-
datafusion/common/src/hash_utils.rs | 136 +-
datafusion/common/src/utils/memory.rs | 200 +-
datafusion/core/Cargo.toml | 5 +
datafusion/core/benches/data_utils/mod.rs | 2 +-
datafusion/core/benches/distinct_query_sql.rs | 4 +-
datafusion/core/benches/filter_query_sql.rs | 2 +-
datafusion/core/benches/map_query_sql.rs | 4 +-
datafusion/core/benches/parquet_query_sql.rs | 4 +-
datafusion/core/benches/physical_plan.rs | 2 +-
.../core/benches/preserve_file_partitioning.rs | 4 +-
datafusion/core/benches/push_down_filter.rs | 6 +-
.../core/benches/range_and_generate_series.rs | 90 +
datafusion/core/benches/scalar.rs | 2 +-
datafusion/core/benches/sort.rs | 66 +-
datafusion/core/benches/sort_limit_query_sql.rs | 3 +-
datafusion/core/benches/spm.rs | 6 +-
datafusion/core/benches/sql_planner.rs | 10 +-
datafusion/core/benches/sql_planner_extended.rs | 2 +-
datafusion/core/benches/sql_query_with_io.rs | 6 +-
datafusion/core/benches/struct_query_sql.rs | 2 +-
datafusion/core/benches/topk_aggregate.rs | 10 +-
datafusion/core/src/bin/print_functions_docs.rs | 10 +-
datafusion/core/src/dataframe/mod.rs | 21 +-
datafusion/core/src/datasource/dynamic_file.rs | 2 +-
datafusion/core/src/datasource/empty.rs | 2 +-
datafusion/core/src/datasource/file_format/avro.rs | 40 +-
datafusion/core/src/datasource/file_format/csv.rs | 159 +-
datafusion/core/src/datasource/file_format/json.rs | 68 +-
datafusion/core/src/datasource/file_format/mod.rs | 29 +-
.../core/src/datasource/file_format/options.rs | 2 +-
.../core/src/datasource/file_format/parquet.rs | 43 +-
datafusion/core/src/datasource/listing/table.rs | 422 +---
.../core/src/datasource/listing_table_factory.rs | 6 +-
datafusion/core/src/datasource/memory_test.rs | 11 +-
datafusion/core/src/datasource/mod.rs | 67 +-
.../core/src/datasource/physical_plan/avro.rs | 90 +-
.../core/src/datasource/physical_plan/csv.rs | 164 +-
.../core/src/datasource/physical_plan/json.rs | 29 +-
.../core/src/datasource/physical_plan/mod.rs | 139 +-
.../core/src/datasource/physical_plan/parquet.rs | 256 +--
datafusion/core/src/datasource/view_test.rs | 36 +-
datafusion/core/src/execution/context/mod.rs | 105 +-
datafusion/core/src/execution/context/parquet.rs | 6 +-
datafusion/core/src/execution/session_state.rs | 76 +-
.../core/src/execution/session_state_defaults.rs | 6 +-
datafusion/core/src/physical_planner.rs | 192 +-
datafusion/core/src/prelude.rs | 2 +-
datafusion/core/src/test/mod.rs | 12 +-
datafusion/core/src/test/object_store.rs | 10 +-
datafusion/core/src/test_util/parquet.rs | 13 +-
datafusion/core/tests/catalog/memory.rs | 10 +-
.../tests/catalog_listing/pruned_partition_list.rs | 4 +-
datafusion/core/tests/config_from_env.rs | 60 +-
datafusion/core/tests/core_integration.rs | 3 -
datafusion/core/tests/custom_sources_cases/mod.rs | 8 +-
.../provider_filter_pushdown.rs | 2 +-
.../core/tests/dataframe/dataframe_functions.rs | 4 +-
datafusion/core/tests/dataframe/describe.rs | 2 +-
datafusion/core/tests/dataframe/mod.rs | 746 +++----
datafusion/core/tests/execution/coop.rs | 24 +-
.../core/tests/execution/datasource_split.rs | 2 +-
datafusion/core/tests/execution/logical_plan.rs | 2 +-
datafusion/core/tests/expr_api/mod.rs | 4 +-
datafusion/core/tests/expr_api/parse_sql_expr.rs | 2 +-
datafusion/core/tests/expr_api/simplification.rs | 22 +-
datafusion/core/tests/fifo/mod.rs | 8 +-
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 37 +-
.../aggregation_fuzzer/context_generator.rs | 2 +-
.../aggregation_fuzzer/data_generator.rs | 6 +-
.../tests/fuzz_cases/aggregation_fuzzer/fuzzer.rs | 4 +-
.../fuzz_cases/aggregation_fuzzer/query_builder.rs | 2 +-
.../tests/fuzz_cases/distinct_count_string_fuzz.rs | 2 +-
.../core/tests/fuzz_cases/equivalence/ordering.rs | 10 +-
.../tests/fuzz_cases/equivalence/projection.rs | 8 +-
.../tests/fuzz_cases/equivalence/properties.rs | 6 +-
.../core/tests/fuzz_cases/equivalence/utils.rs | 8 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 24 +-
datafusion/core/tests/fuzz_cases/limit_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/merge_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/pruning.rs | 4 +-
.../tests/fuzz_cases/record_batch_generator.rs | 20 +-
datafusion/core/tests/fuzz_cases/sort_fuzz.rs | 4 +-
.../fuzz_cases/sort_preserving_repartition_fuzz.rs | 11 +-
.../core/tests/fuzz_cases/sort_query_fuzz.rs | 28 +-
.../spilling_fuzz_in_memory_constrained_env.rs | 40 +-
datafusion/core/tests/fuzz_cases/window_fuzz.rs | 51 +-
.../memory_limit/memory_limit_validation/utils.rs | 6 +-
datafusion/core/tests/memory_limit/mod.rs | 28 +-
.../tests/memory_limit/repartition_mem_limit.rs | 15 +-
datafusion/core/tests/optimizer/mod.rs | 6 +-
datafusion/core/tests/parquet/custom_reader.rs | 6 +-
datafusion/core/tests/parquet/encryption.rs | 4 +-
.../parquet/{schema_adapter.rs => expr_adapter.rs} | 160 +-
.../core/tests/parquet/external_access_plan.rs | 10 +-
datafusion/core/tests/parquet/file_statistics.rs | 12 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 6 +-
datafusion/core/tests/parquet/mod.rs | 29 +-
datafusion/core/tests/parquet/page_pruning.rs | 8 +-
datafusion/core/tests/parquet/schema_coercion.rs | 6 +-
datafusion/core/tests/parquet/utils.rs | 11 +-
.../physical_optimizer/aggregate_statistics.rs | 6 +-
.../combine_partial_final_agg.rs | 8 +-
.../physical_optimizer/enforce_distribution.rs | 830 ++++----
.../tests/physical_optimizer/enforce_sorting.rs | 33 +-
.../enforce_sorting_monotonicity.rs | 2 +-
.../physical_optimizer/filter_pushdown/mod.rs | 455 ++--
.../physical_optimizer/filter_pushdown/util.rs | 35 +-
.../tests/physical_optimizer/join_selection.rs | 25 +-
.../tests/physical_optimizer/limit_pushdown.rs | 148 +-
.../limited_distinct_aggregation.rs | 6 +-
datafusion/core/tests/physical_optimizer/mod.rs | 1 +
.../physical_optimizer/partition_statistics.rs | 201 +-
.../physical_optimizer/projection_pushdown.rs | 292 ++-
.../core/tests/physical_optimizer/pushdown_sort.rs | 672 ++++++
.../replace_with_order_preserving_variants.rs | 10 +-
.../tests/physical_optimizer/sanity_checker.rs | 6 +-
.../core/tests/physical_optimizer/test_utils.rs | 95 +-
.../tests/physical_optimizer/window_optimize.rs | 4 +-
datafusion/core/tests/schema_adapter/mod.rs | 18 -
.../schema_adapter_integration_tests.rs | 752 -------
datafusion/core/tests/sql/aggregates/basic.rs | 8 +-
datafusion/core/tests/sql/aggregates/dict_nulls.rs | 24 +-
datafusion/core/tests/sql/aggregates/mod.rs | 12 +-
datafusion/core/tests/sql/explain_analyze.rs | 55 +-
datafusion/core/tests/sql/joins.rs | 79 +-
datafusion/core/tests/sql/mod.rs | 7 +-
datafusion/core/tests/sql/path_partition.rs | 38 +-
datafusion/core/tests/sql/runtime_config.rs | 92 +
datafusion/core/tests/sql/select.rs | 2 +-
datafusion/core/tests/tracing/asserting_tracer.rs | 2 +-
.../core/tests/tracing/traceable_object_store.rs | 4 +-
datafusion/core/tests/user_defined/expr_planner.rs | 18 +-
.../core/tests/user_defined/insert_operation.rs | 4 +-
.../core/tests/user_defined/relation_planner.rs | 29 +-
.../tests/user_defined/user_defined_aggregates.rs | 80 +-
.../user_defined_async_scalar_functions.rs | 2 +-
.../core/tests/user_defined/user_defined_plan.rs | 73 +-
.../user_defined/user_defined_scalar_functions.rs | 69 +-
.../user_defined/user_defined_table_functions.rs | 16 +-
.../user_defined/user_defined_window_functions.rs | 303 +--
datafusion/datasource-arrow/Cargo.toml | 2 +-
datafusion/datasource-arrow/src/source.rs | 122 +-
datafusion/datasource-avro/Cargo.toml | 2 +-
datafusion/datasource-avro/src/source.rs | 17 -
datafusion/datasource-csv/Cargo.toml | 2 +-
datafusion/datasource-csv/src/file_format.rs | 3 +-
datafusion/datasource-csv/src/source.rs | 31 +-
datafusion/datasource-json/Cargo.toml | 2 +-
datafusion/datasource-json/src/file_format.rs | 1 +
datafusion/datasource-json/src/source.rs | 17 -
datafusion/datasource-parquet/Cargo.toml | 2 +-
datafusion/datasource-parquet/src/mod.rs | 1 +
datafusion/datasource-parquet/src/opener.rs | 662 ++++--
datafusion/datasource-parquet/src/sort.rs | 407 ++++
datafusion/datasource-parquet/src/source.rs | 183 +-
datafusion/datasource/Cargo.toml | 2 +-
datafusion/datasource/src/file.rs | 79 +-
datafusion/datasource/src/file_compression_type.rs | 48 +-
datafusion/datasource/src/file_groups.rs | 18 +-
datafusion/datasource/src/file_scan_config.rs | 141 +-
datafusion/datasource/src/mod.rs | 116 +-
datafusion/datasource/src/schema_adapter.rs | 1065 +---------
datafusion/datasource/src/source.rs | 35 +-
datafusion/datasource/src/test_util.rs | 18 -
datafusion/datasource/src/url.rs | 269 ++-
datafusion/datasource/src/write/mod.rs | 23 +-
datafusion/datasource/src/write/orchestration.rs | 2 +
datafusion/doc/Cargo.toml | 2 +-
datafusion/execution/Cargo.toml | 4 +-
datafusion/execution/src/cache/cache_manager.rs | 114 +-
datafusion/execution/src/cache/list_files_cache.rs | 311 ++-
datafusion/execution/src/config.rs | 7 +
datafusion/execution/src/lib.rs | 1 +
datafusion/execution/src/memory_pool/pool.rs | 34 +-
.../src/metrics/baseline.rs | 8 +-
.../src/metrics/builder.rs | 2 +-
.../src/metrics/custom.rs | 2 +-
.../src/metrics/mod.rs | 22 +-
.../src/metrics/value.rs | 3 +-
datafusion/execution/src/runtime_env.rs | 38 +-
datafusion/expr-common/Cargo.toml | 2 +-
datafusion/expr/Cargo.toml | 2 +-
datafusion/ffi/src/arrow_wrappers.rs | 12 +-
datafusion/ffi/src/catalog_provider.rs | 237 ++-
datafusion/ffi/src/catalog_provider_list.rs | 195 +-
datafusion/ffi/src/execution/task_ctx.rs | 82 +-
datafusion/ffi/src/execution/task_ctx_provider.rs | 56 +-
datafusion/ffi/src/execution_plan.rs | 130 +-
datafusion/ffi/src/physical_expr/mod.rs | 241 ++-
datafusion/ffi/src/physical_expr/partitioning.rs | 4 +-
datafusion/ffi/src/plan_properties.rs | 152 +-
.../ffi/src/proto/logical_extension_codec.rs | 52 +-
.../ffi/src/proto/physical_extension_codec.rs | 23 +-
datafusion/ffi/src/record_batch_stream.rs | 44 +-
datafusion/ffi/src/schema_provider.rs | 245 ++-
datafusion/ffi/src/session/config.rs | 34 +-
datafusion/ffi/src/session/mod.rs | 110 +-
datafusion/ffi/src/table_provider.rs | 291 +--
datafusion/ffi/src/tests/async_provider.rs | 15 +-
datafusion/ffi/src/tests/catalog.rs | 21 +-
datafusion/ffi/src/tests/mod.rs | 27 +-
datafusion/ffi/src/tests/sync_provider.rs | 10 +-
datafusion/ffi/src/tests/udf_udaf_udwf.rs | 42 +-
datafusion/ffi/src/udaf/accumulator.rs | 120 +-
datafusion/ffi/src/udaf/accumulator_args.rs | 105 +-
datafusion/ffi/src/udaf/groups_accumulator.rs | 131 +-
datafusion/ffi/src/udaf/mod.rs | 227 +-
datafusion/ffi/src/udf/mod.rs | 109 +-
datafusion/ffi/src/udf/return_type_args.rs | 2 +-
datafusion/ffi/src/udtf.rs | 168 +-
datafusion/ffi/src/udwf/mod.rs | 155 +-
datafusion/ffi/src/udwf/partition_evaluator.rs | 123 +-
.../ffi/src/udwf/partition_evaluator_args.rs | 104 +-
datafusion/ffi/src/util.rs | 8 +-
datafusion/ffi/tests/ffi_catalog.rs | 11 +-
datafusion/ffi/tests/ffi_integration.rs | 8 +-
datafusion/ffi/tests/ffi_udaf.rs | 73 +-
datafusion/ffi/tests/ffi_udf.rs | 2 +-
datafusion/ffi/tests/ffi_udtf.rs | 9 +-
datafusion/ffi/tests/ffi_udwf.rs | 4 +-
.../function/aggregate => ffi/tests/utils}/mod.rs | 29 +-
datafusion/functions-aggregate-common/Cargo.toml | 2 +-
datafusion/functions-aggregate/Cargo.toml | 2 +-
datafusion/functions-aggregate/src/first_last.rs | 33 +-
datafusion/functions-aggregate/src/median.rs | 54 +-
datafusion/functions-nested/Cargo.toml | 2 +-
datafusion/functions-nested/src/range.rs | 2 +-
datafusion/functions-table/Cargo.toml | 2 +-
datafusion/functions-window-common/Cargo.toml | 2 +-
datafusion/functions-window/Cargo.toml | 2 +-
datafusion/functions/Cargo.toml | 2 +-
datafusion/functions/src/encoding/inner.rs | 745 +++----
datafusion/functions/src/math/ceil.rs | 174 ++
datafusion/functions/src/math/decimal.rs | 111 +
datafusion/functions/src/math/floor.rs | 174 ++
datafusion/functions/src/math/log.rs | 101 +-
datafusion/functions/src/math/mod.rs | 21 +-
datafusion/functions/src/math/monotonicity.rs | 48 -
datafusion/functions/src/math/power.rs | 351 +---
datafusion/functions/src/utils.rs | 137 ++
datafusion/macros/Cargo.toml | 2 +-
datafusion/macros/src/user_doc.rs | 1 +
datafusion/optimizer/Cargo.toml | 2 +-
.../optimizer/src/decorrelate_lateral_join.rs | 2 +-
.../src/decorrelate_predicate_subquery.rs | 6 +-
datafusion/optimizer/src/eliminate_cross_join.rs | 2 +-
.../optimizer/src/eliminate_duplicated_expr.rs | 2 +-
datafusion/optimizer/src/eliminate_filter.rs | 2 +-
datafusion/optimizer/src/eliminate_limit.rs | 4 +-
datafusion/optimizer/src/eliminate_outer_join.rs | 2 +-
.../optimizer/src/extract_equijoin_predicate.rs | 2 +-
datafusion/optimizer/src/lib.rs | 1 +
.../optimizer/src/optimize_projections/mod.rs | 2 +-
datafusion/optimizer/src/optimize_unions.rs | 6 +-
.../optimizer/src/propagate_empty_relation.rs | 2 +-
datafusion/optimizer/src/push_down_filter.rs | 8 +-
datafusion/optimizer/src/push_down_limit.rs | 8 +-
.../optimizer/src/replace_distinct_aggregate.rs | 2 +-
.../optimizer/src/scalar_subquery_to_join.rs | 2 +-
.../src/simplify_expressions/expr_simplifier.rs | 4 +-
.../src/simplify_expressions/simplify_exprs.rs | 20 +-
.../optimizer/src/single_distinct_to_groupby.rs | 2 +-
.../optimizer/tests/optimizer_integration.rs | 315 +--
datafusion/physical-expr-adapter/Cargo.toml | 2 +-
.../physical-expr-adapter/src/schema_rewriter.rs | 4 +-
datafusion/physical-expr-common/Cargo.toml | 2 +-
datafusion/physical-expr-common/src/sort_expr.rs | 114 +
datafusion/physical-expr/Cargo.toml | 6 +-
datafusion/physical-expr/benches/in_list.rs | 42 +-
datafusion/physical-expr/src/partitioning.rs | 586 +++++-
datafusion/physical-expr/src/planner.rs | 33 +-
datafusion/physical-expr/src/projection.rs | 3 +-
datafusion/physical-optimizer/Cargo.toml | 2 +-
.../physical-optimizer/src/coalesce_batches.rs | 20 +-
.../physical-optimizer/src/enforce_distribution.rs | 82 +-
datafusion/physical-optimizer/src/ensure_coop.rs | 8 +-
datafusion/physical-optimizer/src/lib.rs | 1 +
datafusion/physical-optimizer/src/optimizer.rs | 3 +
datafusion/physical-optimizer/src/pushdown_sort.rs | 129 ++
.../physical-optimizer/src/sanity_checker.rs | 3 +-
datafusion/physical-plan/Cargo.toml | 3 +-
.../src/aggregates/group_values/mod.rs | 4 +-
.../group_values/multi_group_by/bytes_view.rs | 121 +-
.../aggregates/group_values/multi_group_by/mod.rs | 24 +-
.../src/aggregates/group_values/row.rs | 9 +-
.../group_values/single_group_by/boolean.rs | 3 +-
.../group_values/single_group_by/bytes.rs | 4 +-
.../group_values/single_group_by/bytes_view.rs | 4 +-
.../group_values/single_group_by/primitive.rs | 8 +-
datafusion/physical-plan/src/aggregates/mod.rs | 499 +++--
.../physical-plan/src/aggregates/no_grouping.rs | 20 +
.../physical-plan/src/aggregates/row_hash.rs | 353 ++--
.../physical-plan/src/aggregates/topk/heap.rs | 88 +-
.../src/aggregates/topk/priority_map.rs | 142 +-
datafusion/physical-plan/src/coalesce_batches.rs | 29 +
.../physical-plan/src/coalesce_partitions.rs | 38 +
datafusion/physical-plan/src/coop.rs | 15 +-
datafusion/physical-plan/src/execution_plan.rs | 28 +-
datafusion/physical-plan/src/joins/cross_join.rs | 24 +-
.../physical-plan/src/joins/hash_join/exec.rs | 196 +-
.../physical-plan/src/joins/hash_join/mod.rs | 2 +-
.../src/joins/hash_join/partitioned_hash_eval.rs | 349 ++-
.../src/joins/hash_join/shared_bounds.rs | 12 +-
.../physical-plan/src/joins/join_hash_map.rs | 6 +
datafusion/physical-plan/src/joins/mod.rs | 2 +-
.../physical-plan/src/joins/nested_loop_join.rs | 168 +-
.../src/joins/piecewise_merge_join/classic_join.rs | 110 +-
.../src/joins/sort_merge_join/tests.rs | 716 +++----
datafusion/physical-plan/src/lib.rs | 2 +
.../src/metrics.rs} | 6 +-
datafusion/physical-plan/src/repartition/mod.rs | 34 +-
datafusion/physical-plan/src/sort_pushdown.rs | 120 ++
datafusion/physical-plan/src/sorts/partial_sort.rs | 130 +-
datafusion/physical-plan/src/sorts/sort.rs | 30 +-
.../src/sorts/sort_preserving_merge.rs | 142 +-
datafusion/physical-plan/src/spill/mod.rs | 206 +-
datafusion/physical-plan/src/spill/spill_pool.rs | 67 +-
datafusion/physical-plan/src/test.rs | 1 +
datafusion/physical-plan/src/unnest.rs | 52 +-
.../src/windows/bounded_window_agg_exec.rs | 58 +-
datafusion/proto-common/Cargo.toml | 2 +-
.../proto-common/proto/datafusion_common.proto | 2 +
datafusion/proto-common/src/from_proto/mod.rs | 2 +
datafusion/proto-common/src/generated/pbjson.rs | 49 +-
datafusion/proto-common/src/generated/prost.rs | 6 +
datafusion/proto-common/src/to_proto/mod.rs | 2 +
datafusion/proto/proto/datafusion.proto | 12 +
datafusion/proto/src/bytes/mod.rs | 8 +-
datafusion/proto/src/bytes/registry.rs | 29 +-
datafusion/proto/src/common.rs | 2 +-
.../proto/src/generated/datafusion_proto_common.rs | 6 +
datafusion/proto/src/generated/mod.rs | 2 +
datafusion/proto/src/generated/pbjson.rs | 225 ++
datafusion/proto/src/generated/prost.rs | 21 +-
datafusion/proto/src/lib.rs | 1 +
datafusion/proto/src/logical_plan/file_formats.rs | 21 +-
datafusion/proto/src/logical_plan/from_proto.rs | 15 +-
datafusion/proto/src/logical_plan/mod.rs | 48 +-
datafusion/proto/src/logical_plan/to_proto.rs | 45 +-
datafusion/proto/src/physical_plan/from_proto.rs | 25 +-
datafusion/proto/src/physical_plan/mod.rs | 57 +-
datafusion/proto/src/physical_plan/to_proto.rs | 24 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 38 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 121 +-
datafusion/proto/tests/cases/serialize.rs | 6 +-
datafusion/pruning/Cargo.toml | 2 +-
datafusion/pruning/src/lib.rs | 1 +
datafusion/pruning/src/pruning_predicate.rs | 4 -
datafusion/session/Cargo.toml | 2 +-
datafusion/session/src/lib.rs | 1 +
datafusion/spark/Cargo.toml | 2 +-
datafusion/spark/src/function/aggregate/mod.rs | 12 +-
datafusion/spark/src/function/aggregate/try_sum.rs | 660 ++++++
datafusion/spark/src/function/datetime/date_add.rs | 88 +-
datafusion/spark/src/function/datetime/next_day.rs | 52 +-
datafusion/spark/src/function/hash/crc32.rs | 57 +-
datafusion/spark/src/function/hash/sha1.rs | 58 +-
datafusion/spark/src/function/string/concat.rs | 69 +-
datafusion/sql/Cargo.toml | 2 +-
datafusion/sql/src/expr/value.rs | 1 -
datafusion/sql/src/select.rs | 8 +-
datafusion/sql/tests/cases/params.rs | 50 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 171 +-
datafusion/sql/tests/sql_integration.rs | 2222 ++++++++++----------
datafusion/sqllogictest/Cargo.toml | 2 +-
datafusion/sqllogictest/bin/postgres_container.rs | 10 +-
datafusion/sqllogictest/bin/sqllogictests.rs | 40 +-
datafusion/sqllogictest/src/engines/conversion.rs | 4 +-
.../src/engines/datafusion_engine/runner.rs | 2 +-
.../runner.rs | 2 +-
datafusion/sqllogictest/src/engines/mod.rs | 4 +-
.../src/engines/postgres_engine/mod.rs | 4 +-
datafusion/sqllogictest/src/filters.rs | 19 +-
datafusion/sqllogictest/src/lib.rs | 4 +-
datafusion/sqllogictest/src/test_context.rs | 62 +-
datafusion/sqllogictest/src/util.rs | 2 +-
.../test_files/agg_func_substitute.slt | 27 +-
datafusion/sqllogictest/test_files/aggregate.slt | 167 +-
.../test_files/aggregate_repartition.slt | 16 +-
.../sqllogictest/test_files/aggregates_topk.slt | 49 +-
datafusion/sqllogictest/test_files/async_udf.slt | 15 +-
.../sqllogictest/test_files/count_star_rule.slt | 14 +-
.../test_files/create_external_table.slt | 2 +-
datafusion/sqllogictest/test_files/cte.slt | 10 +-
datafusion/sqllogictest/test_files/decimal.slt | 75 +-
datafusion/sqllogictest/test_files/distinct_on.slt | 9 +-
.../test_files/dynamic_filter_pushdown_config.slt | 326 +++
datafusion/sqllogictest/test_files/encoding.slt | 143 +-
datafusion/sqllogictest/test_files/explain.slt | 4 +
.../sqllogictest/test_files/explain_tree.slt | 108 +-
datafusion/sqllogictest/test_files/group_by.slt | 177 +-
datafusion/sqllogictest/test_files/grouping.slt | 12 +
.../sqllogictest/test_files/information_schema.slt | 8 +
datafusion/sqllogictest/test_files/insert.slt | 15 +-
.../sqllogictest/test_files/insert_to_external.slt | 10 +-
datafusion/sqllogictest/test_files/join.slt.part | 9 +-
datafusion/sqllogictest/test_files/joins.slt | 130 +-
datafusion/sqllogictest/test_files/limit.slt | 16 +-
datafusion/sqllogictest/test_files/math.slt | 54 +
datafusion/sqllogictest/test_files/metadata.slt | 49 +
datafusion/sqllogictest/test_files/order.slt | 33 +-
datafusion/sqllogictest/test_files/predicates.slt | 32 +-
.../test_files/preserve_file_partitioning.slt | 45 +-
.../sqllogictest/test_files/push_down_filter.slt | 7 +-
datafusion/sqllogictest/test_files/qualify.slt | 25 +-
datafusion/sqllogictest/test_files/repartition.slt | 14 +-
.../test_files/repartition_subset_satisfaction.slt | 526 +++++
datafusion/sqllogictest/test_files/scalar.slt | 98 +-
.../sqllogictest/test_files/schema_evolution.slt | 144 ++
datafusion/sqllogictest/test_files/select.slt | 13 +-
.../sqllogictest/test_files/set_variable.slt | 14 +
.../test_files/spark/aggregate/try_sum.slt | 140 ++
.../sqllogictest/test_files/spark/math/hex.slt | 2 +-
datafusion/sqllogictest/test_files/subquery.slt | 52 +-
.../sqllogictest/test_files/subquery_sort.slt | 9 +-
datafusion/sqllogictest/test_files/timestamps.slt | 6 +-
datafusion/sqllogictest/test_files/topk.slt | 2 +-
.../sqllogictest/test_files/tpch/plans/q1.slt.part | 11 +-
.../test_files/tpch/plans/q10.slt.part | 41 +-
.../test_files/tpch/plans/q11.slt.part | 65 +-
.../test_files/tpch/plans/q12.slt.part | 19 +-
.../test_files/tpch/plans/q13.slt.part | 23 +-
.../test_files/tpch/plans/q14.slt.part | 12 +-
.../test_files/tpch/plans/q15.slt.part | 29 +-
.../test_files/tpch/plans/q16.slt.part | 40 +-
.../test_files/tpch/plans/q17.slt.part | 25 +-
.../test_files/tpch/plans/q18.slt.part | 40 +-
.../test_files/tpch/plans/q19.slt.part | 16 +-
.../sqllogictest/test_files/tpch/plans/q2.slt.part | 103 +-
.../test_files/tpch/plans/q20.slt.part | 56 +-
.../test_files/tpch/plans/q21.slt.part | 64 +-
.../test_files/tpch/plans/q22.slt.part | 43 +-
.../sqllogictest/test_files/tpch/plans/q3.slt.part | 35 +-
.../sqllogictest/test_files/tpch/plans/q4.slt.part | 21 +-
.../sqllogictest/test_files/tpch/plans/q5.slt.part | 63 +-
.../sqllogictest/test_files/tpch/plans/q7.slt.part | 69 +-
.../sqllogictest/test_files/tpch/plans/q8.slt.part | 89 +-
.../sqllogictest/test_files/tpch/plans/q9.slt.part | 63 +-
datafusion/sqllogictest/test_files/union.slt | 122 +-
datafusion/sqllogictest/test_files/unnest.slt | 17 +-
datafusion/sqllogictest/test_files/window.slt | 220 +-
.../sqllogictest/test_files/window_limits.slt | 10 +-
datafusion/substrait/src/extensions.rs | 4 +-
.../consumer/expr/aggregate_function.rs | 6 +-
.../src/logical_plan/consumer/expr/cast.rs | 4 +-
.../logical_plan/consumer/expr/field_reference.rs | 4 +-
.../consumer/expr/function_arguments.rs | 4 +-
.../src/logical_plan/consumer/expr/literal.rs | 40 +-
.../src/logical_plan/consumer/expr/mod.rs | 22 +-
.../logical_plan/consumer/expr/scalar_function.rs | 25 +-
.../logical_plan/consumer/expr/singular_or_list.rs | 4 +-
.../src/logical_plan/consumer/expr/subquery.rs | 8 +-
.../logical_plan/consumer/expr/window_function.rs | 10 +-
.../substrait/src/logical_plan/consumer/plan.rs | 67 +-
.../src/logical_plan/consumer/rel/aggregate_rel.rs | 6 +-
.../src/logical_plan/consumer/rel/exchange_rel.rs | 4 +-
.../src/logical_plan/consumer/rel/fetch_rel.rs | 6 +-
.../src/logical_plan/consumer/rel/join_rel.rs | 4 +-
.../substrait/src/logical_plan/consumer/rel/mod.rs | 6 +-
.../src/logical_plan/consumer/rel/project_rel.rs | 4 +-
.../src/logical_plan/consumer/rel/read_rel.rs | 16 +-
.../src/logical_plan/consumer/rel/sort_rel.rs | 2 +-
.../logical_plan/consumer/substrait_consumer.rs | 12 +-
.../substrait/src/logical_plan/consumer/types.rs | 32 +-
.../substrait/src/logical_plan/consumer/utils.rs | 9 +-
.../src/logical_plan/producer/expr/cast.rs | 6 +-
.../logical_plan/producer/expr/field_reference.rs | 6 +-
.../src/logical_plan/producer/expr/if_then.rs | 2 +-
.../src/logical_plan/producer/expr/literal.rs | 8 +-
.../src/logical_plan/producer/expr/mod.rs | 6 +-
.../logical_plan/producer/expr/scalar_function.rs | 6 +-
.../logical_plan/producer/expr/window_function.rs | 8 +-
.../substrait/src/logical_plan/producer/plan.rs | 4 +-
.../src/logical_plan/producer/rel/aggregate_rel.rs | 8 +-
.../src/logical_plan/producer/rel/exchange_rel.rs | 6 +-
.../src/logical_plan/producer/rel/fetch_rel.rs | 2 +-
.../src/logical_plan/producer/rel/join.rs | 6 +-
.../src/logical_plan/producer/rel/project_rel.rs | 4 +-
.../src/logical_plan/producer/rel/read_rel.rs | 6 +-
.../src/logical_plan/producer/rel/set_rel.rs | 2 +-
.../src/logical_plan/producer/rel/sort_rel.rs | 4 +-
.../logical_plan/producer/substrait_producer.rs | 14 +-
.../substrait/src/logical_plan/producer/types.rs | 10 +-
.../substrait/src/logical_plan/producer/utils.rs | 4 +-
datafusion/substrait/src/physical_plan/consumer.rs | 26 +-
datafusion/substrait/src/physical_plan/producer.rs | 157 +-
.../substrait/tests/cases/aggregation_tests.rs | 16 +-
.../substrait/tests/cases/consumer_integration.rs | 420 ++--
.../substrait/tests/cases/emit_kind_tests.rs | 44 +-
datafusion/substrait/tests/cases/function_test.rs | 8 +-
datafusion/substrait/tests/cases/logical_plans.rs | 65 +-
.../tests/cases/roundtrip_logical_plan.rs | 110 +-
.../tests/cases/roundtrip_physical_plan.rs | 2 +-
datafusion/substrait/tests/cases/serialize.rs | 20 +-
.../substrait/tests/cases/substrait_validations.rs | 24 +-
datafusion/substrait/tests/utils.rs | 20 +-
datafusion/wasmtest/src/lib.rs | 2 +-
docs/source/index.rst | 1 +
.../library-user-guide/extending-operators.md | 7 +-
docs/source/library-user-guide/extending-sql.md | 339 +++
.../library-user-guide/functions/adding-udfs.md | 4 +-
docs/source/library-user-guide/query-optimizer.md | 7 +-
docs/source/library-user-guide/upgrading.md | 94 +-
docs/source/user-guide/configs.md | 252 +--
docs/source/user-guide/sql/scalar_functions.md | 2 +-
rust-toolchain.toml | 2 +-
rustfmt.toml | 2 +-
test-utils/src/array_gen/binary.rs | 2 +-
test-utils/src/array_gen/boolean.rs | 2 +-
test-utils/src/array_gen/decimal.rs | 2 +-
test-utils/src/array_gen/primitive.rs | 4 +-
test-utils/src/array_gen/random_data.rs | 16 +-
test-utils/src/array_gen/string.rs | 2 +-
test-utils/src/string_gen.rs | 2 +-
614 files changed, 21444 insertions(+), 13741 deletions(-)
create mode 100644 datafusion-examples/examples/dataframe/cache_factory.rs
create mode 100644 datafusion/common/benches/with_hashes.rs
create mode 100644 datafusion/core/benches/range_and_generate_series.rs
rename datafusion/core/tests/parquet/{schema_adapter.rs => expr_adapter.rs}
(70%)
create mode 100644 datafusion/core/tests/physical_optimizer/pushdown_sort.rs
delete mode 100644 datafusion/core/tests/schema_adapter/mod.rs
delete mode 100644
datafusion/core/tests/schema_adapter/schema_adapter_integration_tests.rs
create mode 100644 datafusion/datasource-parquet/src/sort.rs
rename datafusion/{physical-plan => execution}/src/metrics/baseline.rs (97%)
rename datafusion/{physical-plan => execution}/src/metrics/builder.rs (99%)
rename datafusion/{physical-plan => execution}/src/metrics/custom.rs (98%)
rename datafusion/{physical-plan => execution}/src/metrics/mod.rs (97%)
rename datafusion/{physical-plan => execution}/src/metrics/value.rs (99%)
copy datafusion/{spark/src/function/aggregate => ffi/tests/utils}/mod.rs (52%)
create mode 100644 datafusion/functions/src/math/ceil.rs
create mode 100644 datafusion/functions/src/math/decimal.rs
create mode 100644 datafusion/functions/src/math/floor.rs
create mode 100644 datafusion/physical-optimizer/src/pushdown_sort.rs
copy datafusion/{common/src/file_options/avro_writer.rs =>
physical-plan/src/metrics.rs} (84%)
create mode 100644 datafusion/physical-plan/src/sort_pushdown.rs
create mode 100644 datafusion/spark/src/function/aggregate/try_sum.rs
create mode 100644
datafusion/sqllogictest/test_files/repartition_subset_satisfaction.slt
create mode 100644
datafusion/sqllogictest/test_files/spark/aggregate/try_sum.slt
create mode 100644 docs/source/library-user-guide/extending-sql.md
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
