This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/substrait-0.59.0
in repository https://gitbox.apache.org/repos/asf/datafusion.git
discard a077e8b074 chore(deps): bump substrait from 0.58.0 to 0.59.0
add d376a32f1f Add `sql_parser.default_null_ordering` config option to
customize the default null ordering (#16963)
add 2a90ff606d Added Example for `Statistical Functions` in Docs (#16927)
add 31c3e58797 test(datafusion-cli): migrate tests to `insta` in
`print_format.rs` (#16993)
add 21362aa53d remove 'spill_record_batch_by_size' api (#16958)
add 6ea01d1336 chore(deps): bump serde_json from 1.0.141 to 1.0.142
(#17006)
add 494851a2bd Feature: Improve hash Expr performance (#16977)
add 9d6f923f85 Add tests for yielding in
`SpillManager::read_spill_as_stream` (#16616)
add 1a575a0835 feat(spark): implement Spark string function like/ilike
(#16962)
add a078cd257d Fix window_functions docs formatting (#17005)
add c0bdff912a Fix 'Analaysis' typo (#17015)
add f4c5a63c6b Fix random extra bullet for 'Analytical Functions' (#17014)
add c37dd5ea45 feat: Cache Parquet metadata in built in parquet reader
(#16971)
add b4e7147c85 fix: `ComposedPhysicalExtensionCodec` does not use the same
codec as encoding when decoding (#16986)
add 6d9b76e4a3 Perf: Port arrow-rs optimization for get_buffer_memory_size
and add fast path for no buffer for gc string view (#17008)
add 66d6995b8f Fix failing documentation check (#17026)
add 2968331e4c #16994 Ensure CooperativeExec#maintains_input_order returns
a Vec of the correct size (#16995)
add fa1f8c192d Upgrade arrow/parquet to 56.0.0 (#16690)
add 36f5f1456d test: Add logic tests for string_agg with order (#17033)
add eb2b8c07b2 fix error result in execute&pre_selection (#16930)
add cc78edc51e docs: Fix failing CI (#17041)
add 4c36226bc4 Docs: Add Examples to Config Options page (#17039)
add f10deb67a3 Add Tuning Guide for small data / short queries (#17040)
add 40015a8345 Docs: Update the crate configuration / build settings page
(#17038)
add 6d0073465a Implement `From<Option<String>>' for `ScalarValue` (#17043)
add e4f16dd60f feat: Add `Arc<ConfigOptions>` to `ScalarFunctionArgs`,
don't copy `ConfigOptions` on each query (#16970)
add 5f26e70f12 chore(deps): bump tokio-util from 0.7.15 to 0.7.16 (#17030)
add bf6f6316ae Add missing Substrait to DataFusion function name mappings
(#16950)
add 71b92bc9e7 refactor: use upstream inline_key_fast (#17044)
add b0c8dd61ac Make `AsyncScalarUDFImpl::invoke_async_with_args`
consistent with `ScalarUDFImpl::invoke_with_args` (#16902)
add 3472aa1349 Implement spark `array` function `array` (#16936)
add 6043be448f feat: spark crc32/sha1 (#17032)
add 79c4c057e6 Address memory over-accounting in array_agg (#16816)
add 183ff6643a Support `centroids` config for
`approx_percentile_cont_with_weight` (#17003)
add 2bbd6a14e0 chore(deps): bump aws-credential-types from 1.2.4 to 1.2.5
(#17053)
add 8147565bf1 Support Substrait functions and_not, xor, and between in
consumer built-in expression builder (#16984)
add 84bbce63af Derive UDWF equality from PartialEq, Hash (#17057)
add a9e6d4be4a feat: Limit the memory used in the file metadata cache
(#17031)
add d7a886cef0 fix return field for `is_null` and `is_not_null` expression
(#17056)
add 20bb7e6901 chore(deps): bump tokio from 1.47.0 to 1.47.1 (#17063)
add df153c2f81 Optimize char expression (#16076)
add 60ac1cc5f9 fix: Remove `datafusion.execution.parquet.cache_metadata`
config (#17062)
add a83237f3a9 Fix equality of parametrizable ArrayAgg function (#17065)
add 7bc9906513 Implement Spark `url` function `parse_url` (#16937)
add f9efba0e2c Add ExecutionPlan::reset_state (#17028)
add ac3a5735ff Derive UDAF equality from Eq, Hash (#17067)
add 541a67d12c Remove elements deprecated since v 45 (#17075)
add 060938bf68 Deprecate ScalarUDF::is_nullable (#17074)
add f0630fb4f7 Re-export `object_store` crate via DataFusion Core and
Common (#17070)
add 173989cc2f Docs: Add Tuning Guide for larger-than-memory queries
(#17069)
add 407a965d37 Link UdfEq and PtrEq to help understand relationship.
(#17082)
add cbc0614209 feat: Dynamic Parquet encryption and decryption properties
(#16779)
add 337ed944e8 chore(deps): bump substrait from 0.58.0 to 0.59.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a077e8b074)
\
N -- N -- N refs/heads/dependabot/cargo/main/substrait-0.59.0
(337ed944e8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
Cargo.lock | 210 +++---
Cargo.toml | 14 +-
benchmarks/Cargo.toml | 2 +-
datafusion-cli/Cargo.toml | 2 +-
datafusion-cli/src/print_format.rs | 310 ++++-----
datafusion-examples/Cargo.toml | 4 +-
datafusion-examples/README.md | 1 +
datafusion-examples/examples/async_udf.rs | 6 +-
.../examples/composed_extension_codec.rs | 76 +--
.../examples/parquet_encrypted_with_kms.rs | 301 +++++++++
datafusion-examples/examples/sql_frontend.rs | 2 +-
datafusion/common/Cargo.toml | 2 +-
datafusion/common/src/config.rs | 140 +++-
datafusion/common/src/encryption.rs | 29 +-
.../common/src/file_options/parquet_writer.rs | 34 +-
datafusion/common/src/scalar/mod.rs | 29 +-
datafusion/common/src/types/native.rs | 5 +-
datafusion/common/src/utils/mod.rs | 116 ----
datafusion/core/src/execution/context/mod.rs | 6 +-
datafusion/core/src/execution/session_state.rs | 31 +-
datafusion/core/src/lib.rs | 11 +-
datafusion/core/src/test/object_store.rs | 24 +-
datafusion/core/tests/execution/coop.rs | 83 ++-
.../core/tests/fuzz_cases/equivalence/ordering.rs | 2 +
.../tests/fuzz_cases/equivalence/projection.rs | 3 +
.../tests/fuzz_cases/equivalence/properties.rs | 2 +
datafusion/core/tests/fuzz_cases/pruning.rs | 11 +-
datafusion/core/tests/memory_limit/mod.rs | 4 +
datafusion/core/tests/optimizer/mod.rs | 2 +-
datafusion/core/tests/parquet/encryption.rs | 254 ++++++-
datafusion/core/tests/parquet/file_statistics.rs | 3 +-
datafusion/core/tests/parquet/mod.rs | 5 +-
datafusion/core/tests/parquet/page_pruning.rs | 4 +-
datafusion/core/tests/parquet/row_group_pruning.rs | 14 +-
.../physical_optimizer/filter_pushdown/mod.rs | 2 +
.../physical_optimizer/projection_pushdown.rs | 4 +
datafusion/core/tests/sql/runtime_config.rs | 33 +
.../tests/user_defined/user_defined_aggregates.rs | 74 +--
.../user_defined/user_defined_scalar_functions.rs | 66 +-
.../user_defined/user_defined_window_functions.rs | 37 +-
.../datasource-avro/src/avro_to_arrow/schema.rs | 2 +
datafusion/datasource-parquet/Cargo.toml | 1 +
datafusion/datasource-parquet/src/file_format.rs | 177 ++++-
datafusion/datasource-parquet/src/opener.rs | 56 +-
datafusion/datasource-parquet/src/reader.rs | 145 +++-
datafusion/datasource-parquet/src/source.rs | 42 +-
datafusion/datasource/Cargo.toml | 2 +-
datafusion/doc/src/lib.rs | 13 +-
datafusion/execution/Cargo.toml | 6 +
datafusion/execution/src/cache/cache_manager.rs | 105 ++-
datafusion/execution/src/cache/cache_unit.rs | 435 +++++++++++-
datafusion/execution/src/cache/lru_queue.rs | 537 +++++++++++++++
datafusion/execution/src/cache/mod.rs | 1 +
datafusion/execution/src/config.rs | 72 +-
datafusion/execution/src/lib.rs | 2 +
datafusion/execution/src/parquet_encryption.rs | 81 +++
datafusion/execution/src/runtime_env.rs | 74 ++-
datafusion/expr-common/src/operator.rs | 9 -
datafusion/expr/src/async_udf.rs | 13 +-
datafusion/expr/src/execution_props.rs | 21 +-
datafusion/expr/src/expr.rs | 10 -
datafusion/expr/src/expr_fn.rs | 122 +---
datafusion/expr/src/lib.rs | 2 +
datafusion/expr/src/ptr_eq.rs | 135 ++++
datafusion/expr/src/udaf.rs | 36 +-
datafusion/expr/src/udf.rs | 38 +-
datafusion/expr/src/udf_eq.rs | 183 ++++++
datafusion/expr/src/udwf.rs | 35 +-
datafusion/expr/src/utils.rs | 22 +-
datafusion/ffi/src/session_config.rs | 20 +-
datafusion/ffi/src/udaf/mod.rs | 44 +-
datafusion/ffi/src/udf/mod.rs | 6 +
datafusion/ffi/src/udwf/mod.rs | 49 +-
datafusion/functions-aggregate-common/src/utils.rs | 61 +-
.../src/approx_percentile_cont.rs | 22 +-
.../src/approx_percentile_cont_with_weight.rs | 139 ++--
datafusion/functions-aggregate/src/array_agg.rs | 32 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 40 +-
datafusion/functions-aggregate/src/first_last.rs | 56 +-
datafusion/functions-aggregate/src/regr.rs | 238 ++++++-
datafusion/functions-aggregate/src/stddev.rs | 24 +-
datafusion/functions-aggregate/src/string_agg.rs | 30 +-
datafusion/functions-nested/benches/map.rs | 12 +-
datafusion/functions-window/src/cume_dist.rs | 15 +-
datafusion/functions-window/src/lead_lag.rs | 53 +-
datafusion/functions-window/src/nth_value.rs | 60 +-
datafusion/functions-window/src/ntile.rs | 18 +-
datafusion/functions-window/src/rank.rs | 80 +--
datafusion/functions-window/src/row_number.rs | 18 +-
datafusion/functions/benches/ascii.rs | 6 +
datafusion/functions/benches/character_length.rs | 6 +
datafusion/functions/benches/chr.rs | 3 +
datafusion/functions/benches/concat.rs | 3 +
datafusion/functions/benches/cot.rs | 4 +
datafusion/functions/benches/date_bin.rs | 9 +-
datafusion/functions/benches/date_trunc.rs | 9 +-
datafusion/functions/benches/encoding.rs | 7 +
datafusion/functions/benches/find_in_set.rs | 7 +
datafusion/functions/benches/gcd.rs | 5 +
datafusion/functions/benches/initcap.rs | 5 +
datafusion/functions/benches/isnan.rs | 4 +
datafusion/functions/benches/iszero.rs | 4 +
datafusion/functions/benches/lower.rs | 9 +
datafusion/functions/benches/ltrim.rs | 4 +
datafusion/functions/benches/make_date.rs | 16 +-
datafusion/functions/benches/nullif.rs | 3 +
datafusion/functions/benches/pad.rs | 3 +
datafusion/functions/benches/random.rs | 6 +-
datafusion/functions/benches/repeat.rs | 3 +
datafusion/functions/benches/reverse.rs | 7 +
datafusion/functions/benches/signum.rs | 4 +
datafusion/functions/benches/strpos.rs | 7 +
datafusion/functions/benches/substr.rs | 3 +
datafusion/functions/benches/substr_index.rs | 8 +-
datafusion/functions/benches/to_char.rs | 13 +-
datafusion/functions/benches/to_hex.rs | 5 +
datafusion/functions/benches/to_timestamp.rs | 10 +-
datafusion/functions/benches/trunc.rs | 5 +
datafusion/functions/benches/upper.rs | 4 +
datafusion/functions/benches/uuid.rs | 5 +
datafusion/functions/src/core/union_extract.rs | 6 +-
datafusion/functions/src/core/union_tag.rs | 3 +
datafusion/functions/src/core/version.rs | 3 +
datafusion/functions/src/datetime/date_bin.rs | 2 +
datafusion/functions/src/datetime/date_trunc.rs | 3 +
datafusion/functions/src/datetime/from_unixtime.rs | 3 +
datafusion/functions/src/datetime/make_date.rs | 2 +
datafusion/functions/src/datetime/to_char.rs | 7 +
datafusion/functions/src/datetime/to_date.rs | 5 +-
datafusion/functions/src/datetime/to_local_time.rs | 3 +
datafusion/functions/src/datetime/to_timestamp.rs | 3 +
datafusion/functions/src/math/log.rs | 11 +
datafusion/functions/src/math/power.rs | 6 +-
datafusion/functions/src/math/signum.rs | 3 +
datafusion/functions/src/regex/regexpcount.rs | 2 +
datafusion/functions/src/regex/regexpinstr.rs | 2 +
datafusion/functions/src/string/concat.rs | 2 +
datafusion/functions/src/string/concat_ws.rs | 3 +
datafusion/functions/src/string/contains.rs | 2 +
datafusion/functions/src/string/lower.rs | 2 +
datafusion/functions/src/string/upper.rs | 2 +
datafusion/functions/src/strings.rs | 39 +-
datafusion/functions/src/unicode/find_in_set.rs | 2 +
datafusion/functions/src/utils.rs | 31 +-
datafusion/optimizer/src/analyzer/mod.rs | 8 -
datafusion/optimizer/src/optimizer.rs | 18 +-
.../src/simplify_expressions/expr_simplifier.rs | 37 +-
.../src/simplify_expressions/simplify_exprs.rs | 1 +
.../optimizer/tests/optimizer_integration.rs | 2 +-
.../physical-expr-common/src/physical_expr.rs | 31 +-
datafusion/physical-expr/src/aggregate.rs | 5 +-
.../physical-expr/src/async_scalar_function.rs | 36 +-
.../physical-expr/src/equivalence/ordering.rs | 4 +
.../physical-expr/src/equivalence/projection.rs | 2 +
.../src/equivalence/properties/dependency.rs | 3 +
datafusion/physical-expr/src/expressions/binary.rs | 147 +++--
.../src/expressions/dynamic_filters.rs | 8 +
.../physical-expr/src/expressions/is_not_null.rs | 5 -
.../physical-expr/src/expressions/is_null.rs | 5 -
datafusion/physical-expr/src/planner.rs | 6 +
datafusion/physical-expr/src/scalar_function.rs | 53 +-
datafusion/physical-expr/src/window/mod.rs | 6 -
.../physical-optimizer/src/join_selection.rs | 52 +-
datafusion/physical-plan/src/async_func.rs | 9 +-
datafusion/physical-plan/src/coalesce/mod.rs | 12 +-
datafusion/physical-plan/src/common.rs | 2 +-
datafusion/physical-plan/src/coop.rs | 2 +-
datafusion/physical-plan/src/execution_plan.rs | 65 +-
datafusion/physical-plan/src/joins/cross_join.rs | 12 +
datafusion/physical-plan/src/joins/hash_join.rs | 20 +
datafusion/physical-plan/src/lib.rs | 1 -
datafusion/physical-plan/src/recursive_query.rs | 5 +-
datafusion/physical-plan/src/sorts/cursor.rs | 214 +-----
datafusion/physical-plan/src/sorts/sort.rs | 80 ++-
datafusion/physical-plan/src/spill/mod.rs | 19 +-
.../physical-plan/src/spill/spill_manager.rs | 28 -
datafusion/physical-plan/src/union.rs | 7 +-
datafusion/physical-plan/src/values.rs | 330 ----------
datafusion/physical-plan/src/work_table.rs | 8 -
.../proto-common/proto/datafusion_common.proto | 40 +-
datafusion/proto-common/src/from_proto/mod.rs | 31 +-
datafusion/proto-common/src/generated/pbjson.rs | 728 +++++++++++++++++----
datafusion/proto-common/src/generated/prost.rs | 66 +-
datafusion/proto-common/src/to_proto/mod.rs | 18 +-
.../proto/src/generated/datafusion_proto_common.rs | 66 +-
datafusion/proto/src/logical_plan/file_formats.rs | 14 -
datafusion/proto/src/physical_plan/from_proto.rs | 98 +--
datafusion/proto/src/physical_plan/mod.rs | 507 +++++++-------
datafusion/proto/tests/cases/mod.rs | 38 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 13 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 5 +-
datafusion/spark/Cargo.toml | 4 +
datafusion/spark/benches/char.rs | 3 +
datafusion/spark/src/function/array/mod.rs | 13 +-
.../src/function/array/spark_array.rs} | 195 +++---
.../spark/src/function/functions_nested_utils.rs | 53 ++
datafusion/spark/src/function/hash/crc32.rs | 134 ++++
datafusion/spark/src/function/hash/mod.rs | 12 +-
datafusion/spark/src/function/hash/sha1.rs | 145 ++++
datafusion/spark/src/function/mod.rs | 1 +
datafusion/spark/src/function/string/char.rs | 58 +-
datafusion/spark/src/function/string/ilike.rs | 173 +++++
datafusion/spark/src/function/string/like.rs | 178 +++++
datafusion/spark/src/function/string/mod.rs | 18 +-
datafusion/spark/src/function/url/mod.rs | 13 +-
datafusion/spark/src/function/url/parse_url.rs | 301 +++++++++
datafusion/spark/src/function/utils.rs | 17 +-
datafusion/sql/src/expr/order_by.rs | 16 +-
datafusion/sql/src/planner.rs | 62 ++
datafusion/sql/src/statement.rs | 4 +-
datafusion/sql/src/unparser/dialect.rs | 11 -
datafusion/sql/src/unparser/expr.rs | 6 +
datafusion/sql/tests/sql_integration.rs | 7 +-
datafusion/sqllogictest/test_files/aggregate.slt | 88 +++
datafusion/sqllogictest/test_files/copy.slt | 1 -
datafusion/sqllogictest/test_files/cte.slt | 55 ++
datafusion/sqllogictest/test_files/expr.slt | 23 +
.../sqllogictest/test_files/information_schema.slt | 8 +-
.../test_files/listing_table_statistics.slt | 2 +-
datafusion/sqllogictest/test_files/order.slt | 92 +++
datafusion/sqllogictest/test_files/parquet.slt | 113 ++++
.../sqllogictest/test_files/parquet_statistics.slt | 16 +-
.../sqllogictest/test_files/repartition_scan.slt | 8 +-
.../sqllogictest/test_files/spark/array/array.slt | 65 +-
.../test_files/spark/datetime/last_day.slt | 1 -
.../sqllogictest/test_files/spark/hash/crc32.slt | 53 +-
.../sqllogictest/test_files/spark/hash/sha.slt | 48 +-
.../sqllogictest/test_files/spark/hash/sha1.slt | 48 +-
.../test_files/spark/predicate/ilike.slt | 51 +-
.../test_files/spark/predicate/like.slt | 61 +-
.../test_files/spark/url/parse_url.slt | 80 ++-
datafusion/substrait/src/extensions.rs | 2 +
.../logical_plan/consumer/expr/scalar_function.rs | 195 ++++--
.../substrait/src/logical_plan/consumer/utils.rs | 2 +
.../logical_plan/producer/expr/scalar_function.rs | 21 +
.../tests/cases/builtin_expr_semantics_tests.rs | 124 ++++
.../substrait/tests/cases/consumer_integration.rs | 68 ++
datafusion/substrait/tests/cases/mod.rs | 1 +
.../tests/cases/roundtrip_logical_plan.rs | 35 +
.../test_plans/scalar_fn_logb_expr.substrait.json | 116 ++++
.../scalar_fn_to_between_expr.substrait.json | 143 ++++
..._to_built_in_binary_expr_and_not.substrait.json | 132 ++++
...r_fn_to_built_in_binary_expr_xor.substrait.json | 132 ++++
dev/update_config_docs.sh | 103 ++-
.../library-user-guide/functions/adding-udfs.md | 13 +-
docs/source/library-user-guide/query-optimizer.md | 2 +-
docs/source/library-user-guide/upgrading.md | 128 ++++
docs/source/user-guide/configs.md | 115 +++-
docs/source/user-guide/crate-configuration.md | 78 ++-
docs/source/user-guide/expressions.md | 42 +-
docs/source/user-guide/sql/aggregate_functions.md | 222 ++++++-
docs/source/user-guide/sql/window_functions.md | 109 ++-
252 files changed, 9200 insertions(+), 3373 deletions(-)
create mode 100644 datafusion-examples/examples/parquet_encrypted_with_kms.rs
create mode 100644 datafusion/execution/src/cache/lru_queue.rs
create mode 100644 datafusion/execution/src/parquet_encryption.rs
create mode 100644 datafusion/expr/src/ptr_eq.rs
create mode 100644 datafusion/expr/src/udf_eq.rs
delete mode 100644 datafusion/physical-plan/src/values.rs
copy datafusion/{functions-nested/src/make_array.rs =>
spark/src/function/array/spark_array.rs} (60%)
create mode 100644 datafusion/spark/src/function/functions_nested_utils.rs
create mode 100644 datafusion/spark/src/function/hash/crc32.rs
create mode 100644 datafusion/spark/src/function/hash/sha1.rs
create mode 100644 datafusion/spark/src/function/string/ilike.rs
create mode 100644 datafusion/spark/src/function/string/like.rs
create mode 100644 datafusion/spark/src/function/url/parse_url.rs
create mode 100644
datafusion/substrait/tests/cases/builtin_expr_semantics_tests.rs
create mode 100644
datafusion/substrait/tests/testdata/test_plans/scalar_fn_logb_expr.substrait.json
create mode 100644
datafusion/substrait/tests/testdata/test_plans/scalar_fn_to_between_expr.substrait.json
create mode 100644
datafusion/substrait/tests/testdata/test_plans/scalar_fn_to_built_in_binary_expr_and_not.substrait.json
create mode 100644
datafusion/substrait/tests/testdata/test_plans/scalar_fn_to_built_in_binary_expr_xor.substrait.json
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@datafusion.apache.org
For additional commands, e-mail: commits-h...@datafusion.apache.org
