This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/apache-avro-0.18.0
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit 9435d858be Merge branch 'main' into
dependabot/cargo/main/apache-avro-0.18.0
omit ad1d9f85e1 chore(deps): bump apache-avro from 0.17.0 to 0.18.0
add e58f86c60a refactor `character_length` impl by unifying null handling
logic (#16877)
add 1e3152ee1e Update dev env documentation to reflect pinned rust version
(#17107)
add 43718d5fff Differentiate 0-row and 1-row EmptyRelation in EXPLAIN
(#17145)
add dd5e0928d7 chore(deps): bump clap from 4.5.43 to 4.5.44 (#17148)
add eaf614d38e (Re)Support old syntax for `approx_percentile_cont` and
`approx_percentile_cont_with_weight` (#16999)
add 0b7186ebaf Pass the input schema to stats_projection for
ProjectionExpr (#17123)
add c2714dbf56 Fix extended tests failure on main by updating
`datafusion-testing` pin (#17176)
add 1961fe79c1 Derive `ScalarUDFImpl` equality, hash from `Eq`, `Hash`
traits (#17164)
add b6d4d3b624 Minor: display filter in HashJoin's tree explain (#17170)
add 3b8fabdefd Add support for `chr(0)` (#17131)
add 048374c91b add test for multi-column topk dynamic filter pushdown
(#17162)
add bb2d90ed15 Test: Add checks to sqllogictest temporary file creations
(#17017)
add 71408cf075 Deprecate unused `ScalarUDF::display_name` (#17168)
add 516f97138c update datafusion-testing submodule (#17187)
add 274cdba753 Normalize `NUL` to `\0` in sqllogictests (#17181)
add 5c370fa620 Simplify `GetFieldFunc`'s display_name`, `schema_name`
(#17167)
add 1206019e1a Rewrite Nested Loop Join executor for 5× speed and 1%
memory usage (#16996)
add 810339666a [branch-49] Update version to 49.0.1 and add changelog
(#17175) (#17191)
add b4c7ef78fb Minor: Fix compiler warning when compiling `datafusion-cli`
(#17205)
add e3d3257fc0 feat: add `datafusion-physical-adapter`, implement
predicate adaptation missing fields of structs (#16589)
add 7e38714fa9 Refactor: Do not silently ignore errors in
`stats_projection` (#17154)
add 0a024a2f0e Miscellaneous cleanups (#17189)
add b250848a17 fix(ci): update `datafusion-physical-expr-adapter` version
to 49.0.1 (#17209)
add 75eb80c1aa [Parquet Metadata Cache] Document the ListingTable cache
(#17133)
add a2f84f3a8d Docs: Consolidate feature proposal content into roadmap
(#17156)
add f3941b207e feat: implement QUALIFY clause (#16933)
add 8f15991f33 nlj upgrade guide (#17202)
add c17116fd51 Remove redundant `plan` from extension's check_invariants
(#17199)
add 420a862faa Fix: Show backtrace for ArrowError (#17204)
add a5c242c214 minor: clean up distinct window code (#17215)
add b84ddfde06 chore: Add drop table test on create_drop.rs (#17219)
add 2ae30af57a Support serializing `generate_series` in `datafusion-proto`
(#17200)
add afc90f727d fix: respect inexact flags in row group metadata (#16412)
add 9eb182e266 Fix broken links in user docs (#17228)
add d750ade6ed chore(deps): bump async-trait from 0.1.88 to 0.1.89 (#17203)
add 92d516cc9b Bump MSRV to 1.86.0 (#17230)
add 3eceedcde3 better err msg when creating file failed (#17217)
add 1d4d74beda Fix dynamic filter pushdown in HashJoinExec (#17201)
add 06547beffb Fix Analyze Exec protobuf roundtrip (#17234)
add 944d8c0f50 Support `distinct` and `ignore_nulls` in window expressions
(#17235)
add 638dc46b00 fix: deserialization error for `FilterExec` (predicates
with inlist) (#17224)
add f27b103dbe FFI_RecordBatchStream was causing a memory leak (#17190)
add afc0537d4b Format `Date32` to string given timestamp specifiers
(#15361)
add 4c7c045bda chore(deps): bump serde_json from 1.0.142 to 1.0.143
(#17240)
add 93412c7b94 chore(deps): bump syn from 2.0.105 to 2.0.106 (#17243)
add 4bc069693d Push dynamic pushdown through CooperativeExec and
ProjectionExec (#17238)
add 8c58f532a1 Configure cli test that requires backtrace to be optional
(#17236)
add 8f30ac279c chore(deps): Update sqlparser to 0.58 (#16456)
add 724be0d2ff chore(deps): bump rustyline from 17.0.0 to 17.0.1 (#17252)
add a26039b01c chore(deps): bump thiserror from 2.0.14 to 2.0.16 (#17257)
add 64bc58d82a Fix HashJoinExec sideways information passing for
partitioned queries (#17197)
add 541fdfbc0d fix: Windows paths crashing core tests (#17231)
add f87763dc9f Fix HashJoinExec test snapshot under
force_hash_collisions=true (#17265)
add d64b7c3b4b Deprecate confusingly named `UserDefinedFunctionPlanner`
(#17247)
add 5592b2e006 Fix: ListingTableFactory paths with dots (#17233)
add 3b7eb267dd fix: sort should always output batches with `batch_size`
rows (#17244)
add 2f61429eef chore(deps): bump tempfile from 3.20.0 to 3.21.0 (#17268)
add 02a7472b79 fix: align `array_has` null buffer for scalar (#17272)
add a0248a98b8 Fix PartialOrd for ScalarUDF (#17182)
add 1529dbd0ec chore(deps): bump url from 2.5.4 to 2.5.6 (#17283)
add cea3ada934 fix determinism of HashJoinExec bounds filter creation
(#17280)
add 9e062dfb41 Improve documentation for Signature, Volatility, and
TypeSignature (#17264)
add f363e38266 Consolidate Parquet Metadata handling into its own module
and struct `DFParquetMetadata` (#17127)
add 3f96c3f3fb [main] Forward port `49.0.2` version and changelog (#17277)
(#17287)
add 18c5a6c82f Only update TopK dynamic filters if the new ones are more
selective (#16433)
add 2f51d5fdc1 Document schema merging. (#17249)
add 73ad4926d9 feat: Make parquet_encryption a non-default feature (#17137)
add bd249ba5b9 Add documentation for UNION schema handling. (#17248)
add cf3b456727 Replace π-related bound constants with next_up/next_down
(#16823)
add be3842b55d chore: add example for how to use TrackConsumersPool
(#17213)
add ee95caea8d feat: allow to `spawn`/`spawn_blocking` on a provided
runtime in `RecordBatchReceiverStreamBuilder` (#17239)
add b63a9607c3 minor: Remove extra line break in explain physical plan
(#17303)
add 85f879d41a Support from-first SQL syntax (#17295)
add 241e47d1c9 Support `avg(distinct)` for `float64` type (#17255)
add 72db63c38d chore: check the error message log (#17308)
add 8c65a41fe4 Expand sql_planner benchmark for benchmarking physical and
logical optimization. (#17276)
add fa265154bd Encapsulate early pruning in its own stream (#17293)
add 4925e6c67e fix: dataframe function count_all with alias (#17282)
add 2c9f42bae7 feat: Support SortMergeJoin proto serde (#17296)
add fd7df66724 feat(spark): implement Spark `bitmap` function
`bitmap_count` (#17179)
add 6368daf0b6 Implement `partition_statistics` API for `RepartitionExec`
(#17061)
add 76559df0cf fix: correct readme field in Cargo.toml to use string
format (#17310)
add ad100fbd66 chore: replace Schema with SchemaRef in
PruningExpressionBuilder (#17216)
add 87b5023776 chore(deps): bump regex-syntax from 0.8.5 to 0.8.6 (#17320)
add 6eb90fdf3c chore(deps): bump indexmap from 2.10.0 to 2.11.0 (#17316)
add 68319c29af fix(doc): update the link of deprecation guidelines
(#17328) (#17329)
add 7cc54d1f2f refactor: Split `SortMergeJoin` into multiple modules
(#17304)
add 1756692218 feat: Track peak value in tracked consumer (#17327)
add c62d67186f fix: lazy case else expression evaluation (#17311)
add 913d7a810a MINOR: add missing examples to example list (#17333)
add 033fc9b0a1 chore: split hash join to smaller modules (#17300)
add 097e893f92 Add `cfg(feature = "avro")` attribute to Avro example in
SQL API docs (#17142)
add 7c85a68b3c chore(deps): bump url from 2.5.6 to 2.5.7 (#17324)
add b6a8a0ec38 chore(deps): bump regex from 1.11.1 to 1.11.2 (#17325)
add 1c86ec7f52 chore: fix typos (#17135)
add df588449fd add a ci job for typo checking (#17339)
add 931aa8ed1f chore(deps): bump clap from 4.5.45 to 4.5.46 (#17338)
add dde595f714 chore(deps): bump korandoru/hawkeye from 6.1.1 to 6.2.0
(#17321)
add 5d66848fb3 chore(deps): bump apache-avro from 0.17.0 to 0.18.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9435d858be)
\
N -- N -- N refs/heads/dependabot/cargo/main/apache-avro-0.18.0
(5d66848fb3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/dev.yml | 2 +-
.github/workflows/rust.yml | 12 +-
Cargo.lock | 501 ++---
Cargo.toml | 18 +-
README.md | 4 +-
benchmarks/README.md | 78 +-
benchmarks/bench.sh | 18 +
benchmarks/src/bin/dfbench.rs | 4 +-
benchmarks/src/bin/external_aggr.rs | 2 +-
benchmarks/src/lib.rs | 1 +
benchmarks/src/nlj.rs | 264 +++
datafusion-cli/Cargo.toml | 3 +-
datafusion-cli/tests/cli_integration.rs | 34 +-
.../cli_quick_test@can_see_indent_format.snap | 3 +-
.../snapshots/cli_top_memory_consum...@top2.snap | 4 +-
.../cli_top_memory_consumers@top3_default.snap | 6 +-
datafusion-examples/Cargo.toml | 3 +-
datafusion-examples/README.md | 4 +
datafusion-examples/examples/advanced_udf.rs | 2 +-
datafusion-examples/examples/async_udf.rs | 2 +-
datafusion-examples/examples/custom_file_casts.rs | 8 +-
.../examples/default_column_values.rs | 6 +-
datafusion-examples/examples/expr_api.rs | 4 +-
datafusion-examples/examples/function_factory.rs | 36 +-
datafusion-examples/examples/json_shredding.rs | 16 +-
.../examples/memory_pool_execution_plan.rs | 303 +++
.../examples/memory_pool_tracking.rs | 127 ++
datafusion-examples/examples/optimizer_rule.rs | 2 +-
datafusion-examples/examples/pruning.rs | 4 +-
datafusion-testing | 2 +-
datafusion/catalog/src/information_schema.rs | 2 +-
datafusion/catalog/src/memory/table.rs | 2 +-
datafusion/catalog/src/streaming.rs | 2 +-
datafusion/common/Cargo.toml | 2 +-
datafusion/common/src/config.rs | 4 +
datafusion/common/src/dfschema.rs | 15 +
datafusion/common/src/encryption.rs | 2 +
datafusion/common/src/error.rs | 2 +-
datafusion/common/src/scalar/consts.rs | 36 +-
datafusion/common/src/scalar/mod.rs | 2 -
datafusion/common/src/test_util.rs | 13 +-
datafusion/core/Cargo.toml | 4 +-
datafusion/core/benches/spm.rs | 4 +-
datafusion/core/benches/sql_planner.rs | 225 ++-
datafusion/core/src/bin/print_functions_docs.rs | 2 +-
.../core/src/datasource/file_format/arrow.rs | 2 +-
.../core/src/datasource/file_format/parquet.rs | 253 +--
datafusion/core/src/datasource/listing/table.rs | 64 +-
.../core/src/datasource/listing_table_factory.rs | 52 +-
.../core/src/datasource/physical_plan/parquet.rs | 8 +-
datafusion/core/src/execution/context/mod.rs | 2 +-
.../core/src/execution/session_state_defaults.rs | 9 +-
datafusion/core/src/lib.rs | 14 +-
datafusion/core/src/physical_planner.rs | 37 +-
datafusion/core/tests/catalog/memory.rs | 2 +-
.../core/tests/dataframe/dataframe_functions.rs | 27 +-
datafusion/core/tests/dataframe/mod.rs | 48 +-
datafusion/core/tests/execution/coop.rs | 17 +-
datafusion/core/tests/execution/logical_plan.rs | 2 +-
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 2 +-
.../core/tests/fuzz_cases/equivalence/utils.rs | 2 +-
datafusion/core/tests/fuzz_cases/pruning.rs | 2 +-
.../core/tests/fuzz_cases/sort_query_fuzz.rs | 2 +-
.../core/tests/fuzz_cases/topk_filter_pushdown.rs | 10 +-
.../memory_limit_validation/sort_mem_validation.rs | 2 +-
datafusion/core/tests/memory_limit/mod.rs | 2 +-
datafusion/core/tests/optimizer/mod.rs | 2 +-
datafusion/core/tests/parquet/custom_reader.rs | 26 +-
datafusion/core/tests/parquet/encryption.rs | 12 +-
datafusion/core/tests/parquet/schema_adapter.rs | 7 +-
.../physical_optimizer/enforce_distribution.rs | 6 +-
.../physical_optimizer/filter_pushdown/mod.rs | 378 +++-
.../physical_optimizer/partition_statistics.rs | 102 +
.../physical_optimizer/projection_pushdown.rs | 2 +-
.../replace_with_order_preserving_variants.rs | 2 +-
.../tests/physical_optimizer/window_optimize.rs | 10 +-
datafusion/core/tests/sql/create_drop.rs | 23 +
datafusion/core/tests/tracing/mod.rs | 8 +-
.../core/tests/user_defined/user_defined_plan.rs | 2 +-
.../user_defined/user_defined_scalar_functions.rs | 42 +-
datafusion/datasource-parquet/Cargo.toml | 1 +
datafusion/datasource-parquet/src/file_format.rs | 396 +---
datafusion/datasource-parquet/src/metadata.rs | 566 ++++++
datafusion/datasource-parquet/src/mod.rs | 1 +
datafusion/datasource-parquet/src/opener.rs | 115 +-
datafusion/datasource-parquet/src/reader.rs | 37 +-
datafusion/datasource-parquet/src/row_filter.rs | 2 +-
datafusion/datasource-parquet/src/source.rs | 6 +-
datafusion/datasource/Cargo.toml | 1 +
datafusion/datasource/src/file_scan_config.rs | 4 +-
datafusion/execution/src/cache/cache_manager.rs | 61 +-
datafusion/execution/src/cache/cache_unit.rs | 45 +-
datafusion/execution/src/memory_pool/pool.rs | 103 +-
datafusion/execution/src/runtime_env.rs | 3 +-
datafusion/expr-common/src/signature.rs | 116 +-
datafusion/expr/src/async_udf.rs | 6 +-
datafusion/expr/src/expr.rs | 113 +-
datafusion/expr/src/expr_fn.rs | 6 +-
datafusion/expr/src/logical_plan/builder.rs | 2 +-
datafusion/expr/src/logical_plan/extension.rs | 12 +-
datafusion/expr/src/logical_plan/invariants.rs | 4 +-
datafusion/expr/src/logical_plan/plan.rs | 44 +-
datafusion/expr/src/udaf.rs | 8 +
datafusion/expr/src/udf.rs | 193 +-
datafusion/expr/src/udf_eq.rs | 26 +-
datafusion/expr/src/udwf.rs | 8 +
datafusion/expr/src/utils.rs | 177 +-
datafusion/ffi/src/catalog_provider.rs | 2 +-
datafusion/ffi/src/record_batch_stream.rs | 16 +
datafusion/ffi/src/schema_provider.rs | 2 +-
datafusion/ffi/src/tests/mod.rs | 2 +-
datafusion/ffi/src/udaf/mod.rs | 2 +-
datafusion/ffi/src/udf/mod.rs | 6 +-
datafusion/ffi/src/udwf/mod.rs | 2 +-
datafusion/ffi/src/util.rs | 4 +-
.../functions-aggregate-common/src/aggregate.rs | 2 +
.../src/aggregate/avg_distinct.rs} | 4 +-
.../src/aggregate/avg_distinct/numeric.rs | 78 +
.../src/aggregate/sum_distinct.rs} | 6 +-
.../src/aggregate/sum_distinct/numeric.rs | 123 ++
.../src/approx_percentile_cont.rs | 19 +-
.../src/approx_percentile_cont_with_weight.rs | 10 +
datafusion/functions-aggregate/src/average.rs | 131 +-
datafusion/functions-aggregate/src/min_max.rs | 4 +-
datafusion/functions-aggregate/src/sum.rs | 89 +-
datafusion/functions-nested/src/array_has.rs | 6 +-
datafusion/functions-nested/src/cardinality.rs | 2 +-
datafusion/functions-nested/src/concat.rs | 6 +-
datafusion/functions-nested/src/dimension.rs | 4 +-
datafusion/functions-nested/src/distance.rs | 2 +-
datafusion/functions-nested/src/empty.rs | 2 +-
datafusion/functions-nested/src/except.rs | 2 +-
datafusion/functions-nested/src/extract.rs | 10 +-
datafusion/functions-nested/src/flatten.rs | 2 +-
datafusion/functions-nested/src/length.rs | 2 +-
datafusion/functions-nested/src/make_array.rs | 2 +-
datafusion/functions-nested/src/map.rs | 2 +-
datafusion/functions-nested/src/map_entries.rs | 2 +-
datafusion/functions-nested/src/map_extract.rs | 2 +-
datafusion/functions-nested/src/map_keys.rs | 2 +-
datafusion/functions-nested/src/map_values.rs | 2 +-
datafusion/functions-nested/src/min_max.rs | 4 +-
datafusion/functions-nested/src/position.rs | 4 +-
datafusion/functions-nested/src/range.rs | 4 +-
datafusion/functions-nested/src/remove.rs | 6 +-
datafusion/functions-nested/src/repeat.rs | 2 +-
datafusion/functions-nested/src/replace.rs | 6 +-
datafusion/functions-nested/src/resize.rs | 2 +-
datafusion/functions-nested/src/reverse.rs | 2 +-
datafusion/functions-nested/src/set_ops.rs | 6 +-
datafusion/functions-nested/src/sort.rs | 2 +-
datafusion/functions-nested/src/string.rs | 4 +-
datafusion/functions-table/src/generate_series.rs | 261 ++-
datafusion/functions/benches/to_char.rs | 173 +-
datafusion/functions/src/core/arrow_cast.rs | 2 +-
datafusion/functions/src/core/arrowtypeof.rs | 2 +-
datafusion/functions/src/core/coalesce.rs | 2 +-
datafusion/functions/src/core/getfield.rs | 10 +-
datafusion/functions/src/core/greatest.rs | 2 +-
datafusion/functions/src/core/least.rs | 2 +-
datafusion/functions/src/core/named_struct.rs | 2 +-
datafusion/functions/src/core/nullif.rs | 2 +-
datafusion/functions/src/core/nvl.rs | 2 +-
datafusion/functions/src/core/nvl2.rs | 2 +-
datafusion/functions/src/core/overlay.rs | 2 +-
datafusion/functions/src/core/struct.rs | 2 +-
datafusion/functions/src/core/union_extract.rs | 2 +-
datafusion/functions/src/core/union_tag.rs | 4 +-
datafusion/functions/src/core/version.rs | 2 +-
datafusion/functions/src/crypto/digest.rs | 2 +-
datafusion/functions/src/crypto/md5.rs | 2 +-
datafusion/functions/src/crypto/sha224.rs | 2 +-
datafusion/functions/src/crypto/sha256.rs | 2 +-
datafusion/functions/src/crypto/sha384.rs | 2 +-
datafusion/functions/src/crypto/sha512.rs | 2 +-
datafusion/functions/src/datetime/current_date.rs | 2 +-
datafusion/functions/src/datetime/current_time.rs | 2 +-
datafusion/functions/src/datetime/date_bin.rs | 2 +-
datafusion/functions/src/datetime/date_part.rs | 2 +-
datafusion/functions/src/datetime/date_trunc.rs | 4 +-
datafusion/functions/src/datetime/from_unixtime.rs | 2 +-
datafusion/functions/src/datetime/make_date.rs | 4 +-
datafusion/functions/src/datetime/mod.rs | 1 +
datafusion/functions/src/datetime/now.rs | 2 +-
.../src/datetime/planner.rs} | 24 +-
datafusion/functions/src/datetime/to_char.rs | 151 +-
datafusion/functions/src/datetime/to_date.rs | 4 +-
datafusion/functions/src/datetime/to_local_time.rs | 2 +-
datafusion/functions/src/datetime/to_timestamp.rs | 22 +-
datafusion/functions/src/datetime/to_unixtime.rs | 2 +-
datafusion/functions/src/encoding/inner.rs | 4 +-
datafusion/functions/src/macros.rs | 4 +-
datafusion/functions/src/math/abs.rs | 2 +-
datafusion/functions/src/math/cot.rs | 2 +-
datafusion/functions/src/math/factorial.rs | 2 +-
datafusion/functions/src/math/gcd.rs | 2 +-
datafusion/functions/src/math/iszero.rs | 2 +-
datafusion/functions/src/math/lcm.rs | 2 +-
datafusion/functions/src/math/log.rs | 2 +-
datafusion/functions/src/math/nans.rs | 2 +-
datafusion/functions/src/math/nanvl.rs | 2 +-
datafusion/functions/src/math/pi.rs | 2 +-
datafusion/functions/src/math/power.rs | 2 +-
datafusion/functions/src/math/random.rs | 2 +-
datafusion/functions/src/math/round.rs | 2 +-
datafusion/functions/src/math/signum.rs | 2 +-
datafusion/functions/src/math/trunc.rs | 2 +-
datafusion/functions/src/planner.rs | 5 +
datafusion/functions/src/regex/regexpcount.rs | 2 +-
datafusion/functions/src/regex/regexpinstr.rs | 2 +-
datafusion/functions/src/regex/regexplike.rs | 2 +-
datafusion/functions/src/regex/regexpmatch.rs | 2 +-
datafusion/functions/src/regex/regexpreplace.rs | 2 +-
datafusion/functions/src/string/ascii.rs | 4 +-
datafusion/functions/src/string/bit_length.rs | 2 +-
datafusion/functions/src/string/btrim.rs | 2 +-
datafusion/functions/src/string/chr.rs | 53 +-
datafusion/functions/src/string/concat.rs | 2 +-
datafusion/functions/src/string/concat_ws.rs | 2 +-
datafusion/functions/src/string/contains.rs | 2 +-
datafusion/functions/src/string/ends_with.rs | 2 +-
datafusion/functions/src/string/levenshtein.rs | 2 +-
datafusion/functions/src/string/lower.rs | 2 +-
datafusion/functions/src/string/ltrim.rs | 2 +-
datafusion/functions/src/string/octet_length.rs | 2 +-
datafusion/functions/src/string/repeat.rs | 2 +-
datafusion/functions/src/string/replace.rs | 2 +-
datafusion/functions/src/string/rtrim.rs | 2 +-
datafusion/functions/src/string/split_part.rs | 2 +-
datafusion/functions/src/string/starts_with.rs | 2 +-
datafusion/functions/src/string/to_hex.rs | 2 +-
datafusion/functions/src/string/upper.rs | 2 +-
datafusion/functions/src/string/uuid.rs | 2 +-
.../functions/src/unicode/character_length.rs | 55 +-
datafusion/functions/src/unicode/find_in_set.rs | 2 +-
datafusion/functions/src/unicode/initcap.rs | 2 +-
datafusion/functions/src/unicode/left.rs | 2 +-
datafusion/functions/src/unicode/lpad.rs | 2 +-
datafusion/functions/src/unicode/mod.rs | 1 +
datafusion/functions/src/{ => unicode}/planner.rs | 34 +-
datafusion/functions/src/unicode/reverse.rs | 2 +-
datafusion/functions/src/unicode/right.rs | 2 +-
datafusion/functions/src/unicode/rpad.rs | 2 +-
datafusion/functions/src/unicode/strpos.rs | 2 +-
datafusion/functions/src/unicode/substr.rs | 2 +-
datafusion/functions/src/unicode/substrindex.rs | 2 +-
datafusion/functions/src/unicode/translate.rs | 2 +-
datafusion/macros/Cargo.toml | 2 +-
datafusion/optimizer/Cargo.toml | 2 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 149 +-
.../optimizer/src/common_subexpr_eliminate.rs | 4 +-
datafusion/optimizer/src/eliminate_filter.rs | 8 +-
.../optimizer/src/eliminate_group_by_constant.rs | 2 +-
datafusion/optimizer/src/eliminate_join.rs | 2 +-
datafusion/optimizer/src/eliminate_limit.rs | 6 +-
.../optimizer/src/optimize_projections/mod.rs | 2 +-
.../optimizer/src/propagate_empty_relation.rs | 10 +-
datafusion/optimizer/src/push_down_filter.rs | 10 +-
.../src/simplify_expressions/expr_simplifier.rs | 20 +-
datafusion/optimizer/src/utils.rs | 10 +-
.../optimizer/tests/optimizer_integration.rs | 4 +-
.../{pruning => physical-expr-adapter}/Cargo.toml | 24 +-
datafusion/physical-expr-adapter/README.md | 8 +
.../mod.rs => physical-expr-adapter/src/lib.rs} | 12 +-
.../src/schema_rewriter.rs | 299 ++-
datafusion/physical-expr-common/src/datum.rs | 21 +-
.../physical-expr-common/src/physical_expr.rs | 6 +-
datafusion/physical-expr/Cargo.toml | 1 +
.../src/equivalence/properties/mod.rs | 2 +-
datafusion/physical-expr/src/expressions/case.rs | 16 +-
.../src/expressions/dynamic_filters.rs | 61 +-
datafusion/physical-expr/src/lib.rs | 3 +-
datafusion/physical-expr/src/scalar_function.rs | 23 +
datafusion/physical-expr/src/utils/mod.rs | 2 +-
.../physical-optimizer/src/filter_pushdown.rs | 2 +-
datafusion/physical-plan/Cargo.toml | 2 +-
datafusion/physical-plan/benches/spill_io.rs | 2 +-
datafusion/physical-plan/src/analyze.rs | 3 +-
datafusion/physical-plan/src/coop.rs | 24 +
datafusion/physical-plan/src/execution_plan.rs | 9 +
datafusion/physical-plan/src/filter_pushdown.rs | 2 +-
.../src/joins/{hash_join.rs => hash_join/exec.rs} | 1052 +++-------
.../src/joins/hash_join}/mod.rs | 11 +-
.../src/joins/hash_join/shared_bounds.rs | 296 +++
.../physical-plan/src/joins/hash_join/stream.rs | 628 ++++++
datafusion/physical-plan/src/joins/join_filter.rs | 10 +-
.../physical-plan/src/joins/nested_loop_join.rs | 2018 ++++++++++++-------
.../exec.rs} | 2046 +-------------------
.../src/joins/sort_merge_join/metrics.rs | 96 +
.../src/joins/sort_merge_join}/mod.rs | 11 +-
.../src/joins/sort_merge_join/stream.rs | 2007 +++++++++++++++++++
.../physical-plan/src/joins/symmetric_hash_join.rs | 7 +-
datafusion/physical-plan/src/joins/utils.rs | 140 +-
datafusion/physical-plan/src/memory.rs | 13 +
datafusion/physical-plan/src/metrics/baseline.rs | 6 +-
datafusion/physical-plan/src/projection.rs | 137 +-
datafusion/physical-plan/src/repartition/mod.rs | 43 +-
datafusion/physical-plan/src/sorts/sort.rs | 338 +++-
datafusion/physical-plan/src/sorts/stream.rs | 2 +-
datafusion/physical-plan/src/spill/mod.rs | 2 +-
datafusion/physical-plan/src/stream.rs | 102 +-
datafusion/physical-plan/src/topk/mod.rs | 182 +-
datafusion/physical-plan/src/windows/mod.rs | 37 +-
datafusion/proto/proto/datafusion.proto | 60 +
datafusion/proto/src/generated/pbjson.rs | 1134 +++++++++++
datafusion/proto/src/generated/prost.rs | 126 +-
datafusion/proto/src/logical_plan/mod.rs | 2 +-
datafusion/proto/src/physical_plan/from_proto.rs | 4 +-
datafusion/proto/src/physical_plan/mod.rs | 455 ++++-
datafusion/proto/src/physical_plan/to_proto.rs | 88 +-
datafusion/proto/tests/cases/mod.rs | 6 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 314 ++-
datafusion/pruning/src/file_pruner.rs | 2 +-
datafusion/pruning/src/pruning_predicate.rs | 27 +-
datafusion/spark/Cargo.toml | 2 +-
datafusion/spark/src/function/array/spark_array.rs | 2 +-
.../spark/src/function/bitmap/bitmap_count.rs | 178 ++
.../spark/src/function/{url => bitmap}/mod.rs | 14 +-
datafusion/spark/src/function/bitwise/bit_count.rs | 2 +-
datafusion/spark/src/function/bitwise/bit_get.rs | 12 +-
datafusion/spark/src/function/datetime/date_add.rs | 2 +-
datafusion/spark/src/function/datetime/date_sub.rs | 2 +-
datafusion/spark/src/function/datetime/last_day.rs | 2 +-
datafusion/spark/src/function/datetime/next_day.rs | 2 +-
datafusion/spark/src/function/hash/crc32.rs | 2 +-
datafusion/spark/src/function/hash/sha1.rs | 2 +-
datafusion/spark/src/function/hash/sha2.rs | 2 +-
datafusion/spark/src/function/math/expm1.rs | 2 +-
datafusion/spark/src/function/math/factorial.rs | 2 +-
datafusion/spark/src/function/math/hex.rs | 2 +-
datafusion/spark/src/function/math/modulus.rs | 4 +-
datafusion/spark/src/function/math/rint.rs | 2 +-
datafusion/spark/src/function/mod.rs | 1 +
datafusion/spark/src/function/string/ascii.rs | 2 +-
datafusion/spark/src/function/string/char.rs | 2 +-
datafusion/spark/src/function/string/ilike.rs | 2 +-
datafusion/spark/src/function/string/like.rs | 2 +-
datafusion/spark/src/function/string/luhn_check.rs | 2 +-
datafusion/spark/src/function/url/parse_url.rs | 2 +-
datafusion/spark/src/lib.rs | 2 +
datafusion/sql/src/expr/function.rs | 17 +-
datafusion/sql/src/expr/mod.rs | 29 +-
datafusion/sql/src/expr/subquery.rs | 2 +-
datafusion/sql/src/expr/substring.rs | 10 +-
datafusion/sql/src/planner.rs | 9 +
datafusion/sql/src/query.rs | 61 +-
datafusion/sql/src/select.rs | 81 +-
datafusion/sql/src/statement.rs | 246 ++-
datafusion/sql/src/unparser/ast.rs | 18 +-
datafusion/sql/src/unparser/expr.rs | 12 +-
datafusion/sql/src/unparser/plan.rs | 3 +-
datafusion/sql/src/utils.rs | 2 +-
datafusion/sql/tests/cases/params.rs | 12 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 4 +-
datafusion/sql/tests/sql_integration.rs | 348 ++--
datafusion/sqllogictest/Cargo.toml | 7 +-
datafusion/sqllogictest/bin/sqllogictests.rs | 79 +
datafusion/sqllogictest/data/1.parquet | Bin 1258 -> 0 bytes
datafusion/sqllogictest/data/2.parquet | Bin 1273 -> 0 bytes
datafusion/sqllogictest/src/engines/conversion.rs | 3 +-
datafusion/sqllogictest/test_files/aggregate.slt | 121 +-
datafusion/sqllogictest/test_files/array.slt | 19 +-
datafusion/sqllogictest/test_files/case.slt | 10 +
.../sqllogictest/test_files/count_star_rule.slt | 2 +-
datafusion/sqllogictest/test_files/cte.slt | 22 +-
datafusion/sqllogictest/test_files/dictionary.slt | 2 +-
datafusion/sqllogictest/test_files/errors.slt | 2 +-
datafusion/sqllogictest/test_files/explain.slt | 8 +-
.../sqllogictest/test_files/explain_tree.slt | 49 +-
datafusion/sqllogictest/test_files/expr.slt | 6 +-
.../{spark/string/ascii.slt => from-first.slt} | 33 +-
.../sqllogictest/test_files/insert_to_external.slt | 35 +
datafusion/sqllogictest/test_files/issue_17138.slt | 4 +-
datafusion/sqllogictest/test_files/join.slt.part | 2 +-
datafusion/sqllogictest/test_files/joins.slt | 54 +-
datafusion/sqllogictest/test_files/limit.slt | 10 +-
.../test_files/listing_table_statistics.slt | 8 +-
datafusion/sqllogictest/test_files/order.slt | 6 +-
.../test_files/parquet_filter_pushdown.slt | 6 +-
.../sqllogictest/test_files/parquet_statistics.slt | 9 +-
datafusion/sqllogictest/test_files/predicates.slt | 6 +-
.../sqllogictest/test_files/push_down_filter.slt | 117 +-
datafusion/sqllogictest/test_files/qualify.slt | 301 +++
.../test_files/simplify_predicates.slt | 4 +-
.../{string/ascii.slt => bitmap/bitmap_count.slt} | 39 +-
.../sqllogictest/test_files/spark/string/char.slt | Bin 969 -> 970 bytes
datafusion/sqllogictest/test_files/subquery.slt | 10 +-
datafusion/sqllogictest/test_files/timestamps.slt | 12 +
datafusion/sqllogictest/test_files/topk.slt | 2 +-
.../sqllogictest/test_files/type_coercion.slt | 32 +-
datafusion/sqllogictest/test_files/union.slt | 14 +-
datafusion/sqllogictest/test_files/unnest.slt | 6 +-
datafusion/sqllogictest/test_files/window.slt | 16 +-
datafusion/substrait/src/variation_const.rs | 4 +-
.../tests/cases/roundtrip_logical_plan.rs | 8 +-
dev/changelog/49.0.1.md | 48 +
dev/changelog/{46.0.1.md => 49.0.2.md} | 19 +-
dev/update_config_docs.sh | 2 +-
.../contributor-guide/development_environment.md | 10 +-
docs/source/contributor-guide/index.md | 71 +-
docs/source/contributor-guide/roadmap.md | 72 +-
.../library-user-guide/extending-operators.md | 2 +-
.../library-user-guide/functions/adding-udfs.md | 8 +-
docs/source/library-user-guide/upgrading.md | 76 +-
.../source/library-user-guide/using-the-sql-api.md | 3 +
.../library-user-guide/working-with-exprs.md | 2 +-
docs/source/user-guide/configs.md | 4 +-
docs/source/user-guide/explain-usage.md | 2 +-
docs/source/user-guide/sql/aggregate_functions.md | 29 +
docs/source/user-guide/sql/scalar_functions.md | 20 +-
docs/source/user-guide/sql/select.md | 23 +
typos.toml | 46 +
412 files changed, 14706 insertions(+), 6650 deletions(-)
create mode 100644 benchmarks/src/nlj.rs
create mode 100644 datafusion-examples/examples/memory_pool_execution_plan.rs
create mode 100644 datafusion-examples/examples/memory_pool_tracking.rs
create mode 100644 datafusion/datasource-parquet/src/metadata.rs
copy
datafusion/{sqllogictest/src/engines/datafusion_substrait_roundtrip_engine/mod.rs
=> functions-aggregate-common/src/aggregate/avg_distinct.rs} (92%)
create mode 100644
datafusion/functions-aggregate-common/src/aggregate/avg_distinct/numeric.rs
copy datafusion/{physical-expr/src/statistics/mod.rs =>
functions-aggregate-common/src/aggregate/sum_distinct.rs} (88%)
create mode 100644
datafusion/functions-aggregate-common/src/aggregate/sum_distinct/numeric.rs
copy datafusion/{common/src/utils/string_utils.rs =>
functions/src/datetime/planner.rs} (58%)
copy datafusion/functions/src/{ => unicode}/planner.rs (57%)
copy datafusion/{pruning => physical-expr-adapter}/Cargo.toml (51%)
create mode 100644 datafusion/physical-expr-adapter/README.md
copy datafusion/{catalog-listing/src/mod.rs =>
physical-expr-adapter/src/lib.rs} (82%)
rename datafusion/{physical-expr =>
physical-expr-adapter}/src/schema_rewriter.rs (67%)
rename datafusion/physical-plan/src/joins/{hash_join.rs => hash_join/exec.rs}
(83%)
copy datafusion/{core/tests/fuzz_cases/equivalence =>
physical-plan/src/joins/hash_join}/mod.rs (86%)
create mode 100644
datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
create mode 100644 datafusion/physical-plan/src/joins/hash_join/stream.rs
rename datafusion/physical-plan/src/joins/{sort_merge_join.rs =>
sort_merge_join/exec.rs} (57%)
create mode 100644
datafusion/physical-plan/src/joins/sort_merge_join/metrics.rs
copy datafusion/{core/tests/fuzz_cases/equivalence =>
physical-plan/src/joins/sort_merge_join}/mod.rs (87%)
create mode 100644 datafusion/physical-plan/src/joins/sort_merge_join/stream.rs
create mode 100644 datafusion/spark/src/function/bitmap/bitmap_count.rs
copy datafusion/spark/src/function/{url => bitmap}/mod.rs (81%)
delete mode 100644 datafusion/sqllogictest/data/1.parquet
delete mode 100644 datafusion/sqllogictest/data/2.parquet
copy datafusion/sqllogictest/test_files/{spark/string/ascii.slt =>
from-first.slt} (80%)
create mode 100644 datafusion/sqllogictest/test_files/qualify.slt
copy datafusion/sqllogictest/test_files/spark/{string/ascii.slt =>
bitmap/bitmap_count.slt} (59%)
create mode 100644 dev/changelog/49.0.1.md
copy dev/changelog/{46.0.1.md => 49.0.2.md} (59%)
create mode 100644 typos.toml
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@datafusion.apache.org
For additional commands, e-mail: commits-h...@datafusion.apache.org
