This is an automated email from the ASF dual-hosted git repository.
korowa pushed a change to branch dependabot/cargo/main/itertools-0.13
in repository https://gitbox.apache.org/repos/asf/datafusion.git
omit fbe3270bd8 Merge remote-tracking branch 'apache/main' into
dependabot/cargo/main/itertools-0.13
omit 0bb0f3508d Avoid deprecated API
omit d048c8f143 Merge remote-tracking branch 'apache/main' into
dependabot/cargo/main/itertools-0.13
omit 2d8c31926b Update Cargo.lock
omit 0a6d362f0e Merge remote-tracking branch 'apache/main' into
dependabot/cargo/main/itertools-0.13
omit bc53eb11c8 Update itertools requirement from 0.12 to 0.13
add 4123ad6ca7 Prepare 40.0.0 Release (#11343)
add fa0191772e Support `NULL` literals in where clause (#11266)
add e65c3e9198 Minor: Add link to blog to main DataFusion website (#11356)
add 3d1792fb9f Implement TPCH substrait integration test, support tpch_6,
tpch_10, tpch_11 (#11349)
add 6cca0f7f37 Fix bug when pushing projection under joins (#11333)
add 7f25d9dd63 Minor: some cosmetics in `filter.rs`, fix clippy due to
logical conflict (#11368)
add 9df393ea55 Update prost-derive requirement from 0.12 to 0.13 (#11355)
add c018c74ae1 Minor: update dashmap (#11335)
add 1e0c06e14a Improve and test dataframe API examples in docs (#11290)
add 16a3148354 Remove redundant `unalias_nested` calls for creating
Filter's (#11340)
add 146b679aa1 Enable `clone_on_ref_ptr` clippy lint on optimizer (#11346)
add 9a8f8b7188 fix: Fix eq properties regression from #10434 (#11363)
add d99002cf0b Update termtree requirement from 0.4.1 to 0.5.0 (#11383)
add b96186fdef Introduce `resources_err!` error macro (#11374)
add 585504a31f Enable clone_on_ref_ptr clippy lint on common (#11384)
add 6038f4cfac Track parquet writer encoding memory usage on MemoryPool
(#11345)
add 32cb3c5a54 Minor: remove clones and unnecessary Arcs in
`from_substrait_rex` (#11337)
add cc7484e0b7 Minor: Change no-statement error message to be clearer
(#11394)
add d3f63728d2 Change `array_agg` to return `null` on no input rather than
empty list (#11299)
add 7a23ea9bce Minor: return "not supported" for `COUNT DISTINCT` with
multiple arguments (#11391)
add 2413155a3e feat: Add `fail_on_overflow` option to `BinaryExpr` (#11400)
add ed65c11065 Enable clone_on_ref_ptr clippy lint on sql (#11380)
add 0b2eb50c0f Move configuration information out of example usage page
(#11300)
add faa1e98fc4 reuse a single function to create the tpch test contexts
(#11396)
add 6692382f22 refactor: change error type for "no statement" (#11411)
add f284e3bb73 feat: add UDF to_local_time() (#11347)
add 1e9f0e1d65 Implement prettier SQL unparsing (more human readable)
(#11186)
add e19dd2d0b9 Add `to_local_time()` in function reference docs (#11401)
add 4402a1a9dd Move `overlay` planning to`ExprPlanner` (#11398)
add d314ced809 Coerce types for all union children plans when eliminating
nesting (#11386)
add 4bed04e4e3 Add customizable equality and hash functions to UDFs
(#11392)
add 5ba634aa4f Implement ScalarFunction `MAKE_MAP` and `MAP` (#11361)
add d542cbda8f Improve `CommonSubexprEliminate` rule with surely and
conditionally evaluated stats (#11357)
add 1dfac86a89 fix(11397): surface proper errors in ParquetSink (#11399)
add 13ddbaf2f7 Minor: Add note about SQLLancer fuzz testing to docs
(#11430)
add c769a70dc1 Trivial: use arrow csv writer's timestamp_tz_format (#11407)
add a2a6458e42 Minor: improve documentation for sql unparsing (#11395)
add dc21a6c258 Minor: Consolidate specificataion doc sections (#11427)
add b075ac471e Minor: consolidate doc roadmap pages (#11426)
add d5367f3ff5 Avoid calling shutdown after failed write of AsyncWrite
(#249) (#250) (#11415)
add 02335ebe2d Short term way to make `AggregateStatistics` still work
when min/max is converted to udaf (#11261)
add bd25e26747 Implement TPCH substrait integration test, support tpch_13,
tpch_14, tpch_16 (#11405)
add 4d04a6ebbb Minor: fix labeler rules (#11428)
add 8f8df07c80 Minor: change internal error to not supported error for
nested field access (#11446)
add 9e4a4a1599 Minor: change Datafusion --> DataFusion in docs (#11439)
add 08fa444aaa fix: make sure JOIN ON expression is boolean type (#11423)
add f5d88d1790 Support serialization/deserialization for custom physical
exprs in proto (#11387)
add a43cf79bf0 remove termtree dependency (#11416)
add a7041feff3 Minor: Add an example for backtrace pretty print (#11450)
add 84758062f8 Add SessionStateBuilder and extract out the registration of
defaults (#11403)
add bfd815622f integrate consumer tests, implement tpch query 18 to 22
(#11462)
add d01301d2ee Docs: Explain the usage of logical expressions for
`create_aggregate_expr` (#11458)
add 0965455486 Return scalar result when all inputs are constants in `map`
and `make_map` (#11461)
add 7bd0e74aaa fix: `regexp_replace` fails when pattern or replacement is
a scalar `NULL` (#11459)
add f204869ff5 Enable `clone_on_ref_ptr` clippy lint on functions* (#11468)
add 2837e02b7e minor: split repartition time and send time metrics (#11440)
add 133128840c Docs: Document creating new extension APIs (#11425)
add 5f0993cf58 Minor: rename `row_groups.rs` to `row_group_filter.rs`
(#11481)
add 55b792a608 Support alternate formats for unparsing `datetime` to
`timestamp` and `interval` (#11466)
add f11bdf08b2 add criterion benchmark for CaseExpr (#11482)
add ccb4baf0fc Initial support for `StringView`, merge changes from
`string-view` development branch (#11402)
add 0c39b4d2ff Replace to_lowercase with to_string in sql exmaple (#11486)
add 169a0d338c chore: switch to using proper Substrait types for
IntervalYearMonth and IntervalDayTime (#11471)
add 82fd6a7de3 Move execute_input_stream (#11449)
add c54a638585 Enable clone_on_ref_ptr clippy lints on proto (#11465)
add 382bf4f3c7 upgrade sqlparser 0.47 -> 0.48 (#11453)
add 81aff944bd feat: support UDWFs in Substrait (#11489)
add 02326998f0 Add extension hooks for encoding and decoding UDAFs and
UDWFs (#11417)
add a979f3e5d4 feat: support `unnest` in GROUP BY clause (#11469)
add d67b0fbf52 Remove element's nullability of array_agg function (#11447)
add de0765a97b Get expr planners when creating new planner (#11485)
add b0925c801e Support alternate format for Utf8 unparsing (CHAR) (#11494)
add fb34ef2cca implement retract_batch for xor accumulator (#11500)
add c95556def3 Refactor: more clearly delineate between
`TableParquetOptions` and `ParquetWriterOptions` (#11444)
add 0021356fb6 feat: support `COUNT()` (#11229)
add 81a6094753 chore: fix typos of common and core packages (#11520)
add adcfd85b77 Move spill related functions to spill.rs (#11509)
add 4b840c0883 Add tests that show the different defaults for
`ArrowWriter` and `TableParquetOptions` (#11524)
add 58da9690f2 Create `datafusion-physical-optimizer` crate (#11507)
add 1b9b35c4a5 Minor: Assert `test_enabled_backtrace` requirements to run
(#11525)
add d91a03f495 Move handlign of NULL literals in where clause to type
coercion pass (#11491)
add dff2f3c3c6 Minor: Clarify which parquet options are used for
reading/writing (#11511)
add b197449687 Update parquet page pruning code to use the
`StatisticsExtractor` (#11483)
add be130b4670 Enable SortMergeJoin LeftAnti filtered fuzz tests (#11535)
add b685e2d4f1 chore: fix typos of expr, functions, optimizer,
physical-expr-common, physical-expr, and physical-plan packages (#11538)
add 4dd8532e6c rm clone (#11532)
add 723a595528 Minor: avoid a clone in type coercion (#11530)
add 12d82c427d Move array `ArrayAgg` to a `UserDefinedAggregate` (#11448)
add 5f0dfbb8e7 Move `MAKE_MAP` to ExprPlanner (#11452)
add 28fa74bf0f feat: Optimize CASE expression for "column or null" use
case (#11534)
add cf9da76830 fix: typos of sql, sqllogictest and substrait packages
(#11548)
add a4c9bb4574 feat: consume and produce Substrait type extensions (#11510)
add af0d2baf02 make unparser Dialect Send + Sync (#11504)
add f195352818 fix: unparser generates wrong sql for derived table with
columns (#17) (#11505)
add 9189a1acdd Prevent bigger files from being checked in (#11508)
add ebe61bae2a fix: make `UnKnownColumn`s not equal to others physical
exprs (#11536)
add 827d0e3a29 Add dialect param to use double precision for float64 in
Postgres (#11495)
add 5da7ab3002 Minor: move `SessionStateDefaults` into its own module
(#11566)
add 7df2bde8fc fix: fixes trig function order by (#11559)
add d232065c3b refactor: rewrite mega type to an enum containing both
cases (#11539)
add 36660fe10d Move `sql_compound_identifier_to_expr ` to `ExprPlanner`
(#11487)
add 2587df09c3 Support `newlines_in_values` CSV option (#11533)
add 63efaee255 Support SortMergeJoin spilling (#11218)
add ecf5323eaa Fix unparser invalid sql for query with order (#11527)
add 12c0a1e2e2 chore: Minor cleanup `simplify_demo()` example (#11576)
add f9457de779 Move Datafusion Query Optimizer to library user guide
(#11563)
add 47d5d1fe1a feat: Error when a SHOW command is passed in with an
accompanying non-existant variable (#11540)
add 5c65efc799 fix: CASE with NULL (#11542)
add 51da92fb9f Provide DataFrame API for `map` and move `map` to
`functions-array` (#11560)
add 81d06f2e10 Move OutputRequirements to datafusion-physical-optimizer
crate (#11579)
add 4417a9404f Minor: move `Column` related tests and rename `column.rs`
(#11573)
add b6e55d7e9c feat: Optimize CASE expression for usage where then and
else values are literals (#11553)
add 7d078d8c11 Fix SortMergeJoin antijoin flaky condition (#11604)
add a2ac00da1b Improve Union Equivalence Propagation (#11506)
add d941dc3f01 Migrate `OrderSensitiveArrayAgg` to be a user defined
aggregate (#11564)
add deef834e7a Minor: Disable flaky antijoin test until perm fix (#11608)
add 77311a5896 support Decimal256 type in datafusion-proto (#11606)
add 7afc99f3d2 Chore/fifo tests cleanup (#11616)
add 67c6ee2fb0 fix typo in doc of Partitioning (#11612)
add f80dde06ec feat: support Map literals in Substrait consumer and
producer (#11547)
add f9ee64a0ec Fix Internal Error for an INNER JOIN query (#11578)
add 5f0ab3100f test: get file size by func metadata (#11575)
add 268be45533 Update scalar_functions.md (#11620)
add 65dd364be4 Improve unparser MySQL compatibility (#11589)
add 9f74dcc999 Change default Parquet writer settings to match arrow-rs
(except for compression & statistics) (#11558)
add 6d3521738b Push scalar functions into cross join (#11528)
add 6d8bd2cb67 Implement physical plan serialization for csv COPY plans ,
add `as_any`, `Debug` to `FileFormatFactory` (#11588)
add fc8e7b9035 Remove ArrayAgg Builtin in favor of UDF (#11611)
add 5c37d008dc refactor: simplify `DFSchema::field_with_unqualified_name`
(#11619)
add c8ef54537c fix: panic and incorrect results in
`LogFunc::output_ordering()` (#11571)
add 72c6491d25 Minor: Use upstream concat_batches (#11615)
add 1e06b91d59 Rename `functions-array` to `functions-nested` (#11602)
add 8945462ed0 Fix : `signum` function bug when `0.0` input (#11580)
add 6efdbe6d4b Enforce uniqueness of `named_struct` field names (#11614)
add e90b3ac5cf Minor: unecessary row_count calculation in `CrossJoinExec`
and `NestedLoopsJoinExec` (#11632)
add 13569340bc ExprBuilder for Physical Aggregate Expr (#11617)
add 5901df58b2 feat: add bounds for unary math scalar functions (#11584)
add bcf715c892 Minor: avoid copying order by exprs in planner (#11634)
add 20b298e9d8 perf: Optimize IsNotNullExpr (#11586)
add 76039fadd9 Unify CI and pre-commit hook settings for clippy (#11640)
add 886e8accda Consistent API to set parameters of aggregate and window
functions (`AggregateExt` --> `ExprFunctionExt`) (#11550)
add c9518245fa Parsing SQL strings to Exprs with the qualified schema
(#11562)
add f12b3db9c3 fix: expose the fluent API fn for approx_distinct instead
of the module (#11644)
add 49d9d45f36 Add some zero column tests covering LIMIT, GROUP BY, WHERE,
JOIN, and WINDOW (#11624)
add 7db4213b71 Refactor/simplify window frame utils (#11648)
add 71903e1b2c Minor: use `ready!` macro to simplify FilterExec poll loop
(#11649)
add 5b7e0aae22 Temporarily pin toolchain version to avoid clippy (#11655)
add 6fd57b2e38 Fix clippy errors for Rust 1.80 (#11654)
add d452d51bac Add parser option enable_options_value_normalization
(#11330)
add dcdcc250cc Add reference to #comet channel in Arrow Rust Discord
server (#11637)
add fab7e238e8 Add `CsvExecBuilder` for creating `CsvExec` (#11633)
add c22e13fa10 chore(deps): update sqlparser requirement from 0.48 to 0.49
(#11630)
add 2eaf1ea8ac Extract catalog API to separate crate (#11516)
add d6e016e691 doc: why nullable of list item is set to true (#11626)
add 64af410d0b Add support for USING to SQL unparser (#11636)
add cdf387e4de fix: dont try to coerce list for regex match (#11646)
add 01dc3f9766 Run CI with latest (Rust 1.80), add ticket references to
commented out tests (#11661)
add 204e1bcbaa Use `AccumulatorArgs::is_reversed` in `NthValueAgg` (#11669)
add 42732b5f40 Docs: adding explicit mention of test_utils to docs (#11670)
add a598739ad7 Ensure statistic defaults in parquet writers are in sync
(#11656)
add 3cfb99d987 Implement physical plan serialization for json Copy plans
(#11645)
add a721be1b1d Minor: improve documentation on SessionState (#11642)
add 5ad60678a2 Add LimitPushdown optimization rule and CoalesceBatchesExec
fetch (#11652)
add 823a007577 Update to arrow/parquet `52.2.0` (#11691)
add 75ced38605 Minor: make field names match metric name (#11478)
add c50fd88faf Rename `ColumnOptions` to `ParquetColumnOptions` (#11512)
add ea8c287855 Update cache key used in rust CI script (#11641)
add a591301a1f Merge `string-view2` branch: reading from parquet up to 2x
faster for some ClickBench queries (not on by default) (#11667)
add 2f5e73c6aa Fix bug in `remove_join_expressions` (#11693)
add 35c2e7e7eb Initial changes to support using udaf min/max for
statistics and opti… (#11696)
add 66a85706f6 Rename `input_type` --> `input_types` on
AggregateFunctionExpr / AccumulatorArgs / StateFieldsArgs (#11666)
add cd786e2751 fix: regr_count now returns Uint64 (#11731)
add 7ca7456d3a Handle nulls in approx_percentile_cont (#11721)
add 8ac50e2dbb Reduce repetition in try_process_group_by_unnest and
try_process_unnest (#11714)
add cc6416e741 Minor: Add example for `ScalarUDF::call` (#11727)
add 929568d600 Use `cargo release` in benchmarks (#11722)
add 6508fa2dc4 expose some fields on session state (#11716)
add abeb8b4f8d Make DefaultSchemaAdapterFactory public (#11709)
add 89677ae66e Check hashes first during probing the aggr hash table
(#11718)
add 2887491fc6 fix: set `null_equals_null` to false when
`convert_cross_join_to_inner_join` (#11738)
add fa50636c6a Implement physical plan serialization for parquet Copy
plans (#11735)
add ae2ca6a0e2 Support cross-timezone `timestamp` comparison via
coercsion (#11711)
add 0f554fa124 Minor: Improve documentation for
AggregateUDFImpl::state_fields (#11740)
add 9dd2cfc801 Do not push down Sorts if it violates the sort requirements
(#11678)
add 4884c08ceb Use upstream StatisticsConveter (#11479)
add 0b8da6d6e7 Rename RepartitionExec metric `repart_time` to
`repartition_time` (#11703)
add 1ce546168d Fix `plan_to_sql`: Add wildcard projection to SELECT
statement if no projection was set (#11744)
add cf98d94c9c Use upstream `DataType::from_str` in arrow-cast (#11254)
add a4ac0829ec Fix documentation warnings, make CsvExecBuilder and
Unparsed pub (#11729)
add 3fe18604e8 Add null test (#11760)
add 921c3b6b18 Add `TrackedMemoryPool` with better error messages on
exhaustion (#11665)
add 6e2ff2955d Derive Debug for logical plan nodes (#11757)
add 45b40c711e Minor: add "clickbench extended" queries to unit tests
(#11763)
add 0d98b99743 Minor: Add comment explaining rationale for hash check
(#11750)
add f044bc8371 Fix bug that `COUNT(DISTINCT)` on StringView panics
(#11768)
add a0ad376840 [Minor] Refactor approx_percentile (#11769)
add 70aba2bd6c minor: always time batch_filter even when the result is an
empty batch (#11775)
add d010ce90f4 refactor(11523): update OOM message provided for a single
failed reservation (#11771)
add df4e6cc4e5 [Minor] Short circuit `ApplyFunctionRewrites` if there are
no function rewrites (#11765)
add 80848f2a0a Fix #11692: Improve doc comments within macros (#11694)
add 5ca4ec3b59 Extract CoalesceBatchesStream to a struct (#11610)
add 0332eb569a refactor: move ExecutionPlan and related structs into
dedicated mod (#11759)
add b89037f0d4 Add references to github issue (#11784)
add de2da34cc5 Add docs and rename param (#11778)
add 6c4c246125 Doc: Add Sail to known users list (#11791)
add 81668f3b2d Support planning `Map` literal (#11780)
add 9e90e17a61 fix: Add additional required expression for natural join
(#11713)
add f4e519f9df Move min and max to user defined aggregate function, remove
`AggregateFunction` / `AggregateFunctionDefinition::BuiltIn` (#11013)
add a4d41d6a66 Support `LogicalPlan` `Debug` differently than `Display`
(#11774)
add c8e5996c47 Remove redundant Aggregate when `DISTINCT` & `GROUP BY` are
in the same query (#11781)
add b4069a65a9 Remove `AggregateFunctionDefinition` (#11803)
add 1d3bdbe14f Minor: add ticket reference and fmt (#11805)
add 336c15e759 Improve MSRV CI check to print out problems to log (#11789)
add c340b6ab7a Skipping partial aggregation when it is not helping for
high cardinality aggregates (#11627)
add 682bc2eff2 Improve log func stability (#11808)
add 6aad19fad3 add valid distinct case for aggregate.slt (#11814)
add 45d85b1d18 Change name of MAX/MIN udaf to lowercase max/min (#11795)
add 5c4254aeb3 Don't implement create_sliding_accumulator repeatedly
(#11813)
add f19d30d526 chore(deps): update rstest requirement from 0.21.0 to
0.22.0 (#11811)
add 0417e543e1 Minor: Update exected output due to logical conflict
(#11824)
add f56a2ef28f doc: Add support for `map` and `make_map` functions (#11799)
add 011a3f3e32 Improve readme page in crates.io (#11809)
add c6f0d3cac9 Pass scalar to `eq` inside `nullif` (#11697)
add fcd907d78e refactor: move `aggregate_statistics` to
`datafusion-physical-optimizer` (#11798)
add eb2b5fe74c feat: Add support for cardinality function on maps (#11801)
add a1645c4c1d Minor: refactor probe check into function
`should_skip_aggregation` (#11821)
add 4a47dcbed0 Minor: move path_partition into `core_integration` (#11831)
add 537774ec1d Move optimizer integration tests to `core_integration`
(#11830)
add bf51babc9c Optionally create name of aggregate expression from
expressions (#11776)
add d8bc7e2e6c Bump deprecated version of SessionState::new_with_config_rt
to 41.0.0 (#11839)
add 1c98e6e640 Fix partial aggregation skipping with Decimal aggregators
(#11833)
add 3d76aa25e4 feat: support `Utf8View` type in `starts_with` function
(#11787)
add 4e278cad19 fix: hash join tests with forced collisions (#11806)
add e19e9825f2 don't create zero sized buffer (#11841)
add 16a3557325 fix: `collect_columns` quadratic complexity (#11843)
add bddb6415a5 Reduce clone of `Statistics` in `ListingTable` and
`PartitionedFile` (#11802)
add 117ab1b8c8 Add `LogicalPlan::CreateIndex` (#11817)
add 1ecdf90078 Update `object_store` to 0.10.2 (#11860)
add 679a85f808 Add metrics for skipped rows (#11706)
add cd322f1145 feat: Expose public method for optimizing physical plans
(#11879)
add 60d1d3a702 Cast `Utf8View` to `Utf8` to support `||` from
`StringViewArray` (#11796)
add d0a1d3030a refactor: remove unneed mut for session context (#11864)
add 053795cd4e Improve nested loop join code (#11863)
add 0bbce5dce2 [Minor]: Refactor to use Result.transpose() (#11882)
add 2521043ddc support `ANY()` op (#11849)
add 1f35b03426 Sync rust docs params for CI and dev (#11890)
add 0ce6d1687e Update ASCII scalar function to support Utf8View #11834
(#11884)
add b9bf6c9ec4 Fix `Duration` vs `Interval` comparisons and `Interval` as
LHS (#11876)
add 786f353ca1 chore: Prepare 41.0.0-rc1 (#11889)
add 193955e0ec Produce clear error message when build runs with
conflicting features (#11895)
add 368df80b7f Add tests for StringView / character functions, fix
`regexp_like` and `regexp_match` to work with StringView (#11753)
add 86030a1ff7 fix: invalid sqls when unparsing derived table with columns
contains calculations, limit/order/distinct (#11756)
add 1c9583ab95 Avoid unecessary copy when reading arrow files (#11840)
add 56be714eca Support NULL literal in Min/Max (#11812)
add 20fbd88521 Remove many `crate::` imports in listing table provider
module (#11887)
add 56f8e3527b feat: expose centroids in approx_percentile_cont fluent api
(#11878)
add 7c41323fa4 Rename `Expr::display_name` to `Expr::schema_name`, make
`UNNEST` naming conform to convention (#11797)
add b5d7931dea Make `CommonSubexprEliminate` top-down like (#11683)
add e088945c38 UDAF refactor: Add PhysicalExpr trait dependency on
`datafusion-expr` and remove logical expressions requirement for creating
physical aggregate expression (#11845)
add 1e93f7fa2c Add `generate_series` tests for arrays (#11921)
add 12aa82c745 Minor: use `lit(true)` and `lit(false)` more (#11904)
add 79fa6f9098 Enforce sorting handle fetchable operators, add option to
repartition based on row count estimates (#11875)
add 27304239ef fix: make ScalarValue::Dictionary with NULL values produce
NULL arrays (#11908)
add ee6910b587 Minor: change wording for PMC membership notice (#11930)
add 64a928057d fix approx_percentile_cont() bug (#11934)
add 7d6cd13d55 fix: throw error on sub-day generate_series increments
(#11907)
add fd237f8705 Ingore shebang at top of file (#11927)
add 63ca7142fb Parse Sqllogictest column types from physical schema
(#11929)
add f2685d3a01 Update INITCAP scalar function to support Utf8View (#11888)
add 5251dc91ae Implement native support StringView for Octet Length
(#11906)
add 032b9c9c2d fix: impl ordering for serialization/deserialization for
AggregateUdf (#11926)
add 8deba02134 Implement native support StringView for Ends With (#11924)
add b60cdc787d Implement native support StringView for Levenshtein (#11925)
add 34ec9d41fa Implement native stringview support for BTRIM (#11920)
add 2cf09566af Move `LimitPushdown` to physical-optimizer crate (#11945)
add e66636d365 Minor: Improve comments in row_hash.rs for skipping
aggregation (#11820)
add 18193e6224 chore: Add SessionState to MockContextProvider just like
SessionContextProvider (#11940)
add ffdc61d335 Update labeler.yml to match crates (#11937)
add 140f7cec78 Support tuples as types (#11896)
add 00ef820470 Support `convert_to_state` for `AVG` accumulator (#11734)
add 7a1a23d5f5 minor: Update release documentation based on 41.0.0 release
(#11947)
add 5d3cda5306 Make `Precision<usize>` copy to make it clear clones are
not expensive (#11828)
add 1d86724a05 fix: Fix various complaints from the latest nightly clippy
(#11958)
add 8e23cba479 Minor: fix outdated link (#11964)
add af75f2d510 Minor: simplify SQL number parsing and add a comment about
unused (#11965)
add 44b9066ea4 Support Arrays for the Map scalar functions (#11712)
add 508da801b6 Update LPAD scalar function to support Utf8View (#11941)
add e8ac93a0ff Add native stringview support for LTRIM & RTRIM (#11948)
add 3438b35530 Move wildcard expansions to the analyzer (#11681)
add 69c99a7e78 Add native stringview support for RIGHT (#11955)
add 482ef4551a register get_field by default (#11959)
add 3c477bf31c Refactor `CoalesceBatches` to use an explicit state machine
(#11966)
add 10fe3c1658 Implement native support StringView for find in set (#11970)
add afa23abc46 fix: move coercion of union from builder to `TypeCoercion`
(#11961)
add e4be013064 test: re-enable window function over parquet with forced
collisions (#11939)
add f98f8a9212 Implement native support StringView for `REPEAT` (#11962)
add 02bfefe7f6 Update RPAD scalar function to support Utf8View (#11942)
add 94034bef93 Improve lpad udf by using a GenericStringBuilder (#11987)
add a00922b33b Implement native support StringView for substr_index
(#11974)
add b9961c3001 Add native stringview support for LEFT (#11976)
add e24a5dd5a5 Improve function documentation (#11996)
add ea2e7ab688 Implement native support StringView for overlay (#11968)
add cb3ec77f28 disable `with_create_default_catalog_and_schema` if the
default catalog exists (#11991)
add 4baa901cec Use tracked-consumers memory pool be the default. (#11949)
add 06bcf33efd Update REVERSE scalar function to support Utf8View (#11973)
add 41f6dd9275 Support partial aggregation skip for boolean functions
(#11847)
add 6b73c4f866 feat/11953: Support StringView for TRANSLATE() fn (#11967)
add c1fb989aed Update SPLIT_PART scalar function to support Utf8View
(#11975)
add 36158b6e6e Handle arguments checking of `min`/`max` function to avoid
crashes (#12016)
add 19ad53d95c Fix: support NULL input for regular expression comparison
operations (#11985)
add 9d1cf74aa7 Remove physical sort parameters on aggregate window
functions (#12009)
add 9f77021c2e Minor: Use execution error in ScalarValue::iter_to_array
for incorrect usage (#11999)
add 58075e2329 Fix: support NULL input for like operations (#12025)
add 57d5e0ec0d Minor: Add error tests for min/max with 2 arguments (#12024)
add 5db036e90f fix: incorrect aggregation result of `bool_and` (#12017)
add bb921812fa Improve performance of REPEAT functions (#12015)
add 300a08c622 support Utf8View (#12019)
add 08f6e54074 Minor: Remove wrong comment on `Accumulator::evaluate` and
`Accumulator::state` (#12001)
add dc84fa56f5 Minor: cleanup `.gitignore` (#12035)
add 2a16704db7 Improve documentation about `ParquetExec` / Parquet
predicate pushdown (#11994)
add b06e8b06b1 catalog.has_header true by default (#11919)
add 72b6a49e74 feat: Add map_extract module and function (#11969)
add e4bc62247c refactor: Move `LimitedDistinctAggregation` to
`physical-optimizer` crate (#12036)
add bd2d4eeaf3 Convert built-in `row_number` to user-defined window
function (#12030)
add 48416e57f4 Fix projection name with DataFrame::with_column and window
functions (#12000)
add 7fa7689c41 Update to `sqlparser-rs` v0.50.0 (#12014)
add 186ba4c3ae Minor: make some physical-plan properties public (#12022)
add 1f90b0060b chore: improve variable naming conventions (#12042)
add e84f3436c0 Fix: handle NULL input for regex match operations (#12028)
add cb1e3f0e5c Fix compilation, change row_number() expr_fn to 0 args
(#12043)
add cd9237f96d Minor: Remove warning when building datafusion-cli from
Dockerfile (#12018)
add 950dc73c7f add getter method for LogicalPlanBuilder.plan (#12038)
add a91be04ced Window UDF signature check (#12045)
add 574dfeb29f Fix: generate_series function support string type (#12002)
add 02eab80cd6 Do not add redundant subquery ordering into plan (#12003)
add c60b798cc0 Fix: Remove Unrelated Fields When Expanding Wildcards in
Functional Dependency Projections (#12060)
add a3c0c19b8a Update async-trait in CLI and catalog crates (#12061)
add b77917747f Improve doc (#12053)
add 159ab179ff Minor: remove clones in `coerce_plan_expr_for_schema`
(#12051)
add 7c5a8ebc32 fix: support min/max for Float16 type (#12050)
add bd482629dc implement utf8_view for replace (#12004)
add c2cbba26df Fix CI check when version changes -- remove checked in file
that is created by doc example (#12034)
add 67cf1d606b Minor: update sqllogictest to treat Utf8View as text
(#12033)
add 37e54ee874 [MINOR] correct document mistakes (#12068)
add 6e34280b2b Plan `LATERAL` subqueries (#11456)
add ecb5a91448 vectorize random() scalar function (#12078)
add 9bd183f417 functions: support strpos with mixed string types (#12072)
add e6e1eb2294 Update to `clap` 4.5.16 (#12064)
add 6786f1592f Fix the schema mismatch between logical and physical for
aggregate function, add `AggregateUDFImpl::is_null` (#11989)
add 846befb6a6 minor: Remove unused create_row_hashes (#12083)
add 1c7209b280 Use `schema_name` to create the `physical_name` (#11977)
add 78f58c8047 Add new user doc to translate logical plan to physical plan
(#12026)
add 7eeac2f5c2 Improve rpad udf by using a GenericStringBuilder (#12070)
add 9d076bde4f fix: Panic non-integer for the second argument of
`nth_value` function (#12076)
add eca71c4cb1 Remove vestigal `datafusion-docs` module compilation
(#12081)
add ad583a8dfa Add test to verify count aggregate function should not be
nullable (#12100)
add 121f330a6c Minor: Extract `BatchCoalescer` to its own module (#12047)
new f110f357ed Update itertools requirement from 0.12 to 0.13
new 6cf4bf11a2 Update Cargo.lock
new 7c046cf15e Avoid deprecated API
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fbe3270bd8)
\
N -- N -- N refs/heads/dependabot/cargo/main/itertools-0.13
(7c046cf15e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitattributes | 1 +
.github/workflows/dev_pr/labeler.yml | 29 +-
.github/workflows/large_files.yml | 55 +
.github/workflows/rust.yml | 45 +-
.gitignore | 46 +-
Cargo.toml | 84 +-
README.md | 2 +-
benchmarks/bench.sh | 3 +-
benchmarks/queries/clickbench/README.md | 2 +-
benchmarks/src/bin/dfbench.rs | 5 +
benchmarks/src/bin/tpch.rs | 5 +
benchmarks/src/clickbench.rs | 8 +-
benchmarks/src/tpch/run.rs | 11 +-
benchmarks/src/util/options.rs | 5 +
ci/scripts/rust_docs.sh | 2 +-
datafusion-cli/Cargo.lock | 758 +++---
datafusion-cli/Cargo.toml | 12 +-
datafusion-cli/Dockerfile | 2 +-
datafusion-cli/examples/cli-session-context.rs | 6 +-
datafusion-cli/src/catalog.rs | 18 +-
datafusion-cli/src/command.rs | 4 +-
datafusion-cli/src/exec.rs | 19 +-
datafusion-cli/src/functions.rs | 4 +-
datafusion-cli/src/main.rs | 57 +-
datafusion-cli/src/pool_type.rs | 2 +-
datafusion-cli/src/print_format.rs | 4 +-
datafusion-examples/Cargo.toml | 2 +-
datafusion-examples/README.md | 6 +-
.../examples/advanced_parquet_index.rs | 6 +-
datafusion-examples/examples/advanced_udwf.rs | 12 +-
datafusion-examples/examples/catalog.rs | 17 +-
.../examples/composed_extension_codec.rs | 80 +-
datafusion-examples/examples/custom_datasource.rs | 5 +-
datafusion-examples/examples/custom_file_format.rs | 15 +-
datafusion-examples/examples/dataframe_subquery.rs | 1 +
datafusion-examples/examples/expr_api.rs | 16 +-
datafusion-examples/examples/parquet_index.rs | 10 +-
datafusion-examples/examples/parse_sql_expr.rs | 9 +
datafusion-examples/examples/plan_to_sql.rs | 24 +-
datafusion-examples/examples/planner_api.rs | 143 ++
datafusion-examples/examples/simple_udtf.rs | 6 +-
datafusion-examples/examples/simple_udwf.rs | 12 +-
{docs => datafusion/catalog}/Cargo.toml | 32 +-
datafusion/catalog/src/catalog.rs | 173 ++
.../src/expressions/mod.rs => catalog/src/lib.rs} | 13 +-
.../{core/src/catalog => catalog/src}/schema.rs | 7 +-
datafusion/catalog/src/session.rs | 138 +
.../provider.rs => catalog/src/table.rs} | 72 +-
datafusion/common-runtime/src/lib.rs | 2 +
datafusion/common/Cargo.toml | 2 +
datafusion/common/src/cast.rs | 11 +
datafusion/common/src/column.rs | 4 +-
datafusion/common/src/config.rs | 142 +-
datafusion/common/src/dfschema.rs | 44 +-
datafusion/common/src/error.rs | 84 +-
datafusion/common/src/file_options/csv_writer.rs | 5 +-
.../common/src/file_options/parquet_writer.rs | 579 ++++-
datafusion/common/src/functional_dependencies.rs | 41 +-
datafusion/common/src/hash_utils.rs | 327 ++-
datafusion/common/src/lib.rs | 16 +-
.../mod.rs => common/src/scalar/consts.rs} | 31 +-
datafusion/common/src/scalar/mod.rs | 285 ++-
datafusion/common/src/scalar/struct_builder.rs | 2 +-
datafusion/common/src/stats.rs | 145 +-
datafusion/common/src/table_reference.rs | 2 +-
datafusion/common/src/tree_node.rs | 11 +-
.../mod.rs => common/src/utils/expr.rs} | 11 +-
datafusion/common/src/utils/memory.rs | 2 +-
datafusion/common/src/utils/mod.rs | 99 +-
datafusion/core/Cargo.toml | 19 +-
datafusion/core/benches/filter_query_sql.rs | 14 +-
datafusion/core/benches/map_query_sql.rs | 93 +
datafusion/core/benches/parquet_statistic.rs | 287 ---
datafusion/core/benches/sort.rs | 2 +-
.../information_schema.rs | 8 +-
.../{catalog => catalog_common}/listing_schema.rs | 4 +-
.../core/src/{catalog => catalog_common}/memory.rs | 9 +-
.../core/src/{catalog => catalog_common}/mod.rs | 171 +-
datafusion/core/src/dataframe/mod.rs | 267 +-
datafusion/core/src/dataframe/parquet.rs | 10 +-
datafusion/core/src/datasource/cte_worktable.rs | 4 +-
datafusion/core/src/datasource/empty.rs | 4 +-
.../core/src/datasource/file_format/arrow.rs | 10 +-
datafusion/core/src/datasource/file_format/avro.rs | 11 +
datafusion/core/src/datasource/file_format/csv.rs | 133 +-
datafusion/core/src/datasource/file_format/json.rs | 15 +-
datafusion/core/src/datasource/file_format/mod.rs | 41 +-
.../core/src/datasource/file_format/options.rs | 22 +
.../core/src/datasource/file_format/parquet.rs | 243 +-
.../core/src/datasource/file_format/write/demux.rs | 2 +-
.../datasource/file_format/write/orchestration.rs | 89 +-
datafusion/core/src/datasource/listing/helpers.rs | 33 +-
datafusion/core/src/datasource/listing/mod.rs | 5 +-
datafusion/core/src/datasource/listing/table.rs | 98 +-
datafusion/core/src/datasource/listing/url.rs | 4 +-
.../core/src/datasource/listing_table_factory.rs | 18 +-
datafusion/core/src/datasource/memory.rs | 7 +-
datafusion/core/src/datasource/mod.rs | 2 +-
.../src/datasource/physical_plan/arrow_file.rs | 34 +-
.../core/src/datasource/physical_plan/avro.rs | 17 +
.../core/src/datasource/physical_plan/csv.rs | 357 ++-
.../datasource/physical_plan/file_scan_config.rs | 6 +-
.../core/src/datasource/physical_plan/json.rs | 16 +
.../core/src/datasource/physical_plan/mod.rs | 2 +-
.../src/datasource/physical_plan/parquet/mod.rs | 173 +-
.../src/datasource/physical_plan/parquet/opener.rs | 31 +-
.../physical_plan/parquet/page_filter.rs | 560 ++---
.../datasource/physical_plan/parquet/row_filter.rs | 205 +-
.../parquet/{row_groups.rs => row_group_filter.rs} | 26 +-
.../datasource/physical_plan/parquet/statistics.rs | 2632 --------------------
datafusion/core/src/datasource/provider.rs | 276 +-
datafusion/core/src/datasource/schema_adapter.rs | 7 +-
datafusion/core/src/datasource/statistics.rs | 164 +-
datafusion/core/src/datasource/stream.rs | 12 +-
datafusion/core/src/datasource/streaming.rs | 13 +-
datafusion/core/src/datasource/view.rs | 46 +-
datafusion/core/src/execution/context/mod.rs | 126 +-
datafusion/core/src/execution/mod.rs | 3 +
datafusion/core/src/execution/session_state.rs | 1070 ++++++--
.../core/src/execution/session_state_defaults.rs | 209 ++
datafusion/core/src/lib.rs | 100 +-
.../src/physical_optimizer/aggregate_statistics.rs | 640 -----
.../src/physical_optimizer/coalesce_batches.rs | 2 +-
.../combine_partial_final_agg.rs | 45 +-
.../src/physical_optimizer/enforce_distribution.rs | 245 +-
.../core/src/physical_optimizer/enforce_sorting.rs | 289 ++-
.../core/src/physical_optimizer/join_selection.rs | 10 +-
.../limited_distinct_aggregation.rs | 611 -----
datafusion/core/src/physical_optimizer/mod.rs | 13 +-
.../core/src/physical_optimizer/optimizer.rs | 33 +-
.../src/physical_optimizer/projection_pushdown.rs | 134 +-
datafusion/core/src/physical_optimizer/pruning.rs | 43 +-
.../replace_with_order_preserving_variants.rs | 32 +-
.../core/src/physical_optimizer/sanity_checker.rs | 6 +-
.../core/src/physical_optimizer/sort_pushdown.rs | 149 +-
.../core/src/physical_optimizer/test_utils.rs | 100 +-
.../src/physical_optimizer/topk_aggregation.rs | 2 +-
.../src/physical_optimizer/update_aggr_exprs.rs | 5 +-
datafusion/core/src/physical_planner.rs | 394 +--
datafusion/core/src/prelude.rs | 4 +-
datafusion/core/src/test/mod.rs | 70 +-
datafusion/core/src/test/object_store.rs | 8 +-
datafusion/core/src/test_util/mod.rs | 115 +-
datafusion/core/tests/core_integration.rs | 6 +
datafusion/core/tests/custom_sources_cases/mod.rs | 11 +-
.../provider_filter_pushdown.rs | 8 +-
.../core/tests/custom_sources_cases/statistics.rs | 5 +-
datafusion/core/tests/data/newlines_in_values.csv | 13 +
.../core/tests/dataframe/dataframe_functions.rs | 41 +-
datafusion/core/tests/dataframe/describe.rs | 4 +-
datafusion/core/tests/dataframe/mod.rs | 49 +-
datafusion/core/tests/expr_api/mod.rs | 4 +-
datafusion/core/tests/expr_api/parse_sql_expr.rs | 22 +-
datafusion/core/tests/expr_api/simplification.rs | 2 +-
datafusion/core/tests/fifo/mod.rs | 174 +-
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 22 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 41 +-
datafusion/core/tests/fuzz_cases/window_fuzz.rs | 102 +-
datafusion/core/tests/memory_limit/mod.rs | 182 +-
.../{optimizer_integration.rs => optimizer/mod.rs} | 5 +-
datafusion/core/tests/parquet/arrow_statistics.rs | 2178 ----------------
datafusion/core/tests/parquet/file_statistics.rs | 6 +-
datafusion/core/tests/parquet/mod.rs | 429 +---
datafusion/core/tests/parquet/schema.rs | 4 +-
.../physical_optimizer/aggregate_statistics.rs | 325 +++
.../tests/physical_optimizer/limit_pushdown.rs | 491 ++++
.../limited_distinct_aggregation.rs | 440 ++++
.../tests/physical_optimizer/mod.rs} | 5 +-
.../core/tests/physical_optimizer/test_util.rs | 57 +
datafusion/core/tests/sql/aggregates.rs | 6 +-
datafusion/core/tests/sql/create_drop.rs | 13 +-
datafusion/core/tests/sql/explain_analyze.rs | 36 +-
datafusion/core/tests/sql/mod.rs | 1 +
datafusion/core/tests/{ => sql}/path_partition.rs | 0
datafusion/core/tests/sql/sql_api.rs | 34 +
.../tests/user_defined/user_defined_aggregates.rs | 79 +-
.../core/tests/user_defined/user_defined_plan.rs | 56 +-
.../user_defined/user_defined_scalar_functions.rs | 142 +-
.../user_defined/user_defined_table_functions.rs | 6 +-
datafusion/execution/src/disk_manager.rs | 6 +-
datafusion/execution/src/memory_pool/mod.rs | 25 +-
datafusion/execution/src/memory_pool/pool.rs | 392 ++-
datafusion/execution/src/object_store.rs | 16 +-
datafusion/execution/src/runtime_env.rs | 14 +-
datafusion/execution/src/task.rs | 6 +-
{docs => datafusion/expr-common}/Cargo.toml | 18 +-
.../{expr => expr-common}/src/accumulator.rs | 125 +-
.../{expr => expr-common}/src/columnar_value.rs | 0
.../src/groups_accumulator.rs | 76 +-
.../src/interval_arithmetic.rs | 67 +-
.../mod.rs => expr-common/src/lib.rs} | 29 +-
datafusion/{expr => expr-common}/src/operator.rs | 230 +-
datafusion/{expr => expr-common}/src/signature.rs | 26 +-
.../{expr => expr-common}/src/sort_properties.rs | 0
.../expr-common/src/type_coercion.rs | 4 +-
.../src/type_coercion/aggregates.rs | 71 +-
.../src/type_coercion/binary.rs | 302 ++-
datafusion/expr/Cargo.toml | 5 +-
datafusion/expr/src/aggregate_function.rs | 168 --
datafusion/expr/src/built_in_window_function.rs | 10 +-
datafusion/expr/src/expr.rs | 1060 +++++---
datafusion/expr/src/expr_fn.rs | 362 ++-
datafusion/expr/src/expr_rewriter/mod.rs | 35 +-
datafusion/expr/src/expr_rewriter/order_by.rs | 18 +-
datafusion/expr/src/expr_schema.rs | 120 +-
datafusion/expr/src/function.rs | 82 +-
datafusion/expr/src/lib.rs | 38 +-
datafusion/expr/src/logical_plan/builder.rs | 211 +-
datafusion/expr/src/logical_plan/ddl.rs | 37 +-
datafusion/expr/src/logical_plan/display.rs | 4 +-
datafusion/expr/src/logical_plan/dml.rs | 18 +-
datafusion/expr/src/logical_plan/mod.rs | 6 +-
datafusion/expr/src/logical_plan/plan.rs | 231 +-
datafusion/expr/src/logical_plan/statement.rs | 8 +-
datafusion/expr/src/logical_plan/tree_node.rs | 1 +
datafusion/expr/src/operation.rs | 222 ++
datafusion/expr/src/partition_evaluator.rs | 2 +-
datafusion/expr/src/planner.rs | 45 +-
datafusion/expr/src/simplify.rs | 2 +-
datafusion/expr/src/test/function_stub.rs | 183 +-
datafusion/expr/src/tree_node.rs | 58 +-
datafusion/expr/src/type_coercion/functions.rs | 108 +-
datafusion/expr/src/type_coercion/mod.rs | 9 +-
datafusion/expr/src/udaf.rs | 309 +--
datafusion/expr/src/udf.rs | 112 +-
datafusion/expr/src/udwf.rs | 180 +-
datafusion/expr/src/utils.rs | 317 ++-
datafusion/expr/src/window_frame.rs | 99 +-
datafusion/expr/src/window_function.rs | 91 +
.../Cargo.toml | 14 +-
.../functions-aggregate-common/src/accumulator.rs | 96 +
.../functions-aggregate-common/src/aggregate.rs | 186 ++
.../src/aggregate/count_distinct.rs} | 1 +
.../src/aggregate/count_distinct/bytes.rs | 65 +-
.../src/aggregate/count_distinct/native.rs | 4 +-
.../src/aggregate/groups_accumulator.rs} | 14 +-
.../src/aggregate/groups_accumulator/accumulate.rs | 10 +-
.../src/aggregate/groups_accumulator/bool_op.rs | 34 +-
.../src/aggregate/groups_accumulator/nulls.rs | 93 +
.../src/aggregate/groups_accumulator/prim_op.rs | 64 +-
.../src/lib.rs} | 27 +-
.../src}/merge_arrays.rs | 0
datafusion/functions-aggregate-common/src/order.rs | 47 +
.../src}/stats.rs | 0
.../src}/tdigest.rs | 76 +-
.../src}/utils.rs | 9 +-
datafusion/functions-aggregate/COMMENTS.md | 77 +
datafusion/functions-aggregate/Cargo.toml | 18 +-
datafusion/functions-aggregate/benches/count.rs | 95 +
datafusion/functions-aggregate/benches/sum.rs | 103 +
.../functions-aggregate/src/approx_distinct.rs | 4 +-
.../functions-aggregate/src/approx_median.rs | 4 +-
.../src/approx_percentile_cont.rs | 180 +-
.../src/approx_percentile_cont_with_weight.rs | 13 +-
.../src/array_agg.rs} | 391 ++-
datafusion/functions-aggregate/src/average.rs | 75 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 56 +-
datafusion/functions-aggregate/src/bool_and_or.rs | 33 +-
datafusion/functions-aggregate/src/correlation.rs | 23 +-
datafusion/functions-aggregate/src/count.rs | 116 +-
datafusion/functions-aggregate/src/covariance.rs | 2 +-
datafusion/functions-aggregate/src/first_last.rs | 60 +-
datafusion/functions-aggregate/src/lib.rs | 18 +-
datafusion/functions-aggregate/src/macros.rs | 6 +-
datafusion/functions-aggregate/src/median.rs | 6 +-
.../src}/min_max.rs | 910 +++++--
datafusion/functions-aggregate/src/nth_value.rs | 73 +-
datafusion/functions-aggregate/src/regr.rs | 15 +-
datafusion/functions-aggregate/src/stddev.rs | 29 +-
datafusion/functions-aggregate/src/string_agg.rs | 32 +-
datafusion/functions-aggregate/src/sum.rs | 23 +-
datafusion/functions-aggregate/src/variance.rs | 2 +-
.../Cargo.toml | 13 +-
.../README.md | 4 +-
.../benches/array_expression.rs | 2 +-
datafusion/functions-nested/benches/map.rs | 104 +
.../src/array_has.rs | 24 +-
.../src/cardinality.rs | 39 +-
.../src/concat.rs | 2 +-
.../src/dimension.rs | 0
.../src/empty.rs | 0
.../src/except.rs | 0
.../src/expr_ext.rs | 8 +-
.../src/extract.rs | 51 +-
.../src/flatten.rs | 4 +-
.../src/length.rs | 0
.../src/lib.rs | 22 +-
.../src/macros.rs | 6 +-
.../src/make_array.rs | 0
datafusion/functions-nested/src/map.rs | 327 +++
datafusion/functions-nested/src/map_extract.rs | 173 ++
.../src/planner.rs | 48 +-
.../src/position.rs | 0
.../src/range.rs | 167 +-
.../src/remove.rs | 2 +-
.../src/repeat.rs | 0
.../src/replace.rs | 0
.../src/resize.rs | 8 +-
.../src/reverse.rs | 4 +-
.../src/set_ops.rs | 12 +-
.../src/sort.rs | 2 +-
.../src/string.rs | 2 +-
.../src/utils.rs | 21 +-
.../Cargo.toml | 15 +-
.../README.md | 5 +-
datafusion/functions-window/src/lib.rs | 58 +
datafusion/functions-window/src/row_number.rs | 183 ++
datafusion/functions/Cargo.toml | 22 +-
datafusion/functions/benches/concat.rs | 3 +-
.../functions/benches/{concat.rs => nullif.rs} | 29 +-
datafusion/functions/benches/pad.rs | 142 ++
.../benches/random.rs} | 44 +-
datafusion/functions/benches/regx.rs | 24 +-
datafusion/functions/benches/repeat.rs | 136 +
datafusion/functions/src/core/arrow_cast.rs | 785 +-----
datafusion/functions/src/core/coalesce.rs | 9 +-
datafusion/functions/src/core/expr_ext.rs | 2 +-
datafusion/functions/src/core/getfield.rs | 27 +-
datafusion/functions/src/core/mod.rs | 8 +
datafusion/functions/src/core/named_struct.rs | 15 +-
datafusion/functions/src/core/nullif.rs | 10 +-
datafusion/functions/src/core/nvl.rs | 7 +-
datafusion/functions/src/core/nvl2.rs | 3 +-
datafusion/functions/src/core/planner.rs | 33 +-
datafusion/functions/src/core/struct.rs | 17 +-
datafusion/functions/src/crypto/mod.rs | 1 +
datafusion/functions/src/datetime/common.rs | 60 +-
datafusion/functions/src/datetime/date_part.rs | 32 +-
datafusion/functions/src/datetime/date_trunc.rs | 25 +-
datafusion/functions/src/datetime/make_date.rs | 4 +-
datafusion/functions/src/datetime/mod.rs | 13 +-
datafusion/functions/src/datetime/to_date.rs | 4 +-
datafusion/functions/src/datetime/to_local_time.rs | 564 +++++
datafusion/functions/src/datetime/to_timestamp.rs | 21 +-
datafusion/functions/src/encoding/mod.rs | 1 +
datafusion/functions/src/lib.rs | 4 +-
datafusion/functions/src/macros.rs | 12 +-
datafusion/functions/src/math/abs.rs | 2 +-
datafusion/functions/src/math/bounds.rs | 108 +
datafusion/functions/src/math/log.rs | 141 +-
datafusion/functions/src/math/mod.rs | 306 ++-
datafusion/functions/src/math/monotonicity.rs | 22 +-
datafusion/functions/src/math/random.rs | 7 +-
datafusion/functions/src/math/round.rs | 2 +-
datafusion/functions/src/math/signum.rs | 215 ++
datafusion/functions/src/math/trunc.rs | 2 +-
datafusion/functions/src/regex/mod.rs | 2 +-
datafusion/functions/src/regex/regexplike.rs | 9 +-
datafusion/functions/src/regex/regexpmatch.rs | 13 +-
datafusion/functions/src/regex/regexpreplace.rs | 240 +-
datafusion/functions/src/string/ascii.rs | 118 +-
datafusion/functions/src/string/btrim.rs | 23 +-
datafusion/functions/src/string/common.rs | 99 +-
datafusion/functions/src/string/ends_with.rs | 33 +-
datafusion/functions/src/string/initcap.rs | 92 +-
datafusion/functions/src/string/levenshtein.rs | 37 +-
datafusion/functions/src/string/ltrim.rs | 20 +-
datafusion/functions/src/string/mod.rs | 3 +-
datafusion/functions/src/string/octet_length.rs | 35 +-
datafusion/functions/src/string/overlay.rs | 182 +-
datafusion/functions/src/string/repeat.rs | 104 +-
datafusion/functions/src/string/replace.rs | 86 +-
datafusion/functions/src/string/rtrim.rs | 20 +-
datafusion/functions/src/string/split_part.rs | 128 +-
datafusion/functions/src/string/starts_with.rs | 92 +-
.../functions/src/unicode/character_length.rs | 133 +-
datafusion/functions/src/unicode/find_in_set.rs | 67 +-
datafusion/functions/src/unicode/left.rs | 43 +-
datafusion/functions/src/unicode/lpad.rs | 655 +++--
datafusion/functions/src/unicode/mod.rs | 2 +-
datafusion/functions/src/unicode/reverse.rs | 106 +-
datafusion/functions/src/unicode/right.rs | 48 +-
datafusion/functions/src/unicode/rpad.rs | 260 +-
datafusion/functions/src/unicode/strpos.rs | 81 +-
datafusion/functions/src/unicode/substr.rs | 124 +-
datafusion/functions/src/unicode/substrindex.rs | 91 +-
datafusion/functions/src/unicode/translate.rs | 70 +-
datafusion/functions/src/utils.rs | 25 +-
datafusion/optimizer/README.md | 318 +--
.../optimizer/src/analyzer/count_wildcard_rule.rs | 48 +-
.../optimizer/src/analyzer/expand_wildcard_rule.rs | 304 +++
.../optimizer/src/analyzer/inline_table_scan.rs | 8 +-
datafusion/optimizer/src/analyzer/mod.rs | 17 +-
datafusion/optimizer/src/analyzer/subquery.rs | 13 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 326 ++-
.../optimizer/src/common_subexpr_eliminate.rs | 1060 ++++----
datafusion/optimizer/src/decorrelate.rs | 51 +-
.../src/decorrelate_predicate_subquery.rs | 6 +-
datafusion/optimizer/src/eliminate_cross_join.rs | 40 +-
datafusion/optimizer/src/eliminate_filter.rs | 11 +-
datafusion/optimizer/src/eliminate_join.rs | 8 +-
datafusion/optimizer/src/eliminate_limit.rs | 7 +-
datafusion/optimizer/src/eliminate_nested_union.rs | 27 +-
datafusion/optimizer/src/eliminate_one_union.rs | 4 +-
datafusion/optimizer/src/eliminate_outer_join.rs | 2 +-
.../optimizer/src/extract_equijoin_predicate.rs | 4 +-
datafusion/optimizer/src/lib.rs | 8 +-
.../optimizer/src/optimize_projections/mod.rs | 95 +-
.../src/optimize_projections/required_indices.rs | 2 +-
datafusion/optimizer/src/optimizer.rs | 8 +-
datafusion/optimizer/src/plan_signature.rs | 4 +-
.../optimizer/src/propagate_empty_relation.rs | 60 +-
datafusion/optimizer/src/push_down_filter.rs | 112 +-
datafusion/optimizer/src/push_down_limit.rs | 89 +-
.../optimizer/src/replace_distinct_aggregate.rs | 93 +-
.../optimizer/src/scalar_subquery_to_join.rs | 166 +-
.../src/simplify_expressions/expr_simplifier.rs | 84 +-
.../src/simplify_expressions/guarantees.rs | 2 +-
.../optimizer/src/simplify_expressions/regex.rs | 1 +
.../src/simplify_expressions/simplify_exprs.rs | 12 +-
.../optimizer/src/single_distinct_to_groupby.rs | 154 +-
datafusion/optimizer/src/test/mod.rs | 10 +-
.../optimizer/src/unwrap_cast_in_comparison.rs | 32 +-
.../optimizer/tests/optimizer_integration.rs | 63 +-
datafusion/physical-expr-common/Cargo.toml | 2 +-
.../physical-expr-common/src/aggregate/mod.rs | 552 ----
datafusion/physical-expr-common/src/binary_map.rs | 32 +-
.../physical-expr-common/src/binary_view_map.rs | 690 +++++
datafusion/physical-expr-common/src/datum.rs | 3 +-
.../physical-expr-common/src/expressions/column.rs | 137 -
datafusion/physical-expr-common/src/lib.rs | 8 +-
.../physical-expr-common/src/physical_expr.rs | 8 +-
datafusion/physical-expr-common/src/sort_expr.rs | 28 +-
datafusion/physical-expr-common/src/utils.rs | 35 +-
.../Cargo.toml | 16 +-
.../src/aggregate.rs | 500 ++++
.../src/lib.rs} | 6 +-
datafusion/physical-expr/Cargo.toml | 10 +
datafusion/physical-expr/benches/case_when.rs | 125 +
datafusion/physical-expr/benches/is_null.rs | 94 +
.../physical-expr/src/aggregate/array_agg.rs | 191 --
.../src/aggregate/array_agg_distinct.rs | 439 ----
datafusion/physical-expr/src/aggregate/build_in.rs | 306 ---
.../physical-expr/src/aggregate/moving_min_max.rs | 335 ---
datafusion/physical-expr/src/analysis.rs | 2 +-
datafusion/physical-expr/src/equivalence/class.rs | 4 +-
datafusion/physical-expr/src/equivalence/mod.rs | 4 +-
.../physical-expr/src/equivalence/properties.rs | 841 +++++--
datafusion/physical-expr/src/expressions/binary.rs | 245 +-
datafusion/physical-expr/src/expressions/case.rs | 205 +-
.../src/expressions/cast.rs | 10 +-
datafusion/physical-expr/src/expressions/column.rs | 104 +-
.../physical-expr/src/expressions/in_list.rs | 10 +-
.../physical-expr/src/expressions/is_not_null.rs | 4 +-
.../physical-expr/src/expressions/is_null.rs | 10 +
.../src/expressions/literal.rs | 7 +-
datafusion/physical-expr/src/expressions/mod.rs | 177 +-
.../physical-expr/src/expressions/try_cast.rs | 2 +-
.../expressions/{column.rs => unknown_column.rs} | 60 +-
.../physical-expr/src/intervals/cp_solver.rs | 2 +-
datafusion/physical-expr/src/lib.rs | 28 +-
datafusion/physical-expr/src/partitioning.rs | 6 +-
datafusion/physical-expr/src/physical_expr.rs | 2 +-
datafusion/physical-expr/src/planner.rs | 22 +-
datafusion/physical-expr/src/scalar_function.rs | 45 +-
datafusion/physical-expr/src/utils/guarantee.rs | 15 +-
datafusion/physical-expr/src/utils/mod.rs | 7 +-
datafusion/physical-expr/src/window/aggregate.rs | 11 +-
datafusion/physical-expr/src/window/built_in.rs | 1 -
datafusion/physical-expr/src/window/mod.rs | 1 -
datafusion/physical-expr/src/window/row_number.rs | 166 --
.../physical-expr/src/window/sliding_aggregate.rs | 5 +-
datafusion/physical-expr/src/window/window_expr.rs | 6 -
.../Cargo.toml | 19 +-
.../README.md | 10 +-
.../physical-optimizer/src/aggregate_statistics.rs | 300 +++
.../src/lib.rs | 10 +-
.../physical-optimizer/src/limit_pushdown.rs | 327 +++
.../src/limited_distinct_aggregation.rs | 192 ++
datafusion/physical-optimizer/src/optimizer.rs | 48 +
.../src}/output_requirements.rs | 26 +-
datafusion/physical-plan/Cargo.toml | 6 +-
.../src/aggregates/group_values/bytes_view.rs | 129 +
.../src/aggregates/group_values/mod.rs | 33 +-
.../src/aggregates/group_values/row.rs | 21 +-
datafusion/physical-plan/src/aggregates/mod.rs | 452 ++--
.../physical-plan/src/aggregates/no_grouping.rs | 1 +
.../physical-plan/src/aggregates/order/full.rs | 8 +-
.../physical-plan/src/aggregates/order/mod.rs | 10 +-
.../physical-plan/src/aggregates/order/partial.rs | 2 +-
.../physical-plan/src/aggregates/row_hash.rs | 251 +-
.../physical-plan/src/aggregates/topk/heap.rs | 86 +-
datafusion/physical-plan/src/analyze.rs | 2 +-
datafusion/physical-plan/src/coalesce/mod.rs | 588 +++++
datafusion/physical-plan/src/coalesce_batches.rs | 327 ++-
.../physical-plan/src/coalesce_partitions.rs | 4 +
datafusion/physical-plan/src/common.rs | 356 +--
datafusion/physical-plan/src/display.rs | 2 +-
.../src/{lib.rs => execution_plan.rs} | 260 +-
datafusion/physical-plan/src/filter.rs | 64 +-
datafusion/physical-plan/src/insert.rs | 77 +-
datafusion/physical-plan/src/joins/cross_join.rs | 43 +-
datafusion/physical-plan/src/joins/hash_join.rs | 96 +-
.../physical-plan/src/joins/nested_loop_join.rs | 113 +-
.../physical-plan/src/joins/sort_merge_join.rs | 472 +++-
.../physical-plan/src/joins/symmetric_hash_join.rs | 6 +-
datafusion/physical-plan/src/joins/utils.rs | 24 +-
datafusion/physical-plan/src/lib.rs | 985 +-------
datafusion/physical-plan/src/limit.rs | 153 +-
datafusion/physical-plan/src/projection.rs | 4 +
datafusion/physical-plan/src/repartition/mod.rs | 26 +-
datafusion/physical-plan/src/sorts/partial_sort.rs | 4 +
datafusion/physical-plan/src/sorts/sort.rs | 48 +-
.../src/sorts/sort_preserving_merge.rs | 19 +
datafusion/physical-plan/src/spill.rs | 186 ++
datafusion/physical-plan/src/streaming.rs | 19 +-
datafusion/physical-plan/src/test/exec.rs | 2 +-
datafusion/physical-plan/src/topk/mod.rs | 8 +-
datafusion/physical-plan/src/union.rs | 123 +-
datafusion/physical-plan/src/unnest.rs | 3 +-
.../src/windows/bounded_window_agg_exec.rs | 29 +-
datafusion/physical-plan/src/windows/mod.rs | 96 +-
.../mod.rs => physical-plan/src/windows/utils.rs} | 32 +-
.../physical-plan/src/windows/window_agg_exec.rs | 21 +-
datafusion/physical-plan/src/work_table.rs | 2 +-
.../proto-common/proto/datafusion_common.proto | 23 +-
datafusion/proto-common/src/from_proto/mod.rs | 43 +-
datafusion/proto-common/src/generated/pbjson.rs | 1289 ++++++----
datafusion/proto-common/src/generated/prost.rs | 77 +-
datafusion/proto-common/src/lib.rs | 2 +
datafusion/proto-common/src/to_proto/mod.rs | 47 +-
datafusion/proto/gen/src/main.rs | 7 +-
datafusion/proto/proto/datafusion.proto | 93 +-
.../proto/src/generated/datafusion_proto_common.rs | 77 +-
datafusion/proto/src/generated/pbjson.rs | 547 ++--
datafusion/proto/src/generated/prost.rs | 129 +-
datafusion/proto/src/lib.rs | 4 +-
datafusion/proto/src/logical_plan/file_formats.rs | 537 +++-
datafusion/proto/src/logical_plan/from_proto.rs | 138 +-
datafusion/proto/src/logical_plan/mod.rs | 57 +-
datafusion/proto/src/logical_plan/to_proto.rs | 105 +-
datafusion/proto/src/physical_plan/from_proto.rs | 28 +-
datafusion/proto/src/physical_plan/mod.rs | 147 +-
datafusion/proto/src/physical_plan/to_proto.rs | 198 +-
datafusion/proto/tests/cases/mod.rs | 99 +
.../proto/tests/cases/roundtrip_logical_plan.rs | 524 ++--
.../proto/tests/cases/roundtrip_physical_plan.rs | 577 +++--
datafusion/proto/tests/cases/serialize.rs | 2 +-
datafusion/sql/Cargo.toml | 1 +
datafusion/sql/examples/sql.rs | 27 +-
datafusion/sql/src/cte.rs | 4 +-
datafusion/sql/src/expr/function.rs | 104 +-
datafusion/sql/src/expr/identifier.rs | 119 +-
datafusion/sql/src/expr/mod.rs | 149 +-
datafusion/sql/src/expr/order_by.rs | 19 +-
datafusion/sql/src/expr/substring.rs | 2 +-
datafusion/sql/src/expr/value.rs | 8 +-
datafusion/sql/src/lib.rs | 12 +-
datafusion/sql/src/parser.rs | 43 +-
datafusion/sql/src/planner.rs | 119 +-
datafusion/sql/src/query.rs | 24 +-
datafusion/sql/src/relation/join.rs | 51 +-
datafusion/sql/src/relation/mod.rs | 94 +-
datafusion/sql/src/select.rs | 295 ++-
datafusion/sql/src/statement.rs | 235 +-
datafusion/sql/src/unparser/ast.rs | 24 +-
datafusion/sql/src/unparser/dialect.rs | 364 ++-
datafusion/sql/src/unparser/expr.rs | 1082 ++++++--
datafusion/sql/src/unparser/mod.rs | 75 +-
datafusion/sql/src/unparser/plan.rs | 291 ++-
datafusion/sql/src/unparser/rewrite.rs | 177 +-
datafusion/sql/src/unparser/utils.rs | 6 +-
datafusion/sql/src/utils.rs | 78 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 366 ++-
datafusion/sql/tests/common/mod.rs | 55 +-
datafusion/sql/tests/sql_integration.rs | 819 +++---
datafusion/sqllogictest/Cargo.toml | 2 +-
datafusion/sqllogictest/README.md | 4 +-
datafusion/sqllogictest/bin/sqllogictests.rs | 30 +-
.../src/engines/datafusion_engine/normalize.rs | 9 +-
.../src/engines/datafusion_engine/runner.rs | 10 +-
datafusion/sqllogictest/src/test_context.rs | 7 +-
datafusion/sqllogictest/test_files/aggregate.slt | 457 +++-
.../test_files/aggregate_skip_partial.slt | 493 ++++
.../sqllogictest/test_files/aggregates_topk.slt | 132 +-
datafusion/sqllogictest/test_files/array.slt | 86 +-
.../sqllogictest/test_files/arrow_typeof.slt | 20 +-
datafusion/sqllogictest/test_files/binary.slt | 2 +-
datafusion/sqllogictest/test_files/binary_view.slt | 202 ++
datafusion/sqllogictest/test_files/case.slt | 110 +
datafusion/sqllogictest/test_files/clickbench.slt | 18 +
datafusion/sqllogictest/test_files/coalesce.slt | 14 +-
datafusion/sqllogictest/test_files/copy.slt | 62 +-
.../sqllogictest/test_files/count_star_rule.slt | 103 +
.../test_files/create_external_table.slt | 14 +-
datafusion/sqllogictest/test_files/cse.slt | 88 +-
datafusion/sqllogictest/test_files/csv_files.slt | 63 +-
datafusion/sqllogictest/test_files/ddl.slt | 7 +-
datafusion/sqllogictest/test_files/encoding.slt | 2 +-
datafusion/sqllogictest/test_files/errors.slt | 4 -
datafusion/sqllogictest/test_files/explain.slt | 45 +-
datafusion/sqllogictest/test_files/expr.slt | 92 +-
datafusion/sqllogictest/test_files/functions.slt | 267 +-
datafusion/sqllogictest/test_files/group_by.slt | 346 ++-
.../sqllogictest/test_files/information_schema.slt | 91 +-
datafusion/sqllogictest/test_files/insert.slt | 6 +-
.../sqllogictest/test_files/insert_to_external.slt | 4 +-
datafusion/sqllogictest/test_files/interval.slt | 2 +-
datafusion/sqllogictest/test_files/join.slt | 192 ++
.../test_files/join_disable_repartition_joins.slt | 64 +-
datafusion/sqllogictest/test_files/joins.slt | 211 +-
datafusion/sqllogictest/test_files/limit.slt | 30 +-
datafusion/sqllogictest/test_files/map.slt | 474 ++++
datafusion/sqllogictest/test_files/math.slt | 8 +-
datafusion/sqllogictest/test_files/misc.slt | 18 +
datafusion/sqllogictest/test_files/nvl.slt | 2 +-
datafusion/sqllogictest/test_files/options.slt | 2 +-
datafusion/sqllogictest/test_files/order.slt | 168 +-
datafusion/sqllogictest/test_files/parquet.slt | 8 +-
.../test_files/parquet_sorted_statistics.slt | 6 +-
datafusion/sqllogictest/test_files/predicates.slt | 4 +-
datafusion/sqllogictest/test_files/projection.slt | 6 +-
.../sqllogictest/test_files/push_down_filter.slt | 40 +-
datafusion/sqllogictest/test_files/regexp.slt | 146 +-
datafusion/sqllogictest/test_files/repartition.slt | 2 +-
datafusion/sqllogictest/test_files/scalar.slt | 56 +-
datafusion/sqllogictest/test_files/select.slt | 152 +-
.../sqllogictest/test_files/sort_merge_join.slt | 32 +-
datafusion/sqllogictest/test_files/string_view.slt | 1222 +++++++++
datafusion/sqllogictest/test_files/strings.slt | 2 +-
datafusion/sqllogictest/test_files/struct.slt | 58 +-
.../sqllogictest/test_files/subquery_sort.slt | 149 ++
datafusion/sqllogictest/test_files/timestamps.slt | 337 ++-
datafusion/sqllogictest/test_files/topk.slt | 10 +-
.../test_files/tpch/create_tables.slt.part | 16 +-
.../sqllogictest/test_files/tpch/q10.slt.part | 110 +-
.../sqllogictest/test_files/tpch/q11.slt.part | 156 +-
.../sqllogictest/test_files/tpch/q13.slt.part | 68 +-
.../sqllogictest/test_files/tpch/q14.slt.part | 33 +-
.../sqllogictest/test_files/tpch/q15.slt.part | 12 +-
.../sqllogictest/test_files/tpch/q16.slt.part | 96 +-
.../sqllogictest/test_files/tpch/q2.slt.part | 236 +-
.../sqllogictest/test_files/tpch/q3.slt.part | 94 +-
.../sqllogictest/test_files/tpch/q9.slt.part | 146 +-
.../sqllogictest/test_files/type_coercion.slt | 176 ++
datafusion/sqllogictest/test_files/union.slt | 255 +-
datafusion/sqllogictest/test_files/unnest.slt | 164 +-
datafusion/sqllogictest/test_files/update.slt | 6 +-
datafusion/sqllogictest/test_files/window.slt | 751 +++---
datafusion/substrait/src/extensions.rs | 157 ++
datafusion/substrait/src/lib.rs | 1 +
datafusion/substrait/src/logical_plan/consumer.rs | 612 +++--
datafusion/substrait/src/logical_plan/producer.rs | 771 +++---
datafusion/substrait/src/variation_const.rs | 24 +-
.../substrait/tests/cases/consumer_integration.rs | 521 +++-
datafusion/substrait/tests/cases/function_test.rs | 2 +-
datafusion/substrait/tests/cases/logical_plans.rs | 6 +-
.../tests/cases/roundtrip_logical_plan.rs | 184 +-
datafusion/substrait/tests/cases/serialize.rs | 4 +-
.../testdata/tpch_substrait_plans/query_10.json | 1257 ++++++++++
.../testdata/tpch_substrait_plans/query_11.json | 1059 ++++++++
.../testdata/tpch_substrait_plans/query_13.json | 624 +++++
.../testdata/tpch_substrait_plans/query_14.json | 924 +++++++
.../testdata/tpch_substrait_plans/query_16.json | 1175 +++++++++
.../testdata/tpch_substrait_plans/query_18.json | 1128 +++++++++
.../testdata/tpch_substrait_plans/query_19.json | 2386 ++++++++++++++++++
.../testdata/tpch_substrait_plans/query_20.json | 1273 ++++++++++
.../testdata/tpch_substrait_plans/query_21.json | 1493 +++++++++++
.../testdata/tpch_substrait_plans/query_22.json | 2034 +++++++++++++++
.../testdata/tpch_substrait_plans/query_6.json | 585 +++++
dev/changelog/40.0.0.md | 371 +++
dev/changelog/41.0.0.md | 363 +++
dev/depcheck/Cargo.toml | 2 +-
dev/depcheck/src/main.rs | 6 +-
dev/release/README.md | 42 +-
dev/release/crate-deps.dot | 91 -
dev/release/crate-deps.svg | 445 ----
dev/release/release-crates.sh | 45 -
dev/update_datafusion_versions.py | 2 +-
docs/source/contributor-guide/architecture.md | 74 +
docs/source/contributor-guide/communication.md | 25 +-
docs/source/contributor-guide/howtos.md | 2 +-
docs/source/contributor-guide/index.md | 16 -
docs/source/contributor-guide/inviting.md | 4 +-
docs/source/contributor-guide/quarterly_roadmap.md | 96 -
docs/source/contributor-guide/roadmap.md | 81 +
.../contributor-guide/specification/index.rst | 10 +
docs/source/contributor-guide/testing.md | 15 +-
docs/source/index.rst | 12 +-
.../library-user-guide/building-logical-plans.md | 187 +-
.../library-user-guide/custom-table-providers.md | 4 +-
docs/source/library-user-guide/index.md | 21 +-
.../source/library-user-guide/query-optimizer.md | 0
.../library-user-guide/using-the-dataframe-api.md | 303 ++-
.../source/library-user-guide/using-the-sql-api.md | 21 +-
docs/source/user-guide/configs.md | 62 +-
docs/source/user-guide/crate-configuration.md | 188 ++
docs/source/user-guide/dataframe.md | 1 +
docs/source/user-guide/example-usage.md | 131 +-
docs/source/user-guide/expressions.md | 4 +-
docs/source/user-guide/introduction.md | 1 +
docs/source/user-guide/sql/scalar_functions.md | 160 +-
docs/src/library_logical_plan.rs | 78 -
pre-commit.sh | 8 +-
694 files changed, 58355 insertions(+), 28895 deletions(-)
create mode 100644 .github/workflows/large_files.yml
create mode 100644 datafusion-examples/examples/planner_api.rs
copy {docs => datafusion/catalog}/Cargo.toml (61%)
create mode 100644 datafusion/catalog/src/catalog.rs
rename datafusion/{physical-expr-common/src/expressions/mod.rs =>
catalog/src/lib.rs} (86%)
rename datafusion/{core/src/catalog => catalog/src}/schema.rs (95%)
create mode 100644 datafusion/catalog/src/session.rs
copy datafusion/{core/src/datasource/provider.rs => catalog/src/table.rs} (86%)
copy datafusion/{core/src/execution/mod.rs => common/src/scalar/consts.rs}
(50%)
copy datafusion/{physical-expr-common/src/aggregate/count_distinct/mod.rs =>
common/src/utils/expr.rs} (78%)
create mode 100644 datafusion/core/benches/map_query_sql.rs
delete mode 100644 datafusion/core/benches/parquet_statistic.rs
rename datafusion/core/src/{catalog => catalog_common}/information_schema.rs
(99%)
rename datafusion/core/src/{catalog => catalog_common}/listing_schema.rs (98%)
rename datafusion/core/src/{catalog => catalog_common}/memory.rs (97%)
rename datafusion/core/src/{catalog => catalog_common}/mod.rs (59%)
rename datafusion/core/src/datasource/physical_plan/parquet/{row_groups.rs =>
row_group_filter.rs} (98%)
delete mode 100644
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs
create mode 100644 datafusion/core/src/execution/session_state_defaults.rs
delete mode 100644
datafusion/core/src/physical_optimizer/aggregate_statistics.rs
delete mode 100644
datafusion/core/src/physical_optimizer/limited_distinct_aggregation.rs
create mode 100644 datafusion/core/tests/data/newlines_in_values.csv
rename datafusion/core/tests/{optimizer_integration.rs => optimizer/mod.rs}
(98%)
delete mode 100644 datafusion/core/tests/parquet/arrow_statistics.rs
create mode 100644
datafusion/core/tests/physical_optimizer/aggregate_statistics.rs
create mode 100644 datafusion/core/tests/physical_optimizer/limit_pushdown.rs
create mode 100644
datafusion/core/tests/physical_optimizer/limited_distinct_aggregation.rs
rename datafusion/{physical-expr/src/aggregate/stats.rs =>
core/tests/physical_optimizer/mod.rs} (89%)
create mode 100644 datafusion/core/tests/physical_optimizer/test_util.rs
rename datafusion/core/tests/{ => sql}/path_partition.rs (100%)
rename {docs => datafusion/expr-common}/Cargo.toml (74%)
rename datafusion/{expr => expr-common}/src/accumulator.rs (54%)
rename datafusion/{expr => expr-common}/src/columnar_value.rs (100%)
rename datafusion/{expr => expr-common}/src/groups_accumulator.rs (68%)
rename datafusion/{expr => expr-common}/src/interval_arithmetic.rs (98%)
copy datafusion/{physical-expr/src/aggregate/groups_accumulator/mod.rs =>
expr-common/src/lib.rs} (56%)
rename datafusion/{expr => expr-common}/src/operator.rs (64%)
rename datafusion/{expr => expr-common}/src/signature.rs (95%)
rename datafusion/{expr => expr-common}/src/sort_properties.rs (100%)
rename docs/src/lib.rs => datafusion/expr-common/src/type_coercion.rs (95%)
rename datafusion/{expr => expr-common}/src/type_coercion/aggregates.rs (82%)
rename datafusion/{expr => expr-common}/src/type_coercion/binary.rs (86%)
delete mode 100644 datafusion/expr/src/aggregate_function.rs
create mode 100644 datafusion/expr/src/operation.rs
create mode 100644 datafusion/expr/src/window_function.rs
copy datafusion/{functions-aggregate => functions-aggregate-common}/Cargo.toml
(80%)
create mode 100644 datafusion/functions-aggregate-common/src/accumulator.rs
create mode 100644 datafusion/functions-aggregate-common/src/aggregate.rs
rename datafusion/{physical-expr-common/src/aggregate/count_distinct/mod.rs =>
functions-aggregate-common/src/aggregate/count_distinct.rs} (95%)
rename datafusion/{physical-expr-common =>
functions-aggregate-common}/src/aggregate/count_distinct/bytes.rs (61%)
rename datafusion/{physical-expr-common =>
functions-aggregate-common}/src/aggregate/count_distinct/native.rs (98%)
rename datafusion/{physical-expr/src/aggregate/groups_accumulator/adapter.rs
=> functions-aggregate-common/src/aggregate/groups_accumulator.rs} (97%)
rename datafusion/{physical-expr-common =>
functions-aggregate-common}/src/aggregate/groups_accumulator/accumulate.rs (98%)
rename datafusion/{physical-expr-common =>
functions-aggregate-common}/src/aggregate/groups_accumulator/bool_op.rs (77%)
create mode 100644
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/nulls.rs
rename datafusion/{physical-expr-common =>
functions-aggregate-common}/src/aggregate/groups_accumulator/prim_op.rs (65%)
rename datafusion/{physical-expr/src/aggregate/groups_accumulator/mod.rs =>
functions-aggregate-common/src/lib.rs} (59%)
rename datafusion/{physical-expr-common/src/aggregate =>
functions-aggregate-common/src}/merge_arrays.rs (100%)
create mode 100644 datafusion/functions-aggregate-common/src/order.rs
rename datafusion/{physical-expr-common/src/aggregate =>
functions-aggregate-common/src}/stats.rs (100%)
rename datafusion/{physical-expr-common/src/aggregate =>
functions-aggregate-common/src}/tdigest.rs (93%)
rename datafusion/{physical-expr-common/src/aggregate =>
functions-aggregate-common/src}/utils.rs (97%)
create mode 100644 datafusion/functions-aggregate/COMMENTS.md
create mode 100644 datafusion/functions-aggregate/benches/count.rs
create mode 100644 datafusion/functions-aggregate/benches/sum.rs
rename datafusion/{physical-expr/src/aggregate/array_agg_ordered.rs =>
functions-aggregate/src/array_agg.rs} (63%)
rename datafusion/{physical-expr/src/aggregate =>
functions-aggregate/src}/min_max.rs (57%)
rename datafusion/{functions-array => functions-nested}/Cargo.toml (87%)
copy datafusion/{functions-array => functions-nested}/README.md (87%)
copy datafusion/{functions-array =>
functions-nested}/benches/array_expression.rs (95%)
create mode 100644 datafusion/functions-nested/benches/map.rs
rename datafusion/{functions-array => functions-nested}/src/array_has.rs (94%)
rename datafusion/{functions-array => functions-nested}/src/cardinality.rs
(72%)
rename datafusion/{functions-array => functions-nested}/src/concat.rs (99%)
rename datafusion/{functions-array => functions-nested}/src/dimension.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/empty.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/except.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/expr_ext.rs (90%)
rename datafusion/{functions-array => functions-nested}/src/extract.rs (93%)
rename datafusion/{functions-array => functions-nested}/src/flatten.rs (97%)
rename datafusion/{functions-array => functions-nested}/src/length.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/lib.rs (89%)
rename datafusion/{functions-array => functions-nested}/src/macros.rs (96%)
rename datafusion/{functions-array => functions-nested}/src/make_array.rs
(100%)
create mode 100644 datafusion/functions-nested/src/map.rs
create mode 100644 datafusion/functions-nested/src/map_extract.rs
rename datafusion/{functions-array => functions-nested}/src/planner.rs (79%)
rename datafusion/{functions-array => functions-nested}/src/position.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/range.rs (71%)
rename datafusion/{functions-array => functions-nested}/src/remove.rs (99%)
rename datafusion/{functions-array => functions-nested}/src/repeat.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/replace.rs (100%)
rename datafusion/{functions-array => functions-nested}/src/resize.rs (96%)
rename datafusion/{functions-array => functions-nested}/src/reverse.rs (98%)
rename datafusion/{functions-array => functions-nested}/src/set_ops.rs (97%)
rename datafusion/{functions-array => functions-nested}/src/sort.rs (99%)
rename datafusion/{functions-array => functions-nested}/src/string.rs (99%)
rename datafusion/{functions-array => functions-nested}/src/utils.rs (94%)
copy datafusion/{functions-aggregate => functions-window}/Cargo.toml (82%)
copy datafusion/{functions-array => functions-window}/README.md (82%)
create mode 100644 datafusion/functions-window/src/lib.rs
create mode 100644 datafusion/functions-window/src/row_number.rs
copy datafusion/functions/benches/{concat.rs => nullif.rs} (60%)
create mode 100644 datafusion/functions/benches/pad.rs
rename datafusion/{functions-array/benches/array_expression.rs =>
functions/benches/random.rs} (50%)
create mode 100644 datafusion/functions/benches/repeat.rs
create mode 100644 datafusion/functions/src/datetime/to_local_time.rs
create mode 100644 datafusion/functions/src/math/bounds.rs
create mode 100644 datafusion/functions/src/math/signum.rs
create mode 100644 datafusion/optimizer/src/analyzer/expand_wildcard_rule.rs
delete mode 100644 datafusion/physical-expr-common/src/aggregate/mod.rs
create mode 100644 datafusion/physical-expr-common/src/binary_view_map.rs
delete mode 100644 datafusion/physical-expr-common/src/expressions/column.rs
copy datafusion/{functions-aggregate =>
physical-expr-functions-aggregate}/Cargo.toml (76%)
create mode 100644
datafusion/physical-expr-functions-aggregate/src/aggregate.rs
rename
datafusion/{physical-expr-common/src/aggregate/groups_accumulator/mod.rs =>
physical-expr-functions-aggregate/src/lib.rs} (87%)
create mode 100644 datafusion/physical-expr/benches/case_when.rs
create mode 100644 datafusion/physical-expr/benches/is_null.rs
delete mode 100644 datafusion/physical-expr/src/aggregate/array_agg.rs
delete mode 100644 datafusion/physical-expr/src/aggregate/array_agg_distinct.rs
delete mode 100644 datafusion/physical-expr/src/aggregate/build_in.rs
delete mode 100644 datafusion/physical-expr/src/aggregate/moving_min_max.rs
rename datafusion/{physical-expr-common =>
physical-expr}/src/expressions/cast.rs (98%)
rename datafusion/{physical-expr-common =>
physical-expr}/src/expressions/literal.rs (95%)
copy datafusion/physical-expr/src/expressions/{column.rs => unknown_column.rs}
(51%)
delete mode 100644 datafusion/physical-expr/src/window/row_number.rs
copy datafusion/{physical-expr-common => physical-optimizer}/Cargo.toml (74%)
rename datafusion/{functions-array => physical-optimizer}/README.md (67%)
create mode 100644 datafusion/physical-optimizer/src/aggregate_statistics.rs
copy datafusion/{common-runtime => physical-optimizer}/src/lib.rs (73%)
create mode 100644 datafusion/physical-optimizer/src/limit_pushdown.rs
create mode 100644
datafusion/physical-optimizer/src/limited_distinct_aggregation.rs
create mode 100644 datafusion/physical-optimizer/src/optimizer.rs
rename datafusion/{core/src/physical_optimizer =>
physical-optimizer/src}/output_requirements.rs (94%)
create mode 100644
datafusion/physical-plan/src/aggregates/group_values/bytes_view.rs
create mode 100644 datafusion/physical-plan/src/coalesce/mod.rs
copy datafusion/physical-plan/src/{lib.rs => execution_plan.rs} (89%)
create mode 100644 datafusion/physical-plan/src/spill.rs
rename datafusion/{physical-expr/src/aggregate/mod.rs =>
physical-plan/src/windows/utils.rs} (57%)
create mode 100644
datafusion/sqllogictest/test_files/aggregate_skip_partial.slt
create mode 100644 datafusion/sqllogictest/test_files/binary_view.slt
create mode 100644 datafusion/sqllogictest/test_files/case.slt
create mode 100644 datafusion/sqllogictest/test_files/count_star_rule.slt
create mode 100644 datafusion/sqllogictest/test_files/string_view.slt
create mode 100644 datafusion/sqllogictest/test_files/subquery_sort.slt
create mode 100644 datafusion/substrait/src/extensions.rs
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_10.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_11.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_13.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_14.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_16.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_18.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_19.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_20.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_21.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_22.json
create mode 100644
datafusion/substrait/tests/testdata/tpch_substrait_plans/query_6.json
create mode 100644 dev/changelog/40.0.0.md
create mode 100644 dev/changelog/41.0.0.md
delete mode 100644 dev/release/crate-deps.dot
delete mode 100644 dev/release/crate-deps.svg
delete mode 100644 dev/release/release-crates.sh
delete mode 100644 docs/source/contributor-guide/quarterly_roadmap.md
copy datafusion/optimizer/README.md =>
docs/source/library-user-guide/query-optimizer.md (100%)
create mode 100644 docs/source/user-guide/crate-configuration.md
delete mode 100644 docs/src/library_logical_plan.rs
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
