timsaucer opened a new pull request, #1763: URL: https://github.com/apache/datafusion-python/pull/1763
# Which issue does this PR close? No issue filed. The gaps were found by auditing the whole Python API against upstream DataFusion 55.1.0 with the `check-upstream` skill. Gaps that already have open issues (#1571–#1575, #1577, #1668, #1669) are deliberately left out. # Rationale for this change Several upstream functions, arguments, and DataFrame methods were not reachable from Python, and the audit turned up two bugs along the way: `F.mean(x, filter=...)` always raised `TypeError`, and chaining a builder method onto an aggregate or window function silently discarded the options the wrapper had already applied. # What changes are included in this PR? One commit per wave: - **New scalar and aggregate functions:** `any_value`; `array_add`, `array_subtract`, `array_scale`, `array_sum`, `array_avg`, `array_product`, `array_first` (each with its `list_*` alias); `input_file_name`, `file_row_index`; aliases `rand`, `substring_index`. - **New Spark functions:** `monthname`, `weekday`, `atan2`, `hypot`, `pow`/`power`, `quote`, `concat_ws`, `substr` (optional `len`), and pyspark aliases `getbit`, `dateadd`, `datediff`, `datepart`, `sha`, `ceiling`, `printf`, `char_length`, `character_length`. - **Optional arguments upstream already supports:** `distinct=` on `bit_and`, `bit_or`, `mean`, `percentile_cont`, `quantile_cont`, `string_agg`; `characters=` on `btrim`/`ltrim`/`rtrim`/`trim`; `null_string=` on `array_to_string` and aliases; `length=` on `substr`; `null_treatment=` on `lead`/`lag`; `show_statistics`, `analyze_level`, `analyze_categories` on `DataFrame.explain`, with new `ExplainAnalyzeLevel` and `ExplainMetricCategory` enums. - **New DataFrame methods:** `fill_nan`, `to_string`. - **FFI:** `ScalarUDF` and `WindowUDF` accept a bare PyCapsule, as `AggregateUDF` already did (#1277). `TableProviderFactory` is added to `datafusion.catalog.__all__`. - **Fixes:** `mean` no longer passes `filter` into `avg`'s `distinct` slot. Builder methods (`.filter()`, `.order_by()`, `.distinct()`, `.partition_by()`, ...) now start from the expression's existing options instead of an empty builder, so e.g. `string_agg(..., order_by=...).distinct().build()` keeps its ordering. A window frame equal to the default for its order-by is re-derived from the final order-by. - **Docs:** the aggregations guide listed `regr_slope` twice and omitted `regr_sxy`; `any_value` is now listed. Every new function has a doctest and pytest coverage. Bare-capsule imports are tested in `examples/datafusion-ffi-example`, which also covers the previously untested `AggregateUDF` capsule path. # Are there any user-facing changes? Yes: new functions, arguments, methods, and enums as listed above. There are two breaking changes, both documented in the 55.0.0 section of `docs/source/user-guide/upgrade-guides.md`: - `distinct` is inserted before `filter` in `bit_and`, `bit_or`, `mean`, `percentile_cont`, `quantile_cont`, and `string_agg`, matching what `sum` and `avg` did in 54.0.0. Code passing `filter` (or `order_by` for `string_agg`) positionally must pass it by keyword. `mean` with a filter never worked, so no working code breaks there. - `spark.last_day`'s parameter is renamed from `col` to `date` to match pyspark. Positional calls are unaffected. The builder fix changes results for code that chained builder methods onto a configured function; that code was previously getting silently wrong results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
