timsaucer opened a new pull request, #1561: URL: https://github.com/apache/datafusion-python/pull/1561
# Which issue does this PR close? N/A — no existing tracking issue. This exposes the Python-side API for lambda and higher-order array functions added upstream in apache/datafusion#21172. # Rationale for this change DataFusion gained lambda expressions and higher-order array functions (apache/datafusion#21172). The Python binding already had the read-only `Lambda`, `HigherOrderFunction`, and `LambdaVariable` wrapper classes, but offered no way to *construct* a lambda or *call* the higher-order functions. This PR adds a Pythonic API for both. # What changes are included in this PR? Rust: new `lambda_`, `lambda_var`, `array_transform`, and `array_any_match` pyfunctions, plus a `ResolveLambdaVariables` analyzer rule. The rule is needed because plans assembled through the expression builder emit unresolved lambda variables, and `LogicalPlan::resolve_lambda_variables()` is never called automatically; the rule resolves them before optimization. Python: `array_transform` and `array_any_match` (with `list_transform`, `any_match`, and `list_any_match` aliases) accept either a Python callable or an explicit lambda built with `lambda_` / `lambda_var`. Callables are introspected so their parameter names become the lambda parameters, which keeps the names readable in displayed plans. Tests (`python/tests/test_lambda.py`) and documentation (the expressions user guide and the agent skill) are included. # Are there any user-facing changes? New public functions: `lambda_`, `lambda_var`, `array_transform`, `list_transform`, `array_any_match`, `any_match`, and `list_any_match`. There are no breaking changes. Two limitations are documented for this initial version: lambda expressions cannot be serialized (`Expr.to_bytes` and pickle raise `Lambda not implemented`, matching the upstream proto limitation), and SQL lambda arrow syntax (`x -> ...`) is only parsed under the DuckDB dialect. The Python expression builder works regardless of dialect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
