vranes opened a new pull request, #59648: URL: https://github.com/apache/spark/pull/59648
### What changes were proposed in this pull request? Add DataFrame APIs for the BIN BY relation operator: - A Scala/Java builder with range, bin width, alignment, distribution, and output-name settings, ending in `build()`. - A PySpark `DataFrame.binBy(...)` method for classic Spark and Spark Connect. - A Spark Connect relation with Scala/Python client and server support. Resolve column expressions through the normal analyzer before BIN BY validation so DataFrame column references retain their identity. Add API documentation, validation tests, and Connect protocol goldens. JIRA: [SPARK-60391](https://issues.apache.org/jira/browse/SPARK-60391) ### Why are the changes needed? DataFrame users need to split timestamp ranges into fixed-width bins and distribute measurements across the overlapping bins. These APIs let them compose BIN BY with other DataFrame operations without registering a view and writing a SQL BIN BY clause. ### Does this PR introduce _any_ user-facing change? Yes. The new APIs expose BIN BY to Scala, Java, and PySpark users, including Spark Connect. Scala: ```scala import org.apache.spark.sql.functions.{col, expr} df.binBy .range(col("start"), col("end")) .binWidth(expr("INTERVAL 5 MINUTES")) .distributeUniform(col("value")) .build() ``` PySpark: ```python from pyspark.sql import functions as sf df.binBy("start", "end", sf.expr("INTERVAL 5 MINUTES"), ["value"]) ``` The result scales the selected FLOAT/DOUBLE values by each bin's overlap ratio and appends `bin_start`, `bin_end`, and `bin_distribute_ratio`. Users can set an alignment origin and rename the appended columns. The APIs use the existing internal, default-off `spark.sql.binByRelationOperator.enabled` gate. It must be enabled in the session to run these examples. ### How was this patch tested? Added coverage for SQL/API equivalence, optional settings, invalid inputs, bound and nested column references, and Spark Connect plan serialization. The following focused local checks passed before the final rebase onto master: - `ResolveBinBySuite` and `BinBySuite`. - `JavaDataFrameSuite`, including `testBinBy`. - BIN BY SQL result and analyzer goldens in `SQLQueryTestSuite`. - BIN BY cases in `ClientDatasetSuite`, `PlanGenerationTestSuite`, `ProtoToParsedPlanTestSuite`, and `SparkConnectProtoSuite`. - PySpark BIN BY tests in classic, Connect parity, and plan-only suites. - Module-scoped Scala formatting checks, Python Ruff checks, and `git diff --check`. After the rebase, `git range-diff` confirms that the patch is unchanged and `git diff --check` passes. The tests have not been rerun on the new base. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex CLI 0.163.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
