zhengruifeng opened a new pull request, #57806: URL: https://github.com/apache/spark/pull/57806
### What changes were proposed in this pull request? For batch `FPGrowthModel.transform` calls, this PR replaces the manually collected association-rule array and user-defined function with DataFrame operations. It joins input transactions with matching rules, aggregates consequents for each input row, and uses `array_except` to exclude items already present in the transaction. The join has no broadcast hint, so Catalyst may use a broadcast nested-loop join only when the plan statistics consider a side small enough. Streaming transforms retain the existing UDF implementation because the join-and-aggregation plan is not a stateless streaming transformation. ### Why are the changes needed? Collecting every association rule before a batch transform creates avoidable driver-memory pressure, particularly for Spark Connect server workloads with many rules. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 mllib/compile` - The focused `FPGrowthSuite` has not been run locally yet. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
