alisha-1000 commented on issue #569: URL: https://github.com/apache/wayang/issues/569#issuecomment-3929012956
Thanks for reporting this , From the description, it seems the current implementation sends each record individually to Python for UDF execution and then returns the result back to Java. This per-element cross-language communication likely introduces significant serialization and IPC overhead, which explains the slowdown. A few possible improvement directions: Batching records before sending them to Python instead of processing one-by-one Using a vectorized execution model (e.g., Arrow-based data exchange if applicable) Reducing round-trips by executing the full join logic on one side (preferably Python if the UDF dominates execution) Profiling serialization/deserialization cost between JVM ↔ Python boundary It would be helpful to clarify: Is this happening for all joins with Python UDFs or only specific execution plans? Is there any existing batching mechanism that is currently disabled or misconfigured? If helpful, I can try to benchmark the current behavior and propose a batched execution prototype. Let me know how you'd like to proceed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
