zhengruifeng commented on PR #40896: URL: https://github.com/apache/spark/pull/40896#issuecomment-1548778624
I think we (@HyukjinKwon @WeichenXu123 and I) have reach to agreement that this PR (which introduces an `@barrier` annotation) is no longer, and we will keep current `barrier` parameter in `MapInPandas/Arrow`. The reason is that barrier UDF is still too specific to `MapInPandas/Arrow`, due to the limitation of underlying `RDDBarrier`: - can not use it within common dataframe operators like `select`, `withColumn` - can not use it within other similar pandas function like `applyInPandas`; I will close it in two days if no more comments. also cc @mengxr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
