cloud-fan commented on PR #40520: URL: https://github.com/apache/spark/pull/40520#issuecomment-1480533198
From a SQL engine's point of view, running all tasks at once or batch by batch doesn't matter. It doesn't change the semantics of the SQL operator, and the optimizer doesn't care about it. However, `mapInPandas` is a public API and you are free to define what's the expectation of the `is_barrier` parameter. To make our life easier, we can just define it as "the tasks of running the pandas function must all be launched at once", and it's not a barrier to the SQL operators. Then I think it's fine to just add a flag to the existing `MapInPandas` operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
