Github user davies commented on the pull request:
https://github.com/apache/spark/pull/10550#issuecomment-168852366
The one socket for PythonRDD is already pretty complicated (considering how
propagate exception and how to re-use it), adding another will make it even
complicated (two socket in different status), so I'd try to avoid that. The new
introduced and not-common-used zipPartititon() should not brings bugs for
existing workload. I'd -1 for current approach, cc @JoshRosen .
Another try could be that only open second socket (not re-used) when
zipPartition() is called, this may reduce the risk for existing workload.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
