Github user avi8tr commented on the issue:
https://github.com/apache/spark/pull/16782
Hi, thanks for explaining that there is a purpose for the retention and
passing of the user-supplied arguments outside of the function call (while not
changing the public api). This fix enabling storage per-instance fits the
usage model for threading in Spark -- one thread creates the pipeline and e.g.
invokes .fit() -- but it stops short of a fix because it leaves in place the
static class variable for all other ML classes that use the wrapper, and those
classes continue to use the static class variable. That is the aspect of the
patch that is not thread-safe. If this branch is merged, one still cannot
reasonably create multiple ML pipelines in a threaded environment because the
elements of the pipeline (its stages) are now known to be subject to the same
bug. (The remaining nit is, what is supposed to happen to arguments, e.g.
stages=, that are changed in the bodies of the wrapped methods? Currently, the
changes are thrown away. This would seem to deserve at least a commen
t placed in the dead code.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
