Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/6679#issuecomment-109493767
Actually, I think that I was misremembering: there's one spot where we
_optionally_ clone a broadcasted configuration to work around some
thread-safety issue, but that sharing / cloning won't be necessary if we can
directly broadcast configurations as part of the tasks rather than as their own
broadcast variables.
One correctness question: does this change behavior if an executor consumes
a deserialized configuration? If I have an option which inherited
environmental defaults on the driver, are those defaults serialized across the
wire and used on the executors? I'm just trying to think of whether there is
any possibility that this could break things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
