Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12248#issuecomment-207562982
I mean making a `Properties` object in the driver, and using a reference to
it in a function that is executed on the executors. That's certainly in scope.
For the example you give that seems equally simple and can be bottled up inside
the library, still.
I understand Josh's use case more. There are certainly tasks and RDDs
entirely internal to some Spark process. But those also won't know anything
about what to do with some custom user properties. Maybe eventually they invoke
a UDF that could use these properties. In many cases that UDF could still just
refer to whatever config you like directly (right?) but I'm probably not
thinking of some case where this fails to work.
I take the point about this already being an API for the caller anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
