Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/4491#issuecomment-74081017
The main issue I see here is we are using private things from hadoop and
mapreduce. We've run into issues with this in the past so I think we should
stay away from that. Either that or we have to get the interfaces made public.
Many of the MapReduce references really aren't needed, just make spark
configs to do the same things for configuration or for the secret key, etc.
I haven't looked at the hadoop encryption stuff in detail so would need to
review that further to give more feedback. Perhaps @sryza has looked at it.
I'm not sure if it would make sense to just copy the code or if its using a
bunch of other private routines.
We will definitely need to update some more documentation if this goes in -
configuration and security docs.
What testing has been performed on this? It affects more then just running
on YARN. Did you test it with the YARN external shuffle service?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
