Any objections? Please articulate your use case. SparkEnv is a weird one
> because it was documented as "private" but not marked as so in class
> visibility.
>
> * NOTE: This is not intended for external use. This is exposed for Shark
> and may be made private
> *
On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan
wrote:
> b) Shuffle manager (to get shuffle reader)
>
What's the use case for shuffle manager/reader? This seems like using super
internal APIs in applications.
Any objections? Please articulate your use case. SparkEnv is a weird one
because it was documented as "private" but not marked as so in class
visibility.
* NOTE: This is not intended for external use. This is exposed for Shark
and may be made private
* in a future release.
I d
We have custom join's that leverage it.
It is used to get to direct shuffle'ed iterator - without needing
sort/aggregate/etc.
IIRC the only way to get to it from ShuffleHandle is via shuffle manager.
Regards,
Mridul
On Wed, Mar 16, 2016 at 3:36 PM, Reynold Xin wrote:
>
> On Wed, Mar 16, 2016 a
is it safe to access SparkEnv.get inside say mapPartitions?
i need to get a Serializer (so SparkEnv.get.serializer)
thanks
