Hi, Shooting in the dark...it's executed on executors (it's old tech RDD-based so not many extra optimizations like in Spark SQL now).
Can you show the code as I'm scared to hear that you're trying to broadcast inside a transformation which I'd believe is impossible. Jacek On 26 Jan 2017 12:18 a.m., "shyla deshpande" <deshpandesh...@gmail.com> wrote: After more reading, I know the state is distributed across the cluster. But If I need to lookup a map in the updatefunction, I need to broadcast it. Just want to make sure I am on the right path. Appreciate your help. Thanks On Wed, Jan 25, 2017 at 2:33 PM, shyla deshpande <deshpandesh...@gmail.com> wrote: > Is it executed on the driver or executor. If I need to lookup a map in > the updatefunction, I need to broadcast it, if mapWithState executed runs > on executor. > > Thanks >
