I am not sure what you are trying to achieve here. Spark is taking care of executing the transformations in a distributed fashion. This means you must not use threads - it does not make sense. Hence, you do not find documentation about it.
> On 12 Feb 2017, at 09:06, Mendelson, Assaf <assaf.mendel...@rsa.com> wrote: > > Hi, > I was wondering if dataframe is considered thread safe. I know the spark > session and spark context are thread safe (and actually have tools to manage > jobs from different threads) but the question is, can I use the same > dataframe in both threads. > The idea would be to create a dataframe in the main thread and then in two > sub threads do different transformations and actions on it. > I understand that some things might not be thread safe (e.g. if I unpersist > in one thread it would affect the other. Checkpointing would cause similar > issues), however, I can’t find any documentation as to what operations (if > any) are thread safe. > > Thanks, > Assaf.
