Join happens on executor. Else spark would not be much of a distributed computing engine :)
Reads happen on executor too. Your options are passed to executors and conn objects are created in executors. On 6 Jul 2015 22:58, "Ashish Soni" <[email protected]> wrote: > Hi All , > > If some one can help me understand as which portion of the code gets > executed on Driver and which portion will be executed on executor from the > below code it would be a great help > > I have to load data from 10 Tables and then use that data in various > manipulation and i am using SPARK SQL for that please let me know if below > code will be executed on the driver or it will be executed in each executor > node. > > And if i do a join on the data frame will it happen on executor or driver ? > > options.put("dbtable", "(select * from t_table1) as > t_table1"); > DataFrame t_gsubmember = > sqlContext.read().format("jdbc").options(options).load(); > t_table1.cache(); > > > > options.put("dbtable", "(select * from t_table2) as > t_table2"); > DataFrame t_sub = > sqlContext.read().format("jdbc").options(options).load(); > t_table2.cache(); > > > > options.put("dbtable", "(select * from t_table3) as > t_table3"); > DataFrame t_pi = > sqlContext.read().format("jdbc").options(options).load(); > t_table3.cache(); > > ....And So on > > Thanks >
