Hi All ,

If some one can help me understand as which portion of the code gets
executed on Driver and which portion will be executed on executor from the
below code it would be a great help

I have to load data from 10 Tables and then use that data in various
manipulation and i am using SPARK SQL for that please let me know if below
code will be executed on the driver or it will be executed in each executor
node.

And if i do a join on the data frame will it happen on executor or driver ?

        options.put("dbtable", "(select * from t_table1) as
t_table1");
        DataFrame t_gsubmember =
sqlContext.read().format("jdbc").options(options).load();
        t_table1.cache();



        options.put("dbtable", "(select * from t_table2) as
t_table2");
        DataFrame t_sub =
sqlContext.read().format("jdbc").options(options).load();
        t_table2.cache();



        options.put("dbtable", "(select * from t_table3) as
t_table3");
        DataFrame t_pi =
sqlContext.read().format("jdbc").options(options).load();
        t_table3.cache();

       ....And So on

Thanks

Reply via email to