Your tableLoad() APIs are not actions. File will be read fully only when an action is performed. If the action is something like table1.join(table2), then I think both files will be read in parallel. Can you try that and look at the execution plan or in 1.4 this is shown in Spark UI.
Srikanth On Wed, Jul 8, 2015 at 11:06 AM, Brandon White <[email protected]> wrote: > The point of running them in parallel would be faster creation of the > tables. Has anybody been able to efficiently parallelize something like > this in Spark? > On Jul 8, 2015 12:29 AM, "Akhil Das" <[email protected]> wrote: > >> Whats the point of creating them in parallel? You can multi-thread it run >> it in parallel though. >> >> Thanks >> Best Regards >> >> On Wed, Jul 8, 2015 at 5:34 AM, Brandon White <[email protected]> >> wrote: >> >>> Say I have a spark job that looks like following: >>> >>> def loadTable1() { >>> val table1 = sqlContext.jsonFile(s"s3://textfiledirectory/") >>> table1.cache().registerTempTable("table1")} >>> def loadTable2() { >>> val table2 = sqlContext.jsonFile(s"s3://testfiledirectory2/") >>> table2.cache().registerTempTable("table2")} >>> >>> def loadAllTables() { >>> loadTable1() >>> loadTable2()} >>> >>> loadAllTables() >>> >>> How do I parallelize this Spark job so that both tables are created at >>> the same time or in parallel? >>> >> >>
