The last two lines are what trigger the operations, and they will each
block until the result is computed and saved. So if you execute this
code as-is, no. You could write a Scala program that invokes these two
operations in parallel, like:
Array((wc1,"titles.out"), (wc2,"tables.out")).par.foreach { case
(wc,path) => wc.saveAsTestFile(path) }
It worked for me and think it's OK to do this if you know you want to.
On Tue, Jul 15, 2014 at 8:38 PM, Wei Tan <w...@us.ibm.com> wrote:
> Hi, I wonder if I do wordcount on two different files, like this:
>
> val file1 = sc.textFile("/...")
> val file2 = sc.textFile("/...")
>
>
> val wc1= file.flatMap(..).reduceByKey(_ + _,1)
> val wc2= file.flatMap(...).reduceByKey(_ + _,1)
>
> wc1.saveAsTextFile("titles.out")
> wc2.saveAsTextFile("tables.out")
>
> Would the two reduceByKey stages run in parallel given sufficient capacity?
>
> Best regards,
> Wei
>
>
> ---------------------------------
> Wei Tan, PhD
> Research Staff Member
> IBM T. J. Watson Research Center
> http://researcher.ibm.com/person/us-wtan
