Re: how to increase parallelism ?

[Silvio Fiorito]

Hi Deepak,

Parallelism is controlled by the number of partitions. In this case, how many 
partitions are there for the details RDD (likely 170).

You can check by running “details.partitions.length”. If you want to increase 
parallelism you can do so by repartitioning, increasing the number of 
partitions: “details.repartition(xxxx)”

Thanks,
Silvio

From: "ÐΞ€ρ@Ҝ (๏̯͡๏)"
Date: Wednesday, June 24, 2015 at 1:57 PM
To: user
Subject: how to increase parallelism ?

I have a filter.map that triggers 170 tasks.  How can i increase it ?

Code:

val viEvents = details.filter(_.get(14).asInstanceOf[Long] != NULL_VALUE).map { 
vi => (vi.get(14).asInstanceOf[Long], vi) }


Deepak