Are you triggering an action within the while loop? How are you loading the
data from jdbc? You need to make sure the job has enough partitions to run
parallel to increase the performance.
Thanks
Best Regards
On Sat, Aug 15, 2015 at 2:41 AM, <saif.a.ell...@wellsfargo.com> wrote:
> Hello all,
>
> I am writing a program which calls from a database. A run a couple
> computations, but in the end I have a while loop, in which I make a
> modification to the persisted thata. eg:
>
> val data = PairRDD... persist()
> var i = 0
> while (i < 10) {
> val data_mod = data.map(_._1 + 1, _._2)
> val data_joined = data.join(data_mod)
> ... do stuff with data_joined
> }
>
> Sadly, the result causes that the shuffle inside the WHILE loop is causing
> a jdbc call and that is very slow. It is not finding the data locally
>
> How can I help myself?
> Saif
>
>
