Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

[Gourav Sengupta]

Hi,

in case the key value store is large then can you give redis a try? SPARK
does work quite well with redis.


Regards,
Gourav Sengupta

On Sat, Apr 7, 2018 at 6:29 PM, klrmowse  wrote:

> okie, well...
>
> i'm working with a pair rdd 
>
> i need to extract the values and store them somehow (maybe a simple
> Array??), which i later parallelize and reuse
>
> since adding to a list is a no-no, what, if any, are the other options?
> (Java Spark, btw)
>
>
>
> thanks
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

[Bryan Jeffrey]

You can just call rdd.flatMap(_._2).collect


Get Outlook for Android<https://aka.ms/ghei36>


From: klrmowse <klrmo...@gmail.com>
Sent: Saturday, April 7, 2018 1:29:34 PM
To: user@spark.apache.org
Subject: Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

okie, well...

i'm working with a pair rdd <key, iterable(value)>

i need to extract the values and store them somehow (maybe a simple
Array??), which i later parallelize and reuse

since adding to a list is a no-no, what, if any, are the other options?
(Java Spark, btw)



thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

[klrmowse]

okie, well... 

i'm working with a pair rdd 

i need to extract the values and store them somehow (maybe a simple
Array??), which i later parallelize and reuse

since adding to a list is a no-no, what, if any, are the other options?
(Java Spark, btw)



thanks



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()

[Jörn Franke]

What are you trying to achieve ? You should not use global variables in a spark 
application. Especially not adding to a list - that makes in most cases no 
sense.

If you want to put everything into a file then you should repartition to 1 .


> On 7. Apr 2018, at 19:07, klrmowse  wrote:
> 
> it gives null pointer exception...
> 
> is there a workaround for adding to an arrayList during .foreach of an rdd?
> 
> 
> thank you
> 
> 
> 
> 
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> 

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org