Re: how to broadcast huge lookup table?

[Adrian Tanase]

have a look at .transformWith, you can specify another RDD.

Sent from my iPhone

On 02 Oct 2015, at 21:50, 
"saif.a.ell...@wellsfargo.com" 
> wrote:

I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions 
inside a map/foreach function of another rdd.

any tips? If going into scala collections I end up with huge memory bottlenecks.

Saif

RE: how to broadcast huge lookup table?

[Saif.A.Ellafi]

Hi, thank you

I would prefer to leave writing-to-disk as a last resort. Is it a last resort?

Saif

From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Friday, October 02, 2015 3:54 PM
To: Ellafi, Saif A.
Cc: user
Subject: Re: how to broadcast huge lookup table?

Have you considered using external storage such as hbase for storing the look 
up table ?

Cheers

On Fri, Oct 2, 2015 at 11:50 AM, 
<saif.a.ell...@wellsfargo.com<mailto:saif.a.ell...@wellsfargo.com>> wrote:
I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions 
inside a map/foreach function of another rdd.

any tips? If going into scala collections I end up with huge memory bottlenecks.

Saif

how to broadcast huge lookup table?

[Saif.A.Ellafi]

I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions 
inside a map/foreach function of another rdd.

any tips? If going into scala collections I end up with huge memory bottlenecks.

Saif

Re: how to broadcast huge lookup table?

[Ted Yu]

Have you considered using external storage such as hbase for storing the
look up table ?

Cheers

On Fri, Oct 2, 2015 at 11:50 AM,  wrote:

> I tried broadcasting a key-value rdd, but then I cannot perform any
> rdd-actions inside a map/foreach function of another rdd.
>
> any tips? If going into scala collections I end up with huge memory
> bottlenecks.
>
> Saif
>
>