Re: Getting the size of a broadcast variable

[Ted Yu]

There is chance that the log message may change in future releases.

Log snooping would be broken.

FYI

On Mon, Feb 1, 2016 at 9:55 PM, Takeshi Yamamuro 
wrote:

> Hi,
>
> Currently, there is no way to check the size except for snooping INFO-logs
> in a driver;
>
> 16/02/02 14:51:53 INFO BlockManagerInfo: Added rdd_2_12 in memory on
> localhost:58536 (size: 40.0 B, free: 510.7 MB)
>
>
>
> On Tue, Feb 2, 2016 at 8:20 AM, apu mishra . rr 
> wrote:
>
>> How can I determine the size (in bytes) of a broadcast variable? Do I
>> need to use the .dump method and then look at the size of the result, or is
>> there an easier way?
>>
>> Using PySpark with Spark 1.6.
>>
>> Thanks!
>>
>> Apu
>>
>
>
>
> --
> ---
> Takeshi Yamamuro
>

Getting the size of a broadcast variable

[apu mishra . rr]

How can I determine the size (in bytes) of a broadcast variable? Do I need
to use the .dump method and then look at the size of the result, or is
there an easier way?

Using PySpark with Spark 1.6.

Thanks!

Apu

Re: Getting the size of a broadcast variable

[Takeshi Yamamuro]

Hi,

Currently, there is no way to check the size except for snooping INFO-logs
in a driver;

16/02/02 14:51:53 INFO BlockManagerInfo: Added rdd_2_12 in memory on
localhost:58536 (size: 40.0 B, free: 510.7 MB)



On Tue, Feb 2, 2016 at 8:20 AM, apu mishra . rr 
wrote:

> How can I determine the size (in bytes) of a broadcast variable? Do I need
> to use the .dump method and then look at the size of the result, or is
> there an easier way?
>
> Using PySpark with Spark 1.6.
>
> Thanks!
>
> Apu
>



-- 
---
Takeshi Yamamuro