Re: Loading a spark dataframe column into T-Digest using java

[Marcelo Vanzin]

The closure in your "foreach" loop runs in a remote executor, no the
local JVM, so it's updating its own copy of the t-digest instance. The
one on the driver side is never touched.

On Sun, Dec 10, 2017 at 10:27 PM, Himasha de Silva  wrote:
> Hi,
>
> I want to load a spark dataframe column into T-Digest using java to
> calculate quantile values. I write this code to do this, but it's giving
> zero for size of tdigest. values are not added to tDigest.
>
> my code - https://gist.github.com/anonymous/1f2e382fdda002580154b5c43fbe9b3a
>
> Thank you.
>
> Himasha De Silva
>
>



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Loading a spark dataframe column into T-Digest using java

[Himasha de Silva]

Hi,

I want to load a spark dataframe column into T-Digest using java to
calculate quantile values. I write this code to do this, but it's giving
zero for size of tdigest. values are not added to tDigest.

my code - https://gist.github.com/anonymous/1f2e382fdda002580154b5c43fbe9b3a

Thank you.

Himasha De Silva