The serializer is created with
val zeroBuffer = SparkEnv.get.serializer.newInstance().serialize(zeroValue)
Which is definitely not the closure serializer and so should respect
what you are setting with spark.serializer.
Maybe you can do a quick bit of debugging to see where that assumption
...@palantir.com, Andrew Ash a...@palantir.com
Subject: Re: JavaRDD Aggregate initial value - Closure-serialized zero
value reasoning?
It looks like this was fixed in
https://issues.apache.org/jira/browse/SPARK-4743
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira
_browse_SPARK
It looks like this was fixed in
https://issues.apache.org/jira/browse/SPARK-4743 /
https://github.com/apache/spark/pull/3605. Can you see whether that patch
fixes this issue for you?
On Tue, Feb 17, 2015 at 8:31 PM, Matt Cheah mch...@palantir.com wrote:
Hi everyone,
I was using
: Re: JavaRDD Aggregate initial value - Closure-serialized zero value
reasoning?
It looks like this was fixed in
https://issues.apache.org/jira/browse/SPARK-4743 /
https://github.com/apache/spark/pull/3605. Can you see whether that patch
fixes this issue for you?
On Tue, Feb 17, 2015 at 8
AM
To: Matt Cheah mch...@palantir.com
Cc: dev@spark.apache.org dev@spark.apache.org, Mingyu Kim
m...@palantir.com, Andrew Ash a...@palantir.com
Subject: Re: JavaRDD Aggregate initial value - Closure-serialized zero
value
reasoning?
It looks like this was fixed in
https
Hi everyone,
I was using JavaPairRDD¹s combineByKey() to compute all of my aggregations
before, since I assumed that every aggregation required a key. However, I
realized I could do my analysis using JavaRDD¹s aggregate() instead and not
use a key.
I have set spark.serializer to use Kryo. As a
