Nicholas Hwang created SPARK-9021:
-------------------------------------
Summary: Have pyspark's RDD.aggregate() make a deepcopy of
zeroValue for each partition
Key: SPARK-9021
URL: https://issues.apache.org/jira/browse/SPARK-9021
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.4.0
Environment: Ubuntu 14.04 LTS
Reporter: Nicholas Hwang
Please see pull request for more information.
I initially "patched" this arguably unexpected behavior by serializing
zeroValue, but ended up mimicking the deepcopy approach used by other RDD
methods. I also contemplated having fold/aggregate accept "zero value generator
functions" instead of an actual object, but that obviously changes the API.
Looking forward to hearing back and/or being educated on how I'm
inappropriately using this functionality (relatively new to Spark and
functional programming). Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
