[ 
https://issues.apache.org/jira/browse/SPARK-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

michael davis tira updated SPARK-17684:
---------------------------------------
    Environment: 
Local environment, virtual box VM running Ubuntu 16.04 with 8 GB of ram.

EMR 3 cores and 1 master m3.xlarge su emr-4.7.2 con spark 1.6.2 su hadoop 2.7.2 

  was:Local environment, virtual box VM running Ubuntu 16.04 with 8 GB of ram. 


> 'null' appears in the data during aggregateByKey action.
> --------------------------------------------------------
>
>                 Key: SPARK-17684
>                 URL: https://issues.apache.org/jira/browse/SPARK-17684
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.2, 2.0.0
>         Environment: Local environment, virtual box VM running Ubuntu 16.04 
> with 8 GB of ram.
> EMR 3 cores and 1 master m3.xlarge su emr-4.7.2 con spark 1.6.2 su hadoop 
> 2.7.2 
>            Reporter: michael davis tira
>
> aggregateByKey issues an unexpected scala.MatchError. 
> The MatchError is triggered in the merging function (the third parameter of 
> aggregateByKey) because some of the records to be merged appear to be 'null':
> bq. scala.MatchError: 
> ((4c5b5fc8-6eb9-40b1-8e6c-a81e4c9869ce07fe38fc-abf2-43b0-b618-ff898cffbad6,1.0),null)
>  (of class scala.Tuple2)
> It is worth noting that no 'null' is actually present in the starting data. 
> The problem only happens when the RDD's partitions are big enough to lead 
> Spark to spill an in-memory map to disk:
> bq. INFO ExternalSorter: Thread 64 spilling in-memory map of 373.6 MB to disk 
> (1 time so far)
> Repartitioning the RDD with a bigger number of partitions is effective to 
> avoid the problem, nevertheless this behavior is worrying.
> I'm using Kryo serializer and, for I guessed a serialization problem, I tried 
> registering the HashMap class with no apparent effect.
> Here is a simplified example I used to reproduce the problem:
> [http://pastebin.com/rmPYb7Mu]
> and here is Spark's stack trace:
> [http://pastebin.com/gkxfYGdT]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to