Burak Yavuz created SPARK-9614:
----------------------------------
Summary: InternalRow representation during
executionPlan.toRdd.aggregete possibly problematic
Key: SPARK-9614
URL: https://issues.apache.org/jira/browse/SPARK-9614
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Burak Yavuz
Priority: Blocker
For example, in FrequentItems.scala, we have a specialized FrequentItems
counter which is backed by a Mutable Map. The keys in the MutableMap are the
elements in a column.
After iterating through a partition, all the keys turn out to be the latest
element seen in that partition!
Assume a partition is composed of the elements ("0","1","2","3","4") for a
column of a dataframe. The expected map is:
{code}
("0" -> 1,"1" -> 1,"2" -> 1,"3" -> 1,"4" -> 1)
{code}
But once you print it out, it turns out to be:
{code}
("4" -> 1,"4" -> 1,"4" -> 1,"4" -> 1,"4" -> 1)
{code}
There might be other instances where such behavior may be observed, and it must
be analyzed before the release.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
