Sameer Agarwal created SPARK-20451:
--------------------------------------
Summary: Filter out nested mapType datatypes from sort order in
randomSplit
Key: SPARK-20451
URL: https://issues.apache.org/jira/browse/SPARK-20451
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0, 2.0.0, 2.2.0
Reporter: Sameer Agarwal
In {{randomSplit}}, It is possible that the underlying dataframe doesn't
guarantee the ordering of rows in its constituent partitions each time a split
is materialized which could result in overlapping splits.
To prevent this, we explicitly sort each input partition to make the ordering
deterministic. Given that MapTypes cannot be sorted they should be explicitly
pruned out from the sort order. Additionally, if the resulting sort order is
empty, we then materialize the dataset to guarantee determinism.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
