Olivier Toupin created SPARK-8048:
-------------------------------------
Summary: Explicit partitionning of an RDD with 0 partition will
yield empty outer join
Key: SPARK-8048
URL: https://issues.apache.org/jira/browse/SPARK-8048
Project: Spark
Issue Type: Bug
Reporter: Olivier Toupin
Priority: Minor
Check this code =>
https://gist.github.com/anonymous/0f935915f2bc182841f0
Because of this => {{.partitionBy(new HashPartitioner(0))}}
The join will return empty result.
Here a normal expected behaviour would the join to crash, cause error, or to
return unjoined results, but instead will yield an empty RDD.
This a trivial exemple, but imagine:
{{.partitionBy(new HashPartitioner(previous.partitions.length))}}.
You join on an empty "previous" rdd, the lookup table is empty, Spark will you
lose all your results, instead of returning unjoined results, and this without
warnings or errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
