Josh Rosen created SPARK-19044:
----------------------------------
Summary: PySpark dropna() can fail with AnalysisException
Key: SPARK-19044
URL: https://issues.apache.org/jira/browse/SPARK-19044
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Reporter: Josh Rosen
Priority: Minor
In PySpark, the following fails with an AnalysisException:
{code}
v1 = spark.range(10)
v2 = v1.crossJoin(v1)
v2.dropna()
{code}
{code}
AnalysisException: u"Reference 'id' is ambiguous, could be: id#66L, id#69L.;"
{code}
However, the equivalent Scala code works fine:
{code}
val v1 = spark.range(10)
val v2 = v1.crossJoin(v1)
v1.na.drop()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
