[
https://issues.apache.org/jira/browse/SPARK-10073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-10073:
------------------------------------
Assignee: Apache Spark (was: Davies Liu)
> Python withColumn for existing column name not consistent with scala
> --------------------------------------------------------------------
>
> Key: SPARK-10073
> URL: https://issues.apache.org/jira/browse/SPARK-10073
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Michael Armbrust
> Assignee: Apache Spark
> Priority: Blocker
>
> The same code as below works in Scala (replacing the old column with the new
> one).
> {code}
> from pyspark.sql import Row
> df = sc.parallelize([Row(a=1)]).toDF()
> df.withColumn("a", df.a).select("a")
> ---------------------------------------------------------------------------
> AnalysisException Traceback (most recent call last)
> <ipython-input-4-d5a4f4132506> in <module>()
> 1 from pyspark.sql import Row
> 2 df = sc.parallelize([Row(a=1)]).toDF()
> ----> 3 df.withColumn("a", df.a).select("a")
> /home/ubuntu/databricks/spark/python/pyspark/sql/dataframe.py in select(self,
> *cols)
> 764 [Row(name=u'Alice', age=12), Row(name=u'Bob', age=15)]
> 765 """
> --> 766 jdf = self._jdf.select(self._jcols(*cols))
> 767 return DataFrame(jdf, self.sql_ctx)
> 768
> /home/ubuntu/databricks/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py
> in __call__(self, *args)
> 536 answer = self.gateway_client.send_command(command)
> 537 return_value = get_return_value(answer, self.gateway_client,
> --> 538 self.target_id, self.name)
> 539
> 540 for temp_arg in temp_args:
> /home/ubuntu/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
> 38 s = e.java_exception.toString()
> 39 if s.startswith('org.apache.spark.sql.AnalysisException:
> '):
> ---> 40 raise AnalysisException(s.split(': ', 1)[1])
> 41 if s.startswith('java.lang.IllegalArgumentException: '):
> 42 raise IllegalArgumentException(s.split(': ', 1)[1])
> AnalysisException: Reference 'a' is ambiguous, could be: a#894L, a#895L.;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
