[
https://issues.apache.org/jira/browse/SPARK-29240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29240.
-----------------------------------
Fix Version/s: 3.0.0
2.4.5
Assignee: Hyukjin Kwon
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/25950
> PySpark 2.4 about sql function 'element_at' param 'extraction'
> --------------------------------------------------------------
>
> Key: SPARK-29240
> URL: https://issues.apache.org/jira/browse/SPARK-29240
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.0
> Reporter: Simon Reon
> Assignee: Hyukjin Kwon
> Priority: Trivial
> Fix For: 2.4.5, 3.0.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I was trying to translate {color:#FF0000}Scala{color} into
> {color:#FF0000}python{color} with {color:#FF0000}PySpark 2.4.0{color} .Codes
> below aims to extract col '{color:#FF0000}list{color}' value using col
> '{color:#FF0000}num{color}' as index.
>
> {code:java}
> x =
> spark.createDataFrame([((1,2,3),1),((4,5,6),2),((7,8,9),3)],['list','num'])
> x.show(){code}
>
> ||list||num||
> |[1,2,3]|1|
> |[4,5,6]|2|
> |[7,8,9]|3|
> I suppose to use new func '{color:#FF0000}element_at{color}' in 2.4.0 .But it
> gives an error:
> {code:java}
> x.withColumn('aa',F.element_at('list',x.num.cast('int')))
> {code}
> _TypeError: Column is not iterable_
>
> Finally ,I have to use {color:#FF0000}udf{color} to solve this problem.
> But in Scala ,it is ok when the second param
> '{color:#FF0000}extraction{color}' in func '{color:#FF0000}element_at{color}'
> is a col name with int type:
> {code:java}
> //Scala
> val y = x.withColumn("aa",element_at('list,'num.cast("int")))
> y.show(){code}
> ||list||num|| aa||
> |[1,2,3]|1| 1|
> |[4,5,6] |2 |5 |
> |[7,8,9] |3 |9 |
> I hope it could be fixed in latest version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
