M Bharat lal created SPARK-11203:
------------------------------------
Summary: UDF doesn't support charType column and lit function
doesn't allow charType as argument
Key: SPARK-11203
URL: https://issues.apache.org/jira/browse/SPARK-11203
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.5.0
Reporter: M Bharat lal
Priority: Minor
We have two issues
1) We cannot create dataframe with Char Type , see below example
scala> val employee =
sqlContext.createDataFrame(Seq((1,"John"))).toDF("id","name")
employee: org.apache.spark.sql.DataFrame = [id: int, name: string]
scala> employee.withColumn("grade",lit('A'))
java.lang.RuntimeException: Unsupported literal type class java.lang.Character A
at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
2) we have a function which takes string and char as input parameters and
returns position of char in given string.
registered function as UDF and called the same UDF with characters literal
which gave the below exception. Literal function doesn't support character as
argument
scala> def strPos(x:String,c:Char):Integer = {x.indexOf(c)}
strPos: (x: String, c: Char)Integer
scala> sqlContext.udf.register("strPos",strPos _)
res13: org.apache.spark.sql.UserDefinedFunction =
UserDefinedFunction(<function2>,IntegerType,List())
scala> df.select( callUDF("strPos",$"name",lit('J')))
java.lang.RuntimeException: Unsupported literal type class java.lang.Character J
at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
Can you please add this support or let us know if there is any other work
around to achieve this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
