The |+| operator only handles numeric data types, you may register you
own concat function like this:
|sqlContext.registerFunction("concat", (s: String, t: String) => s + t)
sqlContext.sql("select concat('$', col1) from tbl")
|
Cheng
On 1/5/15 1:13 PM, RK wrote:
The issue is happening when I try to concatenate column values in the
query like "col1+'
+col3". For some reason, this issue is not manifesting itself when I
do a single IF query.
Is there a concat function in SparkSQL? I can't find anything in the
documentation.
Thanks,
RK
On Sunday, January 4, 2015 7:42 PM, RK <prk...@yahoo.com.INVALID> wrote:
BTW, I am seeing this issue in Spark 1.1.1.
On Sunday, January 4, 2015 7:29 PM, RK <prk...@yahoo.com.INVALID> wrote:
When I use a single IF statement like "select IF(col1 != "", col1+'
+col3, col2+'
+col3) from my_table", it works fine.
However, when I use a nested IF like "select IF(col1 != "", col1+'
+col3, IF(col2 != "", col2+'
+col3, '
)) from my_table", I am getting the following exception.
Exception in thread "main"
org.apache.spark.sql.catalyst.errors.package$TreeNodeException:
Unresolved attributes: if (NOT (col1#1 = )) (CAST((CAST(col1#1,
DoubleType) + CAST($, DoubleType)), DoubleType) + CAST(col3#3,
DoubleType)) else if (NOT (col2#2 = )) (CAST((CAST(col2#2, DoubleType)
+ CAST($, DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else $
AS c0#4, tree:
Project [if (NOT (col1#1 = )) (CAST((CAST(col1#1, DoubleType) +
CAST($, DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else if
(NOT (col2#2 = )) (CAST((CAST(col2#2, DoubleType) + CAST($,
DoubleType)), DoubleType) + CAST(col3#3, DoubleType)) else $ AS c0#4]
Subquery my_table
SparkLogicalPlan (ExistingRdd [DB#0,col1#1,col2#2,col3#3],
MappedRDD[97] at getCallSite at DStream.scala:294)
Does Spark SQL not support nested IF queries or is my query incorrect?
Thanks,
RK
