How can I achieve the following by passing a row to a udf ? val df1 = df.withColumn("col_Z", when($"col_x" === "a", $"col_A") .when($"col_x" === "b", $"col_B") .when($"col_x" === "c", $"col_C") .when($"col_x" === "d", $"col_D") .when($"col_x" === "e", $"col_E") .when($"col_x" === "f", $"col_F") .when($"col_x" === "g", $"col_G") )
As I understand it, only columns can be passed as arguments to a UDF in Scala Spark. I have taken a look at this question: https://stackoverflow.com/questions/31816975/how-to-pass-whole-row-to-udf-spark-dataframe-filter and tried to implement this udf: def myUDF(r:Row) = udf { val z : Float = r.getAs("col_x") match { case "a" => r.getAs("col_A") case "b" => r.getAs("col_B") case other => lit(0.0) } z } but I'm getting a type mismatch error: error: type mismatch; found : String("a") required: Nothing case "a" => r.getAs("col_A") ^ What am I doing wrong ? Sent from my iPhone