[ https://issues.apache.org/jira/browse/SPARK-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank Rosner updated SPARK-12823: --------------------------------- Description: h5. Problem It is not possible to apply a UDF to a column that has a struct data type. Two previous requests to the mailing list remained unanswered. h5. How-To-Reproduce {code} val sql = new org.apache.spark.sql.SQLContext(sc) import sql.implicits._ case class KV(key: Long, value: String) case class Row(kv: KV) val df = sc.parallelize(List(Row(KV(1L, "a")), Row(KV(5L, "b")))).toDF val udf1 = org.apache.spark.sql.functions.udf((kv: KV) => kv.value) df.select(udf1(df("kv"))).show // java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to $line78.$read$$iwC$$iwC$KV val udf2 = org.apache.spark.sql.functions.udf((kv: (Long, String)) => kv._2) df.select(udf2(df("kv"))).show // org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(kv)' due to data type mismatch: argument 1 requires struct<_1:bigint,_2:string> type, however, 'kv' is of struct<key:bigint,value:string> type.; {code} h5. Mailing List Entries - https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCACUahd8M=ipCbFCYDyein_=vqyoantn-tpxe6sq395nh10g...@mail.gmail.com%3E - https://www.mail-archive.com/user@spark.apache.org/msg43092.html h5. Possible Workaround If you create a {{UserDefinedFunction}} manually, not using the {{udf}} helper functions, it works. See https://github.com/FRosner/struct-udf, which exposes the {{UserDefinedFunction}} constructor (public from package private). was: h5. Problem It is not possible to apply a UDF to a column that has a struct data type. Two previous requests to the mailing list remained unanswered. h5. How-To-Reproduce {code} val sql = new org.apache.spark.sql.SQLContext(sc) import sql.implicits._ case class KV(key: Long, value: String) case class Row(kv: KV) val df = sc.parallelize(List(Row(KV(1L, "a")), Row(KV(5L, "b")))).toDF val udf1 = org.apache.spark.sql.functions.udf((kv: KV) => kv.value) df.select(udf1(df("kv"))).show // java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to $line78.$read$$iwC$$iwC$KV val udf2 = org.apache.spark.sql.functions.udf((kv: (Long, String)) => kv._2) df.select(udf2(df("kv"))).show // org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(kv)' due to data type mismatch: argument 1 requires struct<_1:bigint,_2:string> type, however, 'kv' is of struct<key:bigint,value:string> type.; {code} h5. Mailing List Entries - https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCACUahd8M=ipCbFCYDyein_=vqyoantn-tpxe6sq395nh10g...@mail.gmail.com%3E - https://www.mail-archive.com/user@spark.apache.org/msg43092.html h5. Possible Workaround If you create a {{UserDefinedFunction}} manually, not using the {{udf}} helper functions, it works. See https://github.com/FRosner/struct-udf, which exposes the UserDefinedFunction constructor (public from package private). > Cannot create UDF with StructType input > --------------------------------------- > > Key: SPARK-12823 > URL: https://issues.apache.org/jira/browse/SPARK-12823 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.2 > Reporter: Frank Rosner > > h5. Problem > It is not possible to apply a UDF to a column that has a struct data type. > Two previous requests to the mailing list remained unanswered. > h5. How-To-Reproduce > {code} > val sql = new org.apache.spark.sql.SQLContext(sc) > import sql.implicits._ > case class KV(key: Long, value: String) > case class Row(kv: KV) > val df = sc.parallelize(List(Row(KV(1L, "a")), Row(KV(5L, "b")))).toDF > val udf1 = org.apache.spark.sql.functions.udf((kv: KV) => kv.value) > df.select(udf1(df("kv"))).show > // java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast > to $line78.$read$$iwC$$iwC$KV > val udf2 = org.apache.spark.sql.functions.udf((kv: (Long, String)) => kv._2) > df.select(udf2(df("kv"))).show > // org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(kv)' due to > data type mismatch: argument 1 requires struct<_1:bigint,_2:string> type, > however, 'kv' is of struct<key:bigint,value:string> type.; > {code} > h5. Mailing List Entries > - > https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCACUahd8M=ipCbFCYDyein_=vqyoantn-tpxe6sq395nh10g...@mail.gmail.com%3E > - https://www.mail-archive.com/user@spark.apache.org/msg43092.html > h5. Possible Workaround > If you create a {{UserDefinedFunction}} manually, not using the {{udf}} > helper functions, it works. See https://github.com/FRosner/struct-udf, which > exposes the {{UserDefinedFunction}} constructor (public from package private). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org