[ https://issues.apache.org/jira/browse/SPARK-32154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wuyi updated SPARK-32154: ------------------------- Summary: Use ExpressionEncoder for the return type of ScalaUDF to serialize to catalyst type (was: Use ExpressionEncoder to serialize to catalyst type for the return type of ScalaUDF) > Use ExpressionEncoder for the return type of ScalaUDF to serialize to > catalyst type > ----------------------------------------------------------------------------------- > > Key: SPARK-32154 > URL: https://issues.apache.org/jira/browse/SPARK-32154 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.1.0 > Reporter: wuyi > Priority: Major > > Users now could register a UDF with Instant/LocalDate as return type even > with > spark.sql.datetime.java8API.enabled=false. However, the UDF can only be > really used with spark.sql.datetime.java8API.enabled=true. This could make > users confused. > The problem is we use ExpressionEncoder to ser/deser types when registering > the UDF, but use Catalyst converters to ser/deser types, which is under > control of spark.sql.datetime.java8API.enabled, when executing UDF. > If we could also use ExpressionEncoder to ser/deser types, similar to what we > do for input parameter types, the, UDF could support Instant/LocalDate, event > other combined complex types as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org