Option 2 seems fine to me.
2020년 3월 17일 (화) 오후 3:41, Wenchen Fan 님이 작성:
> I don't think option 1 is possible.
>
> For option 2: I think we need to do it anyway. It's kind of a bug that the
> typed Scala UDF doesn't support case class that thus can't support
> struct-type input columns.
>
> For op
I don't think option 1 is possible.
For option 2: I think we need to do it anyway. It's kind of a bug that the
typed Scala UDF doesn't support case class that thus can't support
struct-type input columns.
For option 3: It's a bit risky to add a new API but seems like we have a
good reason. The un
Thanks Sean and Takeshi.
Option 1 seems really impossible. And I'm going to take Option 2 as an
alternative choice.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubscribe e-mail: dev-unsubscr
I don't think it's possible to get the parameters by reflection
anymore -- they are lambdas now in the JVM. At least, indeed, I recall
a few people couldn't find a solution back when we added 2.12 support.
This isn't 'new' in that it has always been the case for Scala 2.12.
If there is a better ide
Ah, I see now what the "broken' means. Thanks, Yi.
I personally think the option 1 is the best for existing Spark users to
support the usecase you suggested above.
So, I think this decision depends on how difficult it is to implement "get
Scala lambda parameter types by reflection"
and the complexi
Hi Takeshi, thanks for your reply.
Before the broken, we only do the null check for primitive types and leave
null value of non-primitive type to UDF itself in case it will be handled
specifically, e.g., a UDF may return something else for null String.
--
Sent from: http://apache-spark-develop
hi, Yi,
Probably, I miss something though, we cannot just wrap the udf with
`if (isnull(x)) null else udf(knownnotnull(x))`?
On Fri, Mar 13, 2020 at 6:22 PM wuyi wrote:
> Hi all, I'd like to raise a discussion here about null-handling of
> primitive-type of untyped Scala UDF [ udf(f: AnyRef, da
Hi all, I'd like to raise a discussion here about null-handling of
primitive-type of untyped Scala UDF [ udf(f: AnyRef, dataType: DataType) ].
After we switch to Scala 2.12 in 3.0, the untyped Scala UDF is broken
because now we can't use reflection to get the parameter types of the Scala
lambda.
T
