[
https://issues.apache.org/jira/browse/SPARK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572115#comment-16572115
]
Sean Owen commented on SPARK-25044:
-----------------------------------
More specific info. In Scala 2.12:
{code:java}
scala> val f = (i: Int, j: Long) => "x"
f: (Int, Long) => String = $$Lambda$1045/927369095@51ec2856
scala> val methods = f.getClass.getMethods.filter(m => m.getName == "apply" &&
!m.isBridge)
methods: Array[java.lang.reflect.Method] = Array(public java.lang.Object
$$Lambda$1045/927369095.apply(java.lang.Object,java.lang.Object))
scala> methods.head.getParameterTypes
res0: Array[Class[_]] = Array(class java.lang.Object, class java.lang.Object)
{code}
Whereas in Scala 2.11 the result is:
{code:java}
...
scala> res0: Array[Class[_]] = Array(int, long){code}
I guess one question for folks like [~lrytz] is, is that 'correct' as far as
Scala is concerned? From reading
[https://docs.oracle.com/javase/8/docs/api/java/lang/invoke/LambdaMetafactory.html]
I got some sense that compilers had some latitude in how the lambda is
implemented, but am just wondering if it makes sense that the {{apply}}
method's signature doesn't seem to match what's expected.
Here is the full list of methods that {{f}}'s class implements; that first one
is the only logical candidate to look for, I think, as it's the only one
returning String.
{code:java}
public java.lang.String
$line3.$read$$iw$$iw$$$Lambda$1045/927369095.apply(java.lang.Object,java.lang.Object)
public java.lang.Object
$line3.$read$$iw$$iw$$$Lambda$1045/927369095.apply(java.lang.Object,java.lang.Object)
public final void java.lang.Object.wait(long,int) throws
java.lang.InterruptedException
public final native void java.lang.Object.wait(long) throws
java.lang.InterruptedException
public final void java.lang.Object.wait() throws java.lang.InterruptedException
public boolean java.lang.Object.equals(java.lang.Object)
public java.lang.String java.lang.Object.toString()
public native int java.lang.Object.hashCode()
public final native java.lang.Class java.lang.Object.getClass()
public final native void java.lang.Object.notify()
public final native void java.lang.Object.notifyAll()
public default scala.Function1 scala.Function2.curried()
public default scala.Function1 scala.Function2.tupled()
public default boolean scala.Function2.apply$mcZDD$sp(double,double)
public default double scala.Function2.apply$mcDDD$sp(double,double)
public default float scala.Function2.apply$mcFDD$sp(double,double)
public default int scala.Function2.apply$mcIDD$sp(double,double)
public default long scala.Function2.apply$mcJDD$sp(double,double)
public default void scala.Function2.apply$mcVDD$sp(double,double)
public default boolean scala.Function2.apply$mcZDI$sp(double,int)
public default double scala.Function2.apply$mcDDI$sp(double,int)
public default float scala.Function2.apply$mcFDI$sp(double,int)
public default int scala.Function2.apply$mcIDI$sp(double,int)
public default long scala.Function2.apply$mcJDI$sp(double,int)
public default void scala.Function2.apply$mcVDI$sp(double,int)
public default boolean scala.Function2.apply$mcZDJ$sp(double,long)
public default double scala.Function2.apply$mcDDJ$sp(double,long)
public default float scala.Function2.apply$mcFDJ$sp(double,long)
public default int scala.Function2.apply$mcIDJ$sp(double,long)
public default long scala.Function2.apply$mcJDJ$sp(double,long)
public default void scala.Function2.apply$mcVDJ$sp(double,long)
public default boolean scala.Function2.apply$mcZID$sp(int,double)
public default double scala.Function2.apply$mcDID$sp(int,double)
public default float scala.Function2.apply$mcFID$sp(int,double)
public default int scala.Function2.apply$mcIID$sp(int,double)
public default long scala.Function2.apply$mcJID$sp(int,double)
public default void scala.Function2.apply$mcVID$sp(int,double)
public default boolean scala.Function2.apply$mcZII$sp(int,int)
public default double scala.Function2.apply$mcDII$sp(int,int)
public default float scala.Function2.apply$mcFII$sp(int,int)
public default int scala.Function2.apply$mcIII$sp(int,int)
public default long scala.Function2.apply$mcJII$sp(int,int)
public default void scala.Function2.apply$mcVII$sp(int,int)
public default boolean scala.Function2.apply$mcZIJ$sp(int,long)
public default double scala.Function2.apply$mcDIJ$sp(int,long)
public default float scala.Function2.apply$mcFIJ$sp(int,long)
public default int scala.Function2.apply$mcIIJ$sp(int,long)
public default long scala.Function2.apply$mcJIJ$sp(int,long)
public default void scala.Function2.apply$mcVIJ$sp(int,long)
public default boolean scala.Function2.apply$mcZJD$sp(long,double)
public default double scala.Function2.apply$mcDJD$sp(long,double)
public default float scala.Function2.apply$mcFJD$sp(long,double)
public default int scala.Function2.apply$mcIJD$sp(long,double)
public default long scala.Function2.apply$mcJJD$sp(long,double)
public default void scala.Function2.apply$mcVJD$sp(long,double)
public default boolean scala.Function2.apply$mcZJI$sp(long,int)
public default double scala.Function2.apply$mcDJI$sp(long,int)
public default float scala.Function2.apply$mcFJI$sp(long,int)
public default int scala.Function2.apply$mcIJI$sp(long,int)
public default long scala.Function2.apply$mcJJI$sp(long,int)
public default void scala.Function2.apply$mcVJI$sp(long,int)
public default boolean scala.Function2.apply$mcZJJ$sp(long,long)
public default double scala.Function2.apply$mcDJJ$sp(long,long)
public default float scala.Function2.apply$mcFJJ$sp(long,long)
public default int scala.Function2.apply$mcIJJ$sp(long,long)
public default long scala.Function2.apply$mcJJJ$sp(long,long)
public default void scala.Function2.apply$mcVJJ$sp(long,long){code}
> Address translation of LMF closure primitive args to Object in Scala 2.12
> -------------------------------------------------------------------------
>
> Key: SPARK-25044
> URL: https://issues.apache.org/jira/browse/SPARK-25044
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
> Affects Versions: 2.4.0
> Reporter: Sean Owen
> Priority: Major
>
> A few SQL-related tests fail in Scala 2.12, such as UDFSuite's "SPARK-24891
> Fix HandleNullInputsForUDF rule". (Details in a sec when I can copy-paste
> them.)
> It seems that the closure that is fed in as a UDF changes behavior, in a way
> that primitive-type arguments are handled differently. For example an Int
> argument, when fed 'null', acts like 0.
> I'm sure it's a difference in the LMF closure and how its types are
> understood, but not exactly sure of the cause yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
