Re: Slight API incompatibility caused by SPARK-4072
One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also): https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 I think it might be reasonable that the Scala trait provides only source compatibitly and the Java class provides binary compatibility. - Patrick On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, Just noticed this when some of our tests started to fail. SPARK-4072 added a new method to the SparkListener trait, and even though it has a default implementation, it doesn't seem like that applies retroactively. Namely, if you have an existing, compiled app that has an implementation of SparkListener, that app won't work on 1.5 without a recompile. You'll get something like this: java.lang.AbstractMethodError at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Now I know that SparkListener is marked as @DeveloperApi, but is this something we should care about? Seems like adding methods to traits is just as backwards-incompatible as adding new methods to Java interfaces. -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Slight API incompatibility caused by SPARK-4072
Actually the java one is a concrete class. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell pwend...@gmail.com wrote: One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also): https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 I think it might be reasonable that the Scala trait provides only source compatibitly and the Java class provides binary compatibility. - Patrick On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, Just noticed this when some of our tests started to fail. SPARK-4072 added a new method to the SparkListener trait, and even though it has a default implementation, it doesn't seem like that applies retroactively. Namely, if you have an existing, compiled app that has an implementation of SparkListener, that app won't work on 1.5 without a recompile. You'll get something like this: java.lang.AbstractMethodError at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Now I know that SparkListener is marked as @DeveloperApi, but is this something we should care about? Seems like adding methods to traits is just as backwards-incompatible as adding new methods to Java interfaces. -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Slight API incompatibility caused by SPARK-4072
Or, alternatively, the bus could catch that error and ignore / log it, instead of stopping the context... On Wed, Jul 15, 2015 at 12:20 PM, Marcelo Vanzin van...@cloudera.com wrote: Hmm, the Java listener was added in 1.3, so I think it will work for my needs. Might be worth it to make it clear in the SparkListener documentation that people should avoid using it directly. Or follow Reynold's suggestion. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell pwend...@gmail.com wrote: One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also): https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 I think it might be reasonable that the Scala trait provides only source compatibitly and the Java class provides binary compatibility. - Patrick On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, Just noticed this when some of our tests started to fail. SPARK-4072 added a new method to the SparkListener trait, and even though it has a default implementation, it doesn't seem like that applies retroactively. Namely, if you have an existing, compiled app that has an implementation of SparkListener, that app won't work on 1.5 without a recompile. You'll get something like this: java.lang.AbstractMethodError at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Now I know that SparkListener is marked as @DeveloperApi, but is this something we should care about? Seems like adding methods to traits is just as backwards-incompatible as adding new methods to Java interfaces. -- Marcelo -- Marcelo -- Marcelo
Re: Slight API incompatibility caused by SPARK-4072
It's bad that expose a trait - even though we want to mixin stuff. We should really audit all of these and expose only abstract classes for anything beyond an extremely simple interface. That itself however would break binary compatibility. On Wed, Jul 15, 2015 at 12:15 PM, Patrick Wendell pwend...@gmail.com wrote: Actually the java one is a concrete class. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell pwend...@gmail.com wrote: One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also): https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 I think it might be reasonable that the Scala trait provides only source compatibitly and the Java class provides binary compatibility. - Patrick On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, Just noticed this when some of our tests started to fail. SPARK-4072 added a new method to the SparkListener trait, and even though it has a default implementation, it doesn't seem like that applies retroactively. Namely, if you have an existing, compiled app that has an implementation of SparkListener, that app won't work on 1.5 without a recompile. You'll get something like this: java.lang.AbstractMethodError at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Now I know that SparkListener is marked as @DeveloperApi, but is this something we should care about? Seems like adding methods to traits is just as backwards-incompatible as adding new methods to Java interfaces. -- Marcelo - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Slight API incompatibility caused by SPARK-4072
Hmm, the Java listener was added in 1.3, so I think it will work for my needs. Might be worth it to make it clear in the SparkListener documentation that people should avoid using it directly. Or follow Reynold's suggestion. On Wed, Jul 15, 2015 at 12:14 PM, Patrick Wendell pwend...@gmail.com wrote: One related note here is that we have a Java version of this that is an abstract class - in the doc it says that it exists more or less to allow for binary compatibility (it says it's for Java users, but really Scala could use this also): https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/JavaSparkListener.java#L23 I think it might be reasonable that the Scala trait provides only source compatibitly and the Java class provides binary compatibility. - Patrick On Wed, Jul 15, 2015 at 11:47 AM, Marcelo Vanzin van...@cloudera.com wrote: Hey all, Just noticed this when some of our tests started to fail. SPARK-4072 added a new method to the SparkListener trait, and even though it has a default implementation, it doesn't seem like that applies retroactively. Namely, if you have an existing, compiled app that has an implementation of SparkListener, that app won't work on 1.5 without a recompile. You'll get something like this: java.lang.AbstractMethodError at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1235) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) Now I know that SparkListener is marked as @DeveloperApi, but is this something we should care about? Seems like adding methods to traits is just as backwards-incompatible as adding new methods to Java interfaces. -- Marcelo -- Marcelo
