Re: Make Scala 2.12 as default Scala version in Spark 3.0
I’d add if folks rely on Twitter in their stack, they might be stuck on
older versions for a while (of their Twitter libs) which might require they
stay on 2.11 for longer than they might otherwise like.
On Friday, November 16, 2018, Marcelo Vanzin
wrote:
> Now that the switch to 2.12 by default has been made, it might be good
> to have a serious discussion about dropping 2.11 altogether. Many of
> the main arguments have already been talked about. But I don't
> remember anyone mentioning how easy it would be to break the 2.11
> build now.
>
> For example, the following works fine in 2.12 but breaks in 2.11:
>
> java.util.Arrays.asList("hi").stream().forEach(println)
>
> We had a similar issue when we supported java 1.6 but the builds were
> all on 1.7 by default. Every once in a while something would silently
> break, because PR builds only check the default. And the jenkins
> builds, which are less monitored, would stay broken for a while.
>
> On Tue, Nov 6, 2018 at 11:13 AM DB Tsai wrote:
> >
> > We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next
> Spark version will be 3.0, so it's a great time to discuss should we make
> Scala 2.12 as default Scala version in Spark 3.0.
> >
> > Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely
> to support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed
> work per discussion in Scala community, https://github.com/scala/
> scala-dev/issues/559#issuecomment-436160166
> >
> > We have initial support of Scala 2.12 in Spark 2.4. If we decide to make
> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on
> bugs and issues that we may run into.
> >
> > What do you think?
> >
> > Thanks,
> >
> > DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
> >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
--
Justin Miller
Senior Data Engineer
*GoSpotCheck*
Direct: 720-517-3979 <+17205173979>
Email: jus...@gospotcheck.com
September 24-26, 2018
Denver, Colorado Learn More and Register
Re: Make Scala 2.12 as default Scala version in Spark 3.0
Now that the switch to 2.12 by default has been made, it might be good
to have a serious discussion about dropping 2.11 altogether. Many of
the main arguments have already been talked about. But I don't
remember anyone mentioning how easy it would be to break the 2.11
build now.
For example, the following works fine in 2.12 but breaks in 2.11:
java.util.Arrays.asList("hi").stream().forEach(println)
We had a similar issue when we supported java 1.6 but the builds were
all on 1.7 by default. Every once in a while something would silently
break, because PR builds only check the default. And the jenkins
builds, which are less monitored, would stay broken for a while.
On Tue, Nov 6, 2018 at 11:13 AM DB Tsai wrote:
>
> We made Scala 2.11 as default Scala version in Spark 2.0. Now, the next Spark
> version will be 3.0, so it's a great time to discuss should we make Scala
> 2.12 as default Scala version in Spark 3.0.
>
> Scala 2.11 is EOL, and it came out 4.5 ago; as a result, it's unlikely to
> support JDK 11 in Scala 2.11 unless we're willing to sponsor the needed work
> per discussion in Scala community,
> https://github.com/scala/scala-dev/issues/559#issuecomment-436160166
>
> We have initial support of Scala 2.12 in Spark 2.4. If we decide to make
> Scala 2.12 as default for Spark 3.0 now, we will have ample time to work on
> bugs and issues that we may run into.
>
> What do you think?
>
> Thanks,
>
> DB Tsai | Siri Open Source Technologies [not a contribution] | Apple,
> Inc
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
--
Marcelo
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Is spark.sql.codegen.factoryMode property really for tests only?
Hi Marco, Many thanks for such a quick response. With that, I'll direct my curiosity into a different direction. Thanks! Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski On Fri, Nov 16, 2018 at 1:44 PM Marco Gaido wrote: > Hi Jacek, > > I do believe it is correct. Please check the method you mentioned > (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant > only if Utils.isTesting. > > Thanks, > Marco > > Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski > ha scritto: > >> Hi, >> >> While reviewing the changes in 2.4 I stumbled >> upon spark.sql.codegen.factoryMode internal configuration property [1]. The >> doc says: >> >> > Note that this config works only for tests. >> >> Is that correct? I've got some doubts. >> >> I found that it's used in UnsafeProjection.create [2] (through >> CodeGeneratorWithInterpretedFallback.createObject) which is used outside >> the tests and so made me think if "this config works only for tests" part >> is correct. >> >> Are my doubts correct? If not, what am I missing? Thanks. >> >> [1] >> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767 >> >> [2] >> https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159 >> >> Pozdrawiam, >> Jacek Laskowski >> >> https://about.me/JacekLaskowski >> Mastering Spark SQL https://bit.ly/mastering-spark-sql >> Spark Structured Streaming https://bit.ly/spark-structured-streaming >> Mastering Kafka Streams https://bit.ly/mastering-kafka-streams >> Follow me at https://twitter.com/jaceklaskowski >> >
Re: Is spark.sql.codegen.factoryMode property really for tests only?
Hi Jacek, I do believe it is correct. Please check the method you mentioned (CodeGeneratorWithInterpretedFallback.createObject): the value is relevant only if Utils.isTesting. Thanks, Marco Il giorno ven 16 nov 2018 alle ore 13:28 Jacek Laskowski ha scritto: > Hi, > > While reviewing the changes in 2.4 I stumbled > upon spark.sql.codegen.factoryMode internal configuration property [1]. The > doc says: > > > Note that this config works only for tests. > > Is that correct? I've got some doubts. > > I found that it's used in UnsafeProjection.create [2] (through > CodeGeneratorWithInterpretedFallback.createObject) which is used outside > the tests and so made me think if "this config works only for tests" part > is correct. > > Are my doubts correct? If not, what am I missing? Thanks. > > [1] > https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767 > > [2] > https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159 > > Pozdrawiam, > Jacek Laskowski > > https://about.me/JacekLaskowski > Mastering Spark SQL https://bit.ly/mastering-spark-sql > Spark Structured Streaming https://bit.ly/spark-structured-streaming > Mastering Kafka Streams https://bit.ly/mastering-kafka-streams > Follow me at https://twitter.com/jaceklaskowski >
Is spark.sql.codegen.factoryMode property really for tests only?
Hi, While reviewing the changes in 2.4 I stumbled upon spark.sql.codegen.factoryMode internal configuration property [1]. The doc says: > Note that this config works only for tests. Is that correct? I've got some doubts. I found that it's used in UnsafeProjection.create [2] (through CodeGeneratorWithInterpretedFallback.createObject) which is used outside the tests and so made me think if "this config works only for tests" part is correct. Are my doubts correct? If not, what am I missing? Thanks. [1] https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L758-L767 [2] https://github.com/apache/spark/blob/v2.4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala#L159 Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski
