One more point: There was a time that we maintain 2 Spark REPL codebase for Scala 2.10 and 2.11, maybe we can do the same for Scala 2.11 and 2.12? if it's too hard to find a common way to do that between different Scala versions.
On Thu, Jun 7, 2018 at 6:20 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > But DB's shell output is on the most recent 2.11, not 2.12, right? > > On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > > I agree that's a little odd, could we not add the bacspace terminal > > character? Regardless even if not, I don't think that should be a blocker > > for 2.12 support especially since it doesn't degrade the 2.11 experience. > > > > On Thu, Jun 7, 2018, 5:53 PM DB Tsai <d_t...@apple.com> wrote: > >> > >> If we decide to initialize Spark in `initializeSynchronous()` in Scala > >> 2.11.12, it will look like the following which is odd. > >> > >> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java > >> 1.8.0_161) > >> Type in expressions to have them evaluated. > >> Type :help for more information. > >> > >> scala> Spark context Web UI available at http://192.168.1.169:4040 > >> Spark context available as 'sc' (master = local[*], app id = > >> local-1528180279528). > >> Spark session available as 'spark’. > >> scala> > >> > >> DB Tsai | Siri Open Source Technologies [not a contribution] | > >> Apple, Inc > >> > >> On Jun 7, 2018, at 5:49 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > >> > >> Tests can just be changed to accept either output too :p > >> > >> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler <deanwamp...@gmail.com> > wrote: > >>> > >>> Do the tests expect a particular console output order? That would annoy > >>> them. ;) You could sort the expected and output lines, then diff... > >>> > >>> Dean Wampler, Ph.D. > >>> VP, Fast Data Engineering at Lightbend > >>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for > >>> Streaming Applications, and other content from O'Reilly > >>> @deanwampler > >>> http://polyglotprogramming.com > >>> https://github.com/deanwampler > >>> > >>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau <hol...@pigscanfly.ca> > >>> wrote: > >>>> > >>>> If the difference is the order of the welcome message I think that > >>>> should be fine. > >>>> > >>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler <deanwamp...@gmail.com> > wrote: > >>>>> > >>>>> I'll point the Scala team to this issue, but it's unlikely to get > fixed > >>>>> any time soon. > >>>>> > >>>>> dean > >>>>> > >>>>> Dean Wampler, Ph.D. > >>>>> VP, Fast Data Engineering at Lightbend > >>>>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for > >>>>> Streaming Applications, and other content from O'Reilly > >>>>> @deanwampler > >>>>> http://polyglotprogramming.com > >>>>> https://github.com/deanwampler > >>>>> > >>>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai <d_t...@apple.com> wrote: > >>>>>> > >>>>>> Thanks Felix for bringing this up. > >>>>>> > >>>>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding > >>>>>> loadFIles() before REPL sees any file since there is no good hook > in Scala > >>>>>> to load our initialization code. > >>>>>> > >>>>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() > >>>>>> method was removed. > >>>>>> > >>>>>> Alternatively, one way we can do in the newer version of Scala is by > >>>>>> overriding initializeSynchronous() suggested by Som Snytt; I have a > working > >>>>>> PR with this approach, > >>>>>> https://github.com/apache/spark/pull/21495 , and this approach > should > >>>>>> work for older version of Scala too. > >>>>>> > >>>>>> However, in the newer version of Scala, the first thing that the > REPL > >>>>>> calls is printWelcome, so in the newer version of Scala, welcome > message > >>>>>> will be shown and then the URL of the SparkUI in this approach. > This will > >>>>>> cause UI inconsistencies between different versions of Scala. > >>>>>> > >>>>>> We can also initialize the Spark in the printWelcome which I feel > more > >>>>>> hacky. It will only work for newer version of Scala since in order > version > >>>>>> of Scala, printWelcome is called in the end of the initialization > process. > >>>>>> If we decide to go this route, basically users can not use Scala > older than > >>>>>> 2.11.9. > >>>>>> > >>>>>> I think this is also a blocker for us to move to newer version of > >>>>>> Scala 2.12.x since the newer version of Scala 2.12.x has the same > issue. > >>>>>> > >>>>>> In my opinion, Scala should fix the root cause and provide a stable > >>>>>> hook for 3rd party developers to initialize their custom code. > >>>>>> > >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | > >>>>>> Apple, Inc > >>>>>> > >>>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung < > felixcheun...@hotmail.com> > >>>>>> > wrote: > >>>>>> > > >>>>>> > +1 > >>>>>> > > >>>>>> > Spoke to Dean as well and mentioned the problem with 2.11.12 > >>>>>> > https://github.com/scala/bug/issues/10913 > >>>>>> > > >>>>>> > _____________________________ > >>>>>> > From: Sean Owen <sro...@gmail.com> > >>>>>> > Sent: Wednesday, June 6, 2018 12:23 PM > >>>>>> > Subject: Re: Scala 2.12 support > >>>>>> > To: Holden Karau <hol...@pigscanfly.ca> > >>>>>> > Cc: Dean Wampler <deanwamp...@gmail.com>, Reynold Xin > >>>>>> > <r...@databricks.com>, dev <dev@spark.apache.org> > >>>>>> > > >>>>>> > > >>>>>> > If it means no change to 2.11 support, seems OK to me for Spark > >>>>>> > 2.4.0. The 2.12 support is separate and has never been mutually > compatible > >>>>>> > with 2.11 builds anyway. (I also hope, suspect that the changes > are minimal; > >>>>>> > tests are already almost entirely passing with no change to the > closure > >>>>>> > cleaner when built for 2.12) > >>>>>> > > >>>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau <hol...@pigscanfly.ca > > > >>>>>> > wrote: > >>>>>> > Just chatted with Dean @ the summit and it sounds like from > Adriaan > >>>>>> > there is a fix in 2.13 for the API change issue that could be > back ported to > >>>>>> > 2.12 so how about we try and get this ball rolling? > >>>>>> > > >>>>>> > It sounds like it would also need a closure cleaner change, which > >>>>>> > could be backwards compatible but since it’s such a core > component and we > >>>>>> > might want to be cautious with it, we could when building for > 2.11 use the > >>>>>> > old cleaner code and for 2.12 use the new code so we don’t break > anyone. > >>>>>> > > >>>>>> > How do folks feel about this? > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>> > >>> > >> > > > > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
