But DB's shell output is on the most recent 2.11, not 2.12, right? On Thu, Jun 7, 2018 at 5:54 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > I agree that's a little odd, could we not add the bacspace terminal > character? Regardless even if not, I don't think that should be a blocker > for 2.12 support especially since it doesn't degrade the 2.11 experience. > > On Thu, Jun 7, 2018, 5:53 PM DB Tsai <d_t...@apple.com> wrote: >> >> If we decide to initialize Spark in `initializeSynchronous()` in Scala >> 2.11.12, it will look like the following which is odd. >> >> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java >> 1.8.0_161) >> Type in expressions to have them evaluated. >> Type :help for more information. >> >> scala> Spark context Web UI available at http://192.168.1.169:4040 >> Spark context available as 'sc' (master = local[*], app id = >> local-1528180279528). >> Spark session available as 'spark’. >> scala> >> >> DB Tsai | Siri Open Source Technologies [not a contribution] | >> Apple, Inc >> >> On Jun 7, 2018, at 5:49 PM, Holden Karau <hol...@pigscanfly.ca> wrote: >> >> Tests can just be changed to accept either output too :p >> >> On Thu, Jun 7, 2018, 5:19 PM Dean Wampler <deanwamp...@gmail.com> wrote: >>> >>> Do the tests expect a particular console output order? That would annoy >>> them. ;) You could sort the expected and output lines, then diff... >>> >>> Dean Wampler, Ph.D. >>> VP, Fast Data Engineering at Lightbend >>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for >>> Streaming Applications, and other content from O'Reilly >>> @deanwampler >>> http://polyglotprogramming.com >>> https://github.com/deanwampler >>> >>> On Thu, Jun 7, 2018 at 5:09 PM, Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>>> >>>> If the difference is the order of the welcome message I think that >>>> should be fine. >>>> >>>> On Thu, Jun 7, 2018, 4:43 PM Dean Wampler <deanwamp...@gmail.com> wrote: >>>>> >>>>> I'll point the Scala team to this issue, but it's unlikely to get fixed >>>>> any time soon. >>>>> >>>>> dean >>>>> >>>>> Dean Wampler, Ph.D. >>>>> VP, Fast Data Engineering at Lightbend >>>>> Author: Programming Scala, 2nd Edition, Fast Data Architectures for >>>>> Streaming Applications, and other content from O'Reilly >>>>> @deanwampler >>>>> http://polyglotprogramming.com >>>>> https://github.com/deanwampler >>>>> >>>>> On Thu, Jun 7, 2018 at 4:27 PM, DB Tsai <d_t...@apple.com> wrote: >>>>>> >>>>>> Thanks Felix for bringing this up. >>>>>> >>>>>> Currently, in Scala 2.11.8, we initialize the Spark by overriding >>>>>> loadFIles() before REPL sees any file since there is no good hook in >>>>>> Scala >>>>>> to load our initialization code. >>>>>> >>>>>> In Scala 2.11.12 and newer version of the Scala 2.12.x, loadFIles() >>>>>> method was removed. >>>>>> >>>>>> Alternatively, one way we can do in the newer version of Scala is by >>>>>> overriding initializeSynchronous() suggested by Som Snytt; I have a >>>>>> working >>>>>> PR with this approach, >>>>>> https://github.com/apache/spark/pull/21495 , and this approach should >>>>>> work for older version of Scala too. >>>>>> >>>>>> However, in the newer version of Scala, the first thing that the REPL >>>>>> calls is printWelcome, so in the newer version of Scala, welcome message >>>>>> will be shown and then the URL of the SparkUI in this approach. This will >>>>>> cause UI inconsistencies between different versions of Scala. >>>>>> >>>>>> We can also initialize the Spark in the printWelcome which I feel more >>>>>> hacky. It will only work for newer version of Scala since in order >>>>>> version >>>>>> of Scala, printWelcome is called in the end of the initialization >>>>>> process. >>>>>> If we decide to go this route, basically users can not use Scala older >>>>>> than >>>>>> 2.11.9. >>>>>> >>>>>> I think this is also a blocker for us to move to newer version of >>>>>> Scala 2.12.x since the newer version of Scala 2.12.x has the same issue. >>>>>> >>>>>> In my opinion, Scala should fix the root cause and provide a stable >>>>>> hook for 3rd party developers to initialize their custom code. >>>>>> >>>>>> DB Tsai | Siri Open Source Technologies [not a contribution] | >>>>>> Apple, Inc >>>>>> >>>>>> > On Jun 7, 2018, at 6:43 AM, Felix Cheung <felixcheun...@hotmail.com> >>>>>> > wrote: >>>>>> > >>>>>> > +1 >>>>>> > >>>>>> > Spoke to Dean as well and mentioned the problem with 2.11.12 >>>>>> > https://github.com/scala/bug/issues/10913 >>>>>> > >>>>>> > _____________________________ >>>>>> > From: Sean Owen <sro...@gmail.com> >>>>>> > Sent: Wednesday, June 6, 2018 12:23 PM >>>>>> > Subject: Re: Scala 2.12 support >>>>>> > To: Holden Karau <hol...@pigscanfly.ca> >>>>>> > Cc: Dean Wampler <deanwamp...@gmail.com>, Reynold Xin >>>>>> > <r...@databricks.com>, dev <dev@spark.apache.org> >>>>>> > >>>>>> > >>>>>> > If it means no change to 2.11 support, seems OK to me for Spark >>>>>> > 2.4.0. The 2.12 support is separate and has never been mutually >>>>>> > compatible >>>>>> > with 2.11 builds anyway. (I also hope, suspect that the changes are >>>>>> > minimal; >>>>>> > tests are already almost entirely passing with no change to the closure >>>>>> > cleaner when built for 2.12) >>>>>> > >>>>>> > On Wed, Jun 6, 2018 at 1:33 PM Holden Karau <hol...@pigscanfly.ca> >>>>>> > wrote: >>>>>> > Just chatted with Dean @ the summit and it sounds like from Adriaan >>>>>> > there is a fix in 2.13 for the API change issue that could be back >>>>>> > ported to >>>>>> > 2.12 so how about we try and get this ball rolling? >>>>>> > >>>>>> > It sounds like it would also need a closure cleaner change, which >>>>>> > could be backwards compatible but since it’s such a core component and >>>>>> > we >>>>>> > might want to be cautious with it, we could when building for 2.11 use >>>>>> > the >>>>>> > old cleaner code and for 2.12 use the new code so we don’t break >>>>>> > anyone. >>>>>> > >>>>>> > How do folks feel about this? >>>>>> > >>>>>> > >>>>>> > >>>>>> >>>>> >>> >> >
-- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
