I also think we should get this in: https://github.com/apache/spark/pull/22841
It's to deprecate a confusing & broken window function API, so we can remove them in 3.0 and redesign a better one. See https://issues.apache.org/jira/browse/SPARK-25841 for more information. On Thu, Oct 25, 2018 at 4:55 PM Sean Owen <sro...@gmail.com> wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical those > are pretty nice to have for 2.4. I think that's doable within the next 12 > hours even. > > @skonto I think there's one last minor thing needed on this PR? > https://github.com/apache/spark/pull/22838/files#r228363727 > > On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <cloud0...@gmail.com> wrote: > >> Any updates on this topic? https://github.com/apache/spark/pull/22827 is >> merged and 2.4 is unblocked. >> >> I'll cut RC5 shortly after the weekend, and it will be great to include >> the change proposed here. >> >> Thanks, >> Wenchen >> >> On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >>> I think it's worth getting in a change to just not enable this module, >>>> which ought to be entirely safe, and avoid two of the issues we >>>> identified. >>>> >>> >>> Besides disabling it, when someone wants to run the tests with 2.12 he >>> should be able to do so. So propagating the Scala profile still makes sense >>> but it is not related to the release other than making sure things work >>> fine. >>> >>> On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <sro...@gmail.com> wrote: >>> >>>> I think it's worth getting in a change to just not enable this module, >>>> which ought to be entirely safe, and avoid two of the issues we >>>> identified. >>>> that said it didn't block RC4 so need not block RC5. >>>> But should happen today if we're doing it. >>>> On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <gatorsm...@gmail.com> wrote: >>>> > >>>> > Hopefully, this will not delay RC5. Since this is not a blocker >>>> ticket, RC5 will start if all the blocker tickets are resolved. >>>> > >>>> > Thanks, >>>> > >>>> > Xiao >>>> > >>>> > Sean Owen <sro...@gmail.com> 于2018年10月25日周四 上午8:44写道： >>>> >> >>>> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 >>>> RC5 :) >>>> >> >>>> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >>>> >> <stavros.kontopou...@lightbend.com> wrote: >>>> >> > >>>> >> > I agree these tests should be manual for now but should be run >>>> somehow before a release to make sure things are working right? >>>> >> > >>>> >> > For the other issue: >>>> https://issues.apache.org/jira/browse/SPARK-25835 . >>>> >> > >>>> >> > >>>> >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < >>>> stavros.kontopou...@lightbend.com> wrote: >>>> >> >> >>>> >> >> I will open a jira for the profile propagation issue and have a >>>> look to fix it. >>>> >> >> >>>> >> >> Stavros >>>> >> >> >>>> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < >>>> eerla...@redhat.com> wrote: >>>> >> >>> >>>> >> >>> >>>> >> >>> I would be comfortable making the integration testing manual for >>>> now. A JIRA for ironing out how to make it reliable for automatic as a >>>> goal for 3.0 seems like a good idea. >>>> >> >>> >>>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <sro...@gmail.com> >>>> wrote: >>>> >> >>>> >>>> >> >>>> Forking this thread. >>>> >> >>>> >>>> >> >>>> Because we'll have another RC, we could possibly address these >>>> two >>>> >> >>>> issues. Only if we have a reliable change of course. >>>> >> >>>> >>>> >> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't >>>> hurt. >>>> >> >>>> >>>> >> >>>> And is it reasonable to essentially 'disable' >>>> >> >>>> kubernetes/integration-tests by removing it from the kubernetes >>>> >> >>>> profile? it doesn't mean it goes away, just means it's run >>>> manually, >>>> >> >>>> not automatically. Is that actually how it's meant to be used >>>> anyway? >>>> >> >>>> in the short term? given the discussion around its requirements >>>> and >>>> >> >>>> minikube and all that? >>>> >> >>>> >>>> >> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem >>>> too) >>>> >> >>>> >>>> >> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <sro...@gmail.com> >>>> wrote: >>>> >> >>>> > >>>> >> >>>> > To be clear I'm currently +1 on this release, with much >>>> commentary. >>>> >> >>>> > >>>> >> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I >>>> think we need to propagate the scala-2.12 build profile to make it work. Go >>>> for it, if you have a lead on what the change is. >>>> >> >>>> > This doesn't block the release as it's an issue for tests, >>>> and only affects 2.12. However if we had a clean fix for this and there >>>> were another RC, I'd include it. >>>> >> >>>> > >>>> >> >>>> > Dongjoon has a good point about the >>>> spark-kubernetes-integration-tests artifact. That doesn't sound like it >>>> should be published in this way, though, of course, we publish the test >>>> artifacts from every module already. This is only a bit odd in being a >>>> non-test artifact meant for testing. But it's special testing! So I also >>>> don't think that needs to block a release. >>>> >> >>>> > >>>> >> >>>> > This happens because the integration tests module is enabled >>>> with the 'kubernetes' profile too, and also this output is copied into the >>>> release tarball at kubernetes/integration-tests/tests. Do we need that in a >>>> binary release? >>>> >> >>>> > >>>> >> >>>> > If these integration tests are meant to be run ad hoc, >>>> manually, not part of a normal test cycle, then I think we can just not >>>> enable it with -Pkubernetes. If it is meant to run every time, then it >>>> sounds like we need a little extra work shown in recent PRs to make that >>>> easier, but then, this test code should just be the 'test' artifact parts >>>> of the kubernetes module, no? >>>> >> >>>> >>>> >> >>>> >>>> --------------------------------------------------------------------- >>>> >> >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >> >>>> >>>> >> >> >>>> >> >> >>>> >> > >>>> >> >>>> >> --------------------------------------------------------------------- >>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >> >>>> >>> >>> >>> >>> -- >>> Stavros Kontopoulos >>> >>> *Senior Software Engineer* >>> *Lightbend, Inc.* >>> >>> *p: +30 6977967274 <%2B1%20650%20678%200020>* >>> *e: stavros.kontopou...@lightbend.com* <dave.mar...@lightbend.com> >>> >>> >>>
