Re: What if anything to fix about k8s for the 2.4.0 RC5?
This is all merged to master/2.4. AFAIK there aren't any items I'm monitoring that are needed for 2.4. On Thu, Oct 25, 2018 at 6:54 PM Sean Owen wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical those > are pretty nice to have for 2.4. I think that's doable within the next 12 > hours even. > > @skonto I think there's one last minor thing needed on this PR? > https://github.com/apache/spark/pull/22838/files#r228363727 > > On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan wrote: > >> Any updates on this topic? https://github.com/apache/spark/pull/22827 is >> merged and 2.4 is unblocked. >> >> I'll cut RC5 shortly after the weekend, and it will be great to include >> the change proposed here. >> >> Thanks, >> Wenchen >> >
Re: What if anything to fix about k8s for the 2.4.0 RC5?
Sean, Yes, I updated the PR and re-run it. On Fri, Oct 26, 2018 at 2:54 AM, Sean Owen wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical those > are pretty nice to have for 2.4. I think that's doable within the next 12 > hours even. > > @skonto I think there's one last minor thing needed on this PR? > https://github.com/apache/spark/pull/22838/files#r228363727 > > On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan wrote: > >> Any updates on this topic? https://github.com/apache/spark/pull/22827 is >> merged and 2.4 is unblocked. >> >> I'll cut RC5 shortly after the weekend, and it will be great to include >> the change proposed here. >> >> Thanks, >> Wenchen >> >> On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >>> I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. >>> >>> Besides disabling it, when someone wants to run the tests with 2.12 he >>> should be able to do so. So propagating the Scala profile still makes sense >>> but it is not related to the release other than making sure things work >>> fine. >>> >>> On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen wrote: >>> I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. that said it didn't block RC4 so need not block RC5. But should happen today if we're doing it. On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: > > Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved. > > Thanks, > > Xiao > > Sean Owen 于2018年10月25日周四 上午8:44写道： >> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) >> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >> wrote: >> > >> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right? >> > >> > For the other issue: https://issues.apache.org/ jira/browse/SPARK-25835 . >> > >> > >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: >> >> >> >> I will open a jira for the profile propagation issue and have a look to fix it. >> >> >> >> Stavros >> >> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < eerla...@redhat.com> wrote: >> >>> >> >>> >> >>> I would be comfortable making the integration testing manual for now. A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea. >> >>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: >> >> Forking this thread. >> >> Because we'll have another RC, we could possibly address these two >> issues. Only if we have a reliable change of course. >> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. >> >> And is it reasonable to essentially 'disable' >> kubernetes/integration-tests by removing it from the kubernetes >> profile? it doesn't mean it goes away, just means it's run manually, >> not automatically. Is that actually how it's meant to be used anyway? >> in the short term? given the discussion around its requirements and >> minikube and all that? >> >> (Actually, this would also 'solve' the Scala 2.12 build problem too) >> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: >> > >> > To be clear I'm currently +1 on this release, with much commentary. >> > >> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is. >> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it. >> > >> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release. >> > >> > This happens because the integration tests module is enabled with the 'kuber
Re: What if anything to fix about k8s for the 2.4.0 RC5?
I also think we should get this in: https://github.com/apache/spark/pull/22841 It's to deprecate a confusing & broken window function API, so we can remove them in 3.0 and redesign a better one. See https://issues.apache.org/jira/browse/SPARK-25841 for more information. On Thu, Oct 25, 2018 at 4:55 PM Sean Owen wrote: > Yep, we're going to merge a change to separate the k8s tests into a > separate profile, and fix up the Scala 2.12 thing. While non-critical those > are pretty nice to have for 2.4. I think that's doable within the next 12 > hours even. > > @skonto I think there's one last minor thing needed on this PR? > https://github.com/apache/spark/pull/22838/files#r228363727 > > On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan wrote: > >> Any updates on this topic? https://github.com/apache/spark/pull/22827 is >> merged and 2.4 is unblocked. >> >> I'll cut RC5 shortly after the weekend, and it will be great to include >> the change proposed here. >> >> Thanks, >> Wenchen >> >> On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >>> I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. >>> >>> Besides disabling it, when someone wants to run the tests with 2.12 he >>> should be able to do so. So propagating the Scala profile still makes sense >>> but it is not related to the release other than making sure things work >>> fine. >>> >>> On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen wrote: >>> I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. that said it didn't block RC4 so need not block RC5. But should happen today if we're doing it. On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: > > Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved. > > Thanks, > > Xiao > > Sean Owen 于2018年10月25日周四 上午8:44写道： >> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) >> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >> wrote: >> > >> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right? >> > >> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . >> > >> > >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: >> >> >> >> I will open a jira for the profile propagation issue and have a look to fix it. >> >> >> >> Stavros >> >> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < eerla...@redhat.com> wrote: >> >>> >> >>> >> >>> I would be comfortable making the integration testing manual for now. A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea. >> >>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: >> >> Forking this thread. >> >> Because we'll have another RC, we could possibly address these two >> issues. Only if we have a reliable change of course. >> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. >> >> And is it reasonable to essentially 'disable' >> kubernetes/integration-tests by removing it from the kubernetes >> profile? it doesn't mean it goes away, just means it's run manually, >> not automatically. Is that actually how it's meant to be used anyway? >> in the short term? given the discussion around its requirements and >> minikube and all that? >> >> (Actually, this would also 'solve' the Scala 2.12 build problem too) >> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: >> > >> > To be clear I'm currently +1 on this release, with much commentary. >> > >> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is. >> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it. >> > >> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-t
Re: What if anything to fix about k8s for the 2.4.0 RC5?
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even. @skonto I think there's one last minor thing needed on this PR? https://github.com/apache/spark/pull/22838/files#r228363727 On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan wrote: > Any updates on this topic? https://github.com/apache/spark/pull/22827 is > merged and 2.4 is unblocked. > > I'll cut RC5 shortly after the weekend, and it will be great to include > the change proposed here. > > Thanks, > Wenchen > > On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> I think it's worth getting in a change to just not enable this module, >>> which ought to be entirely safe, and avoid two of the issues we >>> identified. >>> >> >> Besides disabling it, when someone wants to run the tests with 2.12 he >> should be able to do so. So propagating the Scala profile still makes sense >> but it is not related to the release other than making sure things work >> fine. >> >> On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen wrote: >> >>> I think it's worth getting in a change to just not enable this module, >>> which ought to be entirely safe, and avoid two of the issues we >>> identified. >>> that said it didn't block RC4 so need not block RC5. >>> But should happen today if we're doing it. >>> On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: >>> > >>> > Hopefully, this will not delay RC5. Since this is not a blocker >>> ticket, RC5 will start if all the blocker tickets are resolved. >>> > >>> > Thanks, >>> > >>> > Xiao >>> > >>> > Sean Owen 于2018年10月25日周四 上午8:44写道： >>> >> >>> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 >>> RC5 :) >>> >> >>> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >>> >> wrote: >>> >> > >>> >> > I agree these tests should be manual for now but should be run >>> somehow before a release to make sure things are working right? >>> >> > >>> >> > For the other issue: >>> https://issues.apache.org/jira/browse/SPARK-25835 . >>> >> > >>> >> > >>> >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < >>> stavros.kontopou...@lightbend.com> wrote: >>> >> >> >>> >> >> I will open a jira for the profile propagation issue and have a >>> look to fix it. >>> >> >> >>> >> >> Stavros >>> >> >> >>> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < >>> eerla...@redhat.com> wrote: >>> >> >>> >>> >> >>> >>> >> >>> I would be comfortable making the integration testing manual for >>> now. A JIRA for ironing out how to make it reliable for automatic as a >>> goal for 3.0 seems like a good idea. >>> >> >>> >>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen >>> wrote: >>> >> >>> >> Forking this thread. >>> >> >>> >> Because we'll have another RC, we could possibly address these >>> two >>> >> issues. Only if we have a reliable change of course. >>> >> >>> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't >>> hurt. >>> >> >>> >> And is it reasonable to essentially 'disable' >>> >> kubernetes/integration-tests by removing it from the kubernetes >>> >> profile? it doesn't mean it goes away, just means it's run >>> manually, >>> >> not automatically. Is that actually how it's meant to be used >>> anyway? >>> >> in the short term? given the discussion around its requirements >>> and >>> >> minikube and all that? >>> >> >>> >> (Actually, this would also 'solve' the Scala 2.12 build problem >>> too) >>> >> >>> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen >>> wrote: >>> >> > >>> >> > To be clear I'm currently +1 on this release, with much >>> commentary. >>> >> > >>> >> > OK, the explanation for kubernetes tests makes sense. Yes I >>> think we need to propagate the scala-2.12 build profile to make it work. Go >>> for it, if you have a lead on what the change is. >>> >> > This doesn't block the release as it's an issue for tests, and >>> only affects 2.12. However if we had a clean fix for this and there were >>> another RC, I'd include it. >>> >> > >>> >> > Dongjoon has a good point about the >>> spark-kubernetes-integration-tests artifact. That doesn't sound like it >>> should be published in this way, though, of course, we publish the test >>> artifacts from every module already. This is only a bit odd in being a >>> non-test artifact meant for testing. But it's special testing! So I also >>> don't think that needs to block a release. >>> >> > >>> >> > This happens because the integration tests module is enabled >>> with the 'kubernetes' profile too, and also this output is copied into the >>> release tarball at kubernetes/integration-tests/tests. Do we need that in a >>> binary release? >>> >> > >>> >> > If these integration tests are meant to be
Re: What if anything to fix about k8s for the 2.4.0 RC5?
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked. I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here. Thanks, Wenchen On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > I think it's worth getting in a change to just not enable this module, >> which ought to be entirely safe, and avoid two of the issues we >> identified. >> > > Besides disabling it, when someone wants to run the tests with 2.12 he > should be able to do so. So propagating the Scala profile still makes sense > but it is not related to the release other than making sure things work > fine. > > On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen wrote: > >> I think it's worth getting in a change to just not enable this module, >> which ought to be entirely safe, and avoid two of the issues we >> identified. >> that said it didn't block RC4 so need not block RC5. >> But should happen today if we're doing it. >> On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: >> > >> > Hopefully, this will not delay RC5. Since this is not a blocker ticket, >> RC5 will start if all the blocker tickets are resolved. >> > >> > Thanks, >> > >> > Xiao >> > >> > Sean Owen 于2018年10月25日周四 上午8:44写道： >> >> >> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 >> :) >> >> >> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >> >> wrote: >> >> > >> >> > I agree these tests should be manual for now but should be run >> somehow before a release to make sure things are working right? >> >> > >> >> > For the other issue: >> https://issues.apache.org/jira/browse/SPARK-25835 . >> >> > >> >> > >> >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >> >> >> >> >> I will open a jira for the profile propagation issue and have a >> look to fix it. >> >> >> >> >> >> Stavros >> >> >> >> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson < >> eerla...@redhat.com> wrote: >> >> >>> >> >> >>> >> >> >>> I would be comfortable making the integration testing manual for >> now. A JIRA for ironing out how to make it reliable for automatic as a >> goal for 3.0 seems like a good idea. >> >> >>> >> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen >> wrote: >> >> >> >> Forking this thread. >> >> >> >> Because we'll have another RC, we could possibly address these two >> >> issues. Only if we have a reliable change of course. >> >> >> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't >> hurt. >> >> >> >> And is it reasonable to essentially 'disable' >> >> kubernetes/integration-tests by removing it from the kubernetes >> >> profile? it doesn't mean it goes away, just means it's run >> manually, >> >> not automatically. Is that actually how it's meant to be used >> anyway? >> >> in the short term? given the discussion around its requirements >> and >> >> minikube and all that? >> >> >> >> (Actually, this would also 'solve' the Scala 2.12 build problem >> too) >> >> >> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen >> wrote: >> >> > >> >> > To be clear I'm currently +1 on this release, with much >> commentary. >> >> > >> >> > OK, the explanation for kubernetes tests makes sense. Yes I >> think we need to propagate the scala-2.12 build profile to make it work. Go >> for it, if you have a lead on what the change is. >> >> > This doesn't block the release as it's an issue for tests, and >> only affects 2.12. However if we had a clean fix for this and there were >> another RC, I'd include it. >> >> > >> >> > Dongjoon has a good point about the >> spark-kubernetes-integration-tests artifact. That doesn't sound like it >> should be published in this way, though, of course, we publish the test >> artifacts from every module already. This is only a bit odd in being a >> non-test artifact meant for testing. But it's special testing! So I also >> don't think that needs to block a release. >> >> > >> >> > This happens because the integration tests module is enabled >> with the 'kubernetes' profile too, and also this output is copied into the >> release tarball at kubernetes/integration-tests/tests. Do we need that in a >> binary release? >> >> > >> >> > If these integration tests are meant to be run ad hoc, >> manually, not part of a normal test cycle, then I think we can just not >> enable it with -Pkubernetes. If it is meant to run every time, then it >> sounds like we need a little extra work shown in recent PRs to make that >> easier, but then, this test code should just be the 'test' artifact parts >> of the kubernetes module, no? >> >> >> >> >> - >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> >> >> >> >> >> >> >>
Re: What if anything to fix about k8s for the 2.4.0 RC5?
> > I think it's worth getting in a change to just not enable this module, > which ought to be entirely safe, and avoid two of the issues we > identified. > Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense but it is not related to the release other than making sure things work fine. On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen wrote: > I think it's worth getting in a change to just not enable this module, > which ought to be entirely safe, and avoid two of the issues we > identified. > that said it didn't block RC4 so need not block RC5. > But should happen today if we're doing it. > On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: > > > > Hopefully, this will not delay RC5. Since this is not a blocker ticket, > RC5 will start if all the blocker tickets are resolved. > > > > Thanks, > > > > Xiao > > > > Sean Owen 于2018年10月25日周四 上午8:44写道： > >> > >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 > :) > >> > >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos > >> wrote: > >> > > >> > I agree these tests should be manual for now but should be run > somehow before a release to make sure things are working right? > >> > > >> > For the other issue: https://issues.apache.org/ > jira/browse/SPARK-25835 . > >> > > >> > > >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> >> > >> >> I will open a jira for the profile propagation issue and have a look > to fix it. > >> >> > >> >> Stavros > >> >> > >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson > wrote: > >> >>> > >> >>> > >> >>> I would be comfortable making the integration testing manual for > now. A JIRA for ironing out how to make it reliable for automatic as a > goal for 3.0 seems like a good idea. > >> >>> > >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: > >> > >> Forking this thread. > >> > >> Because we'll have another RC, we could possibly address these two > >> issues. Only if we have a reliable change of course. > >> > >> Is it easy enough to propagate the -Pscala-2.12 profile? can't > hurt. > >> > >> And is it reasonable to essentially 'disable' > >> kubernetes/integration-tests by removing it from the kubernetes > >> profile? it doesn't mean it goes away, just means it's run > manually, > >> not automatically. Is that actually how it's meant to be used > anyway? > >> in the short term? given the discussion around its requirements and > >> minikube and all that? > >> > >> (Actually, this would also 'solve' the Scala 2.12 build problem > too) > >> > >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen > wrote: > >> > > >> > To be clear I'm currently +1 on this release, with much > commentary. > >> > > >> > OK, the explanation for kubernetes tests makes sense. Yes I > think we need to propagate the scala-2.12 build profile to make it work. Go > for it, if you have a lead on what the change is. > >> > This doesn't block the release as it's an issue for tests, and > only affects 2.12. However if we had a clean fix for this and there were > another RC, I'd include it. > >> > > >> > Dongjoon has a good point about the > >> > spark-kubernetes-integration-tests > artifact. That doesn't sound like it should be published in this way, > though, of course, we publish the test artifacts from every module already. > This is only a bit odd in being a non-test artifact meant for testing. But > it's special testing! So I also don't think that needs to block a release. > >> > > >> > This happens because the integration tests module is enabled > with the 'kubernetes' profile too, and also this output is copied into the > release tarball at kubernetes/integration-tests/tests. Do we need that in > a binary release? > >> > > >> > If these integration tests are meant to be run ad hoc, manually, > not part of a normal test cycle, then I think we can just not enable it > with -Pkubernetes. If it is meant to run every time, then it sounds like we > need a little extra work shown in recent PRs to make that easier, but then, > this test code should just be the 'test' artifact parts of the kubernetes > module, no? > >> > >> > - > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> > >> >> > >> >> > >> > > >> > >> - > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> > -- Stavros Kontopoulos *Senior Software Engineer* *Lightbend, Inc.* *p: +30 6977967274 <%2B1%20650%20678%200020>* *e: stavros.kontopou...@lightbend.com*
Re: What if anything to fix about k8s for the 2.4.0 RC5?
I think it's worth getting in a change to just not enable this module, which ought to be entirely safe, and avoid two of the issues we identified. that said it didn't block RC4 so need not block RC5. But should happen today if we're doing it. On Thu, Oct 25, 2018 at 10:47 AM Xiao Li wrote: > > Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 > will start if all the blocker tickets are resolved. > > Thanks, > > Xiao > > Sean Owen 于2018年10月25日周四 上午8:44写道： >> >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) >> >> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos >> wrote: >> > >> > I agree these tests should be manual for now but should be run somehow >> > before a release to make sure things are working right? >> > >> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . >> > >> > >> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos >> > wrote: >> >> >> >> I will open a jira for the profile propagation issue and have a look to >> >> fix it. >> >> >> >> Stavros >> >> >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson >> >> wrote: >> >>> >> >>> >> >>> I would be comfortable making the integration testing manual for now. A >> >>> JIRA for ironing out how to make it reliable for automatic as a goal for >> >>> 3.0 seems like a good idea. >> >>> >> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: >> >> Forking this thread. >> >> Because we'll have another RC, we could possibly address these two >> issues. Only if we have a reliable change of course. >> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. >> >> And is it reasonable to essentially 'disable' >> kubernetes/integration-tests by removing it from the kubernetes >> profile? it doesn't mean it goes away, just means it's run manually, >> not automatically. Is that actually how it's meant to be used anyway? >> in the short term? given the discussion around its requirements and >> minikube and all that? >> >> (Actually, this would also 'solve' the Scala 2.12 build problem too) >> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: >> > >> > To be clear I'm currently +1 on this release, with much commentary. >> > >> > OK, the explanation for kubernetes tests makes sense. Yes I think we >> > need to propagate the scala-2.12 build profile to make it work. Go >> > for it, if you have a lead on what the change is. >> > This doesn't block the release as it's an issue for tests, and only >> > affects 2.12. However if we had a clean fix for this and there were >> > another RC, I'd include it. >> > >> > Dongjoon has a good point about the >> > spark-kubernetes-integration-tests artifact. That doesn't sound like >> > it should be published in this way, though, of course, we publish the >> > test artifacts from every module already. This is only a bit odd in >> > being a non-test artifact meant for testing. But it's special >> > testing! So I also don't think that needs to block a release. >> > >> > This happens because the integration tests module is enabled with the >> > 'kubernetes' profile too, and also this output is copied into the >> > release tarball at kubernetes/integration-tests/tests. Do we need >> > that in a binary release? >> > >> > If these integration tests are meant to be run ad hoc, manually, not >> > part of a normal test cycle, then I think we can just not enable it >> > with -Pkubernetes. If it is meant to run every time, then it sounds >> > like we need a little extra work shown in recent PRs to make that >> > easier, but then, this test code should just be the 'test' artifact >> > parts of the kubernetes module, no? >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> >> >> >> >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: What if anything to fix about k8s for the 2.4.0 RC5?
Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved. Thanks, Xiao Sean Owen 于2018年10月25日周四 上午8:44写道： > Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) > > On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos > wrote: > > > > I agree these tests should be manual for now but should be run somehow > before a release to make sure things are working right? > > > > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . > > > > > > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> > >> I will open a jira for the profile propagation issue and have a look to > fix it. > >> > >> Stavros > >> > >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson > wrote: > >>> > >>> > >>> I would be comfortable making the integration testing manual for now. > A JIRA for ironing out how to make it reliable for automatic as a goal for > 3.0 seems like a good idea. > >>> > >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: > > Forking this thread. > > Because we'll have another RC, we could possibly address these two > issues. Only if we have a reliable change of course. > > Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. > > And is it reasonable to essentially 'disable' > kubernetes/integration-tests by removing it from the kubernetes > profile? it doesn't mean it goes away, just means it's run manually, > not automatically. Is that actually how it's meant to be used anyway? > in the short term? given the discussion around its requirements and > minikube and all that? > > (Actually, this would also 'solve' the Scala 2.12 build problem too) > > On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: > > > > To be clear I'm currently +1 on this release, with much commentary. > > > > OK, the explanation for kubernetes tests makes sense. Yes I think > we need to propagate the scala-2.12 build profile to make it work. Go for > it, if you have a lead on what the change is. > > This doesn't block the release as it's an issue for tests, and only > affects 2.12. However if we had a clean fix for this and there were another > RC, I'd include it. > > > > Dongjoon has a good point about the > spark-kubernetes-integration-tests artifact. That doesn't sound like it > should be published in this way, though, of course, we publish the test > artifacts from every module already. This is only a bit odd in being a > non-test artifact meant for testing. But it's special testing! So I also > don't think that needs to block a release. > > > > This happens because the integration tests module is enabled with > the 'kubernetes' profile too, and also this output is copied into the > release tarball at kubernetes/integration-tests/tests. Do we need that in a > binary release? > > > > If these integration tests are meant to be run ad hoc, manually, > not part of a normal test cycle, then I think we can just not enable it > with -Pkubernetes. If it is meant to run every time, then it sounds like we > need a little extra work shown in recent PRs to make that easier, but then, > this test code should just be the 'test' artifact parts of the kubernetes > module, no? > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > >> > >> > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
Re: What if anything to fix about k8s for the 2.4.0 RC5?
Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :) On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos wrote: > > I agree these tests should be manual for now but should be run somehow before > a release to make sure things are working right? > > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . > > > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos > wrote: >> >> I will open a jira for the profile propagation issue and have a look to fix >> it. >> >> Stavros >> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson wrote: >>> >>> >>> I would be comfortable making the integration testing manual for now. A >>> JIRA for ironing out how to make it reliable for automatic as a goal for >>> 3.0 seems like a good idea. >>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: Forking this thread. Because we'll have another RC, we could possibly address these two issues. Only if we have a reliable change of course. Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. And is it reasonable to essentially 'disable' kubernetes/integration-tests by removing it from the kubernetes profile? it doesn't mean it goes away, just means it's run manually, not automatically. Is that actually how it's meant to be used anyway? in the short term? given the discussion around its requirements and minikube and all that? (Actually, this would also 'solve' the Scala 2.12 build problem too) On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: > > To be clear I'm currently +1 on this release, with much commentary. > > OK, the explanation for kubernetes tests makes sense. Yes I think we > need to propagate the scala-2.12 build profile to make it work. Go for > it, if you have a lead on what the change is. > This doesn't block the release as it's an issue for tests, and only > affects 2.12. However if we had a clean fix for this and there were > another RC, I'd include it. > > Dongjoon has a good point about the spark-kubernetes-integration-tests > artifact. That doesn't sound like it should be published in this way, > though, of course, we publish the test artifacts from every module > already. This is only a bit odd in being a non-test artifact meant for > testing. But it's special testing! So I also don't think that needs to > block a release. > > This happens because the integration tests module is enabled with the > 'kubernetes' profile too, and also this output is copied into the > release tarball at kubernetes/integration-tests/tests. Do we need that > in a binary release? > > If these integration tests are meant to be run ad hoc, manually, not > part of a normal test cycle, then I think we can just not enable it with > -Pkubernetes. If it is meant to run every time, then it sounds like we > need a little extra work shown in recent PRs to make that easier, but > then, this test code should just be the 'test' artifact parts of the > kubernetes module, no? - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: What if anything to fix about k8s for the 2.4.0 RC5?
I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right? For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > I will open a jira for the profile propagation issue and have a look to > fix it. > > Stavros > > On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson > wrote: > >> >> I would be comfortable making the integration testing manual for now. A >> JIRA for ironing out how to make it reliable for automatic as a goal for >> 3.0 seems like a good idea. >> >> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: >> >>> Forking this thread. >>> >>> Because we'll have another RC, we could possibly address these two >>> issues. Only if we have a reliable change of course. >>> >>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. >>> >>> And is it reasonable to essentially 'disable' >>> kubernetes/integration-tests by removing it from the kubernetes >>> profile? it doesn't mean it goes away, just means it's run manually, >>> not automatically. Is that actually how it's meant to be used anyway? >>> in the short term? given the discussion around its requirements and >>> minikube and all that? >>> >>> (Actually, this would also 'solve' the Scala 2.12 build problem too) >>> >>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: >>> > >>> > To be clear I'm currently +1 on this release, with much commentary. >>> > >>> > OK, the explanation for kubernetes tests makes sense. Yes I think we >>> need to propagate the scala-2.12 build profile to make it work. Go for it, >>> if you have a lead on what the change is. >>> > This doesn't block the release as it's an issue for tests, and only >>> affects 2.12. However if we had a clean fix for this and there were another >>> RC, I'd include it. >>> > >>> > Dongjoon has a good point about the spark-kubernetes-integration-tests >>> artifact. That doesn't sound like it should be published in this way, >>> though, of course, we publish the test artifacts from every module already. >>> This is only a bit odd in being a non-test artifact meant for testing. But >>> it's special testing! So I also don't think that needs to block a release. >>> > >>> > This happens because the integration tests module is enabled with the >>> 'kubernetes' profile too, and also this output is copied into the release >>> tarball at kubernetes/integration-tests/tests. Do we need that in a >>> binary release? >>> > >>> > If these integration tests are meant to be run ad hoc, manually, not >>> part of a normal test cycle, then I think we can just not enable it with >>> -Pkubernetes. If it is meant to run every time, then it sounds like we need >>> a little extra work shown in recent PRs to make that easier, but then, this >>> test code should just be the 'test' artifact parts of the kubernetes >>> module, no? >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> > >
Re: What if anything to fix about k8s for the 2.4.0 RC5?
I will open a jira for the profile propagation issue and have a look to fix it. Stavros On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson wrote: > > I would be comfortable making the integration testing manual for now. A > JIRA for ironing out how to make it reliable for automatic as a goal for > 3.0 seems like a good idea. > > On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: > >> Forking this thread. >> >> Because we'll have another RC, we could possibly address these two >> issues. Only if we have a reliable change of course. >> >> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. >> >> And is it reasonable to essentially 'disable' >> kubernetes/integration-tests by removing it from the kubernetes >> profile? it doesn't mean it goes away, just means it's run manually, >> not automatically. Is that actually how it's meant to be used anyway? >> in the short term? given the discussion around its requirements and >> minikube and all that? >> >> (Actually, this would also 'solve' the Scala 2.12 build problem too) >> >> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: >> > >> > To be clear I'm currently +1 on this release, with much commentary. >> > >> > OK, the explanation for kubernetes tests makes sense. Yes I think we >> need to propagate the scala-2.12 build profile to make it work. Go for it, >> if you have a lead on what the change is. >> > This doesn't block the release as it's an issue for tests, and only >> affects 2.12. However if we had a clean fix for this and there were another >> RC, I'd include it. >> > >> > Dongjoon has a good point about the spark-kubernetes-integration-tests >> artifact. That doesn't sound like it should be published in this way, >> though, of course, we publish the test artifacts from every module already. >> This is only a bit odd in being a non-test artifact meant for testing. But >> it's special testing! So I also don't think that needs to block a release. >> > >> > This happens because the integration tests module is enabled with the >> 'kubernetes' profile too, and also this output is copied into the release >> tarball at kubernetes/integration-tests/tests. Do we need that in a >> binary release? >> > >> > If these integration tests are meant to be run ad hoc, manually, not >> part of a normal test cycle, then I think we can just not enable it with >> -Pkubernetes. If it is meant to run every time, then it sounds like we need >> a little extra work shown in recent PRs to make that easier, but then, this >> test code should just be the 'test' artifact parts of the kubernetes >> module, no? >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>
Re: What if anything to fix about k8s for the 2.4.0 RC5?
I would be comfortable making the integration testing manual for now. A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea. On Thu, Oct 25, 2018 at 8:11 AM Sean Owen wrote: > Forking this thread. > > Because we'll have another RC, we could possibly address these two > issues. Only if we have a reliable change of course. > > Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. > > And is it reasonable to essentially 'disable' > kubernetes/integration-tests by removing it from the kubernetes > profile? it doesn't mean it goes away, just means it's run manually, > not automatically. Is that actually how it's meant to be used anyway? > in the short term? given the discussion around its requirements and > minikube and all that? > > (Actually, this would also 'solve' the Scala 2.12 build problem too) > > On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: > > > > To be clear I'm currently +1 on this release, with much commentary. > > > > OK, the explanation for kubernetes tests makes sense. Yes I think we > need to propagate the scala-2.12 build profile to make it work. Go for it, > if you have a lead on what the change is. > > This doesn't block the release as it's an issue for tests, and only > affects 2.12. However if we had a clean fix for this and there were another > RC, I'd include it. > > > > Dongjoon has a good point about the spark-kubernetes-integration-tests > artifact. That doesn't sound like it should be published in this way, > though, of course, we publish the test artifacts from every module already. > This is only a bit odd in being a non-test artifact meant for testing. But > it's special testing! So I also don't think that needs to block a release. > > > > This happens because the integration tests module is enabled with the > 'kubernetes' profile too, and also this output is copied into the release > tarball at kubernetes/integration-tests/tests. Do we need that in a binary > release? > > > > If these integration tests are meant to be run ad hoc, manually, not > part of a normal test cycle, then I think we can just not enable it with > -Pkubernetes. If it is meant to run every time, then it sounds like we need > a little extra work shown in recent PRs to make that easier, but then, this > test code should just be the 'test' artifact parts of the kubernetes > module, no? > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
What if anything to fix about k8s for the 2.4.0 RC5?
Forking this thread. Because we'll have another RC, we could possibly address these two issues. Only if we have a reliable change of course. Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt. And is it reasonable to essentially 'disable' kubernetes/integration-tests by removing it from the kubernetes profile? it doesn't mean it goes away, just means it's run manually, not automatically. Is that actually how it's meant to be used anyway? in the short term? given the discussion around its requirements and minikube and all that? (Actually, this would also 'solve' the Scala 2.12 build problem too) On Tue, Oct 23, 2018 at 2:45 PM Sean Owen wrote: > > To be clear I'm currently +1 on this release, with much commentary. > > OK, the explanation for kubernetes tests makes sense. Yes I think we need to > propagate the scala-2.12 build profile to make it work. Go for it, if you > have a lead on what the change is. > This doesn't block the release as it's an issue for tests, and only affects > 2.12. However if we had a clean fix for this and there were another RC, I'd > include it. > > Dongjoon has a good point about the spark-kubernetes-integration-tests > artifact. That doesn't sound like it should be published in this way, though, > of course, we publish the test artifacts from every module already. This is > only a bit odd in being a non-test artifact meant for testing. But it's > special testing! So I also don't think that needs to block a release. > > This happens because the integration tests module is enabled with the > 'kubernetes' profile too, and also this output is copied into the release > tarball at kubernetes/integration-tests/tests. Do we need that in a binary > release? > > If these integration tests are meant to be run ad hoc, manually, not part of > a normal test cycle, then I think we can just not enable it with > -Pkubernetes. If it is meant to run every time, then it sounds like we need a > little extra work shown in recent PRs to make that easier, but then, this > test code should just be the 'test' artifact parts of the kubernetes module, > no? - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
