Ah these issues were resolved before branch-2.4 is cut, like SPARK-24441 https://github.com/apache/spark/blob/v2.4.0-rc1/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
SPARK-24441 is included to Spark 2.4.0 RC1 but set to 3.0.0. I heard there's a step which version of issues are aligned with new release when branch/RC is being cut, but it doesn't look like happening for some issues. 2018년 9월 21일 (금) 오후 2:10, Holden Karau <hol...@pigscanfly.ca>님이 작성: > So normally during the release process if it's in branch-2.4 but not part > of the current RC we set the resolved version to 2.4.1 and then if roll a > new RC we switch the 2.4.1 issues to 2.4.0. > > On Thu, Sep 20, 2018 at 9:55 PM Jungtaek Lim <kabh...@gmail.com> wrote: > >> I also noticed there're some fixed issues which are included in >> branch-2.4 but its versions are still 3.0.0. Would we want to update >> versions to 2.4.0? If we are not planning to run some automations to >> correct it, I'm happy to fix them. >> >> 2018년 9월 20일 (목) 오후 9:22, Weichen Xu <weichen...@databricks.com>님이 작성: >> >>> We need to merge this. >>> https://github.com/apache/spark/pull/22492 >>> Otherwise mleap cannot build against spark 2.4.0 >>> Thanks! >>> >>> On Wed, Sep 19, 2018 at 1:16 PM Yinan Li <liyinan...@gmail.com> wrote: >>> >>>> FYI: SPARK-23200 has been resolved. >>>> >>>> On Tue, Sep 18, 2018 at 8:49 AM Felix Cheung <felixcheun...@hotmail.com> >>>> wrote: >>>> >>>>> If we could work on this quickly - it might get on to future RCs. >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* Stavros Kontopoulos <stavros.kontopou...@lightbend.com> >>>>> *Sent:* Monday, September 17, 2018 2:35 PM >>>>> *To:* Yinan Li >>>>> *Cc:* Xiao Li; eerla...@redhat.com; van...@cloudera.com.invalid; Sean >>>>> Owen; Wenchen Fan; dev >>>>> *Subject:* Re: [VOTE] SPARK 2.4.0 (RC1) >>>>> >>>>> Hi Xiao, >>>>> >>>>> I just tested it, it seems ok. There are some questions about which >>>>> properties we should keep when restoring the config. Otherwise it looks ok >>>>> to me. >>>>> The reason this should go in 2.4 is that streaming on k8s is something >>>>> people want to try day one (or at least it is cool to try) and since 2.4 >>>>> comes with k8s support being refactored a lot, >>>>> it would be disappointing not to have it in...IMHO. >>>>> >>>>> Best, >>>>> Stavros >>>>> >>>>> On Mon, Sep 17, 2018 at 11:13 PM, Yinan Li <liyinan...@gmail.com> >>>>> wrote: >>>>> >>>>>> We can merge the PR and get SPARK-23200 resolved if the whole point >>>>>> is to make streaming on k8s work first. But given that this is not a >>>>>> blocker for 2.4, I think we can take a bit more time here and get it >>>>>> right. >>>>>> With that being said, I would expect it to be resolved soon. >>>>>> >>>>>> On Mon, Sep 17, 2018 at 11:47 AM Xiao Li <gatorsm...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, Erik and Stavros, >>>>>>> >>>>>>> This bug fix SPARK-23200 is not a blocker of the 2.4 release. It >>>>>>> sounds important for the Streaming on K8S. Could the K8S oriented >>>>>>> committers speed up the reviews? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Xiao >>>>>>> >>>>>>> Erik Erlandson <eerla...@redhat.com> 于2018年9月17日周一 上午11:04写道： >>>>>>> >>>>>>>> >>>>>>>> I have no binding vote but I second Stavros’ recommendation for >>>>>>>> spark-23200 >>>>>>>> >>>>>>>> Per parallel threads on Py2 support I would also like to propose >>>>>>>> deprecating Py2 starting with this 2.4 release >>>>>>>> >>>>>>>> On Mon, Sep 17, 2018 at 10:38 AM Marcelo Vanzin >>>>>>>> <van...@cloudera.com.invalid> wrote: >>>>>>>> >>>>>>>>> You can log in to https://repository.apache.org and see what's >>>>>>>>> wrong. >>>>>>>>> Just find that staging repo and look at the messages. In your case >>>>>>>>> it >>>>>>>>> seems related to your signature. >>>>>>>>> >>>>>>>>> failureMessageNo public key: Key with id: (xxxx) was not able to be >>>>>>>>> located on http://gpg-keyserver.de/. Upload your public key and >>>>>>>>> try >>>>>>>>> the operation again. >>>>>>>>> On Sun, Sep 16, 2018 at 10:00 PM Wenchen Fan <cloud0...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > I confirmed that >>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285 >>>>>>>>> is not accessible. I did it via >>>>>>>>> ./dev/create-release/do-release-docker.sh >>>>>>>>> -d /my/work/dir -s publish , not sure what's going wrong. I didn't >>>>>>>>> see any >>>>>>>>> error message during it. >>>>>>>>> > >>>>>>>>> > Any insights are appreciated! So that I can fix it in the next >>>>>>>>> RC. Thanks! >>>>>>>>> > >>>>>>>>> > On Mon, Sep 17, 2018 at 11:31 AM Sean Owen <sro...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >> >>>>>>>>> >> I think one build is enough, but haven't thought it through. The >>>>>>>>> >> Hadoop 2.6/2.7 builds are already nearly redundant. 2.12 is >>>>>>>>> probably >>>>>>>>> >> best advertised as a 'beta'. So maybe publish a no-hadoop build >>>>>>>>> of it? >>>>>>>>> >> Really, whatever's the easy thing to do. >>>>>>>>> >> On Sun, Sep 16, 2018 at 10:28 PM Wenchen Fan < >>>>>>>>> cloud0...@gmail.com> wrote: >>>>>>>>> >> > >>>>>>>>> >> > Ah I missed the Scala 2.12 build. Do you mean we should >>>>>>>>> publish a Scala 2.12 build this time? Current for Scala 2.11 we have 3 >>>>>>>>> builds: with hadoop 2.7, with hadoop 2.6, without hadoop. Shall we do >>>>>>>>> the >>>>>>>>> same thing for Scala 2.12? >>>>>>>>> >> > >>>>>>>>> >> > On Mon, Sep 17, 2018 at 11:14 AM Sean Owen <sro...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >> >> >>>>>>>>> >> >> A few preliminary notes: >>>>>>>>> >> >> >>>>>>>>> >> >> Wenchen for some weird reason when I hit your key in gpg >>>>>>>>> --import, it >>>>>>>>> >> >> asks for a passphrase. When I skip it, it's fine, gpg can >>>>>>>>> still verify >>>>>>>>> >> >> the signature. No issue there really. >>>>>>>>> >> >> >>>>>>>>> >> >> The staging repo gives a 404: >>>>>>>>> >> >> >>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285/ >>>>>>>>> >> >> 404 - Repository "orgapachespark-1285 (staging: open)" >>>>>>>>> >> >> [id=orgapachespark-1285] exists but is not exposed. >>>>>>>>> >> >> >>>>>>>>> >> >> The (revamped) licenses are OK, though there are some minor >>>>>>>>> glitches >>>>>>>>> >> >> in the final release tarballs (my fault) : there's an extra >>>>>>>>> directory, >>>>>>>>> >> >> and the source release has both binary and source licenses. >>>>>>>>> I'll fix >>>>>>>>> >> >> that. Not strictly necessary to reject the release over >>>>>>>>> those. >>>>>>>>> >> >> >>>>>>>>> >> >> Last, when I check the staging repo I'll get my answer, but, >>>>>>>>> were you >>>>>>>>> >> >> able to build 2.12 artifacts as well? >>>>>>>>> >> >> >>>>>>>>> >> >> On Sun, Sep 16, 2018 at 9:48 PM Wenchen Fan < >>>>>>>>> cloud0...@gmail.com> wrote: >>>>>>>>> >> >> > >>>>>>>>> >> >> > Please vote on releasing the following candidate as Apache >>>>>>>>> Spark version 2.4.0. >>>>>>>>> >> >> > >>>>>>>>> >> >> > The vote is open until September 20 PST and passes if a >>>>>>>>> majority +1 PMC votes are cast, with >>>>>>>>> >> >> > a minimum of 3 +1 votes. >>>>>>>>> >> >> > >>>>>>>>> >> >> > [ ] +1 Release this package as Apache Spark 2.4.0 >>>>>>>>> >> >> > [ ] -1 Do not release this package because ... >>>>>>>>> >> >> > >>>>>>>>> >> >> > To learn more about Apache Spark, please see >>>>>>>>> http://spark.apache.org/ >>>>>>>>> >> >> > >>>>>>>>> >> >> > The tag to be voted on is v2.4.0-rc1 (commit >>>>>>>>> 1220ab8a0738b5f67dc522df5e3e77ffc83d207a): >>>>>>>>> >> >> > https://github.com/apache/spark/tree/v2.4.0-rc1 >>>>>>>>> >> >> > >>>>>>>>> >> >> > The release files, including signatures, digests, etc. can >>>>>>>>> be found at: >>>>>>>>> >> >> > >>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-bin/ >>>>>>>>> >> >> > >>>>>>>>> >> >> > Signatures used for Spark RCs can be found in this file: >>>>>>>>> >> >> > https://dist.apache.org/repos/dist/dev/spark/KEYS >>>>>>>>> >> >> > >>>>>>>>> >> >> > The staging repository for this release can be found at: >>>>>>>>> >> >> > >>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285/ >>>>>>>>> >> >> > >>>>>>>>> >> >> > The documentation corresponding to this release can be >>>>>>>>> found at: >>>>>>>>> >> >> > >>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-docs/ >>>>>>>>> >> >> > >>>>>>>>> >> >> > The list of bug fixes going into 2.4.0 can be found at the >>>>>>>>> following URL: >>>>>>>>> >> >> > >>>>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0 >>>>>>>>> >> >> > >>>>>>>>> >> >> > FAQ >>>>>>>>> >> >> > >>>>>>>>> >> >> > ========================= >>>>>>>>> >> >> > How can I help test this release? >>>>>>>>> >> >> > ========================= >>>>>>>>> >> >> > >>>>>>>>> >> >> > If you are a Spark user, you can help us test this release >>>>>>>>> by taking >>>>>>>>> >> >> > an existing Spark workload and running on this release >>>>>>>>> candidate, then >>>>>>>>> >> >> > reporting any regressions. >>>>>>>>> >> >> > >>>>>>>>> >> >> > If you're working in PySpark you can set up a virtual env >>>>>>>>> and install >>>>>>>>> >> >> > the current RC and see if anything important breaks, in >>>>>>>>> the Java/Scala >>>>>>>>> >> >> > you can add the staging repository to your projects >>>>>>>>> resolvers and test >>>>>>>>> >> >> > with the RC (make sure to clean up the artifact cache >>>>>>>>> before/after so >>>>>>>>> >> >> > you don't end up building with a out of date RC going >>>>>>>>> forward). >>>>>>>>> >> >> > >>>>>>>>> >> >> > =========================================== >>>>>>>>> >> >> > What should happen to JIRA tickets still targeting 2.4.0? >>>>>>>>> >> >> > =========================================== >>>>>>>>> >> >> > >>>>>>>>> >> >> > The current list of open tickets targeted at 2.4.0 can be >>>>>>>>> found at: >>>>>>>>> >> >> > https://issues.apache.org/jira/projects/SPARK and search >>>>>>>>> for "Target Version/s" = 2.4.0 >>>>>>>>> >> >> > >>>>>>>>> >> >> > Committers should look at those and triage. Extremely >>>>>>>>> important bug >>>>>>>>> >> >> > fixes, documentation, and API tweaks that impact >>>>>>>>> compatibility should >>>>>>>>> >> >> > be worked on immediately. Everything else please retarget >>>>>>>>> to an >>>>>>>>> >> >> > appropriate release. >>>>>>>>> >> >> > >>>>>>>>> >> >> > ================== >>>>>>>>> >> >> > But my bug isn't fixed? >>>>>>>>> >> >> > ================== >>>>>>>>> >> >> > >>>>>>>>> >> >> > In order to make timely releases, we will typically not >>>>>>>>> hold the >>>>>>>>> >> >> > release unless the bug in question is a regression from >>>>>>>>> the previous >>>>>>>>> >> >> > release. That being said, if there is something which is a >>>>>>>>> regression >>>>>>>>> >> >> > that has not been correctly targeted please ping me or a >>>>>>>>> committer to >>>>>>>>> >> >> > help target the issue. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Marcelo >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>> >>>>>>>>> >>>>> >>>>> >>>>> -- >>>>> Stavros Kontopoulos >>>>> >>>>> *Senior Software Engineer * >>>>> *Lightbend, Inc. * >>>>> >>>>> *p: +30 6977967274 <%2B1%20650%20678%200020>* >>>>> *e: stavros.kontopou...@lightbend.com* <dave.mar...@lightbend.com> >>>>> >>>>> >>>>> > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
