Re: [ANNOUNCE] Announcing Apache Spark 2.2.3
Congrats, Great work Dongjoon. Dongjoon Hyun 于2019年1月15日周二 下午3:47写道： > We are happy to announce the availability of Spark 2.2.3! > > Apache Spark 2.2.3 is a maintenance release, based on the branch-2.2 > maintenance branch of Spark. We strongly recommend all 2.2.x users to > upgrade to this stable release. > > To download Spark 2.2.3, head over to the download page: > http://spark.apache.org/downloads.html > > To view the release notes: > https://spark.apache.org/releases/spark-release-2-2-3.html > > We would like to acknowledge all community members for contributing to > this release. This release would not have been possible without you. > > Bests, > Dongjoon. > -- Best Regards Jeff Zhang
[ANNOUNCE] Announcing Apache Spark 2.2.3
We are happy to announce the availability of Spark 2.2.3! Apache Spark 2.2.3 is a maintenance release, based on the branch-2.2 maintenance branch of Spark. We strongly recommend all 2.2.x users to upgrade to this stable release. To download Spark 2.2.3, head over to the download page: http://spark.apache.org/downloads.html To view the release notes: https://spark.apache.org/releases/spark-release-2-2-3.html We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you. Bests, Dongjoon.
Re: Apache Spark 2.2.3 ?
Thanks, Dongjoon! I did it. Best, Takeshi On Sun, Jan 13, 2019 at 9:33 AM Dongjoon Hyun wrote: > Hi, Takeshi. > > Please start a new thread by using a new title containing `2.3.3`. > > Bests, > Dongjoon. > > On Sat, Jan 12, 2019 at 3:39 PM Takeshi Yamamuro > wrote: > >> Hi, all >> >> I'm planning to start the release vote for v2.3.3 in the start of the >> next week. >> # I've already checked that all the tests passed in branch-2.3 and >> # there is no problem by the release scripts with dry-run. >> >> If there is any problem, please ping me. >> >> Best, >> Takeshi >> >> >> On Wed, Jan 9, 2019 at 3:16 PM Xiao Li wrote: >> >>> Thank you, Takeshi! >>> >>> Dongjoon Hyun 于2019年1月8日周二 下午10:13写道： >>> >>>> Great! Thank you, Takeshi! :D >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro >>>> wrote: >>>> >>>>> If there is no other volunteer for the release of 2.3.3, I'd like to. >>>>> >>>>> best, >>>>> takeshi >>>>> >>>>> On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun >>>>> wrote: >>>>> >>>>>> Thank you, Sean! >>>>>> >>>>>> Bests, >>>>>> Dongjoon. >>>>>> >>>>>> >>>>>> On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: >>>>>> >>>>>>> Yes, that one's not going to be back-ported to 2.3. I think it's >>>>>>> fine to proceed with a 2.2 release with what's there now and call it >>>>>>> done. >>>>>>> Note that Spark 2.3 would be EOL around September of this year. >>>>>>> >>>>>>> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun < >>>>>>> dongjoon.h...@gmail.com> wrote: >>>>>>> >>>>>>>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>>>>>>> >>>>>>>> >>>>>>>> The following is the update for Apache Spark 2.2.3 release. >>>>>>>> >>>>>>>> For correctness issues, two more patches landed on `branch-2.2`. >>>>>>>> >>>>>>>> SPARK-22951 fix aggregation after dropDuplicates on empty >>>>>>>> dataframes >>>>>>>> SPARK-25591 Avoid overwriting deserialized accumulator >>>>>>>> >>>>>>>> Currently, if we use the following JIRA search query, there exist >>>>>>>> one JIRA issue; SPARK-25206. >>>>>>>> >>>>>>>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, >>>>>>>> 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, >>>>>>>> 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, >>>>>>>> 2.2.0, >>>>>>>> 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>>>>>>> >>>>>>>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) >>>>>>>> has >>>>>>>> >>>>>>>> Affected Version: 2.2.2, 2.3.1 >>>>>>>> Target Versions: 2.3.2, 2.4.0 >>>>>>>> Fixed Version: 2.4.0 >>>>>>>> >>>>>>>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 >>>>>>>> already missed it due to the technical difficulties and risks. Instead, >>>>>>>> it's marked as a known issue. As we see, it's not targeted to 2.3.3, >>>>>>>> too. >>>>>>>> >>>>>>>> I know the correctness issue policy on new releases. However, for >>>>>>>> me, Spark 2.2.3 is a little bit exceptional release since it's a >>>>>>>> farewell >>>>>>>> release and branch-2.2 is already EOL and too far from the active >>>>>>>> branch >>>>>>>> master. >>>>>>>> >>>>>>>> So, I'd like to put SPARK-25206 out of the scope of the farewell >>>>>>>> release and recommend the users to use the other latest release. For >>>>>>>> example, Spark 2.4.0 for SPARK-25206. >>>>>>>> >>>>>>>> How do you think about that? >>>>>>>> >>>>>>>> Bests, >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> >>>>> -- >>>>> --- >>>>> Takeshi Yamamuro >>>>> >>>> >> >> -- >> --- >> Takeshi Yamamuro >> > -- --- Takeshi Yamamuro
Re: Apache Spark 2.2.3 ?
Hi, Takeshi. Please start a new thread by using a new title containing `2.3.3`. Bests, Dongjoon. On Sat, Jan 12, 2019 at 3:39 PM Takeshi Yamamuro wrote: > Hi, all > > I'm planning to start the release vote for v2.3.3 in the start of the next > week. > # I've already checked that all the tests passed in branch-2.3 and > # there is no problem by the release scripts with dry-run. > > If there is any problem, please ping me. > > Best, > Takeshi > > > On Wed, Jan 9, 2019 at 3:16 PM Xiao Li wrote: > >> Thank you, Takeshi! >> >> Dongjoon Hyun 于2019年1月8日周二 下午10:13写道： >> >>> Great! Thank you, Takeshi! :D >>> >>> Bests, >>> Dongjoon. >>> >>> On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro >>> wrote: >>> >>>> If there is no other volunteer for the release of 2.3.3, I'd like to. >>>> >>>> best, >>>> takeshi >>>> >>>> On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun >>>> wrote: >>>> >>>>> Thank you, Sean! >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>> On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: >>>>> >>>>>> Yes, that one's not going to be back-ported to 2.3. I think it's fine >>>>>> to proceed with a 2.2 release with what's there now and call it done. >>>>>> Note that Spark 2.3 would be EOL around September of this year. >>>>>> >>>>>> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun >>>>>> wrote: >>>>>> >>>>>>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>>>>>> >>>>>>> >>>>>>> The following is the update for Apache Spark 2.2.3 release. >>>>>>> >>>>>>> For correctness issues, two more patches landed on `branch-2.2`. >>>>>>> >>>>>>> SPARK-22951 fix aggregation after dropDuplicates on empty >>>>>>> dataframes >>>>>>> SPARK-25591 Avoid overwriting deserialized accumulator >>>>>>> >>>>>>> Currently, if we use the following JIRA search query, there exist >>>>>>> one JIRA issue; SPARK-25206. >>>>>>> >>>>>>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >>>>>>> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >>>>>>> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >>>>>>> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>>>>>> >>>>>>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) >>>>>>> has >>>>>>> >>>>>>> Affected Version: 2.2.2, 2.3.1 >>>>>>> Target Versions: 2.3.2, 2.4.0 >>>>>>> Fixed Version: 2.4.0 >>>>>>> >>>>>>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 >>>>>>> already missed it due to the technical difficulties and risks. Instead, >>>>>>> it's marked as a known issue. As we see, it's not targeted to 2.3.3, >>>>>>> too. >>>>>>> >>>>>>> I know the correctness issue policy on new releases. However, for >>>>>>> me, Spark 2.2.3 is a little bit exceptional release since it's a >>>>>>> farewell >>>>>>> release and branch-2.2 is already EOL and too far from the active branch >>>>>>> master. >>>>>>> >>>>>>> So, I'd like to put SPARK-25206 out of the scope of the farewell >>>>>>> release and recommend the users to use the other latest release. For >>>>>>> example, Spark 2.4.0 for SPARK-25206. >>>>>>> >>>>>>> How do you think about that? >>>>>>> >>>>>>> Bests, >>>>>>> Dongjoon. >>>>>>> >>>>>>>> >>>>>>>> >>>> >>>> -- >>>> --- >>>> Takeshi Yamamuro >>>> >>> > > -- > --- > Takeshi Yamamuro >
Re: Apache Spark 2.2.3 ?
Hi, all I'm planning to start the release vote for v2.3.3 in the start of the next week. # I've already checked that all the tests passed in branch-2.3 and # there is no problem by the release scripts with dry-run. If there is any problem, please ping me. Best, Takeshi On Wed, Jan 9, 2019 at 3:16 PM Xiao Li wrote: > Thank you, Takeshi! > > Dongjoon Hyun 于2019年1月8日周二 下午10:13写道： > >> Great! Thank you, Takeshi! :D >> >> Bests, >> Dongjoon. >> >> On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro >> wrote: >> >>> If there is no other volunteer for the release of 2.3.3, I'd like to. >>> >>> best, >>> takeshi >>> >>> On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun >>> wrote: >>> >>>> Thank you, Sean! >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>> >>>> On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: >>>> >>>>> Yes, that one's not going to be back-ported to 2.3. I think it's fine >>>>> to proceed with a 2.2 release with what's there now and call it done. >>>>> Note that Spark 2.3 would be EOL around September of this year. >>>>> >>>>> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun >>>>> wrote: >>>>> >>>>>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>>>>> >>>>>> >>>>>> The following is the update for Apache Spark 2.2.3 release. >>>>>> >>>>>> For correctness issues, two more patches landed on `branch-2.2`. >>>>>> >>>>>> SPARK-22951 fix aggregation after dropDuplicates on empty >>>>>> dataframes >>>>>> SPARK-25591 Avoid overwriting deserialized accumulator >>>>>> >>>>>> Currently, if we use the following JIRA search query, there exist one >>>>>> JIRA issue; SPARK-25206. >>>>>> >>>>>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >>>>>> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >>>>>> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >>>>>> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>>>>> >>>>>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has >>>>>> >>>>>> Affected Version: 2.2.2, 2.3.1 >>>>>> Target Versions: 2.3.2, 2.4.0 >>>>>> Fixed Version: 2.4.0 >>>>>> >>>>>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already >>>>>> missed it due to the technical difficulties and risks. Instead, it's >>>>>> marked >>>>>> as a known issue. As we see, it's not targeted to 2.3.3, too. >>>>>> >>>>>> I know the correctness issue policy on new releases. However, for me, >>>>>> Spark 2.2.3 is a little bit exceptional release since it's a farewell >>>>>> release and branch-2.2 is already EOL and too far from the active branch >>>>>> master. >>>>>> >>>>>> So, I'd like to put SPARK-25206 out of the scope of the farewell >>>>>> release and recommend the users to use the other latest release. For >>>>>> example, Spark 2.4.0 for SPARK-25206. >>>>>> >>>>>> How do you think about that? >>>>>> >>>>>> Bests, >>>>>> Dongjoon. >>>>>> >>>>>>> >>>>>>> >>> >>> -- >>> --- >>> Takeshi Yamamuro >>> >> -- --- Takeshi Yamamuro
Re: Apache Spark 2.2.3 ?
Thank you, Takeshi! Dongjoon Hyun 于2019年1月8日周二 下午10:13写道： > Great! Thank you, Takeshi! :D > > Bests, > Dongjoon. > > On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro > wrote: > >> If there is no other volunteer for the release of 2.3.3, I'd like to. >> >> best, >> takeshi >> >> On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun >> wrote: >> >>> Thank you, Sean! >>> >>> Bests, >>> Dongjoon. >>> >>> >>> On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: >>> >>>> Yes, that one's not going to be back-ported to 2.3. I think it's fine >>>> to proceed with a 2.2 release with what's there now and call it done. >>>> Note that Spark 2.3 would be EOL around September of this year. >>>> >>>> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun >>>> wrote: >>>> >>>>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>>>> >>>>> >>>>> The following is the update for Apache Spark 2.2.3 release. >>>>> >>>>> For correctness issues, two more patches landed on `branch-2.2`. >>>>> >>>>> SPARK-22951 fix aggregation after dropDuplicates on empty >>>>> dataframes >>>>> SPARK-25591 Avoid overwriting deserialized accumulator >>>>> >>>>> Currently, if we use the following JIRA search query, there exist one >>>>> JIRA issue; SPARK-25206. >>>>> >>>>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >>>>> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >>>>> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >>>>> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>>>> >>>>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has >>>>> >>>>> Affected Version: 2.2.2, 2.3.1 >>>>> Target Versions: 2.3.2, 2.4.0 >>>>> Fixed Version: 2.4.0 >>>>> >>>>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already >>>>> missed it due to the technical difficulties and risks. Instead, it's >>>>> marked >>>>> as a known issue. As we see, it's not targeted to 2.3.3, too. >>>>> >>>>> I know the correctness issue policy on new releases. However, for me, >>>>> Spark 2.2.3 is a little bit exceptional release since it's a farewell >>>>> release and branch-2.2 is already EOL and too far from the active branch >>>>> master. >>>>> >>>>> So, I'd like to put SPARK-25206 out of the scope of the farewell >>>>> release and recommend the users to use the other latest release. For >>>>> example, Spark 2.4.0 for SPARK-25206. >>>>> >>>>> How do you think about that? >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>>> >>>>>> >> >> -- >> --- >> Takeshi Yamamuro >> >
Re: Apache Spark 2.2.3 ?
Great! Thank you, Takeshi! :D Bests, Dongjoon. On Tue, Jan 8, 2019 at 8:47 PM Takeshi Yamamuro wrote: > If there is no other volunteer for the release of 2.3.3, I'd like to. > > best, > takeshi > > On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun > wrote: > >> Thank you, Sean! >> >> Bests, >> Dongjoon. >> >> >> On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: >> >>> Yes, that one's not going to be back-ported to 2.3. I think it's fine to >>> proceed with a 2.2 release with what's there now and call it done. >>> Note that Spark 2.3 would be EOL around September of this year. >>> >>> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun >>> wrote: >>> >>>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>>> >>>> >>>> The following is the update for Apache Spark 2.2.3 release. >>>> >>>> For correctness issues, two more patches landed on `branch-2.2`. >>>> >>>> SPARK-22951 fix aggregation after dropDuplicates on empty >>>> dataframes >>>> SPARK-25591 Avoid overwriting deserialized accumulator >>>> >>>> Currently, if we use the following JIRA search query, there exist one >>>> JIRA issue; SPARK-25206. >>>> >>>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >>>> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >>>> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >>>> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>>> >>>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has >>>> >>>> Affected Version: 2.2.2, 2.3.1 >>>> Target Versions: 2.3.2, 2.4.0 >>>> Fixed Version: 2.4.0 >>>> >>>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already >>>> missed it due to the technical difficulties and risks. Instead, it's marked >>>> as a known issue. As we see, it's not targeted to 2.3.3, too. >>>> >>>> I know the correctness issue policy on new releases. However, for me, >>>> Spark 2.2.3 is a little bit exceptional release since it's a farewell >>>> release and branch-2.2 is already EOL and too far from the active branch >>>> master. >>>> >>>> So, I'd like to put SPARK-25206 out of the scope of the farewell >>>> release and recommend the users to use the other latest release. For >>>> example, Spark 2.4.0 for SPARK-25206. >>>> >>>> How do you think about that? >>>> >>>> Bests, >>>> Dongjoon. >>>> >>>>> >>>>> > > -- > --- > Takeshi Yamamuro >
Re: Apache Spark 2.2.3 ?
If there is no other volunteer for the release of 2.3.3, I'd like to. best, takeshi On Fri, Jan 4, 2019 at 11:49 AM Dongjoon Hyun wrote: > Thank you, Sean! > > Bests, > Dongjoon. > > > On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: > >> Yes, that one's not going to be back-ported to 2.3. I think it's fine to >> proceed with a 2.2 release with what's there now and call it done. >> Note that Spark 2.3 would be EOL around September of this year. >> >> On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun >> wrote: >> >>> Thank you for additional support for 2.2.3, Felix and Takeshi! >>> >>> >>> The following is the update for Apache Spark 2.2.3 release. >>> >>> For correctness issues, two more patches landed on `branch-2.2`. >>> >>> SPARK-22951 fix aggregation after dropDuplicates on empty >>> dataframes >>> SPARK-25591 Avoid overwriting deserialized accumulator >>> >>> Currently, if we use the following JIRA search query, there exist one >>> JIRA issue; SPARK-25206. >>> >>> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >>> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >>> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >>> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >>> >>> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has >>> >>> Affected Version: 2.2.2, 2.3.1 >>> Target Versions: 2.3.2, 2.4.0 >>> Fixed Version: 2.4.0 >>> >>> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already >>> missed it due to the technical difficulties and risks. Instead, it's marked >>> as a known issue. As we see, it's not targeted to 2.3.3, too. >>> >>> I know the correctness issue policy on new releases. However, for me, >>> Spark 2.2.3 is a little bit exceptional release since it's a farewell >>> release and branch-2.2 is already EOL and too far from the active branch >>> master. >>> >>> So, I'd like to put SPARK-25206 out of the scope of the farewell release >>> and recommend the users to use the other latest release. For example, Spark >>> 2.4.0 for SPARK-25206. >>> >>> How do you think about that? >>> >>> Bests, >>> Dongjoon. >>> >>>> >>>> -- --- Takeshi Yamamuro
Re: Apache Spark 2.2.3 ?
Thank you, Sean! Bests, Dongjoon. On Thu, Jan 3, 2019 at 2:50 PM Sean Owen wrote: > Yes, that one's not going to be back-ported to 2.3. I think it's fine to > proceed with a 2.2 release with what's there now and call it done. > Note that Spark 2.3 would be EOL around September of this year. > > On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun > wrote: > >> Thank you for additional support for 2.2.3, Felix and Takeshi! >> >> >> The following is the update for Apache Spark 2.2.3 release. >> >> For correctness issues, two more patches landed on `branch-2.2`. >> >> SPARK-22951 fix aggregation after dropDuplicates on empty dataframes >> SPARK-25591 Avoid overwriting deserialized accumulator >> >> Currently, if we use the following JIRA search query, there exist one >> JIRA issue; SPARK-25206. >> >> Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, >> 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, >> 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, >> 2.2.2, 2.2.3) AND labels in (Correctness, correctness) >> >> SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has >> >> Affected Version: 2.2.2, 2.3.1 >> Target Versions: 2.3.2, 2.4.0 >> Fixed Version: 2.4.0 >> >> Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already >> missed it due to the technical difficulties and risks. Instead, it's marked >> as a known issue. As we see, it's not targeted to 2.3.3, too. >> >> I know the correctness issue policy on new releases. However, for me, >> Spark 2.2.3 is a little bit exceptional release since it's a farewell >> release and branch-2.2 is already EOL and too far from the active branch >> master. >> >> So, I'd like to put SPARK-25206 out of the scope of the farewell release >> and recommend the users to use the other latest release. For example, Spark >> 2.4.0 for SPARK-25206. >> >> How do you think about that? >> >> Bests, >> Dongjoon. >> >>> >>>
Re: Apache Spark 2.2.3 ?
Yes, that one's not going to be back-ported to 2.3. I think it's fine to proceed with a 2.2 release with what's there now and call it done. Note that Spark 2.3 would be EOL around September of this year. On Thu, Jan 3, 2019 at 2:31 PM Dongjoon Hyun wrote: > Thank you for additional support for 2.2.3, Felix and Takeshi! > > > The following is the update for Apache Spark 2.2.3 release. > > For correctness issues, two more patches landed on `branch-2.2`. > > SPARK-22951 fix aggregation after dropDuplicates on empty dataframes > SPARK-25591 Avoid overwriting deserialized accumulator > > Currently, if we use the following JIRA search query, there exist one JIRA > issue; SPARK-25206. > > Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, > 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, > 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, > 2.2.2, 2.2.3) AND labels in (Correctness, correctness) > > SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has > > Affected Version: 2.2.2, 2.3.1 > Target Versions: 2.3.2, 2.4.0 > Fixed Version: 2.4.0 > > Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already > missed it due to the technical difficulties and risks. Instead, it's marked > as a known issue. As we see, it's not targeted to 2.3.3, too. > > I know the correctness issue policy on new releases. However, for me, > Spark 2.2.3 is a little bit exceptional release since it's a farewell > release and branch-2.2 is already EOL and too far from the active branch > master. > > So, I'd like to put SPARK-25206 out of the scope of the farewell release > and recommend the users to use the other latest release. For example, Spark > 2.4.0 for SPARK-25206. > > How do you think about that? > > Bests, > Dongjoon. > >> >>
Re: Apache Spark 2.2.3 ?
Thank you for additional support for 2.2.3, Felix and Takeshi! The following is the update for Apache Spark 2.2.3 release. For correctness issues, two more patches landed on `branch-2.2`. SPARK-22951 fix aggregation after dropDuplicates on empty dataframes SPARK-25591 Avoid overwriting deserialized accumulator Currently, if we use the following JIRA search query, there exist one JIRA issue; SPARK-25206. Query: project = SPARK AND fixVersion in (2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 3.0.0) AND fixVersion not in (2.2.0, 2.2.1, 2.2.2, 2.2.3) AND affectedVersion in (2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3) AND labels in (Correctness, correctness) SPARK-25206 ( https://issues.apache.org/jira/browse/SPARK-25206 ) has Affected Version: 2.2.2, 2.3.1 Target Versions: 2.3.2, 2.4.0 Fixed Version: 2.4.0 Although SPARK-25206 is labeled as a correctness issue, 2.3.2 already missed it due to the technical difficulties and risks. Instead, it's marked as a known issue. As we see, it's not targeted to 2.3.3, too. I know the correctness issue policy on new releases. However, for me, Spark 2.2.3 is a little bit exceptional release since it's a farewell release and branch-2.2 is already EOL and too far from the active branch master. So, I'd like to put SPARK-25206 out of the scope of the farewell release and recommend the users to use the other latest release. For example, Spark 2.4.0 for SPARK-25206. How do you think about that? Bests, Dongjoon. On Thu, Jan 3, 2019 at 12:02 AM Takeshi Yamamuro wrote: > Hi, all, happy new year! > > +1 on the release of 2.2.3/2.3.3. > I checked there is no ongoing issue targeting on 2.3.3, too. > > On Thu, Jan 3, 2019 at 8:50 AM Felix Cheung > wrote: > >> +1 on 2.2.3 of course >> >> >> -- >> *From:* Dongjoon Hyun >> *Sent:* Wednesday, January 2, 2019 12:21 PM >> *To:* Saisai Shao >> *Cc:* Xiao Li; Felix Cheung; Sean Owen; dev >> *Subject:* Re: Apache Spark 2.2.3 ? >> >> Thank you for swift feedbacks and Happy New Year. :) >> For 2.2.3 release on next week, I see two positive opinions (including >> mine) >> and don't see any direct objections. >> >> Apache Spark has a mature, resourceful, and fast-growing community. >> One of the important characteristic of the mature community is >> the expectable behavior where the users are able to depend on. >> For instance, we have a nice tradition to cut the branch as a sign of >> feature freeze. >> The *final* release of a branch is not only good for the end users, but >> also a good sign of the EOL of the branch for all. >> >> As a junior committer of the community, I want to contribute to deliver >> the final 2.2.3 release to the community and to finalize `branch-2.2`. >> >> * For Apache Spark JIRA, I checked that there is no on-going issues >> targeting on `2.2.3`. >> * For commits, I reviewed the newly landed commits after `2.2.2` tag and >> updated a few missing JIRA issues accordingly. >> * Apparently, we can release 2.2.3 next week. >> >> BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases >> before Spark+AI Summit (April) because we did like that usually. >> Please send another email to `dev` mailing list because it's worth to >> receive more attentions and requests. >> >> Bests, >> Dongjoon. >> >> >> On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao >> wrote: >> >>> Agreed to have a new branch-2.3 release, as we already accumulated >>> several fixes. >>> >>> Thanks >>> Saisai >>> >>> Xiao Li 于2019年1月2日周三 下午1:32写道： >>> >>>> Based on the commit history, >>>> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 >>>> contains more critical fixes. Maybe the priority is higher? >>>> >>>> On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung >>>> wrote: >>>> >>>>> Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) >>>>> >>>>> And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 >>>>> branch is stabilizing? >>>>> >>>>> >>>>> -- >>>>> *From:* Sean Owen >>>>> *Sent:* Tuesday, January 1, 2019 8:30 PM >>>>> *To:* Dongjoon Hyun >>>>> *Cc:* dev >>>>> *Subject:* Re: Apache Spark 2.2.3 ? >>>>> >>>>> I agree with that logic, and if you're volunteering to do the legwork, >>>>> I don't see a reason not to cut a final 2
Re: Apache Spark 2.2.3 ?
Hi, all, happy new year! +1 on the release of 2.2.3/2.3.3. I checked there is no ongoing issue targeting on 2.3.3, too. On Thu, Jan 3, 2019 at 8:50 AM Felix Cheung wrote: > +1 on 2.2.3 of course > > > -- > *From:* Dongjoon Hyun > *Sent:* Wednesday, January 2, 2019 12:21 PM > *To:* Saisai Shao > *Cc:* Xiao Li; Felix Cheung; Sean Owen; dev > *Subject:* Re: Apache Spark 2.2.3 ? > > Thank you for swift feedbacks and Happy New Year. :) > For 2.2.3 release on next week, I see two positive opinions (including > mine) > and don't see any direct objections. > > Apache Spark has a mature, resourceful, and fast-growing community. > One of the important characteristic of the mature community is > the expectable behavior where the users are able to depend on. > For instance, we have a nice tradition to cut the branch as a sign of > feature freeze. > The *final* release of a branch is not only good for the end users, but > also a good sign of the EOL of the branch for all. > > As a junior committer of the community, I want to contribute to deliver > the final 2.2.3 release to the community and to finalize `branch-2.2`. > > * For Apache Spark JIRA, I checked that there is no on-going issues > targeting on `2.2.3`. > * For commits, I reviewed the newly landed commits after `2.2.2` tag and > updated a few missing JIRA issues accordingly. > * Apparently, we can release 2.2.3 next week. > > BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases > before Spark+AI Summit (April) because we did like that usually. > Please send another email to `dev` mailing list because it's worth to > receive more attentions and requests. > > Bests, > Dongjoon. > > > On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao wrote: > >> Agreed to have a new branch-2.3 release, as we already accumulated >> several fixes. >> >> Thanks >> Saisai >> >> Xiao Li 于2019年1月2日周三 下午1:32写道： >> >>> Based on the commit history, >>> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 >>> contains more critical fixes. Maybe the priority is higher? >>> >>> On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung >>> wrote: >>> >>>> Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) >>>> >>>> And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 >>>> branch is stabilizing? >>>> >>>> >>>> -- >>>> *From:* Sean Owen >>>> *Sent:* Tuesday, January 1, 2019 8:30 PM >>>> *To:* Dongjoon Hyun >>>> *Cc:* dev >>>> *Subject:* Re: Apache Spark 2.2.3 ? >>>> >>>> I agree with that logic, and if you're volunteering to do the legwork, >>>> I don't see a reason not to cut a final 2.2 release. >>>> >>>> On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun >>>> wrote: >>>> > >>>> > Hi, All. >>>> > >>>> > Apache Spark community has a policy maintaining the feature branch >>>> for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is >>>> released on July 2017. >>>> > >>>> > http://spark.apache.org/versioning-policy.html >>>> > >>>> > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including >>>> security patches). >>>> > >>>> > >>>> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 >>>> > >>>> > If it's okay and there is no further plan on `branch-2.2`, I want to >>>> volunteer to prepare the first RC (early next week?). >>>> > >>>> > Please let me know your opinions about this. >>>> > >>>> > Bests, >>>> > Dongjoon. >>>> >>>> - >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> >>> >>> -- >>> [image: >>> https://databricks.com/sparkaisummit/north-america?utm_source=email_medium=signature] >>> >> -- --- Takeshi Yamamuro
Re: Apache Spark 2.2.3 ?
+1 on 2.2.3 of course From: Dongjoon Hyun Sent: Wednesday, January 2, 2019 12:21 PM To: Saisai Shao Cc: Xiao Li; Felix Cheung; Sean Owen; dev Subject: Re: Apache Spark 2.2.3 ? Thank you for swift feedbacks and Happy New Year. :) For 2.2.3 release on next week, I see two positive opinions (including mine) and don't see any direct objections. Apache Spark has a mature, resourceful, and fast-growing community. One of the important characteristic of the mature community is the expectable behavior where the users are able to depend on. For instance, we have a nice tradition to cut the branch as a sign of feature freeze. The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all. As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`. * For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`. * For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly. * Apparently, we can release 2.2.3 next week. BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually. Please send another email to `dev` mailing list because it's worth to receive more attentions and requests. Bests, Dongjoon. On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao mailto:sai.sai.s...@gmail.com>> wrote: Agreed to have a new branch-2.3 release, as we already accumulated several fixes. Thanks Saisai Xiao Li mailto:lix...@databricks.com>> 于2019年1月2日周三 下午1:32写道： Based on the commit history, https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 contains more critical fixes. Maybe the priority is higher? On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung mailto:felixcheun...@hotmail.com>> wrote: Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing? From: Sean Owen mailto:sro...@gmail.com>> Sent: Tuesday, January 1, 2019 8:30 PM To: Dongjoon Hyun Cc: dev Subject: Re: Apache Spark 2.2.3 ? I agree with that logic, and if you're volunteering to do the legwork, I don't see a reason not to cut a final 2.2 release. On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: > > Hi, All. > > Apache Spark community has a policy maintaining the feature branch for 18 > months. I think it's time for the 2.2.3 release since 2.2.0 is released on > July 2017. > > http://spark.apache.org/versioning-policy.html > > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security > patches). > > https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 > > If it's okay and there is no further plan on `branch-2.2`, I want to > volunteer to prepare the first RC (early next week?). > > Please let me know your opinions about this. > > Bests, > Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> -- [https://databricks.com/sparkaisummit/north-america?utm_source=email_medium=signature]
Re: Apache Spark 2.2.3 ?
Thank you for swift feedbacks and Happy New Year. :) For 2.2.3 release on next week, I see two positive opinions (including mine) and don't see any direct objections. Apache Spark has a mature, resourceful, and fast-growing community. One of the important characteristic of the mature community is the expectable behavior where the users are able to depend on. For instance, we have a nice tradition to cut the branch as a sign of feature freeze. The *final* release of a branch is not only good for the end users, but also a good sign of the EOL of the branch for all. As a junior committer of the community, I want to contribute to deliver the final 2.2.3 release to the community and to finalize `branch-2.2`. * For Apache Spark JIRA, I checked that there is no on-going issues targeting on `2.2.3`. * For commits, I reviewed the newly landed commits after `2.2.2` tag and updated a few missing JIRA issues accordingly. * Apparently, we can release 2.2.3 next week. BTW, I'm +1 for the next 2.3/2.4 and have been expecting those releases before Spark+AI Summit (April) because we did like that usually. Please send another email to `dev` mailing list because it's worth to receive more attentions and requests. Bests, Dongjoon. On Tue, Jan 1, 2019 at 9:35 PM Saisai Shao wrote: > Agreed to have a new branch-2.3 release, as we already accumulated several > fixes. > > Thanks > Saisai > > Xiao Li 于2019年1月2日周三 下午1:32写道： > >> Based on the commit history, >> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 >> contains more critical fixes. Maybe the priority is higher? >> >> On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung >> wrote: >> >>> Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) >>> >>> And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch >>> is stabilizing? >>> >>> >>> ------ >>> *From:* Sean Owen >>> *Sent:* Tuesday, January 1, 2019 8:30 PM >>> *To:* Dongjoon Hyun >>> *Cc:* dev >>> *Subject:* Re: Apache Spark 2.2.3 ? >>> >>> I agree with that logic, and if you're volunteering to do the legwork, >>> I don't see a reason not to cut a final 2.2 release. >>> >>> On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun >>> wrote: >>> > >>> > Hi, All. >>> > >>> > Apache Spark community has a policy maintaining the feature branch for >>> 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released >>> on July 2017. >>> > >>> > http://spark.apache.org/versioning-policy.html >>> > >>> > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including >>> security patches). >>> > >>> > >>> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 >>> > >>> > If it's okay and there is no further plan on `branch-2.2`, I want to >>> volunteer to prepare the first RC (early next week?). >>> > >>> > Please let me know your opinions about this. >>> > >>> > Bests, >>> > Dongjoon. >>> >>> - >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> >> >> -- >> [image: >> https://databricks.com/sparkaisummit/north-america?utm_source=email_medium=signature] >> >
Re: Apache Spark 2.2.3 ?
Agreed to have a new branch-2.3 release, as we already accumulated several fixes. Thanks Saisai Xiao Li 于2019年1月2日周三 下午1:32写道： > Based on the commit history, > https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.3 > contains more critical fixes. Maybe the priority is higher? > > On Tue, Jan 1, 2019 at 9:22 PM Felix Cheung > wrote: > >> Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) >> >> And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch >> is stabilizing? >> >> >> -- >> *From:* Sean Owen >> *Sent:* Tuesday, January 1, 2019 8:30 PM >> *To:* Dongjoon Hyun >> *Cc:* dev >> *Subject:* Re: Apache Spark 2.2.3 ? >> >> I agree with that logic, and if you're volunteering to do the legwork, >> I don't see a reason not to cut a final 2.2 release. >> >> On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun >> wrote: >> > >> > Hi, All. >> > >> > Apache Spark community has a policy maintaining the feature branch for >> 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released >> on July 2017. >> > >> > http://spark.apache.org/versioning-policy.html >> > >> > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including >> security patches). >> > >> > >> https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 >> > >> > If it's okay and there is no further plan on `branch-2.2`, I want to >> volunteer to prepare the first RC (early next week?). >> > >> > Please let me know your opinions about this. >> > >> > Bests, >> > Dongjoon. >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > [image: > https://databricks.com/sparkaisummit/north-america?utm_source=email_medium=signature] >
Re: Apache Spark 2.2.3 ?
Speaking of, it’s been 3 months since 2.3.2... (Sept 2018) And 2 months since 2.4.0 (Nov 2018) - does the community feel 2.4 branch is stabilizing? From: Sean Owen Sent: Tuesday, January 1, 2019 8:30 PM To: Dongjoon Hyun Cc: dev Subject: Re: Apache Spark 2.2.3 ? I agree with that logic, and if you're volunteering to do the legwork, I don't see a reason not to cut a final 2.2 release. On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun wrote: > > Hi, All. > > Apache Spark community has a policy maintaining the feature branch for 18 > months. I think it's time for the 2.2.3 release since 2.2.0 is released on > July 2017. > > http://spark.apache.org/versioning-policy.html > > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security > patches). > > https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 > > If it's okay and there is no further plan on `branch-2.2`, I want to > volunteer to prepare the first RC (early next week?). > > Please let me know your opinions about this. > > Bests, > Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: Apache Spark 2.2.3 ?
I agree with that logic, and if you're volunteering to do the legwork, I don't see a reason not to cut a final 2.2 release. On Tue, Jan 1, 2019 at 9:19 PM Dongjoon Hyun wrote: > > Hi, All. > > Apache Spark community has a policy maintaining the feature branch for 18 > months. I think it's time for the 2.2.3 release since 2.2.0 is released on > July 2017. > > http://spark.apache.org/versioning-policy.html > > After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security > patches). > > > https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 > > If it's okay and there is no further plan on `branch-2.2`, I want to > volunteer to prepare the first RC (early next week?). > > Please let me know your opinions about this. > > Bests, > Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Apache Spark 2.2.3 ?
Hi, All. Apache Spark community has a policy maintaining the feature branch for 18 months. I think it's time for the 2.2.3 release since 2.2.0 is released on July 2017. http://spark.apache.org/versioning-policy.html After 2.2.2 (July 2018), `branch-2.2` has 40 patches (including security patches). https://gitbox.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.2 If it's okay and there is no further plan on `branch-2.2`, I want to volunteer to prepare the first RC (early next week?). Please let me know your opinions about this. Bests, Dongjoon.
