>
> Apache Spark 3.1.0 should be compared with Apache Spark 2.1.0.

I think we made a change in release cadence since Spark 2.3. See the
commit:
https://github.com/apache/spark-website/commit/88990968962e5cc47db8bc2c11a50742d2438daa
Thus, Spark 3.1 might just follow the release cadence of Spark 2.3/2.4, if
we do not want to change the release cadence?

How about moving the code freeze of Spark 3.1 to *Early Dec 2020* and the
RC1 date to* Early Jan 2021*?

Thanks,

Xiao


Dongjoon Hyun <dongjoon.h...@gmail.com> 于2020年10月4日周日 下午12:44写道:

> For Xiao's comment, I want to point out that Apache Spark 3.1.0 is
> different from 2.3 or 2.4.
>
> Apache Spark 3.1.0 should be compared with Apache Spark 2.1.0.
>
> - Apache Spark 2.0.0 was released on July 26, 2016.
> - Apache Spark 2.1.0 was released on December 28, 2016.
>
> Bests,
> Dongjoon.
>
>
> On Sun, Oct 4, 2020 at 10:53 AM Dongjoon Hyun <dongjoon.h...@gmail.com>
> wrote:
>
>> Thank you all.
>>
>> BTW, Xiao and Mridul, I'm wondering what date you have in your mind
>> specifically.
>>
>> Usually, `Christmas and New Year season` doesn't give us much additional
>> time.
>>
>> If you think so, could you make a PR for Apache Spark website according
>> to your expectation?
>>
>> https://spark.apache.org/versioning-policy.html
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Sun, Oct 4, 2020 at 7:18 AM Mridul Muralidharan <mri...@gmail.com>
>> wrote:
>>
>>>
>>> +1 on pushing the branch cut for increased dev time to match previous
>>> releases.
>>>
>>> Regards,
>>> Mridul
>>>
>>> On Sat, Oct 3, 2020 at 10:22 PM Xiao Li <gatorsm...@gmail.com> wrote:
>>>
>>>> Thank you for your updates.
>>>>
>>>> Spark 3.0 got released on Jun 18, 2020. If Nov 1st is the target date
>>>> of the 3.1 branch cut, the feature development time window is less than 5
>>>> months. This is shorter than what we did in Spark 2.3 and 2.4 releases.
>>>>
>>>> Below are three highly desirable feature work I am watching. Hopefully,
>>>> we can finish them before the branch cut.
>>>>
>>>>    - Support push-based shuffle to improve shuffle efficiency:
>>>>    https://issues.apache.org/jira/browse/SPARK-30602
>>>>    - Unify create table syntax:
>>>>    https://issues.apache.org/jira/browse/SPARK-31257
>>>>    - Bloom filter join:
>>>>    https://issues.apache.org/jira/browse/SPARK-32268
>>>>
>>>> Thanks,
>>>>
>>>> Xiao
>>>>
>>>>
>>>> Hyukjin Kwon <gurwls...@gmail.com> 于2020年10月3日周六 下午5:41写道:
>>>>
>>>>> Nice summary. Thanks Dongjoon. One minor correction -> I believe we
>>>>> dropped R 3.5 and below at branch 2.4 as well.
>>>>>
>>>>> On Sun, 4 Oct 2020, 09:17 Dongjoon Hyun, <dongjoon.h...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi, All.
>>>>>>
>>>>>> As of today, master branch (Apache Spark 3.1.0) resolved
>>>>>> 852+ JIRA issues and 606+ issues are 3.1.0-only patches.
>>>>>> According to the 3.1.0 release window, branch-3.1 will be
>>>>>> created on November 1st and enters QA period.
>>>>>>
>>>>>> Here are some notable updates I've been monitoring.
>>>>>>
>>>>>> *Language*
>>>>>> 01. SPARK-25075 Support Scala 2.13
>>>>>>       - Since SPARK-32926, Scala 2.13 build test has
>>>>>>         become a part of GitHub Action jobs.
>>>>>>       - After SPARK-33044, Scala 2.13 test will be
>>>>>>         a part of Jenkins jobs.
>>>>>> 02. SPARK-29909 Drop Python 2 and Python 3.4 and 3.5
>>>>>> 03. SPARK-32082 Project Zen: Improving Python usability
>>>>>>       - 7 of 16 issues are resolved.
>>>>>> 04. SPARK-32073 Drop R < 3.5 support
>>>>>>       - This is done for Spark 3.0.1 and 3.1.0.
>>>>>>
>>>>>> *Dependency*
>>>>>> 05. SPARK-32058 Use Apache Hadoop 3.2.0 dependency
>>>>>>       - This changes the default dist. for better cloud support
>>>>>> 06. SPARK-32981 Remove hive-1.2 distribution
>>>>>> 07. SPARK-20202 Remove references to org.spark-project.hive
>>>>>>       - This will remove Hive 1.2.1 from source code
>>>>>> 08. SPARK-29250 Upgrade to Hadoop 3.2.1 (WIP)
>>>>>>
>>>>>> *Core*
>>>>>> 09. SPARK-27495 Support Stage level resource conf and scheduling
>>>>>>       - 11 of 15 issues are resolved
>>>>>> 10. SPARK-25299 Use remote storage for persisting shuffle data
>>>>>>       - 8 of 14 issues are resolved
>>>>>>
>>>>>> *Resource Manager*
>>>>>> 11. SPARK-33005 Kubernetes GA preparation
>>>>>>       - It is on the way and we are waiting for more feedback.
>>>>>>
>>>>>> *SQL*
>>>>>> 12. SPARK-30648/SPARK-32346 Support filters pushdown
>>>>>>       to JSON/Avro
>>>>>> 13. SPARK-32948/SPARK-32958 Add Json expression optimizer
>>>>>> 14. SPARK-12312 Support JDBC Kerberos w/ keytab
>>>>>>       - 11 of 17 issues are resolved
>>>>>> 15. SPARK-27589 DSv2 was mostly completed in 3.0
>>>>>>       and added more features in 3.1 but still we missed
>>>>>>       - All built-in DataSource v2 write paths are disabled
>>>>>>         and v1 write is used instead.
>>>>>>       - Support partition pruning with subqueries
>>>>>>       - Support bucketing
>>>>>>
>>>>>> We still have one month before the feature freeze
>>>>>> and starting QA. If you are working for 3.1,
>>>>>> please consider the timeline and share your schedule
>>>>>> with the Apache Spark community. For the other stuff,
>>>>>> we can put it into 3.2 release scheduled in June 2021.
>>>>>>
>>>>>> Last not but least, I want to emphasize (7) once again.
>>>>>> We need to remove the forked unofficial Hive eventually.
>>>>>> Please let us know your reasons if you need to build
>>>>>> from Apache Spark 3.1 source code for Hive 1.2.
>>>>>>
>>>>>> https://github.com/apache/spark/pull/29936
>>>>>>
>>>>>> As I wrote in the above PR description, for old releases,
>>>>>> Apache Spark 2.4(LTS) and 3.0 (~2021.12) will provide
>>>>>> Hive 1.2-based distribution.
>>>>>>
>>>>>> Bests,
>>>>>> Dongjoon.
>>>>>>
>>>>>

Reply via email to