[DISCUSS] Deprecate legacy file naming functions in FileCommitProtocol

[Cheng Su]

Hello all,

FileCommitProtocol
 is the class to commit Spark job output (staging file & directory renaming, 
etc). During Spark 3.2 development, we added new functions into this class to 
allow more flexible output file naming (the PR detail is 
here). We didn’t delete the 
existing file naming functions (newTaskTempFile(ext) & 
newTaskTempFileAbsPath(ext)), because we were aware of many other downstream 
projects or codebases already implemented their own custom implementation for 
FileCommitProtocol. Delete the existing functions would be a breaking change 
for them when upgrading Spark version, and we would like to avoid this 
unpleasant surprise for anyone if possible. But we also need to clean up legacy 
as we evolve our codebase. The newly added functions should supersede the 
legacy ones, and the cost to migrate would be fairly minimal.

So for next step, I would like to propose:

  *   Spark 3.3 (now): Add @deprecate annotation to legacy functions in 
FileCommitProtocol - 
newTaskTempFile(ext)
 & 
newTaskTempFileAbsPath(ext).
 So developers depending on the legacy functions would notice this and take 
action to move to new functions.
  *   Next Spark major release (or whenever people feel comfortable): delete 
the legacy functions mentioned above from our codebase.

The PR to add @deprecate annotation is ready for review 
https://github.com/apache/spark/pull/35311 . Feel free to comment here or on 
the PR for further discussion.

Thanks,
Cheng Su (@c21)

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Wenchen Fan]

+1

On Tue, Jan 25, 2022 at 10:13 AM Ruifeng Zheng  wrote:

> +1 (non-binding)
>
>
> -- 原始邮件 --
> *发件人:* "Kent Yao" ;
> *发送时间:* 2022年1月25日(星期二) 上午10:09
> *收件人:* "John Zhuge";
> *抄送:* "dev";
> *主题:* Re: [VOTE] Release Spark 3.2.1 (RC2)
>
> +1， non-binding
>
> John Zhuge  于2022年1月25日周二 06:56写道：
>
>> +1 (non-binding)
>>
>> On Mon, Jan 24, 2022 at 2:28 PM Cheng Su  wrote:
>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> Cheng Su
>>>
>>>
>>>
>>> *From: *Chao Sun 
>>> *Date: *Monday, January 24, 2022 at 2:10 PM
>>> *To: *Michael Heuer 
>>> *Cc: *dev 
>>> *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC2)
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer  wrote:
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>>michael
>>>
>>>
>>>
>>>
>>>
>>> On Jan 24, 2022, at 7:30 AM, Gengliang Wang  wrote:
>>>
>>>
>>>
>>> +1 (non-binding)
>>>
>>>
>>>
>>> On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
>>> wrote:
>>>
>>> +1
>>>
>>>
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
>>> wrote:
>>>
>>>
>>>
>>> +1
>>>
>>>
>>>
>>> Signatures, digests, etc check out fine.
>>>
>>> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>>>
>>>
>>>
>>> Regards,
>>>
>>> Mridul
>>>
>>>
>>>
>>> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>>>
>>> +1 with same result as last time.
>>>
>>>
>>>
>>> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
>>> wrote:
>>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 3.2.1. The vote is open until 8:00pm Pacific time January 25 and passes if
>>> a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1
>>> Release this package as Apache Spark 3.2.1 [ ] -1 Do not release this
>>> package because ... To learn more about Apache Spark, please see
>>> http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
>>> 4f25b3f71238a00508a356591553f2dfa89f8290):
>>> https://github.com/apache/spark/tree/v3.2.1-rc2  The release files,
>>> including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/  Signatures
>>> used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS The staging
>>> repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1398/
>>>   The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/  The
>>> list of bug fixes going into 3.2.1 can be found at the following URL:
>>> https://s.apache.org/yu0cy   This release is using the release script
>>> of the tag v3.2.1-rc2. FAQ = How can I help
>>> test this release? = If you are a Spark user, you
>>> can help us test this release by taking an existing Spark workload and
>>> running on this release candidate, then reporting any regressions. If
>>> you're working in PySpark you can set up a virtual env and install the
>>> current RC and see if anything important breaks, in the Java/Scala you can
>>> add the staging repository to your projects resolvers and test with the RC
>>> (make sure to clean up the artifact cache before/after so you don't end up
>>> building with a out of date RC going forward).
>>> === What should happen to JIRA
>>> tickets still targeting 3.2.1? ===
>>> The current list of open tickets targeted at 3.2.1 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.2.1 Committers should look at those and triage. Extremely
>>> important bug fixes, documentation, and API tweaks that impact
>>> compatibility should be worked on immediately. Everything else please
>>> retarget to an appropriate release. == But my bug isn't
>>> fixed? == In order to make timely releases, we will
>>> typically not hold the release unless the bug in question is a regression
>>> from the previous release. That being said, if there is something which is
>>> a regression that has not been correctly targeted please ping me or a
>>> committer to help target the issue.
>>>
>>>
>>>
>>>
>>
>> --
>> John Zhuge
>>
>

回复： [VOTE] Release Spark 3.2.1 (RC2)

[Ruifeng Zheng]

+1 (non-binding)



--原始邮件--
发件人:
"Kent Yao"  
  http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 
(commit
 
4f25b3f71238a00508a356591553f2dfa89f8290):
 
https://github.com/apache/spark/tree/v3.2.1-rc2; 
 
The release files, including signatures, digests, etc. can be found at:
 
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/; 
 
Signatures used for Spark RCs can be found in this file: 
https://dist.apache.org/repos/dist/dev/spark/KEYS The staging repository for 
this release can be found at:
 
https://repository.apache.org/content/repositories/orgapachespark-1398/
 

 
The documentation corresponding to this release can be found at:
 
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/; 
 
The list of bug fixes going into 3.2.1 can be found at the following URL:
 
https://s.apache.org/yu0cy
 

 
This release is using the release script of the tag v3.2.1-rc2. FAQ  
= How can I help test this release? 
= If you are a Spark user, you can help us test this 
release by taking an existing Spark workload and running on this release 
candidate, then reporting any regressions. If  you're working in PySpark you 
can set up a virtual env and install the current RC and see if anything 
important breaks, in the Java/Scala you can add the staging repository to your 
projects resolvers and test with the RC (make sure to clean up the artifact  
cache before/after so you don't end up building with a out of date RC going 
forward). === What should happen to 
JIRA tickets still targeting 3.2.1? === 
The current list of open  tickets targeted at 3.2.1 can be found at: 
https://issues.apache.org/jira/projects/SPARK  and search for "Target 
Version/s" = 3.2.1 Committers should look at those and triage. Extremely 
important bug fixes, documentation, and API tweaks that impact compatibility 
should be worked on immediately. Everything else please retarget to an 
appropriate  release. == But my bug isn't fixed? 
== In order to make timely releases, we will typically not hold 
the release unless the bug in question is a regression from the previous 
release. That being said, if there is something which  is a regression that has 
not been correctly targeted please ping me or a committer to help target the 
issue.
 
  
  
  
 
  
 
  
 

 
 
  
 
 
 



-- 
John Zhuge

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Kent Yao]

+1， non-binding

John Zhuge  于2022年1月25日周二 06:56写道：

> +1 (non-binding)
>
> On Mon, Jan 24, 2022 at 2:28 PM Cheng Su  wrote:
>
>> +1 (non-binding)
>>
>>
>>
>> Cheng Su
>>
>>
>>
>> *From: *Chao Sun 
>> *Date: *Monday, January 24, 2022 at 2:10 PM
>> *To: *Michael Heuer 
>> *Cc: *dev 
>> *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC2)
>>
>> +1 (non-binding)
>>
>>
>>
>> On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer  wrote:
>>
>> +1 (non-binding)
>>
>>
>>
>>michael
>>
>>
>>
>>
>>
>> On Jan 24, 2022, at 7:30 AM, Gengliang Wang  wrote:
>>
>>
>>
>> +1 (non-binding)
>>
>>
>>
>> On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
>> wrote:
>>
>> +1
>>
>>
>>
>> Dongjoon.
>>
>>
>>
>> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
>> wrote:
>>
>>
>>
>> +1
>>
>>
>>
>> Signatures, digests, etc check out fine.
>>
>> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>>
>>
>>
>> Regards,
>>
>> Mridul
>>
>>
>>
>> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>>
>> +1 with same result as last time.
>>
>>
>>
>> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
>> wrote:
>>
>> Please vote on releasing the following candidate as Apache Spark version
>> 3.2.1. The vote is open until 8:00pm Pacific time January 25 and passes if
>> a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1
>> Release this package as Apache Spark 3.2.1 [ ] -1 Do not release this
>> package because ... To learn more about Apache Spark, please see
>> http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
>> 4f25b3f71238a00508a356591553f2dfa89f8290):
>> https://github.com/apache/spark/tree/v3.2.1-rc2  The release files,
>> including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/  Signatures
>> used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS The staging repository
>> for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1398/   The
>> documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/  The
>> list of bug fixes going into 3.2.1 can be found at the following URL:
>> https://s.apache.org/yu0cy   This release is using the release script of
>> the tag v3.2.1-rc2. FAQ = How can I help test
>> this release? = If you are a Spark user, you can
>> help us test this release by taking an existing Spark workload and running
>> on this release candidate, then reporting any regressions. If you're
>> working in PySpark you can set up a virtual env and install the current RC
>> and see if anything important breaks, in the Java/Scala you can add the
>> staging repository to your projects resolvers and test with the RC (make
>> sure to clean up the artifact cache before/after so you don't end up
>> building with a out of date RC going forward).
>> === What should happen to JIRA
>> tickets still targeting 3.2.1? ===
>> The current list of open tickets targeted at 3.2.1 can be found at:
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.2.1 Committers should look at those and triage. Extremely
>> important bug fixes, documentation, and API tweaks that impact
>> compatibility should be worked on immediately. Everything else please
>> retarget to an appropriate release. == But my bug isn't
>> fixed? == In order to make timely releases, we will
>> typically not hold the release unless the bug in question is a regression
>> from the previous release. That being said, if there is something which is
>> a regression that has not been correctly targeted please ping me or a
>> committer to help target the issue.
>>
>>
>>
>>
>
> --
> John Zhuge
>

Re: [How To] run test suites for specific module

[Qian SUN]

Hi Shen

You can use sbt to run a specific suite.

1. run sbt shell.
   $ bash build/sbt
2. specify project.
   sbt > project core
 You can get project name from properties `sbt.project.name` from
pom.xml
3. Finally, you can run a specific suite
   sbt > testOnly org.apache.spark.scheduler.DAGSchedulerSuite

Hope this helps
Best regards,
Qian Sun

Fangjia Shen  于2022年1月25日周二 07:44写道：

> Hello all,
>
> How do you run Spark's test suites when you want to test the correctness
> of your code? Is there a way to run a specific test suite for Spark? For
> example, running test suite XXXSuite alone, instead of every class under
> the test/ directories.
>
> Here's some background info about what I want to do: I'm a graduate
> student trying to study Spark's design and find ways to improve Spark's
> performance by doing Software/Hardware co-design. I'm relatively new to
> Maven and so far struggling to find to a way to properly run Spark's own
> test suites.
>
> Let's say I did some modifications to a XXXExec node which belongs to the
> org.apache.spark.sql package. I want to see if my design passes the test
> cases. What should I do?
>
>
> What command should I use:
>
>  */build/mvn test *  or  */dev/run-tests*  ?
>
> And where should I run that command:
>
> **  or  ** ? - where  is where
> the modified scala file is located, e.g. "/sql/core/".
>
>
> I tried adding -Dtest=XXXSuite to *mvn test *but still get to run tens of
> thousands of tests. This is taking way too much time and unbearable if I'm
> just modifying a few file in a specific module.
>
> I would really appreciate any suggestion or comment.
>
>
> Best regards,
>
> Fangjia Shen
>
> Purdue University
>
>
>
>

-- 
Best!
Qian SUN

Re: [How To] run test suites for specific module

[Maciej]

Hi,

Please check the relevant section of the developer tools docs:

https://spark.apache.org/developer-tools.html#running-individual-tests

On 1/25/22 00:44, Fangjia Shen wrote:
> Hello all,
> 
> How do you run Spark's test suites when you want to test the correctness
> of your code? Is there a way to run a specific test suite for Spark? For
> example, running test suite XXXSuite alone, instead of every class under
> the test/ directories.
> 
> Here's some background info about what I want to do: I'm a graduate
> student trying to study Spark's design and find ways to improve Spark's
> performance by doing Software/Hardware co-design. I'm relatively new to
> Maven and so far struggling to find to a way to properly run Spark's own
> test suites.
> 
> Let's say I did some modifications to a XXXExec node which belongs to
> the org.apache.spark.sql package. I want to see if my design passes the
> test cases. What should I do?
> 
> 
> What command should I use:
> 
>  */build/mvn test *  or  */dev/run-tests*  ?
> 
> And where should I run that command:
> 
>     **  or  ** ? - where  is where
> the modified scala file is located, e.g. "/sql/core/".
> 
> 
> I tried adding -Dtest=XXXSuite to *mvn test *but still get to run tens
> of thousands of tests. This is taking way too much time and unbearable
> if I'm just modifying a few file in a specific module.
> 
> I would really appreciate any suggestion or comment.
> 
> 
> Best regards,
> 
> Fangjia Shen
> 
> Purdue University
> 
> 
> 


-- 
Best regards,
Maciej Szymkiewicz

Web: https://zero323.net
PGP: A30CEF0C31A501EC


OpenPGP_signature
Description: OpenPGP digital signature

[How To] run test suites for specific module

[Fangjia Shen]

Hello all,

How do you run Spark's test suites when you want to test the correctness 
of your code? Is there a way to run a specific test suite for Spark? For 
example, running test suite XXXSuite alone, instead of every class under 
the test/ directories.


Here's some background info about what I want to do: I'm a graduate 
student trying to study Spark's design and find ways to improve Spark's 
performance by doing Software/Hardware co-design. I'm relatively new to 
Maven and so far struggling to find to a way to properly run Spark's own 
test suites.


Let's say I did some modifications to a XXXExec node which belongs to 
the org.apache.spark.sql package. I want to see if my design passes the 
test cases. What should I do?



What command should I use:

*/build/mvn test *  or */dev/run-tests* ?

And where should I run that command:

**  or ** ? - where  is where the 
modified scala file is located, e.g. "/sql/core/".



I tried adding -Dtest=XXXSuite to *mvn test *but still get to run tens 
of thousands of tests. This is taking way too much time and unbearable 
if I'm just modifying a few file in a specific module.


I would really appreciate any suggestion or comment.


Best regards,

Fangjia Shen

Purdue University

Re: [VOTE] Release Spark 3.2.1 (RC2)

[John Zhuge]

+1 (non-binding)

On Mon, Jan 24, 2022 at 2:28 PM Cheng Su  wrote:

> +1 (non-binding)
>
>
>
> Cheng Su
>
>
>
> *From: *Chao Sun 
> *Date: *Monday, January 24, 2022 at 2:10 PM
> *To: *Michael Heuer 
> *Cc: *dev 
> *Subject: *Re: [VOTE] Release Spark 3.2.1 (RC2)
>
> +1 (non-binding)
>
>
>
> On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer  wrote:
>
> +1 (non-binding)
>
>
>
>michael
>
>
>
>
>
> On Jan 24, 2022, at 7:30 AM, Gengliang Wang  wrote:
>
>
>
> +1 (non-binding)
>
>
>
> On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
> wrote:
>
> +1
>
>
>
> Dongjoon.
>
>
>
> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
> wrote:
>
>
>
> +1
>
>
>
> Signatures, digests, etc check out fine.
>
> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>
>
>
> Regards,
>
> Mridul
>
>
>
> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>
> +1 with same result as last time.
>
>
>
> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 3.2.1. The vote is open until 8:00pm Pacific time January 25 and passes if
> a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1
> Release this package as Apache Spark 3.2.1 [ ] -1 Do not release this
> package because ... To learn more about Apache Spark, please see
> http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
> 4f25b3f71238a00508a356591553f2dfa89f8290):
> https://github.com/apache/spark/tree/v3.2.1-rc2  The release files,
> including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/  Signatures
> used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS The staging repository
> for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1398/   The
> documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/  The
> list of bug fixes going into 3.2.1 can be found at the following URL:
> https://s.apache.org/yu0cy   This release is using the release script of
> the tag v3.2.1-rc2. FAQ = How can I help test
> this release? = If you are a Spark user, you can
> help us test this release by taking an existing Spark workload and running
> on this release candidate, then reporting any regressions. If you're
> working in PySpark you can set up a virtual env and install the current RC
> and see if anything important breaks, in the Java/Scala you can add the
> staging repository to your projects resolvers and test with the RC (make
> sure to clean up the artifact cache before/after so you don't end up
> building with a out of date RC going forward).
> === What should happen to JIRA
> tickets still targeting 3.2.1? ===
> The current list of open tickets targeted at 3.2.1 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.2.1 Committers should look at those and triage. Extremely
> important bug fixes, documentation, and API tweaks that impact
> compatibility should be worked on immediately. Everything else please
> retarget to an appropriate release. == But my bug isn't
> fixed? == In order to make timely releases, we will
> typically not hold the release unless the bug in question is a regression
> from the previous release. That being said, if there is something which is
> a regression that has not been correctly targeted please ping me or a
> committer to help target the issue.
>
>
>
>

-- 
John Zhuge

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Cheng Su]

+1 (non-binding)

Cheng Su

From: Chao Sun 
Date: Monday, January 24, 2022 at 2:10 PM
To: Michael Heuer 
Cc: dev 
Subject: Re: [VOTE] Release Spark 3.2.1 (RC2)
+1 (non-binding)

On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer 
mailto:heue...@gmail.com>> wrote:
+1 (non-binding)

   michael


On Jan 24, 2022, at 7:30 AM, Gengliang Wang 
mailto:ltn...@gmail.com>> wrote:

+1 (non-binding)

On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
mailto:dongjoon.h...@gmail.com>> wrote:
+1

Dongjoon.

On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
mailto:mri...@gmail.com>> wrote:

+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes

Regards,
Mridul

On Fri, Jan 21, 2022 at 9:01 PM Sean Owen 
mailto:sro...@apache.org>> wrote:
+1 with same result as last time.

On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
mailto:huaxin.ga...@gmail.com>> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.2.1. 
The vote is open until 8:00pm Pacific time January 25 and passes if a majority 
+1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this 
package as Apache Spark 3.2.1
[ ] -1 Do not release this package because ... To learn more about Apache 
Spark, please see http://spark.apache.org/ The tag to 
be voted on is v3.2.1-rc2 (commit
4f25b3f71238a00508a356591553f2dfa89f8290):
https://github.com/apache/spark/tree/v3.2.1-rc2
The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/
Signatures used for Spark RCs can be found in this file: 
https://dist.apache.org/repos/dist/dev/spark/KEYS
 The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1398/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/
The list of bug fixes going into 3.2.1 can be found at the following URL:
https://s.apache.org/yu0cy

This release is using the release script of the tag v3.2.1-rc2. FAQ 
= How can I help test this release? 
= If you are a Spark user, you can help us test this 
release by taking an existing Spark workload and running on this release 
candidate, then reporting any regressions. If you're working in PySpark you can 
set up a virtual env and install the current RC and see if anything important 
breaks, in the Java/Scala you can add the staging repository to your projects 
resolvers and test with the RC (make sure to clean up the artifact cache 
before/after so you don't end up building with a out of date RC going forward). 
=== What should happen to JIRA tickets 
still targeting 3.2.1? === The current 
list of open tickets targeted at 3.2.1 can be found at: 
https://issues.apache.org/jira/projects/SPARK
 and search for "Target Version/s" = 3.2.1 Committers should look at those and 
triage. Extremely important bug fixes, documentation, and API tweaks that 
impact compatibility should be worked on immediately. Everything else please 
retarget to an appropriate release. == But my bug isn't fixed? 
== In order to make timely releases, we will typically not hold 
the release unless the bug in question is a regression from the previous 
release. That being said, if there is something which is a regression that has 
not been correctly targeted please ping me or a committer to help target the 
issue.

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Chao Sun]

+1 (non-binding)

On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer  wrote:

> +1 (non-binding)
>
>michael
>
>
> On Jan 24, 2022, at 7:30 AM, Gengliang Wang  wrote:
>
> +1 (non-binding)
>
> On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
> wrote:
>
>> +1
>>
>> Dongjoon.
>>
>> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
>> wrote:
>>
>>>
>>> +1
>>>
>>> Signatures, digests, etc check out fine.
>>> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>>>
>>> Regards,
>>> Mridul
>>>
>>> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>>>
 +1 with same result as last time.

 On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
 wrote:

> Please vote on releasing the following candidate as Apache Spark
> version 3.2.1. The vote is open until 8:00pm Pacific time January 25 and
> passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. 
> [
> ] +1 Release this package as Apache Spark 3.2.1[ ] -1 Do not release
> this package because ... To learn more about Apache Spark, please see
> http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
> 4f25b3f71238a00508a356591553f2dfa89f8290):
> https://github.com/apache/spark/tree/v3.2.1-rc2
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS The staging
> repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1398/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/
> The list of bug fixes going into 3.2.1 can be found at the following
> URL:https://s.apache.org/yu0cy
>
> This release is using the release script of the tag v3.2.1-rc2. FAQ
> = How can I help test this release?
> = If you are a Spark user, you can help us test
> this release by taking an existing Spark workload and running on this
> release candidate, then reporting any regressions. If you're working in
> PySpark you can set up a virtual env and install the current RC and see if
> anything important breaks, in the Java/Scala you can add the staging
> repository to your projects resolvers and test with the RC (make sure to
> clean up the artifact cache before/after so you don't end up building with
> a out of date RC going forward).
> === What should happen to JIRA
> tickets still targeting 3.2.1? ===
> The current list of open tickets targeted at 3.2.1 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.2.1 Committers should look at those and triage. Extremely
> important bug fixes, documentation, and API tweaks that impact
> compatibility should be worked on immediately. Everything else please
> retarget to an appropriate release. == But my bug isn't
> fixed? == In order to make timely releases, we will
> typically not hold the release unless the bug in question is a regression
> from the previous release. That being said, if there is something which is
> a regression that has not been correctly targeted please ping me or a
> committer to help target the issue.
>

>

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Michael Heuer]

+1 (non-binding)

   michael


> On Jan 24, 2022, at 7:30 AM, Gengliang Wang  wrote:
> 
> +1 (non-binding)
> 
> On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun  > wrote:
> +1
> 
> Dongjoon.
> 
> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan  > wrote:
> 
> +1 
> 
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
> 
> Regards,
> Mridul
> 
> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  > wrote:
> +1 with same result as last time.
> 
> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao  > wrote:
> Please vote on releasing the following candidate as Apache Spark version 
> 3.2.1.  The vote is open until 8:00pm Pacific time January 25 and passes if a 
> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.  [ ] +1 Release 
> this package as Apache Spark 3.2.1
> [ ] -1 Do not release this package because ...  To learn more about Apache 
> Spark, please see http://spark.apache.org/   The 
> tag to be voted on is v3.2.1-rc2 (commit 
> 4f25b3f71238a00508a356591553f2dfa89f8290):
> https://github.com/apache/spark/tree/v3.2.1-rc2 
>   
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/ 
>   
> Signatures used for Spark RCs can be found in this file: 
> https://dist.apache.org/repos/dist/dev/spark/KEYS 
>   The staging repository 
> for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1398/ 
> 
> 
> The documentation corresponding to this release can be found at: 
> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/ 
>   
> The list of bug fixes going into 3.2.1 can be found at the following URL:
> https://s.apache.org/yu0cy 
> 
> This release is using the release script of the tag v3.2.1-rc2.   FAQ  
> = How can I help test this release? 
> = If you are a Spark user, you can help us test this 
> release by taking an existing Spark workload and running on this release 
> candidate, then reporting any regressions.  If you're working in PySpark you 
> can set up a virtual env and install the current RC and see if anything 
> important breaks, in the Java/Scala you can add the staging repository to 
> your projects resolvers and test with the RC (make sure to clean up the 
> artifact cache before/after so you don't end up building with a out of date 
> RC going forward).  === What should 
> happen to JIRA tickets still targeting 3.2.1? 
> === The current list of open tickets 
> targeted at 3.2.1 can be found at: 
> https://issues.apache.org/jira/projects/SPARK 
>  and search for "Target 
> Version/s" = 3.2.1  Committers should look at those and triage. Extremely 
> important bug fixes, documentation, and API tweaks that impact compatibility 
> should be worked on immediately. Everything else please retarget to an 
> appropriate release.  == But my bug isn't fixed? 
> == In order to make timely releases, we will typically not 
> hold the release unless the bug in question is a regression from the previous 
> release. That being said, if there is something which is a regression that 
> has not been correctly targeted please ping me or a committer to help target 
> the issue.

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Gengliang Wang]

+1 (non-binding)

On Mon, Jan 24, 2022 at 6:26 PM Dongjoon Hyun 
wrote:

> +1
>
> Dongjoon.
>
> On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
> wrote:
>
>>
>> +1
>>
>> Signatures, digests, etc check out fine.
>> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>>
>> Regards,
>> Mridul
>>
>> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>>
>>> +1 with same result as last time.
>>>
>>> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
>>> wrote:
>>>
 Please vote on releasing the following candidate as Apache Spark
 version 3.2.1. The vote is open until 8:00pm Pacific time January 25 and
 passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [
 ] +1 Release this package as Apache Spark 3.2.1[ ] -1 Do not release
 this package because ... To learn more about Apache Spark, please see
 http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
 4f25b3f71238a00508a356591553f2dfa89f8290):
 https://github.com/apache/spark/tree/v3.2.1-rc2
 The release files, including signatures, digests, etc. can be found at:
 https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/
 Signatures used for Spark RCs can be found in this file:
 https://dist.apache.org/repos/dist/dev/spark/KEYS The staging
 repository for this release can be found at:
 https://repository.apache.org/content/repositories/orgapachespark-1398/

 The documentation corresponding to this release can be found at:
 https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/
 The list of bug fixes going into 3.2.1 can be found at the following
 URL:https://s.apache.org/yu0cy

 This release is using the release script of the tag v3.2.1-rc2. FAQ
 = How can I help test this release?
 = If you are a Spark user, you can help us test
 this release by taking an existing Spark workload and running on this
 release candidate, then reporting any regressions. If you're working in
 PySpark you can set up a virtual env and install the current RC and see if
 anything important breaks, in the Java/Scala you can add the staging
 repository to your projects resolvers and test with the RC (make sure to
 clean up the artifact cache before/after so you don't end up building with
 a out of date RC going forward).
 === What should happen to JIRA
 tickets still targeting 3.2.1? ===
 The current list of open tickets targeted at 3.2.1 can be found at:
 https://issues.apache.org/jira/projects/SPARK and search for "Target
 Version/s" = 3.2.1 Committers should look at those and triage. Extremely
 important bug fixes, documentation, and API tweaks that impact
 compatibility should be worked on immediately. Everything else please
 retarget to an appropriate release. == But my bug isn't
 fixed? == In order to make timely releases, we will
 typically not hold the release unless the bug in question is a regression
 from the previous release. That being said, if there is something which is
 a regression that has not been correctly targeted please ping me or a
 committer to help target the issue.

>>>

Re: [VOTE] Release Spark 3.2.1 (RC2)

[Dongjoon Hyun]

+1

Dongjoon.

On Sat, Jan 22, 2022 at 7:19 AM Mridul Muralidharan 
wrote:

>
> +1
>
> Signatures, digests, etc check out fine.
> Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes
>
> Regards,
> Mridul
>
> On Fri, Jan 21, 2022 at 9:01 PM Sean Owen  wrote:
>
>> +1 with same result as last time.
>>
>> On Thu, Jan 20, 2022 at 9:59 PM huaxin gao 
>> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 3.2.1. The vote is open until 8:00pm Pacific time January 25 and passes if
>>> a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1
>>> Release this package as Apache Spark 3.2.1[ ] -1 Do not release this
>>> package because ... To learn more about Apache Spark, please see
>>> http://spark.apache.org/ The tag to be voted on is v3.2.1-rc2 (commit
>>> 4f25b3f71238a00508a356591553f2dfa89f8290):
>>> https://github.com/apache/spark/tree/v3.2.1-rc2
>>> The release files, including signatures, digests, etc. can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/
>>> Signatures used for Spark RCs can be found in this file:
>>> https://dist.apache.org/repos/dist/dev/spark/KEYS The staging
>>> repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1398/
>>>
>>> The documentation corresponding to this release can be found at:
>>> https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/_site/
>>> The list of bug fixes going into 3.2.1 can be found at the following URL:
>>> https://s.apache.org/yu0cy
>>>
>>> This release is using the release script of the tag v3.2.1-rc2. FAQ
>>> = How can I help test this release?
>>> = If you are a Spark user, you can help us test
>>> this release by taking an existing Spark workload and running on this
>>> release candidate, then reporting any regressions. If you're working in
>>> PySpark you can set up a virtual env and install the current RC and see if
>>> anything important breaks, in the Java/Scala you can add the staging
>>> repository to your projects resolvers and test with the RC (make sure to
>>> clean up the artifact cache before/after so you don't end up building with
>>> a out of date RC going forward).
>>> === What should happen to JIRA
>>> tickets still targeting 3.2.1? ===
>>> The current list of open tickets targeted at 3.2.1 can be found at:
>>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.2.1 Committers should look at those and triage. Extremely
>>> important bug fixes, documentation, and API tweaks that impact
>>> compatibility should be worked on immediately. Everything else please
>>> retarget to an appropriate release. == But my bug isn't
>>> fixed? == In order to make timely releases, we will
>>> typically not hold the release unless the bug in question is a regression
>>> from the previous release. That being said, if there is something which is
>>> a regression that has not been correctly targeted please ping me or a
>>> committer to help target the issue.
>>>
>>