Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Mridul Muralidharan 
+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested with "-Pyarn -Pmesos -Pkubernetes"

As always, the test "SPARK-33084: Add jar support Ivy URI in SQL" in
sql.SQLQuerySuite fails in my env; but other than that, the rest looks good.

Regards,
Mridul


On Tue, Jul 12, 2022 at 3:17 AM Maxim Gekk
 wrote:

> +1
>
> On Tue, Jul 12, 2022 at 11:05 AM Yang,Jie(INF) 
> wrote:
>
>> +1 (non-binding)
>>
>>
>>
>> Yang Jie
>>
>>
>>
>>
>>
>> *发件人**: *Dongjoon Hyun 
>> *日期**: *2022年7月12日 星期二 16:03
>> *收件人**: *dev 
>> *抄送**: *Cheng Su , "Yang,Jie(INF)" <
>> yangji...@baidu.com>, Sean Owen 
>> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>>
>>
>>
>> +1
>>
>>
>>
>> Dongjoon.
>>
>>
>>
>> On Mon, Jul 11, 2022 at 11:34 PM Cheng Su  wrote:
>>
>> +1 (non-binding). Built from source, and ran some scala unit tests on M1
>> mac, with OpenJDK 8 and Scala 2.12.
>>
>>
>>
>> Thanks,
>>
>> Cheng Su
>>
>>
>>
>> On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF) 
>> wrote:
>>
>> Does this happen when running all UTs? I ran this suite several times
>> alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error
>> occurred
>>
>>
>>
>> *发件人**: *Sean Owen 
>> *日期**: *2022年7月12日 星期二 10:45
>> *收件人**: *Dongjoon Hyun 
>> *抄送**: *dev 
>> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>>
>>
>>
>> Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:
>>
>>
>>
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962,
>> tid=0x1603
>> #
>> # JRE version: OpenJDK Runtime Environment (8.0_322) (build
>> 1.8.0_322-bre_2022_02_28_15_01-b00)
>> # Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64
>> compressed oops)
>> # Problematic frame:
>> # V  [libjvm.dylib+0x549ace]
>> #
>> # Failed to write core dump. Core dumps have been disabled. To enable
>> core dumping, try "ulimit -c unlimited" before starting Java again
>> #
>> # An error report file with more information is saved as:
>> # /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
>> ColumnVectorSuite:
>> - boolean
>> - byte
>> Compiled method (nm)  885897 75403 n 0
>> sun.misc.Unsafe::putShort (native)
>>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>>  relocation [0x000102fdab38,0x000102fdab78] = 64
>>  main code  [0x000102fdab80,0x000102fdad48] = 456
>> Compiled method (nm)  885897 75403 n 0
>> sun.misc.Unsafe::putShort (native)
>>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>>  relocation [0x000102fdab38,0x000102fdab78] = 64
>>  main code  [0x000102fdab80,0x000102fdad48] = 456
>>
>>
>>
>> On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
>> wrote:
>>
>> Please vote on releasing the following candidate as Apache Spark version
>> 3.2.2.
>>
>> The vote is open until July 15th 1AM (PST) and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.2.2
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see https://spark.apache.org/
>> 
>>
>> The tag to be voted on is v3.2.2-rc1 (commit
>> 78a5825fe266c0884d2dd18cbca9625fa258d7f7):
>> https://github.com/apache/spark/tree/v3.2.2-rc1
>> 
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/
>> 
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>> 
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1409/
>> 
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/
>> 
>>
>> The list of bug fixes going into 3.2.2 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12351232
>> 
>>
>> This release is using the release script of the tag v3.2.2-rc1.
>>
>> FAQ
>>
>> =
>> How

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread qcsd2011 





Hi dev,We would like raise a discussion about supporting `DataFilter pushed down dynamically`. We are trying to push down dataFilter with PartitionFilter, and prune partitionFilter at runtime before push to parquet, which can push less filter to parquet. More details can be found in 35669.before this patch, the physical plan is:Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters: 
[Or(LessThan(id,2),GreaterThan(id,3))]
after this patch, the physical plan is:Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters: 
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
Please note that PushedFilters is changed to PushedDynamicalFilters. [Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))] means data filter id < 2 is dynamically pushed to partition 0 and data filter id > 3 is dynamically pushed to partition 1.We would like to start a discussion about this PR and whether we can use PushedDynamicalFilters to represent filter dynamic pushdown. Any feedback is welcome.Thanks!Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread qcsd2011 





Hi dev,We would like raise a discussion about supporting `DataFilter pushed down dynamically`. We are trying to push down dataFilter with PartitionFilter, and prune partitionFilter at runtime before push to parquet, which can push less filter to parquet. More details can be found in 35669.before this patch, the physical plan is:Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters: 
[Or(LessThan(id,2),GreaterThan(id,3))]
after this patch, the physical plan is:Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters: 
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
Please note that PushedFilters is changed to PushedDynamicalFilters. [Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))] means data filter id < 2 is dynamically pushed to partition 0 and data filter id > 3 is dynamically pushed to partition 1.We would like to start a discussion about this PR and whether we can use PushedDynamicalFilters to represent filter dynamic pushdown. Any feedback is welcome.Thanks!Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread stc 


Hi dev,


We would like raise a discussion about supporting `DataFilter pushed down 
dynamically`. We are trying to push down dataFilter with PartitionFilter, and 
prune partitionFilter at runtime before push to parquet, which can push less 
filter to parquet. More details can be found in 35669.



before this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters: 
[Or(LessThan(id,2),GreaterThan(id,3))]


after this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters: 
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters: 
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters: 
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]


Please note that PushedFilters is changed to PushedDynamicalFilters. 
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
 means data filter id < 2 is dynamically pushed to partition 0 and data filter 
id > 3 is dynamically pushed to partition 1.

We would like to start a discussion about this PR and whether we can use 
PushedDynamicalFilters to represent filter dynamic pushdown. Any feedback is 
welcome.

Thanks!

Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread Jacky Lee 
Hi dev,

We would like to raise a discussion about supporting `DataFilter pushed
down dynamically`. We are trying to push down dataFilter with
PartitionFilter, and prune partitionFilter at runtime before push to
parquet, which can push less filter to parquet. More details can be found
in 35669 .

before this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters:
[Or(LessThan(id,2),GreaterThan(id,3))]

after this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters:
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]

Please note that PushedFilters is changed to PushedDynamicalFilters.
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
means
data filter id < 2 is dynamically pushed to partition 0 and data filter id
> 3 is dynamically pushed to partition 1.

We would like to start a discussion about this PR and whether we can
use PushedDynamicalFilters
to represent filter dynamic pushdown. Any feedback is welcome.

Thanks!

Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread Jacky Lee 
Hi dev,

We would like to raise a discussion about supporting `DataFilter pushed
down dynamically`. We are trying to push down dataFilter with
PartitionFilter, and prune partitionFilter at runtime before push to
parquet, which can push less filter to parquet. More details can be found
in 35669 .

before this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters:
[Or(LessThan(id,2),GreaterThan(id,3))]

after this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters:
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]

Please note that PushedFilters is changed to PushedDynamicalFilters.
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
means
data filter id < 2 is dynamically pushed to partition 0 and data filter id
> 3 is dynamically pushed to partition 1.

We would like to start a discussion about this PR and whether we can
use PushedDynamicalFilters
to represent filter dynamic pushdown. Any feedback is welcome.

Thanks!

Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread qcsd2011 







Hi dev,We would like to raise a discussion about supporting `DataFilterpushed down dynamically`. We are trying to push down dataFilter withPartitionFilter, and prune partitionFilter at runtime before push toparquet, which can push less filter to parquet. More details can befound in https://github.com/apache/spark/pull/35669.before this patch, the physical plan is:Filter:(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))ParquetScan DataFilters:[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]PartitionFilters:[((part#15 = 0) OR (part#15 = 1))]PushedFilters:[Or(LessThan(id,2),GreaterThan(id,3))]after this patch, the physical plan is:Filter:(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))ParquetScan DataFilters:[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]PartitionFilters:[((part#15 = 0) OR (part#15 = 1))]PushedDynamicalFilters:[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]Please note that PushedFilters is changed to PushedDynamicalFilters.[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]means data filter id < 2 is dynamically pushed to partition 0 and datafilter id > 3 is dynamically pushed to partition 1.We would like to start a discussion about this PR and whether we canuse PushedDynamicalFilters to represent filter dynamic pushdown. Anyfeedback is welcome.Thanks!Jacky Lee 





-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread Jacky Lee 
Hi dev,

We would like raise a discussion about supporting `DataFilter pushed down
dynamically`. We are trying to push down dataFilter with PartitionFilter,
and prune partitionFilter at runtime before push to parquet, which can push
less filter to parquet. More details can be found in 35669
.

before this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters:
[Or(LessThan(id,2),GreaterThan(id,3))]

after this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters:
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]

Please note that PushedFilters is changed to PushedDynamicalFilters.
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
means
data filter id < 2 is dynamically pushed to partition 0 and data filter id
> 3 is dynamically pushed to partition 1.

We would like to start a discussion about this PR and whether we can
use PushedDynamicalFilters
to represent filter dynamic pushdown. Any feedback is welcome.

Thanks!

Jacky Lee

[DISCUSS] DataFilter pushed down dynamically

2022-07-12 Thread Jacky Lee 
Hi dev,

We would like to raise a discussion about supporting `DataFilter
pushed down dynamically`. We are trying to push down dataFilter with
PartitionFilter, and prune partitionFilter at runtime before push to
parquet, which can push less filter to parquet. More details can be
found in 35669.

before this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedFilters:
[Or(LessThan(id,2),GreaterThan(id,3))]

after this patch, the physical plan is:

Filter:
(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))

ParquetScan DataFilters:
[(((id#14 < 2) AND (part#15 = 0)) OR ((id#14 > 3) AND (part#15 = 1)))]

PartitionFilters:
[((part#15 = 0) OR (part#15 = 1))]

PushedDynamicalFilters:
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]

Please note that PushedFilters is changed to PushedDynamicalFilters.
[Or(And(LessThan(id,2),EqualTo(part,0)),And(GreaterThan(id,3),EqualTo(part,1)))]
means data filter id < 2 is dynamically pushed to partition 0 and data
filter id > 3 is dynamically pushed to partition 1.

We would like to start a discussion about this PR and whether we can
use PushedDynamicalFilters to represent filter dynamic pushdown. Any
feedback is welcome.

Thanks!

Jacky Lee

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Maxim Gekk 
+1

On Tue, Jul 12, 2022 at 11:05 AM Yang,Jie(INF)  wrote:

> +1 (non-binding)
>
>
>
> Yang Jie
>
>
>
>
>
> *发件人**: *Dongjoon Hyun 
> *日期**: *2022年7月12日 星期二 16:03
> *收件人**: *dev 
> *抄送**: *Cheng Su , "Yang,Jie(INF)" ,
> Sean Owen 
> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>
>
>
> +1
>
>
>
> Dongjoon.
>
>
>
> On Mon, Jul 11, 2022 at 11:34 PM Cheng Su  wrote:
>
> +1 (non-binding). Built from source, and ran some scala unit tests on M1
> mac, with OpenJDK 8 and Scala 2.12.
>
>
>
> Thanks,
>
> Cheng Su
>
>
>
> On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF) 
> wrote:
>
> Does this happen when running all UTs? I ran this suite several times
> alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error
> occurred
>
>
>
> *发件人**: *Sean Owen 
> *日期**: *2022年7月12日 星期二 10:45
> *收件人**: *Dongjoon Hyun 
> *抄送**: *dev 
> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>
>
>
> Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:
>
>
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962,
> tid=0x1603
> #
> # JRE version: OpenJDK Runtime Environment (8.0_322) (build
> 1.8.0_322-bre_2022_02_28_15_01-b00)
> # Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # V  [libjvm.dylib+0x549ace]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
> ColumnVectorSuite:
> - boolean
> - byte
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
>
>
>
> On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 3.2.2.
>
> The vote is open until July 15th 1AM (PST) and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.2.2
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see https://spark.apache.org/
> 
>
> The tag to be voted on is v3.2.2-rc1 (commit
> 78a5825fe266c0884d2dd18cbca9625fa258d7f7):
> https://github.com/apache/spark/tree/v3.2.2-rc1
> 
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/
> 
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
> 
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1409/
> 
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/
> 
>
> The list of bug fixes going into 3.2.2 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12351232
> 
>
> This release is using the release script of the tag v3.2.2-rc1.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Yang,Jie(INF) 
+1 (non-binding)

Yang Jie


发件人: Dongjoon Hyun 
日期: 2022年7月12日 星期二 16:03
收件人: dev 
抄送: Cheng Su , "Yang,Jie(INF)" , Sean 
Owen 
主题: Re: [VOTE] Release Spark 3.2.2 (RC1)

+1

Dongjoon.

On Mon, Jul 11, 2022 at 11:34 PM Cheng Su 
mailto:scnj...@gmail.com>> wrote:
+1 (non-binding). Built from source, and ran some scala unit tests on M1 mac, 
with OpenJDK 8 and Scala 2.12.

Thanks,
Cheng Su

On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF) 
mailto:yangji...@baidu.com>> wrote:
Does this happen when running all UTs? I ran this suite several times alone 
using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error occurred

发件人: Sean Owen mailto:sro...@gmail.com>>
日期: 2022年7月12日 星期二 10:45
收件人: Dongjoon Hyun mailto:dongjoon.h...@gmail.com>>
抄送: dev mailto:dev@spark.apache.org>>
主题: Re: [VOTE] Release Spark 3.2.2 (RC1)

Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962, tid=0x1603
#
# JRE version: OpenJDK Runtime Environment (8.0_322) (build 
1.8.0_322-bre_2022_02_28_15_01-b00)
# Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64 compressed 
oops)
# Problematic frame:
# V  [libjvm.dylib+0x549ace]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
ColumnVectorSuite:
- boolean
- byte
Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort 
(native)
 total in heap  [0x000102fdaa10,0x000102fdad48] = 824
 relocation [0x000102fdab38,0x000102fdab78] = 64
 main code  [0x000102fdab80,0x000102fdad48] = 456
Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort 
(native)
 total in heap  [0x000102fdaa10,0x000102fdad48] = 824
 relocation [0x000102fdab38,0x000102fdab78] = 64
 main code  [0x000102fdab80,0x000102fdad48] = 456

On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
mailto:dongjoon.h...@gmail.com>> wrote:
Please vote on releasing the following candidate as Apache Spark version 3.2.2.

The vote is open until July 15th 1AM (PST) and passes if a majority +1 PMC 
votes are cast, with a minimum of 3 +1 votes.

[ ] +1 Release this package as Apache Spark 3.2.2
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see 
https://spark.apache.org/

The tag to be voted on is v3.2.2-rc1 (commit 
78a5825fe266c0884d2dd18cbca9625fa258d7f7):
https://github.com/apache/spark/tree/v3.2.2-rc1

The release files, including signatures, digests, etc. can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/

Signatures used for Spark RCs can be found in this file:
https://dist.apache.org/repos/dist/dev/spark/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1409/

The documentation corresponding to this release can be found at:
https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/

The list of bug fixes going into 3.2.2 can be found at the following URL:
https://issues.apache.org/jira/projects/SPARK/versions/12351232

This release is using the release script of the tag v3.2.2-rc1.

FAQ

=
How can I help test this release?
=

If you are a Spark user, you can help us test this release by taking
an existing Spark workload and running on this release candidate, then
reporting any regressions.

If you're working in PySpark you can set up a virtual env and install
the current RC and see if anything important breaks, in the Java/Scala
you can add the staging repository to your projects resolvers and test
with the RC (make sure to clean up the artifact cache before/after so
you don't end up building with a out of date RC going forward).

===
What should happen to JIRA tickets still targeting

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Dongjoon Hyun 
+1

Dongjoon.

On Mon, Jul 11, 2022 at 11:34 PM Cheng Su  wrote:

> +1 (non-binding). Built from source, and ran some scala unit tests on M1
> mac, with OpenJDK 8 and Scala 2.12.
>
> Thanks,
> Cheng Su
>
> On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF) 
> wrote:
>
>> Does this happen when running all UTs? I ran this suite several times
>> alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error
>> occurred
>>
>>
>>
>> *发件人**: *Sean Owen 
>> *日期**: *2022年7月12日 星期二 10:45
>> *收件人**: *Dongjoon Hyun 
>> *抄送**: *dev 
>> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>>
>>
>>
>> Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:
>>
>>
>>
>> #
>> # A fatal error has been detected by the Java Runtime Environment:
>> #
>> #  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962,
>> tid=0x1603
>> #
>> # JRE version: OpenJDK Runtime Environment (8.0_322) (build
>> 1.8.0_322-bre_2022_02_28_15_01-b00)
>> # Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64
>> compressed oops)
>> # Problematic frame:
>> # V  [libjvm.dylib+0x549ace]
>> #
>> # Failed to write core dump. Core dumps have been disabled. To enable
>> core dumping, try "ulimit -c unlimited" before starting Java again
>> #
>> # An error report file with more information is saved as:
>> # /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
>> ColumnVectorSuite:
>> - boolean
>> - byte
>> Compiled method (nm)  885897 75403 n 0
>> sun.misc.Unsafe::putShort (native)
>>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>>  relocation [0x000102fdab38,0x000102fdab78] = 64
>>  main code  [0x000102fdab80,0x000102fdad48] = 456
>> Compiled method (nm)  885897 75403 n 0
>> sun.misc.Unsafe::putShort (native)
>>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>>  relocation [0x000102fdab38,0x000102fdab78] = 64
>>  main code  [0x000102fdab80,0x000102fdad48] = 456
>>
>>
>>
>> On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
>> wrote:
>>
>> Please vote on releasing the following candidate as Apache Spark version
>> 3.2.2.
>>
>> The vote is open until July 15th 1AM (PST) and passes if a majority +1
>> PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.2.2
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see https://spark.apache.org/
>> 
>>
>> The tag to be voted on is v3.2.2-rc1 (commit
>> 78a5825fe266c0884d2dd18cbca9625fa258d7f7):
>> https://github.com/apache/spark/tree/v3.2.2-rc1
>> 
>>
>> The release files, including signatures, digests, etc. can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/
>> 
>>
>> Signatures used for Spark RCs can be found in this file:
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>> 
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1409/
>> 
>>
>> The documentation corresponding to this release can be found at:
>> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/
>> 
>>
>> The list of bug fixes going into 3.2.2 can be found at the following URL:
>> https://issues.apache.org/jira/projects/SPARK/versions/12351232
>> 
>>
>> This release is using the release script of the tag v3.2.2-rc1.
>>
>> FAQ
>>
>> =
>> How can I help test this release?
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>> an existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>> the current RC and see if anything important breaks, in the Java/Scala
>> you can add the staging repository to your projects resolvers and test
>> with the RC (make sure to clean up the artifact cache before/after so
>> you don't end up building with a out of date RC going forward).
>>
>> ===
>> What should happen to JIRA tickets still targeting 3.2.2?
>>

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Dongjoon Hyun 
I also didn't hit that issue (Temurin Java 8 on Intel MacOS).

$ java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)

$ sw_vers
ProductName: macOS
ProductVersion: 12.4

Dongjoon.

On Mon, Jul 11, 2022 at 10:30 PM Yang,Jie(INF)  wrote:

> Does this happen when running all UTs? I ran this suite several times
> alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error
> occurred
>
>
>
> *发件人**: *Sean Owen 
> *日期**: *2022年7月12日 星期二 10:45
> *收件人**: *Dongjoon Hyun 
> *抄送**: *dev 
> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>
>
>
> Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:
>
>
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962,
> tid=0x1603
> #
> # JRE version: OpenJDK Runtime Environment (8.0_322) (build
> 1.8.0_322-bre_2022_02_28_15_01-b00)
> # Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # V  [libjvm.dylib+0x549ace]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
> ColumnVectorSuite:
> - boolean
> - byte
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
>
>
>
> On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 3.2.2.
>
> The vote is open until July 15th 1AM (PST) and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.2.2
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see https://spark.apache.org/
> 
>
> The tag to be voted on is v3.2.2-rc1 (commit
> 78a5825fe266c0884d2dd18cbca9625fa258d7f7):
> https://github.com/apache/spark/tree/v3.2.2-rc1
> 
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/
> 
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
> 
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1409/
> 
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/
> 
>
> The list of bug fixes going into 3.2.2 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12351232
> 
>
> This release is using the release script of the tag v3.2.2-rc1.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 3.2.2?
>

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Cheng Su 
+1 (non-binding). Built from source, and ran some scala unit tests on M1
mac, with OpenJDK 8 and Scala 2.12.

Thanks,
Cheng Su

On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF)  wrote:

> Does this happen when running all UTs? I ran this suite several times
> alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error
> occurred
>
>
>
> *发件人**: *Sean Owen 
> *日期**: *2022年7月12日 星期二 10:45
> *收件人**: *Dongjoon Hyun 
> *抄送**: *dev 
> *主题**: *Re: [VOTE] Release Spark 3.2.2 (RC1)
>
>
>
> Is anyone seeing this error? I'm on OpenJDK 8 on a Mac:
>
>
>
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x000101ca8ace, pid=11962,
> tid=0x1603
> #
> # JRE version: OpenJDK Runtime Environment (8.0_322) (build
> 1.8.0_322-bre_2022_02_28_15_01-b00)
> # Java VM: OpenJDK 64-Bit Server VM (25.322-b00 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # V  [libjvm.dylib+0x549ace]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /private/tmp/spark-3.2.2/sql/core/hs_err_pid11962.log
> ColumnVectorSuite:
> - boolean
> - byte
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
> Compiled method (nm)  885897 75403 n 0   sun.misc.Unsafe::putShort
> (native)
>  total in heap  [0x000102fdaa10,0x000102fdad48] = 824
>  relocation [0x000102fdab38,0x000102fdab78] = 64
>  main code  [0x000102fdab80,0x000102fdad48] = 456
>
>
>
> On Mon, Jul 11, 2022 at 4:58 PM Dongjoon Hyun 
> wrote:
>
> Please vote on releasing the following candidate as Apache Spark version
> 3.2.2.
>
> The vote is open until July 15th 1AM (PST) and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.2.2
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see https://spark.apache.org/
> 
>
> The tag to be voted on is v3.2.2-rc1 (commit
> 78a5825fe266c0884d2dd18cbca9625fa258d7f7):
> https://github.com/apache/spark/tree/v3.2.2-rc1
> 
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-bin/
> 
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
> 
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1409/
> 
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.2.2-rc1-docs/
> 
>
> The list of bug fixes going into 3.2.2 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12351232
> 
>
> This release is using the release script of the tag v3.2.2-rc1.
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 3.2.2?
> ===
>
> The current list of open tickets targeted at 3.2.2 can be found at:
> https://issues.apache.org/jira/projects/SPARK
>