Re: Welcome two new Apache Spark committers

[XiDuo You]

Thank you all !

Jia Fan  于2023年8月7日周一 11:31写道：
>
> Congratulations!
> 
>
> Jia Fan
>
>
> 2023年8月7日 11:28，Ye Xianjin  写道：
>
> Congratulations!
>
> Sent from my iPhone
>
> On Aug 7, 2023, at 11:16 AM, Yuming Wang  wrote:
>
> 
>
> Congratulations!
>
> On Mon, Aug 7, 2023 at 11:11 AM Kent Yao  wrote:
>>
>> Congrats! Peter and Xiduo!
>>
>> Cheng Pan  于2023年8月7日周一 11:01写道：
>> >
>> > Congratulations! Peter and Xiduo!
>> >
>> > Thanks,
>> > Cheng Pan
>> >
>> >
>> > > On Aug 7, 2023, at 10:58, Gengliang Wang  wrote:
>> > >
>> > > Congratulations! Peter and Xiduo!
>> >
>> >
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Welcome two new Apache Spark committers

[Jia Fan]

Congratulations!


Jia Fan


> 2023年8月7日 11:28，Ye Xianjin  写道：
> 
> Congratulations!
> 
> Sent from my iPhone
> 
>> On Aug 7, 2023, at 11:16 AM, Yuming Wang  wrote:
>> 
>> 
>> 
>> Congratulations!
>> 
>> On Mon, Aug 7, 2023 at 11:11 AM Kent Yao > > wrote:
>>> Congrats! Peter and Xiduo!
>>> 
>>> Cheng Pan mailto:pan3...@gmail.com>> 于2023年8月7日周一 
>>> 11:01写道：
>>> >
>>> > Congratulations! Peter and Xiduo!
>>> >
>>> > Thanks,
>>> > Cheng Pan
>>> >
>>> >
>>> > > On Aug 7, 2023, at 10:58, Gengliang Wang >> > > > wrote:
>>> > >
>>> > > Congratulations! Peter and Xiduo!
>>> >
>>> >
>>> >
>>> > -
>>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org 
>>> > 
>>> >
>>> 
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org 
>>> 
>>> 

Re: Welcome two new Apache Spark committers

[Ye Xianjin]

Congratulations!Sent from my iPhoneOn Aug 7, 2023, at 11:16 AM, Yuming Wang  wrote:Congratulations!On Mon, Aug 7, 2023 at 11:11 AM Kent Yao  wrote:Congrats! Peter and Xiduo!

Cheng Pan  于2023年8月7日周一 11:01写道：
>
> Congratulations! Peter and Xiduo!
>
> Thanks,
> Cheng Pan
>
>
> > On Aug 7, 2023, at 10:58, Gengliang Wang  wrote:
> >
> > Congratulations! Peter and Xiduo!
>
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

[yangjie01]

I submitted a PR last week to try and solve this issue: 
https://github.com/apache/spark/pull/42236.

发件人: Sean Owen 
日期: 2023年8月7日 星期一 11:05
收件人: Yuanjian Li 
抄送: Spark dev list 
主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC1)


【外部邮件】信息安全要牢记，账号密码不传递！


Let's keep testing 3.5.0 of course while that change is going in. (See 
https://github.com/apache/spark/pull/42364#issuecomment-1666878287
 )

Otherwise testing is pretty much as usual, except I get this test failure in 
Connect, which is new. Anyone else? this is Java 8, Scala 2.13, Debian 12.

- from_protobuf_messageClassName_options *** FAILED ***
  org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_PROTOBUF_CLASS] Could 
not load Protobuf class with name org.apache.spark.connect.proto.StorageLevel. 
org.apache.spark.connect.proto.StorageLevel does not extend shaded Protobuf 
Message class org.sparkproject.spark_protobuf.protobuf.Message. The jar with 
Protobuf classes needs to be shaded (com.google.protobuf.* --> 
org.sparkproject.spark_protobuf.protobuf.*).
  at 
org.apache.spark.sql.errors.QueryCompilationErrors$.protobufClassLoadError(QueryCompilationErrors.scala:3554)
  at 
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptorFromJavaClass(ProtobufUtils.scala:198)
  at 
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:156)
  at 
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor$lzycompute(ProtobufDataToCatalyst.scala:58)
  at 
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor(ProtobufDataToCatalyst.scala:57)
  at 
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType$lzycompute(ProtobufDataToCatalyst.scala:43)
  at 
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType(ProtobufDataToCatalyst.scala:42)
  at 
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:194)
  at 
org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:73)
  at scala.collection.immutable.List.map(List.scala:246)

On Sat, Aug 5, 2023 at 5:42 PM Sean Owen 
mailto:sro...@gmail.com>> wrote:
I'm still testing other combinations, but it looks like tests fail on Java 17 
after building with Java 8, which should be a normal supported configuration.
This is described at 
https://github.com/apache/spark/pull/41943
 and looks like it is resolved by moving back to Scala 2.13.8 for now.
Unless I'm missing something we need to fix this for 3.5 or it's not clear the 
build will run on Java 17.

On Fri, Aug 4, 2023 at 5:45 PM Yuanjian Li 
mailto:xyliyuanj...@gmail.com>> wrote:

Please vote on releasing the following candidate(RC1) as Apache Spark version 
3.5.0.


The vote is open until 11:59pm Pacific time Aug 9th and passes if a majority +1 
PMC votes are cast, with a minimum of 3 +1 votes.


[ ] +1 Release this package as Apache Spark 3.5.0

[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see 
http://spark.apache.org/


The tag to be voted on is v3.5.0-rc1 (commit 
7e862c01fc9a1d3b47764df8b6a4b5c4cafb0807):

https://github.com/apache/spark/tree/v3.5.0-rc1


The release files, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-bin/


Signatures used for Spark RCs can be found in this file:

https://dist.apache.org/repos/dist/dev/spark/KEYS


The staging repository for this release can be found at:

https://repository.apache.org/content/repositories/orgapachespark-1444


The documentation corresponding to this release can be found at:

https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-docs/


The list of bug fixes going into 3.5.0 can be found at the following URL:

https://issues.apache.org/jira/projects/SPARK/versions/12352848

Re: Welcome two new Apache Spark committers

[Yuming Wang]

Congratulations!

On Mon, Aug 7, 2023 at 11:11 AM Kent Yao  wrote:

> Congrats! Peter and Xiduo!
>
> Cheng Pan  于2023年8月7日周一 11:01写道：
> >
> > Congratulations! Peter and Xiduo!
> >
> > Thanks,
> > Cheng Pan
> >
> >
> > > On Aug 7, 2023, at 10:58, Gengliang Wang  wrote:
> > >
> > > Congratulations! Peter and Xiduo!
> >
> >
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Welcome two new Apache Spark committers

[Kent Yao]

Congrats! Peter and Xiduo!

Cheng Pan  于2023年8月7日周一 11:01写道：
>
> Congratulations! Peter and Xiduo!
>
> Thanks,
> Cheng Pan
>
>
> > On Aug 7, 2023, at 10:58, Gengliang Wang  wrote:
> >
> > Congratulations! Peter and Xiduo!
>
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Welcome two new Apache Spark committers

[Cheng Pan]

Congratulations! Peter and Xiduo! 

Thanks,
Cheng Pan


> On Aug 7, 2023, at 10:58, Gengliang Wang  wrote:
> 
> Congratulations! Peter and Xiduo! 



-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Welcome two new Apache Spark committers

[Gengliang Wang]

Congratulations! Peter and Xiduo!

On Sun, Aug 6, 2023 at 7:37 PM Jungtaek Lim 
wrote:

> Congrats Peter and Xiduo!
>
> On Mon, Aug 7, 2023 at 11:33 AM yangjie01 
> wrote:
>
>> Congratulations, Peter and Xiduo ~
>>
>>
>>
>> *发件人**: *Hyukjin Kwon 
>> *日期**: *2023年8月7日 星期一 10:30
>> *收件人**: *Ruifeng Zheng 
>> *抄送**: *Xiao Li , Debasish Das <
>> debasish.da...@gmail.com>, Wenchen Fan , Spark dev
>> list 
>> *主题**: *Re: Welcome two new Apache Spark committers
>>
>>
>>
>> Woohoo!
>>
>>
>>
>> On Mon, 7 Aug 2023 at 11:28, Ruifeng Zheng  wrote:
>>
>> Congratulations! Peter and Xiduo!
>>
>>
>>
>> On Mon, Aug 7, 2023 at 10:13 AM Xiao Li  wrote:
>>
>> Congratulations, Peter and Xiduo!
>>
>>
>>
>>
>>
>>
>>
>> Debasish Das  于2023年8月6日周日 19:08写道：
>>
>> Congratulations Peter and Xidou.
>>
>> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
>>
>> Hi all,
>>
>>
>>
>> The Spark PMC recently voted to add two new committers. Please join me in
>> welcoming them to their new role!
>>
>>
>>
>> - Peter Toth (Spark SQL)
>>
>> - Xiduo You (Spark SQL)
>>
>>
>>
>> They consistently make contributions to the project and clearly showed
>> their expertise. We are very excited to have them join as committers.
>>
>>

Re: Welcome two new Apache Spark committers

[Jungtaek Lim]

Congrats Peter and Xiduo!

On Mon, Aug 7, 2023 at 11:33 AM yangjie01 
wrote:

> Congratulations, Peter and Xiduo ~
>
>
>
> *发件人**: *Hyukjin Kwon 
> *日期**: *2023年8月7日 星期一 10:30
> *收件人**: *Ruifeng Zheng 
> *抄送**: *Xiao Li , Debasish Das <
> debasish.da...@gmail.com>, Wenchen Fan , Spark dev
> list 
> *主题**: *Re: Welcome two new Apache Spark committers
>
>
>
> Woohoo!
>
>
>
> On Mon, 7 Aug 2023 at 11:28, Ruifeng Zheng  wrote:
>
> Congratulations! Peter and Xiduo!
>
>
>
> On Mon, Aug 7, 2023 at 10:13 AM Xiao Li  wrote:
>
> Congratulations, Peter and Xiduo!
>
>
>
>
>
>
>
> Debasish Das  于2023年8月6日周日 19:08写道：
>
> Congratulations Peter and Xidou.
>
> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
>
> Hi all,
>
>
>
> The Spark PMC recently voted to add two new committers. Please join me in
> welcoming them to their new role!
>
>
>
> - Peter Toth (Spark SQL)
>
> - Xiduo You (Spark SQL)
>
>
>
> They consistently make contributions to the project and clearly showed
> their expertise. We are very excited to have them join as committers.
>
>

Re: Welcome two new Apache Spark committers

[Binjie Yang]

Congratulations! Peter and Xiduo!

On 2023/08/07 02:26:09 Ruifeng Zheng wrote:
> Congratulations! Peter and Xiduo!
> 
> On Mon, Aug 7, 2023 at 10:13 AM Xiao Li  wrote:
> 
> > Congratulations, Peter and Xiduo!
> >
> >
> >
> > Debasish Das  于2023年8月6日周日 19:08写道：
> >
> >> Congratulations Peter and Xidou.
> >>
> >> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
> >>
> >>> Hi all,
> >>>
> >>> The Spark PMC recently voted to add two new committers. Please join me
> >>> in welcoming them to their new role!
> >>>
> >>> - Peter Toth (Spark SQL)
> >>> - Xiduo You (Spark SQL)
> >>>
> >>> They consistently make contributions to the project and clearly showed
> >>> their expertise. We are very excited to have them join as committers.
> >>>
> >>
> 

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Welcome two new Apache Spark committers

[yangjie01]

Congratulations, Peter and Xiduo ~

发件人: Hyukjin Kwon 
日期: 2023年8月7日 星期一 10:30
收件人: Ruifeng Zheng 
抄送: Xiao Li , Debasish Das , 
Wenchen Fan , Spark dev list 
主题: Re: Welcome two new Apache Spark committers

Woohoo!

On Mon, 7 Aug 2023 at 11:28, Ruifeng Zheng 
mailto:ruife...@apache.org>> wrote:
Congratulations! Peter and Xiduo!

On Mon, Aug 7, 2023 at 10:13 AM Xiao Li 
mailto:gatorsm...@gmail.com>> wrote:
Congratulations, Peter and Xiduo!



Debasish Das mailto:debasish.da...@gmail.com>> 
于2023年8月6日周日 19:08写道：
Congratulations Peter and Xidou.
On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan 
mailto:cloud0...@gmail.com>> wrote:
Hi all,

The Spark PMC recently voted to add two new committers. Please join me in 
welcoming them to their new role!

- Peter Toth (Spark SQL)
- Xiduo You (Spark SQL)

They consistently make contributions to the project and clearly showed their 
expertise. We are very excited to have them join as committers.

Re: Welcome two new Apache Spark committers

[Hyukjin Kwon]

Woohoo!

On Mon, 7 Aug 2023 at 11:28, Ruifeng Zheng  wrote:

> Congratulations! Peter and Xiduo!
>
> On Mon, Aug 7, 2023 at 10:13 AM Xiao Li  wrote:
>
>> Congratulations, Peter and Xiduo!
>>
>>
>>
>> Debasish Das  于2023年8月6日周日 19:08写道：
>>
>>> Congratulations Peter and Xidou.
>>>
>>> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
>>>
 Hi all,

 The Spark PMC recently voted to add two new committers. Please join me
 in welcoming them to their new role!

 - Peter Toth (Spark SQL)
 - Xiduo You (Spark SQL)

 They consistently make contributions to the project and clearly showed
 their expertise. We are very excited to have them join as committers.

>>>

Re: Welcome two new Apache Spark committers

[Ruifeng Zheng]

Congratulations! Peter and Xiduo!

On Mon, Aug 7, 2023 at 10:13 AM Xiao Li  wrote:

> Congratulations, Peter and Xiduo!
>
>
>
> Debasish Das  于2023年8月6日周日 19:08写道：
>
>> Congratulations Peter and Xidou.
>>
>> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
>>
>>> Hi all,
>>>
>>> The Spark PMC recently voted to add two new committers. Please join me
>>> in welcoming them to their new role!
>>>
>>> - Peter Toth (Spark SQL)
>>> - Xiduo You (Spark SQL)
>>>
>>> They consistently make contributions to the project and clearly showed
>>> their expertise. We are very excited to have them join as committers.
>>>
>>

Re: Welcome two new Apache Spark committers

[Xiao Li]

Congratulations, Peter and Xiduo!



Debasish Das  于2023年8月6日周日 19:08写道：

> Congratulations Peter and Xidou.
>
> On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:
>
>> Hi all,
>>
>> The Spark PMC recently voted to add two new committers. Please join me in
>> welcoming them to their new role!
>>
>> - Peter Toth (Spark SQL)
>> - Xiduo You (Spark SQL)
>>
>> They consistently make contributions to the project and clearly showed
>> their expertise. We are very excited to have them join as committers.
>>
>

Re: Welcome two new Apache Spark committers

[Debasish Das]

Congratulations Peter and Xidou.

On Sun, Aug 6, 2023, 7:05 PM Wenchen Fan  wrote:

> Hi all,
>
> The Spark PMC recently voted to add two new committers. Please join me in
> welcoming them to their new role!
>
> - Peter Toth (Spark SQL)
> - Xiduo You (Spark SQL)
>
> They consistently make contributions to the project and clearly showed
> their expertise. We are very excited to have them join as committers.
>

Welcome two new Apache Spark committers

[Wenchen Fan]

Hi all,

The Spark PMC recently voted to add two new committers. Please join me in
welcoming them to their new role!

- Peter Toth (Spark SQL)
- Xiduo You (Spark SQL)

They consistently make contributions to the project and clearly showed
their expertise. We are very excited to have them join as committers.

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

[Sean Owen]

Let's keep testing 3.5.0 of course while that change is going in. (See
https://github.com/apache/spark/pull/42364#issuecomment-1666878287 )

Otherwise testing is pretty much as usual, except I get this test failure
in Connect, which is new. Anyone else? this is Java 8, Scala 2.13, Debian
12.

- from_protobuf_messageClassName_options *** FAILED ***
  org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_PROTOBUF_CLASS]
Could not load Protobuf class with name
org.apache.spark.connect.proto.StorageLevel.
org.apache.spark.connect.proto.StorageLevel does not extend shaded Protobuf
Message class org.sparkproject.spark_protobuf.protobuf.Message. The jar
with Protobuf classes needs to be shaded (com.google.protobuf.* -->
org.sparkproject.spark_protobuf.protobuf.*).
  at
org.apache.spark.sql.errors.QueryCompilationErrors$.protobufClassLoadError(QueryCompilationErrors.scala:3554)
  at
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptorFromJavaClass(ProtobufUtils.scala:198)
  at
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:156)
  at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor$lzycompute(ProtobufDataToCatalyst.scala:58)
  at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor(ProtobufDataToCatalyst.scala:57)
  at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType$lzycompute(ProtobufDataToCatalyst.scala:43)
  at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType(ProtobufDataToCatalyst.scala:42)
  at
org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:194)
  at
org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:73)
  at scala.collection.immutable.List.map(List.scala:246)

On Sat, Aug 5, 2023 at 5:42 PM Sean Owen  wrote:

> I'm still testing other combinations, but it looks like tests fail on Java
> 17 after building with Java 8, which should be a normal supported
> configuration.
> This is described at https://github.com/apache/spark/pull/41943 and looks
> like it is resolved by moving back to Scala 2.13.8 for now.
> Unless I'm missing something we need to fix this for 3.5 or it's not clear
> the build will run on Java 17.
>
> On Fri, Aug 4, 2023 at 5:45 PM Yuanjian Li  wrote:
>
>> Please vote on releasing the following candidate(RC1) as Apache Spark
>> version 3.5.0.
>>
>> The vote is open until 11:59pm Pacific time Aug 9th and passes if a
>> majority +1 PMC votes are cast, with a minimum of 3 +1 votes.
>>
>> [ ] +1 Release this package as Apache Spark 3.5.0
>>
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is v3.5.0-rc1 (commit
>> 7e862c01fc9a1d3b47764df8b6a4b5c4cafb0807):
>>
>> https://github.com/apache/spark/tree/v3.5.0-rc1
>>
>> The release files, including signatures, digests, etc. can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-bin/
>>
>> Signatures used for Spark RCs can be found in this file:
>>
>> https://dist.apache.org/repos/dist/dev/spark/KEYS
>>
>> The staging repository for this release can be found at:
>>
>> https://repository.apache.org/content/repositories/orgapachespark-1444
>>
>> The documentation corresponding to this release can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/spark/v3.5.0-rc1-docs/
>>
>> The list of bug fixes going into 3.5.0 can be found at the following URL:
>>
>> https://issues.apache.org/jira/projects/SPARK/versions/12352848
>>
>> This release is using the release script of the tag v3.5.0-rc1.
>>
>>
>> FAQ
>>
>> =
>>
>> How can I help test this release?
>>
>> =
>>
>> If you are a Spark user, you can help us test this release by taking
>>
>> an existing Spark workload and running on this release candidate, then
>>
>> reporting any regressions.
>>
>> If you're working in PySpark you can set up a virtual env and install
>>
>> the current RC and see if anything important breaks, in the Java/Scala
>>
>> you can add the staging repository to your projects resolvers and test
>>
>> with the RC (make sure to clean up the artifact cache before/after so
>>
>> you don't end up building with an out of date RC going forward).
>>
>> ===
>>
>> What should happen to JIRA tickets still targeting 3.5.0?
>>
>> ===
>>
>> The current list of open tickets targeted at 3.5.0 can be found at:
>>
>> https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.5.0
>>
>> Committers should look at those and triage. Extremely important bug
>>
>> fixes, documentation, and API tweaks that impact compatibility should
>>
>> be worked on immediately. Everything else please retarget to an
>>
>> appropriate release.
>>
>> ==
>>
>> But my bug isn't fixed?
>>
>> ==
>>
>> In order to make