Re:[ANNOUNCE] Apache Spark 3.4.2 released

[beliefer]

Congratulations!







At 2023-12-01 01:23:55, "Dongjoon Hyun"  wrote:

We are happy to announce the availability of Apache Spark 3.4.2!

Spark 3.4.2 is a maintenance release containing many fixes including
security and correctness domains. This release is based on the
branch-3.4 maintenance branch of Spark. We strongly
recommend all 3.4 users to upgrade to this stable release.

To download Spark 3.4.2, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-4-2.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun

Unsubscribe

[Devarshi Vyas]

unsubscribe

[Sandeep Vinayak]

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

[Kumar K]

+1

On Fri, Nov 10, 2023 at 8:51 PM Khalid Mammadov 
wrote:

> +1
>
> On Fri, 10 Nov 2023, 15:23 Peter Toth,  wrote:
>
>> +1
>>
>> On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen 
>> wrote:
>>
>>> +1
>>>
>>> fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu :
>>>
 just curious what happened on google’s spark operator?

 On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko  wrote:

> +1
>
> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue  wrote:
>
>> +1
>>
>> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala 
>> wrote:
>>
>>> +1 for creating an official Kubernetes operator for Apache Spark
>>>
>>> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao 
>>> wrote:
>>>
 +1

>>>
 On Thu, Nov 9, 2023 at 3:14 PM DB Tsai  wrote:

> +1
>
> To be completely transparent, I am employed in the same department
> as Zhou at Apple.
>
> I support this proposal, provided that we witness community
> adoption following the release of the Flink Kubernetes operator,
> streamlining Flink deployment on Kubernetes.
>
> A well-maintained official Spark Kubernetes operator is essential
> for our Spark community as well.
>
> DB Tsai  |  https://www.dbtsai.com/
> 
>  |  PGP 42E5B25A8F7A82C1
>
> On Nov 9, 2023, at 12:05 PM, Zhou Jiang 
> wrote:
>
> Hi Spark community,
> I'm reaching out to initiate a conversation about the possibility
> of developing a Java-based Kubernetes operator for Apache Spark. 
> Following
> the operator pattern (
> https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
> ),
> Spark users may manage applications and related components seamlessly 
> using
> native tools like kubectl. The primary goal is to simplify the Spark 
> user
> experience on Kubernetes, minimizing the learning curve and 
> operational
> complexities and therefore enable users to focus on the Spark 
> application
> development.
> Although there are several open-source Spark on Kubernetes
> operators available, none of them are officially integrated into the 
> Apache
> Spark project. As a result, these operators may lack active support 
> and
> development for new features. Within this proposal, our aim is to 
> introduce
> a Java-based Spark operator as an integral component of the Apache 
> Spark
> project. This solution has been employed internally at Apple for 
> multiple
> years, operating millions of executors in real production 
> environments. The
> use of Java in this solution is intended to accommodate a wider user 
> and
> contributor audience, especially those who are familiar with Scala.
> Ideally, this operator should have its dedicated repository,
> similar to Spark Connect Golang or Spark Docker, allowing it to 
> maintain a
> loose connection with the Spark release cycle. This model is also 
> followed
> by the Apache Flink Kubernetes operator.
> We believe that this project holds the potential to evolve into a
> thriving community project over the long run. A comparison can be 
> drawn
> with the Flink Kubernetes Operator: Apple has open-sourced internal 
> Flink
> Kubernetes operator, making it a part of the Apache Flink project (
> https://github.com/apache/flink-kubernetes-operator
> ).
> This move has gained wide industry adoption and contributions 

[ANNOUNCE] Apache Spark 3.4.2 released

[Dongjoon Hyun]

We are happy to announce the availability of Apache Spark 3.4.2!

Spark 3.4.2 is a maintenance release containing many fixes including
security and correctness domains. This release is based on the
branch-3.4 maintenance branch of Spark. We strongly
recommend all 3.4 users to upgrade to this stable release.

To download Spark 3.4.2, head over to the download page:
https://spark.apache.org/downloads.html

To view the release notes:
https://spark.apache.org/releases/spark-release-3-4-2.html

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun

[VOTE][RESULT] Release Spark 3.4.2 (RC1)

[Dongjoon Hyun]

The vote passes with 6 +1s (3 binding +1s) and one non-binding -1.
Thanks to all who helped with the release!

(* = binding)
+1:
- Dongjoon Hyun *
- Kent Yao
- Yang Jie
- Mridul Muralidharan *
- Liang-Chi Hsieh *
- Jia Fan

+0: None

-1:
- Marc Le Bihan

Re: [VOTE] Release Spark 3.4.2 (RC1)

[Dongjoon Hyun]

Thank you all.

This vote passed. I will conclude this vote.

Dongjoon.

On 2023/11/30 09:53:17 Jia Fan wrote:
> +1
> 
> L. C. Hsieh  于2023年11月30日周四 12:33写道：
> 
> > +1
> >
> > Thanks Dongjoon!
> >
> > On Wed, Nov 29, 2023 at 7:53 PM Mridul Muralidharan 
> > wrote:
> > >
> > > +1
> > >
> > > Signatures, digests, etc check out fine.
> > > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes
> > >
> > > Regards,
> > > Mridul
> > >
> > > On Wed, Nov 29, 2023 at 5:08 AM Yang Jie  wrote:
> > >>
> > >> +1(non-binding)
> > >>
> > >> Jie Yang
> > >>
> > >> On 2023/11/29 02:08:04 Kent Yao wrote:
> > >> > +1(non-binding)
> > >> >
> > >> > Kent Yao
> > >> >
> > >> > On 2023/11/27 01:12:53 Dongjoon Hyun wrote:
> > >> > > Hi, Marc.
> > >> > >
> > >> > > Given that it exists in 3.4.0 and 3.4.1, I don't think it's a
> > release
> > >> > > blocker for Apache Spark 3.4.2.
> > >> > >
> > >> > > When the patch is ready, we can consider it for 3.4.3.
> > >> > >
> > >> > > In addition, note that we categorized release-blocker-level issues
> > by
> > >> > > marking 'Blocker' priority with `Target Version` before the vote.
> > >> > >
> > >> > > Best,
> > >> > > Dongjoon.
> > >> > >
> > >> > >
> > >> > > On Sat, Nov 25, 2023 at 12:01 PM Marc Le Bihan <
> > mlebiha...@gmail.com> wrote:
> > >> > >
> > >> > > > -1 If you can wait that the last remaining problem with Generics
> > (?) is
> > >> > > > entirely solved, that causes this exception to be thrown :
> > >> > > >
> > >> > > > java.lang.ClassCastException: class [Ljava.lang.Object; cannot be
> > cast to class [Ljava.lang.reflect.TypeVariable; ([Ljava.lang.Object; and
> > [Ljava.lang.reflect.TypeVariable; are in module java.base of loader
> > 'bootstrap')
> > >> > > > at
> > org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:116)
> > >> > > > at
> > org.apache.spark.sql.catalyst.JavaTypeInference$.$anonfun$encoderFor$1(JavaTypeInference.scala:140)
> > >> > > > at
> > scala.collection.ArrayOps$.map$extension(ArrayOps.scala:929)
> > >> > > > at
> > org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:138)
> > >> > > > at
> > org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:60)
> > >> > > > at
> > org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:53)
> > >> > > > at
> > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
> > >> > > > at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
> > >> > > > at org.apache.spark.sql.Encoders.bean(Encoders.scala)
> > >> > > >
> > >> > > >
> > >> > > > https://issues.apache.org/jira/browse/SPARK-45311
> > >> > > >
> > >> > > > Thanks !
> > >> > > >
> > >> > > > Marc Le Bihan
> > >> > > >
> > >> > > >
> > >> > > > On 25/11/2023 11:48, Dongjoon Hyun wrote:
> > >> > > >
> > >> > > > Please vote on releasing the following candidate as Apache Spark
> > version
> > >> > > > 3.4.2.
> > >> > > >
> > >> > > > The vote is open until November 30th 1AM (PST) and passes if a
> > majority +1
> > >> > > > PMC votes are cast, with a minimum of 3 +1 votes.
> > >> > > >
> > >> > > > [ ] +1 Release this package as Apache Spark 3.4.2
> > >> > > > [ ] -1 Do not release this package because ...
> > >> > > >
> > >> > > > To learn more about Apache Spark, please see
> > https://spark.apache.org/
> > >> > > >
> > >> > > > The tag to be voted on is v3.4.2-rc1 (commit
> > >> > > > 0c0e7d4087c64efca259b4fb656b8be643be5686)
> > >> > > > https://github.com/apache/spark/tree/v3.4.2-rc1
> > >> > > >
> > >> > > > The release files, including signatures, digests, etc. can be
> > found at:
> > >> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.2-rc1-bin/
> > >> > > >
> > >> > > > Signatures used for Spark RCs can be found in this file:
> > >> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> > >> > > >
> > >> > > > The staging repository for this release can be found at:
> > >> > > >
> > https://repository.apache.org/content/repositories/orgapachespark-1450/
> > >> > > >
> > >> > > > The documentation corresponding to this release can be found at:
> > >> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.2-rc1-docs/
> > >> > > >
> > >> > > > The list of bug fixes going into 3.4.2 can be found at the
> > following URL:
> > >> > > > https://issues.apache.org/jira/projects/SPARK/versions/12353368
> > >> > > >
> > >> > > > This release is using the release script of the tag v3.4.2-rc1.
> > >> > > >
> > >> > > > FAQ
> > >> > > >
> > >> > > > =
> > >> > > > How can I help test this release?
> > >> > > > =
> > >> > > >
> > >> > > > If you are a Spark user, you can help us test this release by
> > taking
> > >> > > > an existing Spark workload and running on this release candidate,
> > then
> > >> > > > reporting any regressions.
> > >> > > >
> > >> > > > If you're working in PySpark 

Re: [VOTE] Release Spark 3.4.2 (RC1)

[Jia Fan]

+1

L. C. Hsieh  于2023年11月30日周四 12:33写道：

> +1
>
> Thanks Dongjoon!
>
> On Wed, Nov 29, 2023 at 7:53 PM Mridul Muralidharan 
> wrote:
> >
> > +1
> >
> > Signatures, digests, etc check out fine.
> > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes
> >
> > Regards,
> > Mridul
> >
> > On Wed, Nov 29, 2023 at 5:08 AM Yang Jie  wrote:
> >>
> >> +1(non-binding)
> >>
> >> Jie Yang
> >>
> >> On 2023/11/29 02:08:04 Kent Yao wrote:
> >> > +1(non-binding)
> >> >
> >> > Kent Yao
> >> >
> >> > On 2023/11/27 01:12:53 Dongjoon Hyun wrote:
> >> > > Hi, Marc.
> >> > >
> >> > > Given that it exists in 3.4.0 and 3.4.1, I don't think it's a
> release
> >> > > blocker for Apache Spark 3.4.2.
> >> > >
> >> > > When the patch is ready, we can consider it for 3.4.3.
> >> > >
> >> > > In addition, note that we categorized release-blocker-level issues
> by
> >> > > marking 'Blocker' priority with `Target Version` before the vote.
> >> > >
> >> > > Best,
> >> > > Dongjoon.
> >> > >
> >> > >
> >> > > On Sat, Nov 25, 2023 at 12:01 PM Marc Le Bihan <
> mlebiha...@gmail.com> wrote:
> >> > >
> >> > > > -1 If you can wait that the last remaining problem with Generics
> (?) is
> >> > > > entirely solved, that causes this exception to be thrown :
> >> > > >
> >> > > > java.lang.ClassCastException: class [Ljava.lang.Object; cannot be
> cast to class [Ljava.lang.reflect.TypeVariable; ([Ljava.lang.Object; and
> [Ljava.lang.reflect.TypeVariable; are in module java.base of loader
> 'bootstrap')
> >> > > > at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:116)
> >> > > > at
> org.apache.spark.sql.catalyst.JavaTypeInference$.$anonfun$encoderFor$1(JavaTypeInference.scala:140)
> >> > > > at
> scala.collection.ArrayOps$.map$extension(ArrayOps.scala:929)
> >> > > > at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:138)
> >> > > > at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:60)
> >> > > > at
> org.apache.spark.sql.catalyst.JavaTypeInference$.encoderFor(JavaTypeInference.scala:53)
> >> > > > at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.javaBean(ExpressionEncoder.scala:62)
> >> > > > at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179)
> >> > > > at org.apache.spark.sql.Encoders.bean(Encoders.scala)
> >> > > >
> >> > > >
> >> > > > https://issues.apache.org/jira/browse/SPARK-45311
> >> > > >
> >> > > > Thanks !
> >> > > >
> >> > > > Marc Le Bihan
> >> > > >
> >> > > >
> >> > > > On 25/11/2023 11:48, Dongjoon Hyun wrote:
> >> > > >
> >> > > > Please vote on releasing the following candidate as Apache Spark
> version
> >> > > > 3.4.2.
> >> > > >
> >> > > > The vote is open until November 30th 1AM (PST) and passes if a
> majority +1
> >> > > > PMC votes are cast, with a minimum of 3 +1 votes.
> >> > > >
> >> > > > [ ] +1 Release this package as Apache Spark 3.4.2
> >> > > > [ ] -1 Do not release this package because ...
> >> > > >
> >> > > > To learn more about Apache Spark, please see
> https://spark.apache.org/
> >> > > >
> >> > > > The tag to be voted on is v3.4.2-rc1 (commit
> >> > > > 0c0e7d4087c64efca259b4fb656b8be643be5686)
> >> > > > https://github.com/apache/spark/tree/v3.4.2-rc1
> >> > > >
> >> > > > The release files, including signatures, digests, etc. can be
> found at:
> >> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.2-rc1-bin/
> >> > > >
> >> > > > Signatures used for Spark RCs can be found in this file:
> >> > > > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >> > > >
> >> > > > The staging repository for this release can be found at:
> >> > > >
> https://repository.apache.org/content/repositories/orgapachespark-1450/
> >> > > >
> >> > > > The documentation corresponding to this release can be found at:
> >> > > > https://dist.apache.org/repos/dist/dev/spark/v3.4.2-rc1-docs/
> >> > > >
> >> > > > The list of bug fixes going into 3.4.2 can be found at the
> following URL:
> >> > > > https://issues.apache.org/jira/projects/SPARK/versions/12353368
> >> > > >
> >> > > > This release is using the release script of the tag v3.4.2-rc1.
> >> > > >
> >> > > > FAQ
> >> > > >
> >> > > > =
> >> > > > How can I help test this release?
> >> > > > =
> >> > > >
> >> > > > If you are a Spark user, you can help us test this release by
> taking
> >> > > > an existing Spark workload and running on this release candidate,
> then
> >> > > > reporting any regressions.
> >> > > >
> >> > > > If you're working in PySpark you can set up a virtual env and
> install
> >> > > > the current RC and see if anything important breaks, in the
> Java/Scala
> >> > > > you can add the staging repository to your projects resolvers and
> test
> >> > > > with the RC (make sure to clean up the artifact cache
> before/after so
> >> > > > you don't end up building with a out of date RC going forward).
> >> > > >
> >>