Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Fri, Apr 26, 2024 at 10:01 AM Dongjoon Hyun wrote: > > I'll start with my +1. > > Dongjoon. > > On 2024/04/26 16:45:51 Dongjoon Hyun wrote: > > Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault > > to `false` by default. The technical scope is defined in the

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Thu, Apr 25, 2024 at 8:16 PM Yuming Wang wrote: > +1 > > On Fri, Apr 26, 2024 at 8:25 AM Nimrod Ofek wrote: > >> Of course, I can't think of a scenario of thousands of tables with single >> in memory Spark cluster with in memory catalog. >> Thanks for the help! >> >> בתאריך יום ה׳, 25

Re: [FYI] SPARK-47993: Drop Python 3.8

+1 On Thu, Apr 25, 2024 at 11:19 AM Maciej wrote: > > +1 > > Best regards, > Maciej Szymkiewicz > > Web: https://zero323.net > PGP: A30CEF0C31A501EC > > On 4/25/24 6:21 PM, Reynold Xin wrote: > > +1 > > On Thu, Apr 25, 2024 at 9:01 AM Santosh Pingale > wrote: >> >> +1 >> >> On Thu, Apr 25,

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 On Tue, Apr 16, 2024 at 4:08 AM Wenchen Fan wrote: > > +1 > > On Mon, Apr 15, 2024 at 12:31 PM Dongjoon Hyun wrote: >> >> I'll start with my +1. >> >> - Checked checksum and signature >> - Checked Scala/Java/R/Python/SQL Document's Spark version >> - Checked published Maven artifacts >> -

[VOTE][RESULT] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

Hi all, The vote passes with 7+1s (5 binding +1s). (* = binding) +1: Dongjoon Hyun(*) Liang-Chi Hsieh(*) Huaxin Gao(*) Bo Yang Xiao Li(*) Chao Sun(*) Hussein Awala +0: None -1: None Thanks. - To unsubscribe e-mail:

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Sat, Apr 13, 2024 at 4:12 PM Hyukjin Kwon wrote: > > +1 > > On Sun, Apr 14, 2024 at 7:46 AM Chao Sun wrote: >> >> +1. >> >> This feature is very helpful for guarding against correctness issues, such >> as null results due to invalid input or math overflows. It’s been there for >> a

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

> Dongjoon. > > On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > > Hi all, > > > > Thanks for all discussions in the thread of "Versioning of Spark > > Operator": https://lists.apache.org/thread/zhc7nb2sxm8jjxdppq8qjcmlf4rcsthh > > > &

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

+1 I believe ANSI mode is well developed after many releases. No doubt it could be used. Since it is very easy to disable it to restore to current behavior, I guess the impact could be limited. Do we have known the possible impacts such as what are the major changes (e.g., what kind of

[VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

Hi all, Thanks for all discussions in the thread of "Versioning of Spark Operator": https://lists.apache.org/thread/zhc7nb2sxm8jjxdppq8qjcmlf4rcsthh I would like to create this vote to get the consensus for versioning of the Spark Kubernetes Operator. The proposal is to use an independent

Re: SPIP: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

+1 for Wenchen's point. I don't see a strong reason to pull these transformations into Spark instead of keeping them in third party packages/projects. On Wed, Apr 10, 2024 at 5:32 AM Wenchen Fan wrote: > > It's good to reduce duplication between different native accelerators of > Spark, and

Re: Versioning of Spark Operator

This approach makes sense to me. If Spark K8s operator is aligned with Spark versions, for example, it uses 4.0.0 now. Because these JIRA tickets are not actually targeting Spark 4.0.0, it will cause confusion and more questions, like when we are going to cut Spark release, should we include

Re: Versioning of Spark Operator

there is no release at all and no activity since last 6 months. > > >> It seems to be the first time for Apache Spark community to consider > > >> these sister repositories (Go and K8s Operator). > > >> > > >> https://github.com/apache/spark-connect

Re: Versioning of Spark Operator

024 at 10:09 AM Dongjoon Hyun > > <mailto:dongj...@apache.org>> wrote: > > > > >> Hi, Liang-Chi. > > > > >> > > > > >> Thank you for leading Apache Spark K8s operator as a shepherd. > > > > >> > > >

Versioning of Spark Operator

Hi all, We've opened the dedicated repository of Spark Kubernetes Operator, and the first PR is created. Thank you for the review from the community so far. About the versioning of Spark Operator, there are questions. As we are using Spark JIRA, when we are going to merge PRs, we need to choose

Re: Apache Spark 3.4.3 (?)

+1 Thanks Dongjoon! On Sun, Apr 7, 2024 at 1:56 AM Kent Yao wrote: > > +1, thank you, Dongjoon > > > Kent > > Holden Karau 于2024年4月7日周日 14:54写道： > > > > Sounds good to me :) > > > > Twitter: https://twitter.com/holdenkarau > > Books (Learning Spark, High Performance Spark, etc.): > >

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Thanks Hyukjin. On Sun, Mar 31, 2024 at 10:52 PM Dongjoon Hyun wrote: > > +1 > > Thank you, Hyukjin. > > Dongjoon > > On Sun, Mar 31, 2024 at 19:07 Haejoon Lee > wrote: >> >> +1 >> >> On Mon, Apr 1, 2024 at 10:15 AM Hyukjin Kwon wrote: >>> >>> Hi all, >>> >>> I'd like to start the vote

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

operator in spark project, it would be the best. >>>> 3. The new Spark Operator should continue being spark agnostic and >>>> continue having this lightweight/separate layer of submission worker. >>>> We've seen scalability issues caused by the heavy JVM during spark-subm

The dedicated repository for Kubernetes Operator for Apache Spark

Hi all, For the passed SPIP: An Official Kubernetes Operator for Apache Spark, the developers have been working on code cleaning and refactoring for open source in the last few months. They are ready to contribute the code to Spark now. As we discussed, I will go to create a dedicated repository

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 On Tue, Mar 12, 2024 at 8:20 AM Chao Sun wrote: > +1 > > On Tue, Mar 12, 2024 at 8:03 AM Xiao Li > wrote: > >> +1 >> >> On Tue, Mar 12, 2024 at 6:09 AM Holden Karau >> wrote: >> >>> +1 >>> >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark,

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

+1 On Wed, Jan 10, 2024 at 9:06 AM Bhuwan Sahni wrote: > +1. This is a good addition. > > > *Bhuwan Sahni* > Staff Software Engineer > > bhuwan.sa...@databricks.com > 500 108th Ave. NE > Bellevue, WA 98004 > USA > > > On Wed, Jan 10, 2024 at 9:00 AM Burak Yavuz

Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2

+1 I left some comments in the SPIP doc and got replies quickly. The new API looks good and more comprehensive. I think it will help Spark Structured Streaming to be more useful in more complicated streaming use cases. On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz wrote: > > I'm also a +1 on the

Re: [VOTE] Release Spark 3.3.4 (RC1)

+1 On Sun, Dec 10, 2023 at 6:15 PM Kent Yao wrote: > > +1(non-binding > > Kent Yao > > Yuming Wang 于2023年12月11日周一 09:33写道： > > > > +1 > > > > On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun wrote: > >> > >> +1 > >> > >> Dongjoon > >> > >> On 2023/12/08 21:41:00 Dongjoon Hyun wrote: > >> > Please

Re: Apache Spark 3.3.4 EOL Release?

+1 Thanks Dongjoon! On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > Jie Yang > > On 2023/12/04 15:08:25 Tom Graves wrote: > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > Tom > > On Friday, December 1, 2023 at 02:48:22 PM CST,

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Thanks Dongjoon! On Wed, Nov 29, 2023 at 7:53 PM Mridul Muralidharan wrote: > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: >> >>

[VOTE][RESULT] SPIP: An Official Kubernetes Operator for Apache Spark

Hi all, The vote passes with 19 +1s (11 binding +1s). Thanks to all who reviews the SPIP doc and votes! (* = binding) +1: - Ye Zhou - L. C. Hsieh (*) - Chao Sun (*) - Vakaris Baškirov - DB Tsai (*) - Holden Karau (*) - Lucian Neghina - Mridul Muralidharan (*) - Huaxin Gao (*) - Cheng Pan

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Tue, Nov 14, 2023 at 9:46 AM Ye Zhou wrote: > > +1(Non-binding) > > On Tue, Nov 14, 2023 at 9:42 AM L. C. Hsieh wrote: >> >> Hi all, >> >> I’d like to start a vote for SPIP: An Official Kubernetes Operator for >> Apache Spark. >> >

[VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

Hi all, I’d like to start a vote for SPIP: An Official Kubernetes Operator for Apache Spark. The proposal is to develop an official Java-based Kubernetes operator for Apache Spark to automate the deployment and simplify the lifecycle management and orchestration of Spark applications and Spark

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks for all the support from the community for the SPIP proposal. Since all questions/discussion are settled down (if I didn't miss any major ones), if no more questions or concerns, I'll be the shepherd for this SPIP proposal and call for a vote tomorrow. Thank you all! On Mon, Nov 13, 2023

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 7:57 PM Chao Sun wrote: > > +1 > > > On Thu, Nov 9, 2023 at 6:36 PM Xiao Li wrote: > > > > +1 > > > > huaxin gao 于2023年11月9日周四 16:53写道： > >> > >> +1 > >> > >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >>> > >>> +1 > >>> > >>> To be completely transparent, I am

Re: Apache Spark 3.4.2 (?)

+1 On Tue, Nov 7, 2023 at 4:56 PM Dongjoon Hyun wrote: > > Thank you all! > > Dongjoon > > On Mon, Nov 6, 2023 at 6:03 PM Holden Karau wrote: >> >> +1 >> >> On Mon, Nov 6, 2023 at 4:30 PM yangjie01 wrote: >>> >>> +1 >>> >>> >>> >>> 发件人: Yuming Wang >>> 日期: 2023年11月7日 星期二 07:00 >>> 收件人:

Re: [VOTE] SPIP: State Data Source - Reader

+1 On Mon, Oct 23, 2023 at 6:31 PM Anish Shrigondekar wrote: > > +1 (non-binding) > > Thanks, > Anish > > On Mon, Oct 23, 2023 at 5:01 PM Wenchen Fan wrote: >> >> +1 >> >> On Mon, Oct 23, 2023 at 4:03 PM Jungtaek Lim >> wrote: >>> >>> Starting with my +1 (non-binding). Thanks! >>> >>> On Mon,

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 Thanks Yuming. On Thu, Aug 10, 2023 at 3:24 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On 2023/08/10 07:14:07 yangjie01 wrote: > > +1 > > Thanks, Jie Yang > > > > > > 发件人: Yuming Wang > > 日期: 2023年8月10日 星期四 13:33 > > 收件人: Dongjoon Hyun > > 抄送: dev > > 主题: Re: [VOTE] Release Apache

Re: Welcome two new Apache Spark committers

Congratulations! On Mon, Aug 7, 2023 at 9:44 AM huaxin gao wrote: > > Congratulations! Peter and Xiduo! > > On Mon, Aug 7, 2023 at 9:40 AM Dongjoon Hyun wrote: >> >> Congratulations, Peter and Xiduo. :) >> >> Dongjoon. >> >> On Sun, Aug 6, 2023 at 10:08 PM XiDuo You wrote: >>> >>> Thank you

Re: Time for Spark v3.5.0 release

+1 Thanks Yuanjian. On Tue, Jul 4, 2023 at 7:45 AM yangjie01 wrote: > > +1 > > > > 发件人: Maxim Gekk > 日期: 2023年7月4日 星期二 17:24 > 收件人: Kent Yao > 抄送: "dev@spark.apache.org" > 主题: Re: Time for Spark v3.5.0 release > > > > +1 > > On Tue, Jul 4, 2023 at 11:55 AM Kent Yao wrote: > > +1， thank you

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon! On Fri, Jun 23, 2023 at 7:10 PM Hyukjin Kwon wrote: > > Thanks! > > On Sat, Jun 24, 2023 at 11:01 AM Mridul Muralidharan wrote: >> >> >> Thanks Dongjoon ! >> >> Regards, >> Mridul >> >> On Fri, Jun 23, 2023 at 6:58 PM Dongjoon Hyun wrote: >>> >>> We are happy to announce the

Re: [VOTE][SPIP] PySpark Test Framework

+1 On Thu, Jun 22, 2023 at 3:10 PM Xinrong Meng wrote: > > +1 > > Thanks for driving that! > > On Wed, Jun 21, 2023 at 10:25 PM Ruifeng Zheng wrote: >> >> +1 >> >> On Thu, Jun 22, 2023 at 1:11 PM Dongjoon Hyun >> wrote: >>> >>> +1 >>> >>> Dongjoon >>> >>> On Wed, Jun 21, 2023 at 8:56 PM

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1 On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun wrote: > > +1 > > Dongjoon > > On 2023/06/20 02:51:32 Jia Fan wrote: > > +1 > > > > Dongjoon Hyun 于2023年6月20日周二 10:41写道： > > > > > Please vote on releasing the following candidate as Apache Spark version > > > 3.4.1. > > > > > > The vote is open

Re: [VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

+1 On Mon, Jun 12, 2023 at 11:06 AM huaxin gao wrote: > > +1 > > On Mon, Jun 12, 2023 at 11:05 AM Dongjoon Hyun wrote: >> >> +1 >> >> Dongjoon >> >> On 2023/06/12 18:00:38 Dongjoon Hyun wrote: >> > Please vote on the release plan for Apache Spark 4.0.0. >> > >> > The vote is open until June

Re: Apache Spark 3.4.1 Release?

+1 Thanks Dongjoon for driving this. On Thu, Jun 8, 2023 at 2:25 PM Dongjoon Hyun wrote: > > Hi, All. > > `branch-3.4` already has 77 commits since v3.4.0 tag. > > https://github.com/apache/spark/releases/v3.4.0 (Tagged on April 6th) > > $ git log --oneline v3.4.0..HEAD | wc -l >

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 Thanks Dongjoon On Sun, Apr 9, 2023 at 5:20 PM Dongjoon Hyun wrote: > > I'll start with my +1. > > I verified the checksum, signatures of the artifacts, and documentations. > Also, ran the tests with YARN and K8s modules. > > Dongjoon. > > On 2023/04/09 23:46:10 Dongjoon Hyun wrote: > >

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 Thanks Xinrong. On Sat, Apr 8, 2023 at 8:23 AM yangjie01 wrote: > > +1 > > > > 发件人: Sean Owen > 日期: 2023年4月8日 星期六 20:27 > 收件人: Xinrong Meng > 抄送: dev > 主题: Re: [VOTE] Release Apache Spark 3.4.0 (RC7) > > > > +1 form me, same result as last time. > > > > On Fri, Apr 7, 2023 at 6:30 PM

Re: Apache Spark 3.2.4 EOL Release?

+1 Sounds good and thanks Dongjoon for driving this. On 2023/04/04 17:24:54 Dongjoon Hyun wrote: > Hi, All. > > Since Apache Spark 3.2.0 passed RC7 vote on October 12, 2021, branch-3.2 > has been maintained and served well until now. > > - https://github.com/apache/spark/releases/tag/v3.2.0

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

+1 Thanks Xinrong. On Mon, Apr 3, 2023 at 12:35 PM Dongjoon Hyun wrote: > > +1 > > I also verified that RC5 has SBOM artifacts. > > https://repository.apache.org/content/repositories/orgapachespark-1439/org/apache/spark/spark-core_2.12/3.4.0/spark-core_2.12-3.4.0-cyclonedx.json >

[VOTE][RESULT][SPIP] Lazy Materialization for Parquet Read Performance Improvement

The vote passes with 9 +1s (4 binding +1s). Thanks to all who reviews the SPIP doc and votes! (* = binding) +1: - Dongjoon Hyun (*) - Huaxin Gao (*) - Mich Talebzadeh - L. C. Hsieh (*) - Prem Sahoo - Yuming Wang - Guo Weijie - DB Tsai (*) - Kazuyuki Tanimura +0: None -1: None Thanks

[ANNOUNCE] Apache Spark 3.3.2 released

We are happy to announce the availability of Apache Spark 3.3.2! Spark 3.3.2 is a maintenance release containing stability fixes. This release is based on the branch-3.3 maintenance branch of Spark. We strongly recommend all 3.3 users to upgrade to this stable release. To download Spark 3.3.2,

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

; +1 >> >> Yuming Wang 于2023年2月14日周二 15:58写道： >>> >>> +1 >>> >>> On Tue, Feb 14, 2023 at 11:27 AM Prem Sahoo wrote: >>>> >>>> +1 >>>> >>>> On Mon, Feb 13, 2023 at 8:13 PM L. C.

[VOTE][RESULT] Release Spark 3.3.2 (RC1)

The vote passes with 12 +1s (4 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Mridul Muralidharan (*) - Dongjoon Hyun (*) - Sean Owen (*) - Enrico Minack - Bjørn Jørgensen - Yikun Jiang - Yang Jie - Yuming Wang - John Zhuge - William Hyun - Chao Sun - L. C. Hsieh

Re: [VOTE] Release Spark 3.3.2 (RC1)

ch-3.3. >> >> We need to talk. :) >> >> Bests, >> Dongjoon. >> >> >> On Mon, Feb 13, 2023 at 9:31 AM Chao Sun wrote: >>> >>> +1 >>> >>> On Mon, Feb 13, 2023 at 9:20 AM L. C. Hsieh wrote: >>> > >

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

oss, damage or destruction. > > > > > On Mon, 13 Feb 2023 at 23:18, huaxin gao wrote: > >> +1 >> >> On Mon, Feb 13, 2023 at 3:09 PM Dongjoon Hyun >> wrote: >> >>> +1 >>> >>> Dongjoon >>> >>> On 2023/02/1

[VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

Hi all, I'd like to start the vote for SPIP: Lazy Materialization for Parquet Read Performance Improvement. The high summary of the SPIP is that it proposes an improvement to the Parquet reader with lazy materialization which only materializes (i.e. decompress, de-code, etc...) necessary values.

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

> On Mon, 13 Feb 2023 at 20:41, kazuyuki tanimura > wrote: > >> Thank you Liang-Chi! >> >> Kazu >> >> On Feb 11, 2023, at 7:12 PM, L. C. Hsieh wrote: >> >> Thanks all for your feedback. >> >> Given this positive feedback, if there is no

Re: [VOTE] Release Spark 3.3.2 (RC1)

gjie01 : >>>> >>>> Which Python version do you use for testing? When I use the latest Python >>>> 3.11, I can reproduce similar test failures (43 tests of sql module fail), >>>> but when I use python 3.10, they will succeed >>>> >>&g

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

Thanks all for your feedback. Given this positive feedback, if there is no other comments/discussion, I will go to start a vote in the next few days. Thank you again! On Thu, Feb 2, 2023 at 10:12 AM kazuyuki tanimura wrote: > Thank you all for +1s and reviewing the SPIP doc. > > Kazu > > On

Re: [VOTE] Release Spark 3.3.2 (RC1)

-- > [INFO] BUILD FAILURE > [INFO] > -------- > [INFO] Total time: 02:30 h > [INFO] Finished at: 2023-02-11T17:32:45+01:00 > > lør. 11. feb. 2023 kl. 06:01 skrev L. C. Hsieh : >> >>

Re: [VOTE] Release Spark 3.3.2 (RC1)

ignatures, digests, etc check out fine. > > Built and tested with "-Phive -Pyarn -Pmesos -Pkubernetes". > > Regards, > Mridul > > > > > On Fri, Feb 10, 2023 at 11:01 PM L. C. Hsieh wrote: >> >> Please vote on releasing the following candid

[VOTE] Release Spark 3.3.2 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.3.2. The vote is open until Feb 15th 9AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.2 [ ] -1 Do not release this package because ...

Time for release v3.3.2

Hi Spark devs, As you know, it has been 4 months since Spark 3.3.1 was released on 2022/10, it seems a good time to think about next maintenance release, i.e. Spark 3.3.2. I'm thinking of the release of Spark 3.3.2 this Feb (2023/02). What do you think? I am willing to volunteer for Spark

Re: [DISCUSS] Deprecate DStream in 3.4

+1 On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim wrote: > > Yes, exactly. I'm sorry to bring confusion - should have clarified action > items on the proposal. > > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: >> >> Then, could you elaborate `the proposed code change` specifically? >>

Re: Time for Spark 3.4.0 release?

+1 Thank you! On Wed, Jan 4, 2023 at 9:13 AM Chao Sun wrote: > +1, thanks! > > Chao > > On Wed, Jan 4, 2023 at 1:56 AM Mridul Muralidharan > wrote: > >> >> +1, Thanks ! >> >> Regards, >> Mridul >> >> On Wed, Jan 4, 2023 at 2:20 AM Gengliang Wang wrote: >> >>> +1, thanks for driving the

Re: [ANNOUNCE] Apache Spark 3.2.3 released

Thanks, Chao! On Wed, Nov 30, 2022 at 9:58 AM huaxin gao wrote: > > Thanks Chao for driving the release! > > On Wed, Nov 30, 2022 at 9:24 AM Dongjoon Hyun wrote: >> >> Thank you, Chao! >> >> On Wed, Nov 30, 2022 at 8:16 AM Yang,Jie(INF) wrote: >>> >>> Thanks, Chao! >>> >>> >>> >>> 发件人: Maxim

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 Thanks Chao. On Mon, Nov 14, 2022 at 6:55 PM Dongjoon Hyun wrote: > > +1 > > Thank you, Chao. > > On Mon, Nov 14, 2022 at 4:12 PM Chao Sun wrote: >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.2.3. >> >> The vote is open until 11:59pm Pacific time Nov

Re: [ANNOUNCE] Apache Spark 3.3.1 released

Thank you for driving the release of Apache Spark 3.3.1, Yuming! On Tue, Oct 25, 2022 at 11:38 PM Dongjoon Hyun wrote: > > It's great. Thank you so much, Yuming! > > Dongjoon > > On Tue, Oct 25, 2022 at 11:23 PM Yuming Wang wrote: >> >> We are happy to announce the availability of Apache Spark

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 Thanks Yuming! On Tue, Oct 18, 2022 at 11:28 AM Dongjoon Hyun wrote: > > +1 > > Thank you, Yuming and all! > > Dongjoon. > > > On Tue, Oct 18, 2022 at 9:22 AM Yang,Jie(INF) wrote: >> >> Use maven to test Java 17 + Scala 2.13 and test passed, +1 for me >> >> >> >> 发件人: Sean Owen >> 日期:

Re: Apache Spark 3.2.3 Release?

+1 Thanks Chao! On Tue, Oct 18, 2022 at 11:30 AM Dongjoon Hyun wrote: > > +1 > > Thank you for volunteering, Chao! > > Dongjoon. > > > On Tue, Oct 18, 2022 at 9:55 AM Sean Owen wrote: >> >> OK by me, if someone is willing to drive it. >> >> On Tue, Oct 18, 2022 at 11:47 AM Chao Sun wrote: >>>

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

+1 Thanks Dongjoon. On Wed, Oct 5, 2022 at 3:11 PM Jungtaek Lim wrote: > > +1 > > On Thu, Oct 6, 2022 at 5:59 AM Chao Sun wrote: >> >> +1 >> >> > and specifically may allow us to finally move off of the ancient version >> > of Guava (?) >> >> I think the Guava issue comes from Hive 2.3

Re: Time for Spark 3.3.1 release?

+1 Thanks Yuming! On Mon, Sep 12, 2022 at 11:50 AM Dongjoon Hyun wrote: > > +1 > > Thanks, > Dongjoon. > > On Mon, Sep 12, 2022 at 6:38 AM Yuming Wang wrote: >> >> Hi, All. >> >> >> >> Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including 7 >> correctness patches arrived

Re: Welcoming three new PMC members

Congrats! On Tue, Aug 9, 2022 at 5:38 PM Chao Sun wrote: > > Congrats everyone! > > On Tue, Aug 9, 2022 at 5:36 PM Dongjoon Hyun wrote: > > > > Congrat to all! > > > > Dongjoon. > > > > On Tue, Aug 9, 2022 at 5:13 PM Takuya UESHIN wrote: > > > > > > Congratulations! > > > > > > On Tue, Aug 9,

Re: Update Spark 3.4 Release Window?

I'm also +1 for Feb. 2023 (RC) and Jan. 2023 (Code freeze). Liang-Chi On Wed, Jul 20, 2022 at 2:02 PM Dongjoon Hyun wrote: > > I fixed typos :) > > +1 for February 2023 (Release Candidate) and January 2023 (Code freeze). > > On 2022/07/20 20:59:30 Dongjoon Hyun wrote: > > Thank you for

Re: [VOTE] Release Spark 3.2.2 (RC1)

+1 On Mon, Jul 11, 2022 at 4:50 PM Hyukjin Kwon wrote: > > +1 > > On Tue, 12 Jul 2022 at 06:58, Dongjoon Hyun wrote: >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.2.2. >> >> The vote is open until July 15th 1AM (PST) and passes if a majority +1 PMC >>

Re: [VOTE][SPIP] Spark Connect

+1 On Mon, Jun 13, 2022 at 5:41 PM Chao Sun wrote: > > +1 (non-binding) > > On Mon, Jun 13, 2022 at 5:11 PM Hyukjin Kwon wrote: >> >> +1 >> >> On Tue, 14 Jun 2022 at 08:50, Yuming Wang wrote: >>> >>> +1. >>> >>> On Tue, Jun 14, 2022 at 2:20 AM Matei Zaharia >>> wrote: +1, very

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 On Mon, Jun 13, 2022 at 5:07 PM Holden Karau wrote: > > +1 > > On Mon, Jun 13, 2022 at 4:51 PM Yuming Wang wrote: >> >> +1 (non-binding) >> >> On Tue, Jun 14, 2022 at 7:41 AM Dongjoon Hyun >> wrote: >>> >>> +1 >>> >>> Thanks, >>> Dongjoon. >>> >>> On Mon, Jun 13, 2022 at 3:54 PM Chris

Re: [VOTE] Release Spark 3.3.0 (RC5)

+1 Liang-Chi On Tue, Jun 7, 2022 at 1:03 PM Gengliang Wang wrote: > > +1 (non-binding) > > Gengliang > > On Tue, Jun 7, 2022 at 12:24 PM Thomas Graves wrote: >> >> +1 >> >> Tom Graves >> >> On Sat, Jun 4, 2022 at 9:50 AM Maxim Gekk >> wrote: >> > >> > Please vote on releasing the following

Re: [VOTE] Release Spark 3.3.0 (RC4)

It's fixed at https://github.com/apache/spark/pull/36762. On Fri, Jun 3, 2022 at 2:20 PM Sean Owen wrote: > > Ah yeah, I think it's this change from 15 hrs ago. That needs to be .toSeq: > >

Re: Introducing "Pandas API on Spark" component in JIRA, and use "PS" PR title component

+1. Thanks Hyukjin. On Thu, May 19, 2022 at 10:14 AM Bryan Cutler wrote: > > +1, sounds good > > On Wed, May 18, 2022 at 9:16 PM Dongjoon Hyun wrote: >> >> +1 >> >> Thank you for the suggestion, Hyukjin. >> >> Dongjoon. >> >> On Wed, May 18, 2022 at 11:08 AM Bjørn Jørgensen >> wrote: >>> >>>

Re: SIGMOD System Award for Apache Spark

This is awesome! Great congrats to everyone in the Spark community! On Fri, May 13, 2022 at 10:57 AM Manolis Gemeliaris < gemeliarismano...@gmail.com> wrote: > Congratulations everyone ! > > Στις Παρ 13 Μαΐ 2022 στις 8:06 μ.μ., ο/η Xingbo Jiang < > jiangxb1...@gmail.com> έγραψε: > >>

Re: [VOTE] SPIP: Catalog API for view metadata

+1 On Thu, Feb 3, 2022 at 7:25 PM Chao Sun wrote: > > +1 (non-binding). Looking forward to this feature! > > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue wrote: >> >> +1 for the SPIP. I think it's well designed and it has worked quite well at >> Netflix for a long time. >> >> On Thu, Feb 3, 2022

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thanks Huaxin for the 3.2.1 release! On Fri, Jan 28, 2022 at 10:14 PM Dongjoon Hyun wrote: > > Thank you again, Huaxin! > > Dongjoon. > > On Fri, Jan 28, 2022 at 6:23 PM DB Tsai wrote: >> >> Thank you, Huaxin for the 3.2.1 release! >> >> Sent from my iPhone >> >> On Jan 28, 2022, at 5:45 PM,

Re: [Apache Spark Jenkins] build system shutting down Dec 23th, 2021

Thank you, Shane. On Mon, Dec 6, 2021 at 4:27 PM Holden Karau wrote: > > Shane you kick ass thank you for everything you’ve done for us :) Keep on > rocking :) > > On Mon, Dec 6, 2021 at 4:24 PM Hyukjin Kwon wrote: >> >> Thanks, Shane. >> >> On Tue, 7 Dec 2021 at 09:19, Dongjoon Hyun wrote:

Re: [VOTE][RESULT] SPIP: Row-level operations in Data Source V2

* = binding On Tue, Nov 16, 2021 at 9:37 AM L. C. Hsieh wrote: > > Hi all, > > The vote passed with the following 12 +1 votes and no -1 or +0 votes: > > Liang-Chi Hsieh* > Anton Okolnychyi > DB Tsai* > Huaxin Gao > Dongjoon Hyun* > Russell Spitzer > Mich Talebzadeh

[VOTE][RESULT] SPIP: Row-level operations in Data Source V2

Hi all, The vote passed with the following 12 +1 votes and no -1 or +0 votes: Liang-Chi Hsieh* Anton Okolnychyi DB Tsai* Huaxin Gao Dongjoon Hyun* Russell Spitzer Mich Talebzadeh Ryan Blue Chao Sun John Zhuge Wenchen Fan* Gengliang Wang * = binding Thank you guys all for your feedback and

[VOTE] SPIP: Row-level operations in Data Source V2

Hi all, I’d like to start a vote for SPIP: Row-level operations in Data Source V2. The proposal is to add support for executing row-level operations such as DELETE, UPDATE, MERGE for v2 tables (SPARK-35801). The execution should be the same across data sources and the best way to do that is to

Re: [DISCUSS] SPIP: Row-level operations in Data Source V2

Hi all, I think mostly we are in favor for the SPIP as I've seen. If not more comments or discussion on the SPIP doc, I will raise a vote soon. Thanks. On Tue, Nov 2, 2021 at 9:58 AM L. C. Hsieh wrote: > > +1 for the idea to commit the work earlier. > > I think we will raise the

Re: [DISCUSS] SPIP: Row-level operations in Data Source V2

> > On Thu, Oct 28, 2021 at 12:53 AM L. C. Hsieh wrote: >> >> >> Thanks for the initial feedback. >> >> I think previously the community is busy on the works related to Spark 3.2 >> release. >> As 3.2 release was done, I'd like to bring this up

Re: [VOTE] SPIP: Storage Partitioned Join for Data Source V2

I'll start with my +1. On 2021/10/29 17:30:03, L. C. Hsieh wrote: > Hi all, > > I’d like to start a vote for SPIP: Storage Partitioned Join for Data Source > V2. > > The proposal is to support a new type of join: storage partitioned join which > covers bucket join sup

[VOTE] SPIP: Storage Partitioned Join for Data Source V2

Hi all, I’d like to start a vote for SPIP: Storage Partitioned Join for Data Source V2. The proposal is to support a new type of join: storage partitioned join which covers bucket join support for DataSourceV2 but is more general. The goal is to let Spark leverage distribution properties

Re: [DISCUSS] SPIP: Storage Partitioned Join for Data Source V2

Thanks all for your inputs here! Seems the discussion already settles, I will be the shepherd for the SPIP and call for a vote on the SPIP moving forward in a new thread. On 2021/10/28 13:05:53, Wenchen Fan wrote: > Thanks for the explanation! It makes sense to always resolve the logical >

Re: [DISCUSS] SPIP: Row-level operations in Data Source V2

Thu, Jun 24, 2021 at 9:42 PM L. C. Hsieh wrote: > > > Thanks Anton. I'm voluntarily to be the shepherd of the SPIP. This is also > > my first time to shepherd a SPIP, so please let me know if anything I can > > improve. > > > > This looks great features and th

Re: [DISCUSS] SPIP: Storage Partitioned Join for Data Source V2

+1 for the SPIP. This is a great improvement and optimization! On 2021/10/26 19:01:03, Erik Krogen wrote: > It's great to see this SPIP going live. Once this is complete, it will > really help Spark to play nicely with a broader data ecosystem (Hive, > Iceberg, Trino, etc.), and it's great to

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 Looks good. Liang-Chi On 2021/10/08 16:16:12, Kent Yao wrote: > +1 (non-binding) BR > > > > > > > font{ > line-height: 1.6; > } > > > > font{ > line-height: 1.6; > } > > > > font{ >

Re: [ANNOUNCE] Apache Spark 3.0.3 released

Thanks Yi for the work! On 2021/06/25 05:51:38, Yi Wu wrote: > We are happy to announce the availability of Spark 3.0.3! > > Spark 3.0.3 is a maintenance release containing stability fixes. This > release is based on the branch-3.0 maintenance branch of Spark. We strongly > recommend all 3.0

Re: [DISCUSS] SPIP: Row-level operations in Data Source V2

Thanks Anton. I'm voluntarily to be the shepherd of the SPIP. This is also my first time to shepherd a SPIP, so please let me know if anything I can improve. This looks great features and the rationale claimed by the proposal makes sense. These operations are getting more common and more