Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Tue, Apr 30, 2024 at 3:31 PM Ye Xianjin wrote: > +1 > Sent from my iPhone > > On Apr 30, 2024, at 3:23 PM, DB Tsai wrote: > >  > +1 > > On Apr 29, 2024, at 8:01 PM, Wenchen Fan wrote: > >  > To add more color: > > Spark data source table and Hive Serde table are both stored in the

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Fri, Apr 26, 2024 at 8:25 AM Nimrod Ofek wrote: > Of course, I can't think of a scenario of thousands of tables with single > in memory Spark cluster with in memory catalog. > Thanks for the help! > > בתאריך יום ה׳, 25 באפר׳ 2024, 23:56, מאת Mich Talebzadeh ‏< >

Re: Please unlock Jira ticket for SPARK-24815, Dynamic resource allocation for structured streaming

Unlocked. On Tue, Feb 27, 2024 at 11:47 AM Mich Talebzadeh wrote: > > Hi, > > Can a committer please unlock this SPIP? It is for Dynamic resource > allocation for structured streaming that has got 6 votes. it was locked > because of inactivity by GitHub actions > > [SPARK-24815] Structured

Re: [VOTE] Release Spark 3.3.4 (RC1)

+1 On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun wrote: > +1 > > Dongjoon > > On 2023/12/08 21:41:00 Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > > 3.3.4. > > > > The vote is open until December 15th 1AM (PST) and passes if a majority > +1

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Wed, Nov 15, 2023 at 2:44 AM Holden Karau wrote: > +1 > > On Tue, Nov 14, 2023 at 10:21 AM DB Tsai wrote: > >> +1 >> >> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >> >> On Nov 14, 2023, at 10:14 AM, Vakaris Baškirov < >> vakaris.bashki...@gmail.com> wrote: >> >> +1

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Fri, Nov 10, 2023 at 10:01 AM Ilan Filonenko wrote: > +1 > > On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: >> >>> +1 for creating an official Kubernetes operator for Apache Spark >>> >>> On Fri, Nov 10, 2023 at 12:38 AM

Re: Apache Spark 3.4.2 (?)

+1 On Tue, Nov 7, 2023 at 3:55 AM Santosh Pingale wrote: > Makes sense given the nature of those commits. > > On Mon, Nov 6, 2023, 7:52 PM Dongjoon Hyun > wrote: > >> Hi, All. >> >> Apache Spark 3.4.1 tag was created on Jun 19th and `branch-3.4` has 103 >> commits including important security

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1. On Tue, Sep 12, 2023 at 10:57 AM yangjie01 wrote: > +1 > > > > *发件人**: *Jia Fan > *日期**: *2023年9月12日 星期二 10:08 > *收件人**: *Ruifeng Zheng > *抄送**: *Hyukjin Kwon , Xiao Li , > Mridul Muralidharan , Peter Toth , > Spark dev list , Yuanjian Li > > *主题**: *Re: [VOTE] Release Apache Spark 3.5.0

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1. On Thu, Sep 7, 2023 at 10:33 PM yangjie01 wrote: > +1 > > > > *发件人**: *Gengliang Wang > *日期**: *2023年9月7日 星期四 12:53 > *收件人**: *Yuanjian Li > *抄送**: *Xiao Li , "her...@databricks.com.invalid" > , Spark dev list > *主题**: *Re: [VOTE] Release Apache Spark 3.5.0 (RC4) > > > > +1 > > > > On

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

It seems can not check signature: yumwang@G9L07H60PK Downloads % gpg --keyserver hkps://keys.openpgp.org --recv-key FC3AE3A7EAA1BAC98770840E7E1ABCC53AAA2216 gpg: key 7E1ABCC53AAA2216: no user ID gpg: Total number processed: 1 yumwang@G9L07H60PK Downloads % gpg --batch --verify

[ANNOUNCE] Apache Spark 3.3.3 released

We are happy to announce the availability of Apache Spark 3.3.3! Spark 3.3.3 is a maintenance release containing stability fixes. This release is based on the branch-3.3 maintenance branch of Spark. We strongly recommend all 3.3 users to upgrade to this stable release. To download Spark 3.3.3,

[VOTE][RESULT] Release Spark 3.3.3 (RC1)

The vote passes with 7 +1s (4 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Yuming Wang * - Jie Yang - Dongjoon Hyun * - Liang-Chi Hsieh * - Cheng Pan - Mridul Muralidharan * - Jia Fan +0: None -1: None

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

e: >> https://github.com/apache/spark/pull/42167. The reason for the failure >> is that the branch daily test and the master use the same yml file. >> >> >> >> Jie Yang >> >> >> >> *发件人**: *Dongjoon Hyun >> *日期**: *2023年8月8日 星期二 00

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

-1. I found a NoClassDefFoundError bug: https://issues.apache.org/jira/browse/SPARK-44719. On Mon, Aug 7, 2023 at 11:24 AM yangjie01 wrote: > > > I submitted a PR last week to try and solve this issue: > https://github.com/apache/spark/pull/42236. > > > > *发件人**: *Sean Owen > *日期**: *2023年8月7日

Re: Welcome two new Apache Spark committers

Congratulations! On Mon, Aug 7, 2023 at 11:11 AM Kent Yao wrote: > Congrats! Peter and Xiduo! > > Cheng Pan 于2023年8月7日周一 11:01写道： > > > > Congratulations! Peter and Xiduo! > > > > Thanks, > > Cheng Pan > > > > > > > On Aug 7, 2023, at 10:58, Gengliang Wang wrote: > > > > > > Congratulations!

[VOTE] Release Apache Spark 3.3.3 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.3.3. The vote is open until 11:59pm Pacific time August 10th and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.3 [ ] -1 Do not release this

Re: Time for Spark 3.3.3 release?

Thank you. I will prepare 3.3.3-rc1 soon. On Sun, Jul 30, 2023 at 12:15 AM Dongjoon Hyun wrote: > +1 > > Thank you for volunteering, Yuming. > > Dongjoon > > > On Fri, Jul 28, 2023 at 11:35 AM Yuming Wang wrote: > >> Hi Spark devs, >> >> Since

Time for Spark 3.3.3 release?

Hi Spark devs, Since Apache Spark 3.3.2 tag creation (Feb 11), 60 patches have arrived at branch-3.3. Shall we make a new release, Apache Spark 3.3.3, as the third release at branch-3.3? I'd like to volunteer as the release manager

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1. On Thu, Jun 22, 2023 at 4:41 PM Jacek Laskowski wrote: > +1 > > Builds and runs fine on Java 17, macOS. > > $ ./dev/change-scala-version.sh 2.13 > $ mvn \ > -Pkubernetes,hadoop-cloud,hive,hive-thriftserver,scala-2.13,volcano,connect > \ > -DskipTests \ > clean install > > $ python/run-tests

Re: Apache Spark 4.0.0 Dev Item Planning (SPARK-44111)

Thank you Dongjoon. I'd like to add these items. *Support for more SQL syntax* SPARK-31561 Add QUALIFY clause SPARK-24497 Support recursive SQL SPARK-32064

Re: Apache Spark 3.4.1 Release?

+1. On Fri, Jun 9, 2023 at 7:14 AM Chao Sun wrote: > +1 too > > On Thu, Jun 8, 2023 at 2:34 PM kazuyuki tanimura > wrote: > > > > +1 (non-binding), Thank you Dongjoon > > > > Kazu > > > > - > To unsubscribe e-mail:

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1. On Tue, Apr 11, 2023 at 12:17 AM Mridul Muralidharan wrote: > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Phive -Pyarn -Pmesos -Pkubernetes > > Regards, > Mridul > > > On Mon, Apr 10, 2023 at 10:34 AM huaxin gao > wrote: > >> +1 >> >> On Mon,

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1. On Tue, Apr 11, 2023 at 9:14 AM Yikun Jiang wrote: > +1 (non-binding) > > Also ran the docker image related test (signatures/standalone/k8s) with > rc7: https://github.com/apache/spark-docker/pull/32 > > Regards, > Yikun > > > On Tue, Apr 11, 2023 at 4:44 AM Jacek Laskowski wrote: > >> +1

sbt build is broken because repo is not available

Hi all, sbt build is broken because repo is not available. Please see: https://github.com/sbt/sbt/issues/7202.

Re: Apache Spark 3.2.4 EOL Release?

+1 On Wed, Apr 5, 2023 at 9:09 AM Xinrong Meng wrote: > +1 > > Hyukjin Kwon 于2023年4月4日 周二下午5:31写道： > >> +1 >> >> On Wed, 5 Apr 2023 at 07:31, Mridul Muralidharan >> wrote: >> >>> >>> +1 >>> Sounds good to me. >>> >>> Thanks, >>> Mridul >>> >>> >>> On Tue, Apr 4, 2023 at 1:39 PM huaxin gao >>>

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

+1 On Tue, Apr 4, 2023 at 3:46 AM L. C. Hsieh wrote: > +1 > > Thanks Xinrong. > > On Mon, Apr 3, 2023 at 12:35 PM Dongjoon Hyun > wrote: > > > > +1 > > > > I also verified that RC5 has SBOM artifacts. > > > > >

Re: please help why there is big difference in partitionFilter in spark2.4.2 and spark3.1.3.

Hi Liyun, This is because of this change: https://issues.apache.org/jira/browse/SPARK-27638. You can set spark.sql.legacy.typeCoercion.datetimeToString.enabled to true to restore the old behavior. On Mon, Mar 6, 2023 at 10:27 AM zhangliyun wrote: > Hi all > > > i have a spark sql , before

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

+1 On Tue, Feb 14, 2023 at 11:27 AM Prem Sahoo wrote: > +1 > > On Mon, Feb 13, 2023 at 8:13 PM L. C. Hsieh wrote: > >> +1 >> >> On Mon, Feb 13, 2023 at 3:49 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> +1 for me >>> >>> >>> >>>view my Linkedin profile >>>

Re: [VOTE] Release Spark 3.3.2 (RC1)

+1. On Mon, Feb 13, 2023 at 11:52 AM yangjie01 wrote: > +1, Test 3.3.2-rc1 with Java 17 + Scala 2.13 + Python 3.10, all test > passed. > > > > Yang Jie > > > > *发件人**: *Yikun Jiang > *日期**: *2023年2月13日 星期一 11:47 > *收件人**: *Spark dev list > *抄送**: *"L. C. Hsieh" > *主题**: *Re: [VOTE] Release

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

+1. On Wed, Feb 1, 2023 at 7:42 AM kazuyuki tanimura wrote: > Great! Much appreciated, Mitch! > > Kazu > > On Jan 31, 2023, at 3:07 PM, Mich Talebzadeh > wrote: > > Thanks, Kazu. > > I followed that template link and indeed as you pointed out it is a common > template. If it works then it is

Re: Time for release v3.3.2

+1 On Tue, Jan 31, 2023 at 12:18 PM yangjie01 wrote: > +1 Thanks Liang-Chi! > > > > YangJie > > > > *发件人**: *huaxin gao > *日期**: *2023年1月31日 星期二 10:03 > *收件人**: *Dongjoon Hyun > *抄送**: *Hyukjin Kwon , Chao Sun , > "L. C. Hsieh" , Spark dev list > *主题**: *Re: Time for release v3.3.2 > > > >

Re: [VOTE] Release Spark 3.2.3 (RC1)

+1 On Wed, Nov 16, 2022 at 2:28 PM Yang,Jie(INF) wrote: > I switched Scala 2.13 to Scala 2.12 today. The test is still in progress > and it has not been hung. > > > > Yang Jie > > > > *发件人**: *Dongjoon Hyun > *日期**: *2022年11月16日 星期三 01:17 > *收件人**: *"Yang,Jie(INF)" > *抄送**: *huaxin gao , "L.

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

+1, non-binding On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) wrote: > +1, non-binding > > > > Yang Jie > > > > *发件人**: *Mridul Muralidharan > *日期**: *2022年11月16日 星期三 17:35 > *收件人**: *Kent Yao > *抄送**: *Gengliang Wang , dev > *主题**: *Re: [VOTE][SPIP] Better Spark UI scalability and Driver

[ANNOUNCE] Apache Spark 3.3.1 released

We are happy to announce the availability of Apache Spark 3.3.1! Spark 3.3.1 is a maintenance release containing stability fixes. This release is based on the branch-3.3 maintenance branch of Spark. We strongly recommend all 3.3 users to upgrade to this stable release. To download Spark 3.3.1,

[VOTE][RESULT] Release Spark 3.3.1 (RC4)

The vote passes with 11 +1s (6 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Sean Owen (*) - Yang,Jie - Dongjoon Hyun (*) - L. C. Hsieh (*) - Gengliang Wang (*) - Thomas graves (*) - Chao Sun - Wenchen Fan (*) - Yikun Jiang - Cheng Pan - Yuming Wang +0: None -1

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 for me. Passed Delta unit tests <https://github.com/delta-io/delta/pull/1382>, Iceberg unit tests <https://github.com/apache/iceberg/pull/5783> and Hudi unit tests <https://github.com/apache/hudi/pull/6707>. On Sat, Oct 22, 2022 at 8:30 PM Yuming Wang wrote: > > @Mrid

Re: [VOTE] Release Spark 3.3.1 (RC4)

Cheng Pan >>>> >>>> On Wed, Oct 19, 2022 at 9:13 AM Yikun Jiang >>>> wrote: >>>> > >>>> > +1, also test passed with spark-docker workflow (downloading rc4 tgz, >>>> extract, build image, run K8s IT) >>>> > >>>> &

Re: Apache Spark 3.2.3 Release?

+1 On Wed, Oct 19, 2022 at 4:17 AM kazuyuki tanimura wrote: > +1 Thanks Chao! > > > Kazu > > On Oct 18, 2022, at 11:48 AM, Gengliang Wang wrote: > > +1. Thanks Chao! > > On Tue, Oct 18, 2022 at 11:45 AM huaxin gao > wrote: > >> +1 Thanks Chao! >> >> Huaxin >> >> On Tue, Oct 18, 2022 at 11:29

[VOTE] Release Spark 3.3.1 (RC4)

Please vote on releasing the following candidate as Apache Spark version 3.3.1. The vote is open until 11:59pm Pacific time October 21th and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.1 [ ] -1 Do not release this

Re: Welcome Yikun Jiang as a Spark committer

Congratulations Yikun! On Sat, Oct 8, 2022 at 12:40 PM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Yikun Jiang as a committer on the project. > Yikun is the major contributor of the infrastructure and GitHub Actions in > Apache Spark as well as Kubernates and PySpark. > He

Re: [VOTE] Release Spark 3.3.1 (RC2)

d my local > .m2 recently) > > Other than this, rest looks good to me. > > Regards, > Mridul > > > On Wed, Sep 28, 2022 at 2:56 PM Sean Owen wrote: > >> +1 from me, same result as last RC. >> >> On Wed, Sep 28, 2022 at 12:21 AM Yuming Wang wrote: >&

[VOTE] Release Spark 3.3.1 (RC2)

Please vote on releasing the following candidate as Apache Spark version 3.3.1. The vote is open until 11:59pm Pacific time October 3th and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.1 [ ] -1 Do not release this

Re: [VOTE] Release Spark 3.3.1 (RC1)

ested Scala 2.13 + Java 11 on Ubuntu 22.04. I get the same >> results as usual. >> > >> > On Sat, Sep 17, 2022 at 2:42 AM Yuming Wang wrote: >> >> >> >> Please vote on releasing the following candidate as Apache Spark >> version 3.3.1.

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

+1. On Mon, Sep 19, 2022 at 9:44 AM Kent Yao wrote: > +1 > > Gengliang Wang 于2022年9月19日周一 09:23写道： > > > > +1, thanks for the work! > > > > On Sun, Sep 18, 2022 at 6:20 PM Hyukjin Kwon > wrote: > >> > >> +1 > >> > >> On Mon, 19 Sept 2022 at 09:15, Yikun Jiang wrote: > >>> > >>> Hi, all > >>>

[VOTE] Release Spark 3.3.1 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.3.1. The vote is open until 11:59pm Pacific time September 22th and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.1 [ ] -1 Do not release this

Re: Time for Spark 3.3.1 release?

n Kwon >> *日期**: *2022年9月13日 星期二 08:19 >> *收件人**: *Gengliang Wang >> *抄送**: *"L. C. Hsieh" , Dongjoon Hyun < >> dongjoon.h...@gmail.com>, Yuming Wang , dev < >> dev@spark.apache.org> >> *主题**: *Re: Time for Spark 3.3.1 release? >> &

Time for Spark 3.3.1 release?

Hi, All. Since Apache Spark 3.3.0 tag creation (Jun 10), new 138 patches including 7 correctness patches arrived at branch-3.3. Shall we make a new release, Apache Spark 3.3.1, as the second release at branch-3.3? I'd like to volunteer as the release manager for Apache Spark 3. 3.1. All

Re: Welcoming three new PMC members

Congratulations! On Wed, Aug 10, 2022 at 4:35 PM Yikun Jiang wrote: > Congratulations! > > Regards, > Yikun > > > On Wed, Aug 10, 2022 at 3:19 PM Maciej wrote: > >> Congratulations! >> >> On 8/10/22 08:14, Yi Wu wrote: >> > Congrats everyone! >> > >> > >> > >> > On Wed, Aug 10, 2022 at 11:33

Re: Apache Spark 3.2.2 Release?

+1 On Thu, Jul 7, 2022 at 5:53 AM Maxim Gekk wrote: > +1 > > On Thu, Jul 7, 2022 at 12:26 AM John Zhuge wrote: > >> +1 Thanks for the effort! >> >> On Wed, Jul 6, 2022 at 2:23 PM Bjørn Jørgensen >> wrote: >> >>> +1 >>> >>> ons. 6. jul. 2022, 23:05 skrev Hyukjin Kwon : >>> Yeah +1

Re: [PSA] Please rebase and sync your master branch in your forked repository

Thank you Hyukjin. On Tue, Jun 21, 2022 at 7:46 AM Hyukjin Kwon wrote: > After https://github.com/apache/spark/pull/36922 gets merged, it requires > your fork's master branch to be synced to the latest master branch in > Apache Spark. Otherwise, builds would not be triggered in your PR. > >

Re: [ANNOUNCE] Apache Spark 3.3.0 released

Congrats and thanks. On Fri, Jun 17, 2022 at 7:46 PM Maxim Gekk wrote: > We are happy to announce the availability of Spark 3.3.0! > > Apache Spark 3.3.0 is the fourth release of the 3.x line. With tremendous > contribution from the open-source community, this release managed to > resolve in

Re: [VOTE][SPIP] Spark Connect

+1. On Tue, Jun 14, 2022 at 2:20 AM Matei Zaharia wrote: > +1, very excited about this direction. > > Matei > > On Jun 13, 2022, at 11:07 AM, Herman van Hovell < > her...@databricks.com.INVALID> wrote: > > Let me kick off the voting... > > +1 > > On Mon, Jun 13, 2022 at 2:02 PM Herman van

Re: [VOTE] Release Spark 3.3.0 (RC6)

+1 (non-binding) On Tue, Jun 14, 2022 at 7:41 AM Dongjoon Hyun wrote: > +1 > > Thanks, > Dongjoon. > > On Mon, Jun 13, 2022 at 3:54 PM Chris Nauroth wrote: > >> +1 (non-binding) >> >> I repeated all checks I described for RC5: >> >>

Re: [VOTE] Release Spark 3.3.0 (RC5)

+1. Ran and verified results through internal tests. On Mon, Jun 6, 2022 at 3:43 PM Dongjoon Hyun wrote: > +1. > > I double-checked the following additionally. > > - Run unit tests on Apple Silicon with Java 17/Python 3.9.11/R 4.1.2 > - Run unit tests on Linux with Java11/Scala 2.12/2.13 > -

Re: [VOTE] Release Spark 3.3.0 (RC2)

-1. There is a regression: https://github.com/apache/spark/pull/36595 On Wed, May 18, 2022 at 4:11 PM Martin Grigorov wrote: > Hi, > > [X] +1 Release this package as Apache Spark 3.3.0 > > Tested: > - make local distribution from sources (with ./dev/make-distribution.sh > --tgz --name

Re: [VOTE] Spark 3.1.3 RC4

+1 (non-binding). On Tue, Feb 15, 2022 at 10:22 AM Ruifeng Zheng wrote: > +1 (non-binding) > > checked the release script issue Dongjoon mentioned: > > curl -s > https://dist.apache.org/repos/dist/dev/spark/v3.1.3-rc4-bin/spark-3.1.3-bin-hadoop2.7.tgz > | tar tz | grep hadoop-common >

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thank you Huaxin. On Sat, Jan 29, 2022 at 9:08 AM huaxin gao wrote: > We are happy to announce the availability of Spark 3.2.1! > > Spark 3.2.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.2 maintenance branch of Spark. We strongly > recommend all

Re: [VOTE] Release Spark 3.2.1 (RC2)

+1 (non-binding) On Tue, Jan 25, 2022 at 12:44 PM Wenchen Fan wrote: > +1 > > On Tue, Jan 25, 2022 at 10:13 AM Ruifeng Zheng > wrote: > >> +1 (non-binding) >> >> >> -- 原始邮件 -- >> *发件人:* "Kent Yao" ; >> *发送时间:* 2022年1月25日(星期二) 上午10:09 >> *收件人:* "John Zhuge"; >>

Re: [FYI] Build and run tests on Java 17 for Apache Spark 3.3

Cool, thank you Dongjoon. On Sat, Nov 13, 2021 at 4:09 AM shane knapp ☠ wrote: > woot! nice work everyone! :) > > On Fri, Nov 12, 2021 at 11:37 AM Dongjoon Hyun > wrote: > >> Hi, All. >> >> Apache Spark community has been working on Java 17 support under the >> following JIRA. >> >>

Re: [ANNOUNCE] Apache Spark 3.2.0

Congrats and thanks! On Tue, Oct 19, 2021 at 10:17 PM Gengliang Wang wrote: > Hi all, > > Apache Spark 3.2.0 is the third release of the 3.x line. With tremendous > contribution from the open-source community, this release managed to > resolve in excess of 1,700 Jira tickets. > > We'd like to

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 (non-binding). On Fri, Oct 8, 2021 at 1:02 PM Dongjoon Hyun wrote: > +1 for Apache Spark 3.2.0 RC7. > > It looks good to me. I tested with EKS 1.21 additionally. > > Cheers, > Dongjoon. > > > On Thu, Oct 7, 2021 at 7:46 PM 郑瑞峰 wrote: > >> +1 (non-binding) >> >> >> -- 原始邮件

Re: [VOTE] Release Spark 3.2.0 (RC6)

+1 (non-bindng). On Wed, Sep 29, 2021 at 12:03 AM Michael Heuer wrote: > +1 (non-bindng) > > Works for us, as with previous RCs. > >michael > > > On Sep 28, 2021, at 10:45 AM, Gengliang Wang wrote: > > Starting with my +1(non-binding) > > Thanks, > Gengliang > > On Tue, Sep 28, 2021 at

Re: [VOTE] Release Spark 3.0.3 (RC1)

+1 Tested a batch of production query with Thrift Server. On Sat, Jun 19, 2021 at 3:04 PM Mridul Muralidharan wrote: > > +1 > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos > -Pkubernetes > > Regards, > Mridul > > PS: Might be

Re: [VOTE] Release Spark 3.1.2 (RC1)

+1 (non-binding) On Wed, May 26, 2021 at 11:27 PM Maxim Gekk wrote: > +1 (non-binding) > > On Mon, May 24, 2021 at 9:14 AM Dongjoon Hyun > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 3.1.2. >> >> The vote is open until May 27th 1AM (PST) and passes

Re: [ANNOUNCE] Apache Spark 2.4.8 released

Great work, Liang-Chi! On Tue, May 18, 2021 at 3:57 PM Jungtaek Lim wrote: > Thanks for the huge efforts on driving the release! > > On Tue, May 18, 2021 at 4:53 PM Wenchen Fan wrote: > >> Thank you, Liang-Chi! >> >> On Tue, May 18, 2021 at 1:32 PM Dongjoon Hyun >> wrote: >> >>> Finally!

Re: Apache Spark 3.1.2 Release?

+1. On Tue, May 18, 2021 at 9:06 AM Hyukjin Kwon wrote: > +1 thanks for driving me > > On Tue, 18 May 2021, 09:33 Holden Karau, wrote: > >> +1 and thanks for volunteering to be the RM :) >> >> On Mon, May 17, 2021 at 4:09 PM Takeshi Yamamuro >> wrote: >> >>> Thank you, Dongjoon~ sgtm, too.

Re: beeline spark thrift server issue

Unable to access log(https://pastebin.com/G5Mwaw7E). On Thu, May 13, 2021 at 11:23 PM Suryansh Agnihotri < sagnihotri2...@gmail.com> wrote: > Hi > I was trying to access spark sql through jdbc but facing some error. I am > trying to run beeline > > ! /usr/lib/spark/bin/beeline -u >

Re: [DISCUSS] Add error IDs

+1 for this proposal. On Fri, Apr 16, 2021 at 5:15 AM Karen wrote: > We could leave space in the numbering system, but a more flexible method > may be to have the severity as a field associated with the error class - > the same way we would associate error ID with SQLSTATE, or with whether an >

Re: [DISCUSS] Build error message guideline

+1 LGTM. On Thu, Apr 15, 2021 at 1:50 AM Karen wrote: > That makes sense to me - given that an assert failure throws an > AssertException, I would say that the same guidelines should apply for > asserts. > > On Tue, Apr 13, 2021 at 7:41 PM Yuming Wang wrote: > >> D

Re: [DISCUSS] Build error message guideline

Do we have plans to apply these guidelines to assert? For example: https://github.com/apache/spark/blob/5b478416f8e3fe2f015af1b6c8faa7fe9f15c05d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala#L136-L138

Re: Welcoming six new Apache Spark committers

Congrats! On Sat, Mar 27, 2021 at 7:13 AM Takeshi Yamamuro wrote: > Congrats, all~ > > On Sat, Mar 27, 2021 at 7:46 AM Jungtaek Lim > wrote: > >> Congrats all! >> >> 2021년 3월 27일 (토) 오전 6:56, Liang-Chi Hsieh 님이 작성: >> >>> Congrats! Welcome! >>> >>> >>> Matei Zaharia wrote >>> > Hi all, >>> >

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Great work, Hyukjin! On Wed, Mar 3, 2021 at 9:50 AM Hyukjin Kwon wrote: > We are excited to announce Spark 3.1.1 today. > > Apache Spark 3.1.1 is the second release of the 3.x line. This release adds > Python type annotations and Python dependency management support as part > of Project Zen. >

Re: [VOTE] Release Spark 3.1.1 (RC3)

+1 @Sean Owen I do not have this issue: [info] SparkSQLEnvSuite: 19:45:15.430 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19:45:56.366 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of

Re: Apache Spark 3.0.2 Release ?

+1. On Sat, Feb 13, 2021 at 10:38 AM Takeshi Yamamuro wrote: > +1, too. Thanks, Dongjoon! > > 2021/02/13 11:07、Xiao Li のメール: > >  > +1 > > Happy Lunar New Year! > > Xiao > > On Fri, Feb 12, 2021 at 5:33 PM Hyukjin Kwon wrote: > >> Yeah, +1 too >> >> 2021년 2월 13일 (토) 오전 4:49, Dongjoon Hyun 님이

Re: [VOTE] Release Spark 3.1.1 (RC2)

:* "Sean Owen"; > *抄送:* "Hyukjin Kwon";"Yuming Wang" >;"dev"; > *主题:* Re: [VOTE] Release Spark 3.1.1 (RC2) > > +1 > > On Tue, Feb 9, 2021 at 1:39 PM Sean Owen wrote: > >> Same result as last time for me, +1. Tested with Java 11. >&

Re: [VOTE] Release Spark 3.1.1 (RC2)

Many tickets do not have correct assignee: https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20in%20(3.1.0%2C%203.1.1)%20AND%20(assignee%20is%20EMPTY%20or%20assignee%20%3D%20apachespark) On Tue, Feb 9, 2021 at 9:05 AM

Re: [VOTE] Release Spark 3.1.1 (RC1)

+1. On Tue, Jan 19, 2021 at 7:54 AM Hyukjin Kwon wrote: > I forgot to say :). I'll start with my +1. > > On Mon, 18 Jan 2021, 21:06 Hyukjin Kwon, wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 3.1.1. >> >> The vote is open until January 22nd 4PM PST and

Re: [build system] WE'RE LIVE!

Thank you, Shane. On Wed, Dec 2, 2020 at 8:55 AM shane knapp ☠ wrote: > https://amplab.cs.berkeley.edu/jenkins/ > > i cleared the build queue, so you'll need to retrigger your PRs. there > will be occasional downtime over the next few days and weeks as we uncover > system-level errors and more

Re: jenkins downtime tomorrow evening/weekend

Hi Shane, Did you set :export LANG=en_US.UTF-8? Some test seems failed because of this issue: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131631/testReport/ Please see https://issues.apache.org/jira/browse/SPARK-27177 for more details. On Tue, Nov 24, 2020 at 8:23 AM

Re: 回复： [DISCUSS] Apache Spark 3.0.1 Release

Another correctness issue: https://issues.apache.org/jira/browse/SPARK-32659 On Tue, Aug 25, 2020 at 11:25 PM Sean Owen wrote: > That isn't a blocker (see comments - not a regression). > That said I think we have a fix ready to merge now, if there are no > objections. > > On Tue, Aug 25, 2020

[ANNOUNCE] Announcing Apache Spark 3.0.0-preview2

Hi all, To enable wide-scale community testing of the upcoming Spark 3.0 release, the Apache Spark community has posted a new preview release of Spark 3.0. This preview is *not a stable release in terms of either API or functionality*, but it is meant to give the community early access to try the

Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

I'm +1 for this SPIP for these two reasons: 1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636 . 2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger

[VOTE][RESULT] SPARK 3.0.0-preview2 (RC2)

Hi, All. The vote passes. Thanks to all who helped with this release 3.0.0-preview2! I'll follow up later with a release announcement once everything is published. +1 (* = binding): - Sean Owen * - Dongjoon Hyun * - Takeshi Yamamuro * - Wenchen Fan * +0: None -1: None Regards, Yuming

Re: [VOTE] SPARK 3.0.0-preview2 (RC2)

Please go to td28549 to vote, this voting link is incorrect. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

[VOTE] SPARK 3.0.0-preview2 (RC2)

Please vote on releasing the following candidate as Apache Spark version 3.0.0-preview2. The vote is open until December 20 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0-preview2 [ ] -1 Do not release this

[VOTE] SPARK 3.0.0-preview2 (RC2)

Please vote on releasing the following candidate as Apache Spark version 3.0 .0-preview2. The vote is open until December 20 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0-preview2 [ ] -1 Do not release this

Re: Packages to release in 3.0.0-preview

er. >> >> Thanks, >> Dongjoon. >> >> >> On Sun, Oct 27, 2019 at 7:31 AM Sean Owen wrote: >> >>> Is the Spark artifact actually any different between those builds? I >>> thought it just affected what else was included in the

Re: Packages to release in 3.0.0-preview

Do we need to publish the Scala 2.12 + hadoop 3.2 jar packages to the Maven repository? Otherwise it will throw a NoSuchMethodError on Java 11. Here is an example: https://github.com/wangyum/test-spark-jdk11/blob/master/src/test/scala/test/spark/HiveTableSuite.scala#L34-L38

jenkins locale issue

Why jenkins locale is: LANG= LC_CTYPE="POSIX" LC_NUMERIC="POSIX" LC_TIME="POSIX" LC_COLLATE="POSIX" LC_MONETARY="POSIX" LC_MESSAGES="POSIX" LC_PAPER="POSIX" LC_NAME="POSIX" LC_ADDRESS="POSIX" LC_TELEPHONE="POSIX" LC_MEASUREMENT="POSIX" LC_IDENTIFICATION="POSIX Hadoop will throw

[DISCUSS] Upgrade built-in Hive to 2.3.4

Dear Spark Developers and Users, Hyukjin and I plan to upgrade the built-in Hive from 1.2.1-spark2 to 2.3.4 to solve some critical issues, such as support Hadoop 3.x,

run docker-integration-tests in jenkins

hi, shane Can we run docker-integration-tests in jenkins? see discuss here . Looking forward to hear your feedback, Thanks.

Re: [VOTE] Spark 2.1.2 (RC1)

Yes, It doesn’t work in 2.1.0 and 2.1.1, I create a PR for this: https://github.com/apache/spark/pull/19259 . > 在 2017年9月17日，16:14，Sean Owen 写道： > > So, didn't work in 2.1.0 or 2.1.1? If it's not a regression and not critical,

Is there something wrong with jenkins?

Hi All, Is there something wrong with jenkins? # To activate this environment, use: # $ source activate /tmp/tmp.tWAUGnH6wZ/3.5 # # To deactivate this environment, use: # $ source deactivate # discarding /home/anaconda/bin from PATH prepending /tmp/tmp.tWAUGnH6wZ/3.5/bin to PATH Fetching

Remove old Hive support

Do we have a plan to remove old Hive support?

Re: Request for comments: Java 7 removal

There is a way only Spark use Java 8, Hadoop still use Java 7: spark-conf.jpg (58K) By the way, I have a way to install any spark version on CM5.4 - CM5.7 by custom CSD

Re: [VOTE] Apache Spark 2.1.0 (RC5)

I hope https://github.com/apache/spark/pull/16252 can be fixed until release 2.1.0. It's a fix for broadcast cannot fit in memory. On Sat, Dec 17, 2016 at 10:23 AM, Joseph Bradley wrote: > +1 > > On Fri, Dec 16, 2016 at 3:21 PM, Herman van Hövell tot Westerflier < >

Apply patch to 1.2.1.spark2

Hi, Is there a way to apply this patch to ?

ml.feature.Word2Vec.transform() very slow issue

Hi I found org.apache.spark.ml.feature.Word2Vec.transform() very slow. I think we should not read broadcast every sentence, so I fixed on my forked. https://github.com/979969786/spark/commit/a9f894df3671bb8df2f342de1820dab3185598f3 I have use 2 number rows test it. Original version