Re: [DISCUSS] Release Apache Spark 3.5.7

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Sep 12, 2025, at 10:14 PM, Peter Toth wrote: > > Thank you all for the positive feedback! > > On Fri, Sep 12, 2025 at 7:26 AM Jungtaek Lim <mailto:kabhwan.opensou...@gmail.com>> wrote: >> +1 sou

Re: [VOTE] SPIP: Monthly preview release

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jul 2, 2025, at 9:37 PM, Hyukjin Kwon wrote: > > Hi all, > > I would like to start a vote on the monthly preview releases. > > Discussion thread: > https://lists.apache.org/thread/1hmsb3g7lm5k2f9

Re: [DISCUSS] SPIP: Monthly preview release

new features Stronger contributor engagement Quicker community feedback cycles Easier debugging and testing through smaller, incremental changes Thanks, DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jul 1, 2025, at 2:17 PM, Dongjoon Hyun wrote: > > Thank you so muc

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 looking forward to seeing real-time mode.Sent from my iPhoneOn Jun 1, 2025, at 9:47 PM, Xiao Li wrote:+1huaxin gao 于2025年6月1日周日 20:00写道：+1On Sun, Jun 1, 2025 at 7:50 PM Tathagata Das wrote:+1 (binding)super excited about this!On Sun, Jun 1

Re: [VOTE] Release Apache Spark Connect Swift Client 0.3.0 (RC1)

+1 Sent from my iPhone > On Jun 1, 2025, at 2:32 AM, Yang Jie wrote: > > +1 > >> On 2025/06/01 08:09:38 Peter Toth wrote: >> +1 >> >>> On Sun, Jun 1, 2025 at 9:00 AM L. C. Hsieh wrote: >>> >>> +1 >>> >>> On Sat, May 31, 2025 at 9:18 PM Dongjoon Hyun >>> wrote: Please vote on r

Re: [VOTE] Release Apache Spark K8s Operator 0.3.0 (RC1)

+1 Sent from my iPhone > On Jun 1, 2025, at 2:32 AM, Yang Jie wrote: > > +1 > >> On 2025/06/01 08:10:10 Peter Toth wrote: >> +1 >> >>> On Sun, Jun 1, 2025 at 9:01 AM L. C. Hsieh wrote: >>> >>> +1 >>> >>> On Sat, May 31, 2025 at 10:38 PM Dongjoon Hyun >>> wrote: Please vote on

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1Sent from my iPhoneOn May 29, 2025, at 12:15 AM, John Zhuge wrote:+1 Nice featureOn Wed, May 28, 2025 at 9:53 PM Yuanjian Li wrote:+1Kent Yao 于2025年5月28日周三 19:31写道：+1, LGTM.Kent在 2025年5月29日星期四，Chao Sun 写道：+1. Super excited by this

Re: [VOTE] Release Apache Spark K8s Operator 0.1.0 (RC1)

+1On May 5, 2025, at 1:10 AM, Gabor Somogyi wrote:+1 (non-binding)GOn Mon, May 5, 2025 at 8:59 AM huaxin gao wrote:+1 Thanks Dongjoon.On Sun, May 4, 2025 at 7:36 PM Rozov, Vlad wrote:+1 (not binding) Checked checksum and signatures, build,  and confirmed that binary fil

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1, it’s exciting to see Spark Connect Swift client, showcasing Spark Connect as a truly language-agnostic protocol, and also powering Swift users to use Spark!Sent from my iPhoneOn May 5, 2025, at 1:11 AM, Gabor Somogyi wrote:+1 (non-binding)GOn Mon, May 5, 2025 at 8:35 AM huaxin gao

Re: [VOTE] SPIP: Add geospatial types to Spark

+1, geospatial types will be a great feature for Spark. Thanks for working on it. > On May 5, 2025, at 11:04 AM, Menelaos Karavelas > wrote: > > I started the discussion on adding geospatial types to Spark on March 28th. > Since then there has been some discussion in the dev mailing list, as

Re: [VOTE] SPIP: Declarative Pipelines

+1. Really exciting to see the community actively investing on decorative pipeline APIs and materialized views. Thanks. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Apr 9, 2025, at 9:43 AM, Jules Damji wrote: > > +1 (non-binding) > > Excuse the thumb t

Re: [VOTE] SPIP: Support NanoSecond Timestamps

Thanks!!! DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Mar 27, 2025, at 3:56 PM, Qi Tan wrote: > > Thanks DB, > > I just noticed a few more comments came in after I initiated the vote. I'm > going to postpone the voting process and address tho

Re: [VOTE] SPIP: Support NanoSecond Timestamps

etails for data sources are missing. These points still need discussion. I appreciate your efforts in putting the doc together and look forward to your contribution! Thanks, DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Mar 27, 2025, at 1:24 PM, huaxin gao wrote: >

Re: [VOTE] SPIP: Constraints in DSv2

+1Sent from my iPhoneOn Mar 21, 2025, at 2:25 PM, Szehon Ho wrote:+1 (non binding)Agree with Anton, data sources like the open table formats define the requirement, and definitely need engines to write to it accordingly.Thanks,SzehonOn Fri, Mar 21, 2025 at 1:31 PM Anton Okolnychyi

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1Sent from my iPhoneOn Mar 14, 2025, at 11:28 PM, Szehon Ho wrote:+1 to the idea as well, as Iceberg V3 is coming with time with nanos, and Spark would not be able to read this type without this.ThanksSzehonOn Fri, Mar 14, 2025 at 3:34 PM Wenchen Fan wrote:In general, I thi

Re: [DISCUSS] New Spark Connect Client repository for Swift language

as a system language beyond app development. Additionally, Swift's interactive capabilities make it well-suited for integration with Jupyter Notebooks for Spark. Thanks, DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Mar 10, 2025, at 10:47 AM, Mich Talebzadeh >

Re: [DISCUSS] SPIP: Constraints in DSv2

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 13, 2025, at 10:21 PM, Gengliang Wang wrote: > > +1, the proposal will unify constraint management in DSv2 and reduce > redundant work across connectors > > On Thu, Feb 13, 2025 at 9:20 P

Re: [VOTE] Publish additional Spark distribution with Spark Connect enabled

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 4, 2025, at 11:13 PM, Xiao Li wrote: > > +1 > > Xiao > > Jungtaek Lim <mailto:kabhwan.opensou...@gmail.com>> 于2025年2月4日周二 23:11写道： >> +1 (non-binding) >> >> Sounds like

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

Spark 4.0 release. If these PRs do not land before the RC cut, we should ensure they are properly documented. Thanks, DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 4, 2025, at 7:23 AM, Dongjoon Hyun wrote: > > Many new feature `Connect` patches are still landin

Re: [DISCUSS] Publish additional Spark distribution with Spark Connect enabled

+1 This enables users to easily experiment with and provide feedback on Spark Connect, while also facilitating broader adoption and development in other languages like Rust, Go, or Scala 3. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 3, 2025, at 11:29 PM, Wenc

Re: [VOTE] Use plain text logs by default

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jan 9, 2025, at 12:27 AM, Peter Toth wrote: > > +1 > > On Thu, Jan 9, 2025 at 8:37 AM Allison Wang <mailto:allisonw...@apache.org>> wrote: >> +1 >> >> On Thu, Jan 9, 2025 at 2:45

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Apr 29, 2024, at 8:01 PM, Wenchen Fan wrote:To add more color:Spark data source table and Hive Serde table are both stored in the Hive metastore and keep the data files in the table directory. The only difference is they have different "table provider", which means Spark will use different

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1Sent from my iPhoneOn Apr 16, 2024, at 3:11 PM, bo yang wrote:+1On Tue, Apr 16, 2024 at 1:38 PM Hyukjin Kwon wrote:+1On Wed, Apr 17, 2024 at 3:57 AM L. C. Hsieh wrote:+1 On Tue, Apr 16, 2024 at 4:08 AM Wenchen Fan wrote: > > +1 >

Re: Versioning of Spark Operator

Aligning with Spark releases is sensible, as it allows us to guarantee that the Spark operator functions correctly with the new version while also maintaining support for previous versions. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Apr 9, 2024, at 9:45 AM, Mri

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Nov 14, 2023, at 10:14 AM, Vakaris Baškirov > wrote: > > +1 (non-binding) > > On Tue, Nov 14, 2023 at 8:03 PM Chao Sun <mailto:sunc...@apache.org>> wrote: >> +1 >> >>

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Kubernetes operator is essential for our Spark community as well. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Nov 9, 2023, at 12:05 PM, Zhou Jiang wrote: > > Hi Spark community, > I'm reaching out to initiate a conversation about the possibility of &g

Re: [VOTE][SPIP] Lazy Materialization for Parquet Read Performance Improvement

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Feb 14, 2023, at 8:29 AM, Guo Weijie wrote: > > +1 > > Yuming Wang mailto:wgy...@gmail.com>> 于2023年2月14日周二 > 15:58写道： >> +1 >> >> On Tue, Feb 14, 2023 at 11:27 AM Prem Sahoo >

Re: [DISCUSS] SPIP: Lazy Materialization for Parquet Read Performance Improvement

+1Sent from my iPhoneOn Jan 31, 2023, at 4:16 PM, Yuming Wang wrote:+1.On Wed, Feb 1, 2023 at 7:42 AM kazuyuki tanimura wrote:Great! Much appreciated, Mitch! KazuOn Jan 31, 2023, at 3:07 PM, Mich Talebzadeh wrote:Thanks, Kazu.I followed that template link and indeed a

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thank you, Huaxin for the 3.2.1 release! Sent from my iPhone > On Jan 28, 2022, at 5:45 PM, Chao Sun wrote: > >  > Thanks Huaxin for driving the release! > >> On Fri, Jan 28, 2022 at 5:37 PM Ruifeng Zheng wrote: >> It's Great! >> Congrats and thanks, huaxin! >> >> >> -- 原始邮

Re: Apache Spark Jenkins Infra 2022

Thank you, Dongjoon for driving the build infra. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jan 9, 2022, at 6:38 PM, shane knapp ☠ wrote: > > > apache spark jenkins lives on! > > @dongjoon, let me know if there's anything you need

Re: [VOTE] SPIP: Row-level operations in Data Source V2

----- > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >> > >> > > - > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1

Re: [VOTE] SPIP: Storage Partitioned Join for Data Source V2

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 On Fri, Oct 29, 2021 at 11:42 AM Ryan Blue wrote: > +1 > > On Fri, Oct 29, 2021 at 11:06 AM huaxin gao > wrote: > >> +1 >> >> On Fri, Oct 29, 2021 at 10:59 AM Dongjoon Hyun >> wrote: >&g

Re: [DISCUSS] SPIP: Storage Partitioned Join for Data Source V2

forward to it as a new feature in Spark 3.3 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 On Fri, Oct 22, 2021 at 12:18 PM Chao Sun wrote: > > Hi, > > Ryan and I drafted a design doc to support a new type of join: storage > partitioned join which covers bucket j

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 On Mon, Oct 11, 2021 at 6:01 AM Almeida, (Ricardo) wrote: > > +1 (non-binding) > > > > Ricardo Almeida > > > > From: Xiao Li > Sent: Monday, October 11, 2021 9:09 AM > To: Yi Wu > Cc: Ho

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hello Xiao, there are multiple patches in Spark 3.2 depending on parquet 1.12, so it might be easier to wait for the fix in parquet community instead of reverting all the related changes. The fix in parquet community is very trivial, and we hope that it will not take too long. Thanks. DB Tsai

Re: [DISCUSS] Rename hadoop-3.2/hadoop-2.7 profile to hadoop-3/hadoop-2?

+1 on renaming. DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > On Jun 24, 2021, at 11:41 AM, Chao Sun wrote: > > Hi, > > As Spark master has upgraded to Hadoop-3.3.1, the current Maven profile name > hadoop-3.2 is no longer accurate, and it may confuse Spa

Re: [VOTE] Release Spark 2.4.8 (RC3)

+1 (binding) > On Apr 28, 2021, at 9:26 AM, Liang-Chi Hsieh wrote: > > > Please vote on releasing the following candidate as Apache Spark version > 2.4.8. > > The vote is open until May 4th at 9AM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Relea

Re: [VOTE] Release Spark 2.4.8 (RC2)

+1 (binding) DB Tsai | ACS Spark Core |  Apple, Inc. > On Apr 14, 2021, at 10:42 AM, Wenchen Fan wrote: > > +1 (binding) > > On Thu, Apr 15, 2021 at 12:22 AM Maxim Gekk <mailto:maxim.g...@databricks.com>> wrote: > +1 (non-binding) > > On Wed, Apr

Re: [DISCUSS] Add RocksDB StateStore

For second >>> one, we propose (SPARK-34198) to add it as an external module to relieve the >>> dependency concern. >>> >>> Because it was pushed back previously, I'm going to raise this discussion to >>> know what people think about it now, in advance

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

/edit >> >> Active discussions on the jira and SPIP document have settled. >> >> I will leave the vote open until Friday (the 18th September 2020), 5pm >> CST. >> >> [ ] +1: Accept the proposal as an official SPIP >> [ ] +0 >> [ ] -1: I don't think this is a good idea because ... >> >> >> Thanks, >> Mridul >> > -- Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1

Re: [VOTE] Decommissioning SPIP

; >>> +1 for having this feature in Spark >>> >>> >>> >>> -- >>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>> >>> ----- &

Re: Revisiting the idea of a Spark 2.5 transitional release

. >> At the job level sure, but upgrading large jobs, possibly written in Scala >> 2.11, whole-hog as it currently stands is not a small matter. >> >> On Fri, Jun 12, 2020 at 9:40 PM DB Tsai wrote: >> +1 for a 2.x release with DSv2, JDK11, and Scala 2.11 supp

Re: Revisiting the idea of a Spark 2.5 transitional release

can still move forward using new features. Afterall, the reason why we are working on OSS is we like people to use our code, isn't it? Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Fri, Jun 12, 2020

Re: [vote] Apache Spark 3.0 RC3

+1 (binding) Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Mon, Jun 8, 2020 at 1:03 PM Dongjoon Hyun wrote: > > +1 > > Thanks, > Dongjoon. > > On Mon, Jun 8, 2020 at 6:37 AM Russ

Re: [VOTE] Release Spark 2.4.6 (RC8)

+1 (binding), thanks! On Sun, May 31, 2020 at 9:23 PM Wenchen Fan wrote: > +1 (binding), although I don't know why we jump from RC 3 to RC 8... > > On Mon, Jun 1, 2020 at 7:47 AM Holden Karau wrote: > >> Please vote on releasing the following candidate as Apache Spark >> version 2.4.6. >> >> Th

Re: [VOTE] Release Spark 2.4.6 (RC3)

' code when upgrading from Scala 2.11 to Scala 2.12. Thanks, Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 Sincerely, DB Tsai -- Web:

Re: [VOTE] Release Spark 2.4.6 (RC3)

+1 as well. Thanks. On Sun, May 17, 2020 at 7:39 AM Sean Owen wrote: > +1 , same response as to the last RC. > This looks like it includes the fix discussed last time, as well as a > few more small good fixes. > > On Sat, May 16, 2020 at 12:08 AM Holden Karau > wrote: > > > > Please vote on rel

Re: [VOTE] Release Apache Spark 2.4.5 (RC1)

+1 Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Tue, Jan 14, 2020 at 11:08 AM Sean Owen wrote: > > Yeah it's something about the env I spun up, but I don't know what. It >

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 Thanks! On Wed, Aug 28, 2019 at 7:14 AM Wenchen Fan wrote: > +1, no more blocking issues that I'm aware of. > > On Wed, Aug 28, 2019 at 8:33 PM Sean Owen wrote: > >> +1 from me again. >> >> On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun >> wrote: >> > >> > Please vote on releasing the follo

[DISCUSSION]JDK11 for Apache 2.x?

is not desired in minor release? Thanks. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.3.4 (RC1)

+1 Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Tue, Aug 27, 2019 at 11:31 AM Dongjoon Hyun wrote: > > +1. > > I also verified SHA/GPG and tested UTs on AdoptOpenJDKu8_222/CentOS6.9 wit

Re: JDK11 Support in Apache Spark

Congratulations on the great work! Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Sat, Aug 24, 2019 at 8:11 AM Dongjoon Hyun wrote: > > Hi, All. > > Thanks to your many many contributions, &g

Re: Release Apache Spark 2.4.4

+1 On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun wrote: > > Hi, All. > > Spark 2.4.3 was released three months ago (8th May). > As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` > since 2.4.3. > > It would be great if we can have Spark 2.4.4. > Shall we start `2.4.4 RC1`

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

umnar processing support, I can imagine that the heavy lifting parts of ML applications (such as computing the objective functions) can be written as columnar expressions that leverage on SIMD architectures to get a good speedup. Sincerely, DB

[ANNOUNCE] Announcing Apache Spark 2.4.1

+user list We are happy to announce the availability of Spark 2.4.1! Apache Spark 2.4.1 is a maintenance release, based on the branch-2.4 maintenance branch of Spark. We strongly recommend all 2.4.0 users to upgrade to this stable release. In Apache Spark 2.4.1, Scala 2.12 support is GA, and it'

[ANNOUNCE] Announcing Apache Spark 2.4.1

ithout you. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

This vote passes! +1: Wenchen Fan (binding) Sean Owen (binding) Mihaly Toth DB Tsai (binding) Jonatan Jäderberg Xiao Li (binding) Denny Lee Felix Cheung (binding) +0: None -1: None It's the largest RC ever; I will follow up with an official release announcement soon. Thank you all for

Re: [VOTE] Release Apache Spark 2.4.1 (RC9)

+1 from myself On Thu, Mar 28, 2019 at 3:14 AM Mihaly Toth wrote: > +1 (non-binding) > > Thanks, Misi > > Sean Owen ezt írta (időpont: 2019. márc. 28., Cs, > 0:19): > >> +1 from me - same as last time. >> >> On Wed, Mar 27, 2019 at 1:31 PM DB Tsai wrote:

[VOTE] Release Apache Spark 2.4.1 (RC9)

es, we will typically not hold the release unless the bug in question is a regression from the previous release. That being said, if there is something which is a regression that has not been correctly targeted please ping me or a committer to help target the issue. DB Tsai | Siri Open S

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

RC9 was just cut. Will send out another thread once the build is finished. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Mon, Mar 25, 2019 at 5:10 PM Sean Owen wrote: > > That's all merged n

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

I am going to cut a 2.4.1 rc9 soon tonight. Besides SPARK-26961 https://github.com/apache/spark/pull/24126 , anything critical that we have to wait for 2.4.1 release? Thanks! Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

Hello Sean, By looking at SPARK-26961 PR, seems it's ready to go. Do you think we can merge it into 2.4 branch soon? Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Sat, Mar 23, 2019 at 12:04 PM Sean

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

-1 I will fail RC8, and cut another RC9 on Monday to include SPARK-27160, SPARK-27178, SPARK-27112. Please let me know if there is any critical PR that has to be back-ported into branch-2.4. Thanks. Sincerely, DB Tsai -- Web: https

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

branch-2.4, can you make anther PR against branch-2.4 so we can include the ORC fix in 2.4.1? Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Wed, Mar 20, 2019 at 9:11 PM Felix Cheung wrote

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

differences between RC8 and 2.4.0 are big? If an issue is found to justify to fail RC8, we can include SPARK-27112 and SPARK-27160 in next cut. Thus, even we decide to cut another RC, it will be easier to test. Thanks. Sincerely, DB Tsai -- Web

[VOTE] Release Apache Spark 2.4.1 (RC8)

es, we will typically not hold the release unless the bug in question is a regression from the previous release. That being said, if there is something which is a regression that has not been correctly targeted please ping me or a committer to help target the issue. DB Tsai | Siri Open S

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Since rc8 was already cut without the k8s client upgrade; the build is ready to vote, and including k8s client upgrade in 2.4.1 implies that we will drop the old-but-not-that-old K8S versions as Sean mentioned, should we include this upgrade in 2.4.2? Thanks. Sincerely, DB Tsai

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

As we have many important fixes in 2.4 branch which we want to release asap, and this is is not a regression from Spark 2.4; as a result, 2.4.1 will be not blocked by this. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Since I can not find the commit of `Preparing development version 2.4.2-SNAPSHOT` after rc6 cut, it's very risky to fix the branch and do a force-push. I'll follow Marcelo's suggestion to have another rc7 cut. Thus, this vote fails. DB Tsai | Siri Open Source Technologies [not

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

Okay, I see the problem. rc6 tag is not in the 2.4 branch. It's very weird. It must be overwritten by a force push. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Mar 8, 2019, at 11:39 AM, DB Tsai wrote: > > I was using `./do-release-docker

Re: [VOTE] Release Apache Spark 2.4.1 (RC6)

of using the same commit causing this issue. Should we create a new rc7? DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Mar 8, 2019, at 10:54 AM, Marcelo Vanzin > wrote: > > I personally find it a little weird to not have the commit i

[VOTE] Release Apache Spark 2.4.1 (RC6)

es, we will typically not hold the release unless the bug in question is a regression from the previous release. That being said, if there is something which is a regression that has not been correctly targeted please ping me or a committer to help target the issue. DB Tsai | Siri Open Source Technol

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

spark-streaming-flume-assembly_2.11-2.4.1-tests.jar', check the logs.* I am sure my key is in the key server, and the weird thing is that it fails on different jars each time I ran the publish script. Sincerely, DB Tsai -- Web: https://www.

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

I am cutting a new rc4 with fix from Felix. Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0359BC9965359766 On Thu, Feb 21, 2019 at 8:57 AM Felix Cheung wrote: > > I merged the fix

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

Okay. Let's fail rc2, and I'll prepare rc3 with SPARK-26859. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Feb 20, 2019, at 12:11 PM, Marcelo Vanzin > wrote: > > Just wanted to point out that > https://issues.apache.org/jira/bro

[VOTE] Release Apache Spark 2.4.1 (RC2)

e will typically not hold the release unless the bug in question is a regression from the previous release. That being said, if there is something which is a regression that has not been correctly targeted please ping me or a committer to help target the issue. DB Tsai | Siri Open Source Technol

Re: Time to cut an Apache 2.4.1 release?

Great. I'll prepare the release for voting. Thanks! DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Feb 12, 2019, at 4:11 AM, Wenchen Fan wrote: > > +1 for 2.4.1 > > On Tue, Feb 12, 2019 at 7:55 PM Hyukjin Kwon wrote: > +1 for 2.4

Time to cut an Apache 2.4.1 release?

Hello all, I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4. The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20D

Re: [VOTE] Release Apache Spark 2.3.3 (RC1)

-1 Agreed with Anton that this bug will potentially corrupt the data silently. As he is ready to submit a PR, I'll suggest to wait to include the fix. Thanks! Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0x5CED8B896A6

Re: [VOTE] SPARK 2.2.3 (RC1)

+1 Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0x5CED8B896A6BDFA0 On Tue, Jan 8, 2019 at 11:14 AM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.2.3. >

Re: Automated formatting

I like the idea of checking only the diff. Even I am sometimes confused about the right style in Spark since I am working on multiple projects with slightly different coding styles. On Wed, Nov 21, 2018 at 1:36 PM Sean Owen wrote: > I know the PR builder runs SBT, but I presume this would just b

Re: Make Scala 2.12 as default Scala version in Spark 3.0

+1 on removing Scala 2.11 support for 3.0 given Scala 2.11 is already EOL. On Tue, Nov 20, 2018 at 2:53 PM Sean Owen wrote: > PS: pull request at https://github.com/apache/spark/pull/23098 > Not going to merge it until there's clear agreement. > > On Tue, Nov 20, 2018 at 10:16 AM Ryan Blue wro

Re: Make Scala 2.12 as default Scala version in Spark 3.0

Most of the time in the PR build is on running tests. How about we also add Scala 2.11 compilation for both main and test without running the tests in the PR build? Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID

Re: Make Scala 2.12 as default Scala version in Spark 3.0

later if we want to change the alternative Scala version to 2.13 and drop 2.11 if we just want to support two Scala versions at one time. Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0x5CED8B896A6BDFA0 On Wed, Nov 7,

Re: Make Scala 2.12 as default Scala version in Spark 3.0

Ideally, supporting only Scala 2.12 in Spark 3 will be ideal. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Nov 6, 2018, at 2:55 PM, Felix Cheung wrote: > > So to clarify, only scala 2.12 is supported in Spark 3? > > > From: Ryan Blu

Re: Make Scala 2.12 as default Scala version in Spark 3.0

agree with Sean that this can make the decencies really complicated; hence I support to drop Scala 2.11 in Spark 3.0 directly. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Nov 6, 2018, at 11:38 AM, Sean Owen wrote: > > I think we should make Scala

Re: Test and support only LTS JDK release?

OpenJDK will follow Oracle's release cycle, https://openjdk.java.net/projects/jdk/ <https://openjdk.java.net/projects/jdk/>, a strict six months model. I'm not familiar with other non-Oracle VMs and Redhat support. DB Tsai | Siri Open Source Technologies [not a contribution]

Test and support only LTS JDK release?

Given Oracle's new 6-month release model, I feel the only realistic option is to only test and support JDK such as JDK 11 LTS and future LTS release. I would like to have a discussion on this in Spark community. Thanks, DB Tsai | Siri Open Source Technologies [not a contrib

Make Scala 2.12 as default Scala version in Spark 3.0

have ample time to work on bugs and issues that we may run into. What do you think? Thanks, DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Java 11 support

Given Oracle's new 6-month release model, I think the only realistic option is to only support and test LTS JDK. I'll send out two separate emails to dev to facilitate the discussion. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Nov 6, 2018, at 9

Re: [VOTE] SPARK 2.4.0 (RC5)

selected simultaneously. https://issues.apache.org/jira/browse/SPARK-25879 If we decide to not fix it in 2.4, we should at least document it in the release note to let users know. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID

Re: Starting to make changes for Spark 3 -- what can we delete?

I'll +1 on removing those legacy mllib code. Many users are confused about the APIs, and some of them have weird behaviors (for example, in gradient descent, the intercept is regularized which supports not to). DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple

Re: Scala 2.12 support

018 at 5:54 PM, Holden Karau >> wrote: >> > I agree that's a little odd, could we not add the bacspace terminal >> > character? Regardless even if not, I don't think that should be a >> blocker >> > for 2.12 support especially since it doesn'

Re: Scala 2.12 support

ark context Web UI available at http://192.168.1.169:4040 Spark context available as 'sc' (master = local[*], app id = local-1528180279528). Spark session available as 'spark’. scala> DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Jun 7,

Re: Scala 2.12 support

blocker for us to move to newer version of Scala 2.12.x since the newer version of Scala 2.12.x has the same issue. In my opinion, Scala should fix the root cause and provide a stable hook for 3rd party developers to initialize their custom code. DB Tsai | Siri Open Source Technologies [not a

Re: [MLLib] Logistic Regression and standadization

the result should match R. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Apr 20, 2018, at 5:56 PM, Weichen Xu wrote: > > Right. If regularization item isn't zero, then enable/disable standardization > will get different result. > But,

Re: Will higher order functions in spark SQL be pushed upstream?

, this effort is primarily tracked via SPARK-4502 (see >> https://github.com/apache/spark/pull/16578) and is currently targeted for >> 2.3. -- Sincerely, DB Tsai -- PGP Key ID: 0x5CED8B896A6BDFA0 - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Welcoming Tejas Patil as a Spark committer

Congratulations! On Wed, Oct 4, 2017 at 6:55 PM, Liwei Lin wrote: > Congratulations! > > Cheers, > Liwei > > On Wed, Oct 4, 2017 at 2:27 PM, Yuval Itzchakov wrote: >> >> Congratulations and Good luck! :) >> >> >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >

Re: [VOTE] Spark 2.1.2 (RC4)

+1 Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0x5CED8B896A6BDFA0 On Fri, Oct 6, 2017 at 7:46 AM, Felix Cheung wrote: > Thanks Nick, Hyukjin. Yes this seems to be a longer standing issue on RHEL > with resp

Re: [VOTE] Apache Spark 2.1.1 (RC2)

I backported the fix into both branch-2.1 and branch-2.0. Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0x5CED8B896A6BDFA0 On Mon, Apr 10, 2017 at 4:20 PM, Ryan Blue wrote: > DB, > > This vote already f

  1   2   3   >