Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Cheng Su
+1 (non-binding). Built from source, and ran some scala unit tests on M1 mac, with OpenJDK 8 and Scala 2.12. Thanks, Cheng Su On Mon, Jul 11, 2022 at 10:31 PM Yang,Jie(INF) wrote: > Does this happen when running all UTs? I ran this suite several times > alone using OpenJDK(zulu) 8u3

Re: Apache Spark 3.2.2 Release?

2022-07-06 Thread Cheng Su
+1 (non-binding) Thanks, Cheng Su On Wed, Jul 6, 2022 at 6:01 PM Yuming Wang wrote: > +1 > > On Thu, Jul 7, 2022 at 5:53 AM Maxim Gekk > wrote: > >> +1 >> >> On Thu, Jul 7, 2022 at 12:26 AM John Zhuge wrote: >> >>> +1 Thanks for the effo

Re: [VOTE] Release Spark 3.3.0 (RC6)

2022-06-13 Thread Cheng Su
+1 (non-binding). Thanks, Cheng Su From: L. C. Hsieh Date: Monday, June 13, 2022 at 5:13 PM To: dev Subject: Re: [VOTE] Release Spark 3.3.0 (RC6) +1 On Mon, Jun 13, 2022 at 5:07 PM Holden Karau wrote: > > +1 > > On Mon, Jun 13, 2022 at 4:51 PM Yuming Wang wrote: >>

Re: [VOTE] Release Spark 3.3.0 (RC5)

2022-06-07 Thread Cheng Su
+1 (non-binding). Built and ran some internal test for Spark SQL. Thanks, Cheng Su From: L. C. Hsieh Date: Tuesday, June 7, 2022 at 1:23 PM To: dev Subject: Re: [VOTE] Release Spark 3.3.0 (RC5) +1 Liang-Chi On Tue, Jun 7, 2022 at 1:03 PM Gengliang Wang wrote: > > +1 (non-b

[DISCUSS] Deprecate legacy file naming functions in FileCommitProtocol

2022-01-24 Thread Cheng Su
y functions mentioned above from our codebase. The PR to add @deprecate annotation is ready for review https://github.com/apache/spark/pull/35311 . Feel free to comment here or on the PR for further discussion. Thanks, Cheng Su (@c21)

Re: [VOTE] Release Spark 3.2.1 (RC2)

2022-01-24 Thread Cheng Su
+1 (non-binding) Cheng Su From: Chao Sun Date: Monday, January 24, 2022 at 2:10 PM To: Michael Heuer Cc: dev Subject: Re: [VOTE] Release Spark 3.2.1 (RC2) +1 (non-binding) On Mon, Jan 24, 2022 at 6:32 AM Michael Heuer mailto:heue...@gmail.com>> wrote: +1 (non-binding) michael

Re: [VOTE] Release Spark 3.2.1 (RC1)

2022-01-11 Thread Cheng Su
+1 (non-binding). Checked commit history and ran some local tests. Thanks, Cheng Su From: Qian Sun Date: Tuesday, January 11, 2022 at 7:55 PM To: huaxin gao Cc: dev Subject: Re: [VOTE] Release Spark 3.2.1 (RC1) +1 Looks good. All integration tests passed. Qian 2022年1月11日 上午2:09,huaxin gao

Re: [VOTE] SPIP: Storage Partitioned Join for Data Source V2

2021-10-29 Thread Cheng Su
+1 Thanks, Cheng Su From: Holden Karau Date: Friday, October 29, 2021 at 12:41 PM To: DB Tsai Cc: Dongjoon Hyun , Ryan Blue , dev , huaxin gao Subject: Re: [VOTE] SPIP: Storage Partitioned Join for Data Source V2 +1 On Fri, Oct 29, 2021 at 3:07 PM DB Tsai wrote: +1 DB Tsai | https

Re: [DISCUSS] SPIP: Storage Partitioned Join for Data Source V2

2021-10-26 Thread Cheng Su
want to bring to our attention that it would be great to consider aggregate as well when doing this proposal. 1. Any major use cases in mind except Hive bucketed table? Just curious if there’s any other use cases we are targeting as part of SPIP. Thanks, Cheng Su From: Ryan Blue

Re: [VOTE] Release Spark 3.2.0 (RC7)

2021-10-08 Thread Cheng Su
+1 (non-binding). Thanks, Cheng Su From: Reynold Xin Date: Thursday, October 7, 2021 at 11:57 PM To: Yuming Wang Cc: Dongjoon Hyun , 郑瑞峰 , Sean Owen , Gengliang Wang , dev Subject: Re: [VOTE] Release Spark 3.2.0 (RC7) +1 On Thu, Oct 07, 2021 at 11:54 PM, Yuming Wang mailto:wgy

Re: Should AggregationIterator.initializeBuffer be moved down to SortBasedAggregationIterator?

2021-05-26 Thread Cheng Su
is refactored away and no longer exists. Cheng Su From: Jacek Laskowski Date: Tuesday, May 25, 2021 at 6:35 AM To: dev Subject: Should AggregationIterator.initializeBuffer be moved down to SortBasedAggregationIterator? Hi, Just found out that the only purpose

Re: [VOTE] Release Spark 3.1.2 (RC1)

2021-05-25 Thread Cheng Su
+1 (non-binding) Checked the related commits in commit history manually. Thanks! Cheng Su From: Takeshi Yamamuro Date: Tuesday, May 25, 2021 at 4:47 PM To: Dongjoon Hyun , dev Subject: Re: [VOTE] Release Spark 3.1.2 (RC1) +1 (non-binding) I ran the tests, checked the related jira tickets

Re: Apache Spark 3.1.2 Release?

2021-05-17 Thread Cheng Su
+1 for a new release, thanks Dongjoon! Cheng Su On 5/17/21, 2:44 PM, "Liang-Chi Hsieh" wrote: +1 sounds good. Thanks Dongjoon for volunteering on this! Liang-Chi Dongjoon Hyun-2 wrote > Hi, All. > > Since Apache Spark 3.1.1 tag creation (Feb

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2021-04-27 Thread Cheng Su
Hi Ximo, sorry for a super late reply. Just FYI https://github.com/apache/spark/pull/32198 is merged in master, and you can try it out if building with latest master. Let me know if it works or not for your case, thanks. Cheng Su From: Cheng Su Date: Monday, September 21, 2020 at 8:54 AM

Re: modifying spark's optimizer for research

2021-04-22 Thread Cheng Su
u have done? So this can help give people some basic ideas and provide help more specifically. Thanks, Cheng Su From: Walter Cai Date: Wednesday, April 21, 2021 at 6:09 PM To: "dev@spark.apache.org" Subject: modifying spark's optimizer for research Hi, I'm Walter, a PhD student a

Re: Welcoming six new Apache Spark committers

2021-03-26 Thread Cheng Su
Congratulations to all! From: huaxin gao Date: Friday, March 26, 2021 at 4:33 PM To: Yuming Wang Cc: Takeshi Yamamuro , Spark dev list Subject: Re: Welcoming six new Apache Spark committers Congratulations to you all!! On Fri, Mar 26, 2021 at 4:22 PM Yuming Wang mailto:wgy...@gmail.com>>

Re: Apache Spark 3.2 Expectation

2021-02-26 Thread Cheng Su
ideally want to finish the feature in 3.2. For most of features here, we already developed internally and rolled out to production. Thanks, Cheng Su From: Dongjoon Hyun Date: Friday, February 26, 2021 at 4:06 PM To: Hyukjin Kwon Cc: huaxin gao , Xiao Li , dev Subject: Re: Apache Spark 3.2

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-23 Thread Cheng Su
+1 (non-binding) From: Takeshi Yamamuro Date: Tuesday, February 23, 2021 at 3:30 PM To: Hyukjin Kwon , dev Subject: Re: [VOTE] Release Spark 3.1.1 (RC3) +1 On Wed, Feb 24, 2021 at 2:07 AM John Zhuge mailto:jzh...@apache.org>> wrote: +1 (non-binding) On Mon, Feb 22, 2021 at 10:19 PM

Re: [VOTE] Release Spark 3.1.1 (RC2)

2021-02-08 Thread Cheng Su
+1 for this release candidate. Thanks, Cheng Su From: 郑瑞峰 Date: Monday, February 8, 2021 at 10:58 PM To: Gengliang Wang , Sean Owen Cc: gurwls223 , Yuming Wang , dev Subject: 回复: [VOTE] Release Spark 3.1.1 (RC2) +1 (non-binding) Thank you, Hyukjin -- 原始邮件

Re: [DISCUSS] Add RocksDB StateStore

2021-02-08 Thread Cheng Su
+1 for (2) adding to external module. I think this feature is useful and popular in practice, and option 2 is not conflict with previous concern for dependency. Thanks, Cheng Su From: Dongjoon Hyun Date: Monday, February 8, 2021 at 10:39 AM To: Jacek Laskowski Cc: Liang-Chi Hsieh , dev

[SS] full outer stream-stream join

2020-11-20 Thread real-cheng-su
blocker to add FULL OUTER stream-stream join? Asking in dev mailing list in case we miss anything besides PR review participation, thanks. Cheng Su -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-21 Thread Cheng Su
Hi Ximo, sorry for delaying, was busy with other stuff. I will raise a PR in this week, let me ping you for review to leverage your help, thanks. Cheng Su On Sep 21, 2020, at 8:16 AM, XIMO GUANTER GONZALBEZ wrote:  Hi Cheng, I think there still isn’t a PR for this, right? Do you need any

Re: [DISCUSS] Time to evaluate "continuous mode" in SS?

2020-09-16 Thread Cheng Su
. Thanks, Cheng Su From: Jungtaek Lim Date: Tuesday, September 15, 2020 at 5:04 PM To: Joseph Torres Cc: Sean Owen , dev Subject: Re: [DISCUSS] Time to evaluate "continuous mode" in SS? Yeah I realized there's a proposal for push-based shuffle, and I agree that may unblock the architect

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-09 Thread Cheng Su
no objection. Thanks, Cheng Su From: XIMO GUANTER GONZALBEZ Date: Sunday, September 6, 2020 at 10:55 PM To: Cheng Su , Reynold Xin Cc: Spark Dev List Subject: RE: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter > 1.If number of writers exceeds a pre-defi

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-05 Thread Cheng Su
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcOutputWriter.scala#L54 and https://github.com/apache/orc/blob/master/java/mapreduce/src/java/org/apache/orc/mapreduce/OrcMapreduceRecordWriter.java#L53-L56 Cheng Su From: kalyan

Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter

2020-09-04 Thread Cheng Su
to get more opinion on this. Thanks. Cheng Su From: Reynold Xin Date: Friday, September 4, 2020 at 10:33 AM To: XIMO GUANTER GONZALBEZ Cc: Spark Dev List Subject: Re: Avoiding unnnecessary sort in FileFormatWriter/DynamicPartitionDataWriter The issue is memory overhead. Writing files create