Re: [VOTE] Release Spark 3.5.7 (RC1)

+1 (non-binding) Thanks for driving the release! On Fri, Sep 19, 2025 at 5:49 PM Max Gekk wrote: > +1 > > On Thu, Sep 18, 2025 at 7:09 PM Kousuke Saruta wrote: > >> +1 >> >> 2025年9月19日(金) 1:03 huaxin gao : >> >>> +1 >>> Thanks Peter for driving the release! >>> >>> Huaxin >>> >>> On Thu, Sep 18

Re: [VOTE] SPIP: Add llms.txt files to Spark Documentation

(I missed to clarify, my +1 is non-binding, just to make easier to count) On Fri, Sep 19, 2025 at 8:44 AM Jungtaek Lim wrote: > +1 this sounds promising given the trends of reliance in AI. > > On Wed, Sep 17, 2025 at 4:04 PM Kousuke Saruta wrote: > >> +1 >> >> 202

Re: [VOTE] SPIP: Add llms.txt files to Spark Documentation

+1 this sounds promising given the trends of reliance in AI. On Wed, Sep 17, 2025 at 4:04 PM Kousuke Saruta wrote: > +1 > > 2025年9月17日(水) 15:17 Dongjoon Hyun : > >> +1 >> >> Dongjoon >> >> On 2025/09/16 02:30:33 Jules Damji wrote: >> > + 1 (non-binding) >> > — >> > Sent from my iPhone >> > Pardo

Re: [DISCUSS] Release Apache Spark 3.5.7

+1 sounds like a plan. On Wed, Sep 10, 2025 at 6:02 PM Kousuke Saruta wrote: > +1 > > 2025年9月10日(水) 17:44 Max Gekk : > >> +1 >> >> On Wed, Sep 10, 2025 at 8:13 AM Shaoyun Chen wrote: >> >>> +1 >>> >>> SPARK-46941[1] also fixed an issue with incorrect results. >>> >>> 1. https://issues.apache.or

Re: [VOTE] Release Spark 4.0.1 (RC1)

+1 (non-binding) Thanks, Jungtaek Lim (HeartSaVioR) On Wed, Sep 3, 2025 at 9:16 AM kazuyuki tanimura wrote: > > +1 (non-binding) > > Thanks > Kazu > > > On Sep 2, 2025, at 2:17 PM, Holden Karau wrote: > > +1 > > On Tue, Sep 2, 2025 at 11:56 AM Rozov,

Re: Apache Spark 4.0.1 ?

+1 I was thinking of doing this since we had several major fixes on the new API transformWithState, but forgot about it. Thanks for raising this! On Thu, Aug 28, 2025 at 6:36 AM Jules Damji wrote: > +1 non-binding > — > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 26, 2025, a

Re: [VOTE] Release Spark 4.1.0-preview1 (RC1)

ient Error: Not Found for url: > https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz > for URL > https://dist.apache.org/repos/dist/dev/spark/v4.1.0-preview1-rc1-bin/pyspark-4.1.0-preview1.tar.gz > > Can you check? > > > Ken

Re: [VOTE] Release Spark 4.1.0-preview1 (RC1)

+1 (non-binding) Let's give it a try! On Thu, Jul 10, 2025 at 12:24 AM Anton Okolnychyi wrote: > +1 (non-binding) > > On Wed, Jul 9, 2025 at 8:07 AM Max Gekk wrote: > >> +1 >> >> On Wed, Jul 9, 2025 at 4:04 PM Sandy Ryza >> wrote: >> >>> +1 (non-binding) >>> >>> On Wed, Jul 9, 2025 at 6:57 AM

Re: [VOTE] SPIP: Monthly preview release

+1 (non-binding) It would be great outcome regardless of whether this effort would be successful or not. We may even find the way to do official release more often, which would be a huge win. 2025년 7월 4일 (금) 오전 3:13, Szehon Ho 님이 작성: > +1 (non-binding) > > Thanks for the proposal, hope one day t

Re: [DISCUSS] SPIP: Monthly preview release

IMHO it's probably better to avoid trying to figure out a workaround if that is the "policy" of ASF. Btw, I guess the main point here is, what is the definition of "acceptable" release. For example, do we feel overloaded to just verify the signature per month? I think we wouldn't. The reason the t

Re: [DISCUSS] SPIP: Monthly preview release

do we intend to try it out while it's still in development? > > We won't apply any compatibility policies here I believe. Should be > treated as the same as our past preview releases which does not guarantee > anything. > > > On Wed, 2 Jul 2025 at 11:54, Jungtaek Lim

Re: [DISCUSS] SPIP: Monthly preview release

Thanks for the proposal. The direction is awesome - we have such a long interval between minor releases and this would help us address some of the issues from the long release cadence. I'd like to understand a couple of things. 1. Since we "release" the preview, we will go through the VOTE proces

Re: [DISCUSS] Dropping LevelDB support in Spark

tions of data reconstruction or re-parsing > have already existed. > > On 2025/06/09 01:08:05 Jungtaek Lim wrote: > > Thanks for the valuable input. > > > > I think it's more about the case where upgrading would surprise the end > > users. If we simply remo

Re: [DISCUSS] Dropping LevelDB support in Spark

S directory, multiple threads re-parsing to rebuild listing.rdb > takes ~15mins. > > Thanks, > Cheng Pan > > > > On Jun 6, 2025, at 15:36, Jungtaek Lim > wrote: > > IMHO, it's probably dependent on how long the rewrite will take, from > reading the event log.

Re: [DISCUSS] Dropping LevelDB support in Spark

rsonally, I haven't found a > proper way to migrate data from LevelDB to RocksDB, as their storage > structures are different. Should we wait until a reasonable migration > solution becomes available before moving forward with this? > > Jungtaek Lim 于2025年5月28日周三 15:41写道： > &

Re: [DISCUSS] Automation of RC email

nt and password in GitHub Secrets. So I plan to use that to send > an email. > I will probably add a note that the email was auto generated .. > > On Fri, 6 Jun 2025 at 11:37, Jungtaek Lim > wrote: > >> One question: is it possible for the automation to send the mail on >> b

Re: [DISCUSS] Automation of RC email

One question: is it possible for the automation to send the mail on behalf of release manager? Or will we simply send the mail as specific mail account (mostly dedicated one for automated)? Maybe latter doesn’t even matter, but it might be less clear about who is driving the release, from automate

Re: [VOTE] Release Apache Spark Connect Swift Client 0.3.0 (RC1)

+1 (non-binding) On Tue, Jun 3, 2025 at 12:18 AM Denny Lee wrote: > +1 (non-binding) > > > On Mon, Jun 2, 2025 at 07:44 Sandy Ryza > wrote: > >> +1 (non-binding) >> >> On Mon, Jun 2, 2025 at 7:20 AM Dongjoon Hyun wrote: >> >>> +1 >>> >>> Dongjoon >>> >>> On 2025/06/02 13:13:45 "Rozov, Vlad" wr

Re: [VOTE] Release Apache Spark K8s Operator 0.3.0 (RC1)

+1 (non-binding) On Mon, Jun 2, 2025 at 11:18 PM Dongjoon Hyun wrote: > +1 > > Dongjoon > > On 2025/06/02 13:12:46 "Rozov, Vlad" wrote: > > +1 (non-binding) > > > > Thank you, > > > > Vlad > > > > On Jun 1, 2025, at 7:44 PM, Liu Cao wrote: > > > > +1 (non-binding) > > > > On Sun, Jun 1, 2025 at

Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

+1 (non-binding) On Mon, Jun 2, 2025 at 11:09 PM Wenchen Fan wrote: > +1 > > On Mon, Jun 2, 2025 at 8:55 PM Peter Toth wrote: > >> +1 >> >> On Mon, Jun 2, 2025 at 2:33 PM xianjin wrote: >> >>> +1. >>> Sent from my iPhone >>> >>> On Jun 2, 2025, at 12:50 PM, DB Tsai wrote: >>> >>> +1 looking

Re: [DISCUSS] SPIP: Real-Time Mode in Apache Spark Structured Streaming

t;> need this >>>>>>>>> > >>>>>>>> Spark Structured Streaming proposal. >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> The proposal is prec

Re: [DISCUSS][MINOR] Fix broken link in spark-website for SS Programming Guide

I’m +1 to fix this in website for 4.0.0 immediately. I got some inputs about this and they were unable to figure out the correct page url. I’m mostly sure it will happen to many users as well. We could also fix this in the next maintenance release, but since we just released Apache Spark 4.0.0, i

Re: [PSA] GitHub Actions for releasing Apache Spark

Awesome work! I also can't see the screenshots. Maybe we'd need to have this in the release guide page in Apache Spark website anyway? On Thu, May 29, 2025 at 4:24 PM Yuanjian Li wrote: > Thanks, Hyukjin—this made the release 10 times easier (maybe even more)! > > (Not sure if it’s just me, but

Re: [DISCUSS] Dropping LevelDB support in Spark

Thanks for initiating this. I wonder if we don't have any compatibility issue on every component - SS area does not have an issue, but I don't quite remember if the history server would be OK with this. What is the story of the migration if they had been using leveldb? I guess it could be probably

Re: [VOTE] Release Spark 3.5.6 (RC1)

+1 (non-binding) pending the discussion on CVEs (not the performance improvement) in the current version of Apache Parquet. On Tue, May 27, 2025 at 11:19 AM L. C. Hsieh wrote: > +1 > > On Mon, May 26, 2025 at 6:51 PM Wenchen Fan wrote: > > > > +1. When this release is out, let's also update the

Re: [VOTE] Release Spark 3.5.6 (RC1)

Let's not couple the performance improvement here from upgrading the library in the bugfix version of Apache Spark. The major focus on bugfix versions should be something users can upgrade to gain more safety and reliability - upgrading and downgrading Spark could be painful on the setup env and it

Re: [VOTE] New Spark Connect Client Repository for Rust

s and releases. >> >> CC’ing @Renjie Liu & @sjrus...@gmail.com >> to keep me honest here and to add any details I may >> have missed from other discussions. >> >> Jungtaek Lim 于2025年5月19日周一 15:41写道： >> >>> Just one question - who will be in charge o

Re: [VOTE] Release Spark 4.0.0 (RC7)

+1 (non-binding) On Tue, May 20, 2025 at 8:47 AM Ruifeng Zheng wrote: > +1 > > On Tue, May 20, 2025 at 7:04 AM Hyukjin Kwon wrote: > >> +1 >> >> On Mon, 19 May 2025 at 21:27, Wenchen Fan wrote: >> >>> Same as before, I'll start with my own +1. >>> >>> On Mon, May 19, 2025 at 8:25 PM Wenchen Fa

Re: [VOTE] New Spark Connect Client Repository for Rust

Just one question - who will be in charge of maintaining the new repo? The committers and PMC members could help on merging, but who will author the code and who will review? Also how many active committers + PMC members are familiar with Rust language? On Tue, May 20, 2025 at 4:15 AM Yuanjian Li

Re: [VOTE] Release Spark 4.0.0 (RC6)

UPDATE: The fix is merged. @Adam Binford if you could help verifying this in latest branch-4.0, that would be great. Thanks for reporting the issue! On Sat, May 17, 2025 at 3:05 AM Anish Shrigondekar wrote: > Hi Adam, > > Thanks for reporting the issue. @Eric Marnadi > has a PR for the fix her

Re: [VOTE] Release Spark 4.0.0 (RC6)

+1 (non-binding) On Wed, May 14, 2025 at 7:29 AM Wenchen Fan wrote: > Please vote on releasing the following candidate as Apache Spark version > 4.0.0. > > The vote is open until May 16 (PST) and passes if a majority +1 PMC votes > are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this

Re: [VOTE] Release Spark 4.0.0 (RC5)

+1 (non-binding) Thanks Wenchen for driving the release! On Tue, May 13, 2025 at 11:35 AM Yang Jie wrote: > +1, thank you Wenchen > > On 2025/05/13 02:11:02 "Rozov, Vlad" wrote: > > +1 (non-binding) > > > > Thank you, > > > > Vlad > > > > On May 12, 2025, at 5:44 PM, huaxin gao wrote: > > > >

Re: [VOTE] Release Apache Spark Connect Swift Client 0.1.0 (RC1)

+1 (non-binding) Nice addition on Spark Connect! On Tue, May 6, 2025 at 5:47 PM Peter Toth wrote: > +1 > > On Tue, May 6, 2025 at 9:59 AM Yang Jie wrote: > >> +1, A big thank you to Dongjoon for all the hard work you've put into >> this! >> >> On 2025/05/05 18:19:33 DB Tsai wrote: >> > +1, it’s

Re: [VOTE][RESULT] SPIP: Declarative Pipelines

ust (*) > Peter Toth > L.C. Hsieh (*) > Chao Sun (*) > Denny Lee > Martin Grund > Gengliang Wang (*) > Mich Talebzadeh > Mark Hamstra (*) > Holden Karau (*) > Hyukjin Kwon (*) > Jungtaek Lim > Anton Okolnychyi > Szehon Ho > Cheng Pan > Yang Jie (*) > Kent Yao

Re: [DISCUSS] SPIP: Declarative Pipelines

+1 looking forward to seeing this make progress! On Wed, Apr 9, 2025 at 11:32 AM Yang Jie wrote: > +1 > > On 2025/04/09 01:07:57 Hyukjin Kwon wrote: > > +1 > > > > I am actually pretty excited to have this. Happy to see this being > proposed. > > > > On Wed, 9 Apr 2025 at 01:55, Chao Sun wrote:

Re: [VOTE] SPIP: Declarative Pipelines

Btw who is going to shephard this SPIP? I don't see this in the doc/JIRA/discussion thread. I understand there are PMC members in the author list, but probably good to be explicit about "who" is shepherding this SPIP. On Wed, Apr 9, 2025 at 11:22 PM Sandy Ryza wrote: > We started to get some vot

Re: [VOTE] SPIP: Declarative Pipelines

+1 (non-binding) On Wed, Apr 9, 2025 at 11:22 PM Sandy Ryza wrote: > We started to get some votes on the discussion thread, so I'd like to move > to a formal vote on adding support for declarative pipelines. > > *Discussion thread: * > https://lists.apache.org/thread/lsv8f829ps0bog41fjoqc45xk7m5

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Maybe I will just update the VOTE result, since the rationale of this VOTE, and the VOTE result is public. On Tue, Mar 18, 2025 at 10:00 PM Jungtaek Lim wrote: > I'm definitely OK with modifying migration logic to exclude "databricks" > if people think it is better. I

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

+1 on explanation that it is not happening only to Vlad but always happening as a normal process. Vlad, if we are very strict about ASF voting policy, we have to have three +1s without -1 to merge the code change. I don't think the major projects in ASF follow it - instead, they (including Spark)

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

> introduce new tests that do not skip if jar does not exist. Such test will >>>>> break only during release. >>>>> >>>>> IMO, it is necessary to see if the source code for test jars is >>>>> available or can be reconstructed. If not, i

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

; They aren't included in binary releases in any event so removal on every >> source release should work. >> >> On Tue, 25 Mar 2025 at 10:51, Jungtaek Lim >> wrote: >> >>> Let's make this very clear - do we not have a source code to build a >>>

Re: [DISCUSS] SPARK-51318: Remove `jar` files from Apache Spark repository and disable affected tests

o they can help report any test failures. That said, >>> since these tests are quite old and stable, failures are unlikely. >>> >>> Thanks, >>> Wenchen >>> >>> On Thu, Mar 13, 2025 at 12:15 AM Rozov, Vlad >>> wrote: >>> &g

Re: [VOTE] SPIP: Constraints in DSv2

+1 (non-binding) Thanks for initiating this! On Sun, Mar 23, 2025 at 3:45 AM serge rielau.com wrote: > +1 (non binding) > > On Mar 21, 2025, at 12:52 PM, Jules Damji wrote: > > +1 (non-binding) > — > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Mar 21, 2025, at 11:47 AM, Anton O

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

ut IMHO it should be OK as long as we address this back in Spark 4.0.1 and higher. On Sun, Mar 16, 2025 at 5:24 PM Jungtaek Lim wrote: > Yeah... maybe 1 is simpler if there is no side effect, and probably the > latter pattern we have is long enough to figure out aliases without full > te

[VOTE][RESULT][FINAL] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

The vote passes with 8 +1s (4 binding +1s) and no -1s. Thanks to all who helped with the vote! (* = binding) +1: - Sean R. Owen * - Jungtaek Lim - Nicholas Chammas - Wenchen Fan * - Adam Binford - Russell Jurney - Yang Jie * - Mridul Muralidharan * NOTE1: The community discussed the type of vote

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

ternative is proposed. > > > On Mon, Mar 17, 2025 at 3:37 PM Hyukjin Kwon wrote: > >> The vote passes with 5 +1s (4 binding +1s) and 3 -1s (3 binding -1s). >> >> (* = binding) >> +1: >> - Mark Hamstra * >> - Jungtaek Lim >> - Wenchen Fan * >> - Reynold Xin * >> - Yuanjian Li * >> >> -1: >> - Holden Karau * >> - Hyukjin Kwon * >> - Dongjoon Hyun * >> >> Thanks. >> >>

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

hat decisions are being swayed by affiliations rather than >> technical arguments, that would set a concerning precedent. We should >> strive to engage in discussions constructively and focus on what is best >> for the project and its users. >> >> Jungtaek L

[PROPOSE] Fix the procedural steps on the incorrect config fixes

r -> 4.0 -> 3.5. I haven't heard any strong reason to not follow the normal procedure, so I'd rather say this is what we should have done, rather than what happened now. Looking forward to hearing the voice. Thanks, Jungtaek Lim (HeartSaVioR)

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

The problem you are mentioning is arguably resolvable if we leave a config name as it is, and add withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan"). Let's not nitpick just from reverting the PR. We have to revert the PR "semantically". Btw, from what I understa

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

ision is up to the > community. > > On Mon, Mar 17, 2025 at 10:19 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> OK, let's be super honest. >> >> Again, I think you agree that *"both" proposals are "technically" >> c

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

in code, otherwise my +1 cannot merge the PR), and revert unless anyone objects with valid reason. On Mon, Mar 17, 2025 at 12:31 PM Jungtaek Lim wrote: > Holden, I think I have some workaround like I posted in dev@ (link > <https://lists.apache.org/thread/xsq58800smtc5xo15kfzyj5kfw5y

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

sensus but it's clear that this breaking change PR has failed to achieve >>> consensus. >>> >>> I hope we now have a clear foundation for discussing solutions. As it >>> stands, the misnamed configuration will be released in 4.0.0. I like >>> Jungtaek’s prop

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Holden and Dongjoon, Let me make this vote super simple. I never got the answer from Dongjoon about this question. This is super important because if he's casting veto "to block", it is a strong indication that he was intended to play with me, which I am seriously considering escalating the proble

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

ighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > > On Sun, Mar 16, 2025 at 6:28 PM Jun

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Holden, I believe you should already know "both" approaches are "technically" correct. It's not about which one you have a preference for, no, this VOTE is not intended to extend the debate. Again, what you are encouraged to do here is, not exposing your preference of two approaches, but exposing

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I was trying hard to stay away from this VOTE, but I should have reminded everyone about "what" we are going to VOTE. Dongjoon casted a VETO against code change VOTE. That VETO is described in ASF Voting Process page: https://www.apache.org/foundation/voting.html#Veto A -1 vote by a qualified vo

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

-- this vote thread is very > clearly about VETOing the change for Spark 4. > > I think that accusing each-other of acting in bad faith is unproductive to > resolving this dispute. > > On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim > wrote: > >> Holden and Dongjoon, >&

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

\u0072\u0069\u0063\u006b\u0073 instead of > “databricks” might also be an option if including “databricks” in the code > is believed to be so offensive. > > > On Sun, Mar 16, 2025 at 12:52 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Hi dev, >>

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Apologize for quick fix, This won't go to VOTE "unless" there are "valid" objections. The message was sent accidentally before I double checked. On Sun, Mar 16, 2025 at 5:00 PM Jungtaek Lim wrote: > Just to clarify: This won't go to VOTE as long as there ar

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Just to clarify: This won't go to VOTE as long as there are "valid" objections. Let's not waste more time on our end. On Sun, Mar 16, 2025 at 4:52 PM Jungtaek Lim wrote: > Hi dev, > > I'm really tired of the discussion which does not move forward because the

[DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

OK to allow a bit of indirect checking on the logic to just remove out the debate, I'm happy to do that. It's obvious that we can just leave this migration logic much longer than what I was proposing, because we eliminate the main concern. I'm open to hear about support and objections. Thanks, Jungtaek Lim (HeartSaVioR)

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

said. Please stop it. On Sun, Mar 16, 2025 at 3:14 PM Jungtaek Lim wrote: > Dongjoon, > > I'm now OK with whatever you think, but I argue your vote is technically > moot since it's about your vote justification, and I have no binding vote > to counter you. Let's be

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Dongjoon, I'm now OK with whatever you think, but I argue your vote is technically moot since it's about your vote justification, and I have no binding vote to counter you. Let's be fair. On Sun, Mar 16, 2025 at 3:07 PM Dongjoon Hyun wrote: > Thank you for focusing on this, Mark. > > I also agr

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

osed to be there in the first place, especially in a major > release like Spark 4? > > Rather than going through the entire email chain, I’d appreciate a quick > confirmation: I support keeping it as is in Spark 3.5 and removing it in > Spark 4. Would that be considered a -1 or a +1

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

it is a code change vote, a -1 can be seen as a veto, blocking >>>the change unless specific conditions are met (like the PMC overriding >>> the >>>veto). >>>2. If it is just a procedural vote, a -1 might simply be a >>>dissenting vo

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Slight correction (to avoid nitpick): “vote” in the last sentence is “veto”. I think community will understand correctly, but just to avoid any voice making this to be sidetracked. 2025년 3월 16일 (일) 오전 11:20, Jungtaek Lim 님이 작성: > For sure, I’m +1 (non-binding). > > I believe I don’

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

an be seen as a veto, blocking >>>the change unless specific conditions are met (like the PMC overriding >>> the >>>veto). >>>2. If it is just a procedural vote, a -1 might simply be a >>>dissenting vote, *not necessarily carrying the powe

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

For sure, I’m +1 (non-binding). I believe I don’t need to explain more and I spent whole weekend and we have a history about my justification in the history of mailing list. I’m open to summary my justification again, but as a tl;dr, I have a strong evidence that he knew we never had a consensus

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

osal, you should have just said "we were already decided" and you had to give the evidence. I haven't heard any, so I had to initiate DISCUSS. On Sat, Mar 15, 2025 at 11:18 PM Jungtaek Lim wrote: > > according to the ASF process, the Apache Spark community made the > conc

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

have confirmed that Dongjoon's -1 is not a veto. I think the VOTE result is correct as it is. I'll proceed with the next steps. On Fri, Mar 14, 2025 at 11:19 AM Jungtaek Lim wrote: > The vote passes with 7 +1s (3 binding +1s) and 1 -1s (1 binding -1s). > Thanks to all who helped with th

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

Starting from my +1 (non-binding). In addition, I propose to retain migration logic till Spark 4.1.x and remove it in Spark 4.2.0. On Mon, Mar 10, 2025 at 9:44 PM Jungtaek Lim wrote: > Hi dev, > > Please vote to retain migration logic of incorrect `spark.databricks.*` > configurat

[VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

The vote passes with 7 +1s (3 binding +1s) and 1 -1s (1 binding -1s). Thanks to all who helped with the vote! I'm going to make a code change in branch-4.0 quickly so that we don't have to trigger another RC for Spark 4.0.0 just because of this. (* = binding) +1: - Sean R. Owen * - Ju

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

ppening in private@, as private@ is not meant to be used for discussion which could have been done in public. On Sat, Mar 15, 2025 at 11:21 PM Jungtaek Lim wrote: > small missing on link: > > 4. I claimed I wanted to proceed with migration logic for branch-4.0 PR, > and hadn't go

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

interpretation, according to the ASF process, the > Apache Spark community made the conclusion to unblock the Apache Spark > 4.0.0 release with the AS-IS code with the improved Spark 4.0 migration > guide because I provided a technical justification for my vote via the > concrete alt

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

are here to do the opposite, saving users' life. On Sat, Mar 15, 2025 at 9:51 PM Jungtaek Lim wrote: > > That's the reason why you proposed the vote procedure and we agreed. > > Didn’t you see the part “we agreed”? Who is we in the context? > > I don’t think he answered my

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

this feels like you are trying to manipulate the vote > procedure by misrepresenting Dongjoon, and you are quickly losing my > confidence in your ability to administer a fair voting procedure. > > I still consider the proposal to be vetoed. > > > On Fri, Mar 14, 2025 at 6

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

th the AS-IS code with the improved Spark 4.0 migration > guide because I provided a technical justification for my vote via the > concrete alternative based on the existing Spark 3.5.5, AS-IS code base, > and the suggested better migration guide way in order to eliminate the > affected streamin

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

t; >>> >>> On Thu, Mar 13, 2025 at 9:35 PM Mark Hamstra >>> wrote: >>> >>>> Absolutely not! >>>> >>>> This is clearly a vote on a code change, not on a procedural issue or >>>> a package release. The code change ha

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

01 PM Jungtaek Lim wrote: > If we were not intended to block the VOTE but just to express the > disagreement, please say "-1" instead of representing it as "veto". When > saying veto, you intend to kill the process unless you are not persuaded or > you are not

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

14, 2025 at 6:27 PM Jungtaek Lim wrote: > Thanks for the update. > > Though I have to clarify that "What all of us agree on is that the > previous code base is okay." is not true. > > Wenchen summarized what happened in other thread which I think it's more > pr

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

rk, imo this is a qualified veto. >>> We should give Dongjoon the opportunity to give his clarification, if >>> any. >>> >>> I do realize this delays the RC process, but this deserves to be looked >>> into carefully. >>> >>> Thanks, >&

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

alid. Please stop asserting irrelevant timeframes > and extraneous issues. > > The end of this week appears more than adequate and fair to me. > > On Thu, Mar 13, 2025 at 9:46 PM Jungtaek Lim > wrote: > > > > I love to hear what is the reasonable time here. If you say 1

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

shouldn't be diminished or ignored. > > Please stop trying to claim control over the process, and let's > patiently await Dongjoon's clarification of his technical issues with > the proposal -- or his failure to do so. > > On Thu, Mar 13, 2025 at 10:04 PM Jungtaek Li

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

to to the proposal. The > technical justification he gave was challenged or asserted to be > invalid. We should either see his response to the challenge or at > least wait a reasonable time for that response before declaring the > veto invalid. > > On Thu, Mar 13, 2025 at 8:43 PM Jung

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

ance to explain > why there is technical justification for it. > > On Thu, Mar 13, 2025 at 7:21 PM Jungtaek Lim > wrote: > > > > The vote passes with 7 +1s (3 binding +1s) and 1 -1s (1 binding -1s). > > Thanks to all who helped with the vote! > > > > I&#

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

There > is little to no harm in giving him some more time to respond to the > recent challenge to his veto. > > > On Thu, Mar 13, 2025 at 8:17 PM Jungtaek Lim > wrote: > > > > Actually, this has been initially triggered from 3 weeks ago, not just a > week we have spent.

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

affects >> performance, etc. )." It has been less than 24 hours since Dongjoon's >> veto was called into question. He should be given a chance to explain >> why there is technical justification for it. >> >> On Thu, Mar 13, 2025 at 7:21 PM Jungtaek Lim >>

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

. I don't want to block the release because of the above. So, let's change the current codebase the way we discussed and voted here. Reverting this decision should require another VOTE. Thanks to everyone who voted! On Thu, Mar 13, 2025 at 4:54 PM Jungtaek Lim wrote: > Thanks t

Re: [Discuss] SPIP: Support NanoSecond Timestamps

Hi, would you mind allowing comments on the doc? Thanks! On Fri, Mar 14, 2025 at 8:50 AM Qi Tan wrote: > Hello everybody, > > I would like to start a discussion on SPARK-50532 > to enable Spark to > support nanoseconds. Here attached the spip d

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

de > to the next maintenance release before moving to a new feature or major > release? > > Thanks, > Dongjoon. > > > On Thu, Mar 13, 2025 at 12:58 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Thanks to everyone who participated and voted! &g

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

ctively saying +1 to his proposal, which makes zero sense to me. On Fri, Mar 14, 2025 at 5:57 AM Jungtaek Lim wrote: > I do believe there are two ways of considering -1 vote. Valid -1 votes are > not restricted to technical objections, but in that case, it must not be > considered as veto,

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

to > upgrade to the next maintenance release before moving to a new feature or > major release? > >> > >> Thanks, > >> Dongjoon. > >> > >> > >> On Thu, Mar 13, 2025 at 12:58 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote:

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

> > > > On Wed, Mar 12, 2025 at 11:18 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> > > wrote: > > > > > Russell, > > > > > > Of course, we hear people' voices who aren't having binding votes as > well. > > > Person

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

ts VOTE, so I can do whatever simplest and easiest and fastest way to just solve the issue, including just cancelling the VOTE if we have consensus. Hope we avoid spending time on this longer. Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Mar 10, 2025 at 11:53 PM Dongjoon Hyun wrote: > -1 because

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

gt;>>> of >>>> > removing them because we understand the challenges of upgrading Spark >>>> > versions. Our goal has always been to make upgrades easier, even if it >>>> > means carrying some technical debt. I don’t think we want to chang

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

x the checkpoint. Hope this clarifies your question. Thanks, Jungtaek Lim (HeartSaVioR) On Mon, Mar 10, 2025 at 10:06 PM Adam Binford wrote: > As someone who has a lot of streams that have been restarted with 3.5.4, I > would prefer not to have to restart everything with 3.5.5 but it'

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

you are maintaining streaming queries in your daily job. Thanks! On Tue, Mar 11, 2025 at 12:18 PM Jungtaek Lim wrote: > Thanks for looking into the issue in depth. What you described is right. > > I also understand the concern why we keep the buggy behavior, but the QO > is

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

bvious that this Databricks' mistake already causes a huge >> communication cost in the Apache Spark community and is suggesting a burden >> to enforce us to handle at least two more PRs at 4.0.0 and 4.1.0. >> > >> > Given that, I don't think >> > - T

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

t if > this bug is causing breaking problems for your query you probably would > need to restart it from scratch anyway. > > At the end of the day there's probably no reason not to include 4 lines of > code if it makes a few peoples' lives easier, not sure how many people are &

Re: [DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in Spark 4.0.0+

or perspective. This is really just due diligence of the mistake, because I feel I need to do that. > Without a direct response to this, I think this discussion should be > considered to just be what it is on it's face -- a solution to a > vendors mistake and should not be ported

  1   2   3   4   5   6   >