Final recap: SPIP: Support Customized Kubernetes Scheduler

Last month, I synced some progress on "Support Customized Kubernetes Scheduler" [1] at 24. Feb. 2022. Another month has passed, with the cut of the 3.3 release, there are also some changes on SPIP. I'd like to share in here:

Re: Tools for regression testing

At the wikipedia regression testing page https://en.wikipedia.org/wiki/Regression_testing Under use " Regression tests can be broadly categorized as functional tests or unit tests

Re: Tools for regression testing

good point. I just wanted to know when we do changes to releases or RC, is there some mechanism that ensures the Spark release still functions as expected after any code changes, updates etc? For example there was a recent discussion about Kafka upgrade to 3.x with Spark upgrade to 3.x and its

Re: Tools for regression testing

Hm, then what are you looking for besides all the tests in Spark? On Thu, Mar 24, 2022, 2:34 PM Mich Talebzadeh wrote: > Thanks > > I know what unit testing is. The question was not about unit testing. it > was specific to regression testing >

Re: Tools for regression testing

Thanks I know what unit testing is. The question was not about unit testing. it was specific to regression testing artifacts . cheers,

Re: Tools for regression testing

Yes, Spark uses unit tests. https://app.codecov.io/gh/apache/spark https://en.wikipedia.org/wiki/Unit_testing man. 21. mar. 2022 kl. 15:46 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > Hi, > > As a matter of interest do Spark releases deploy a specific regression > testing tool? > >

Probable bug in async commit of Kafka offset in DirectKafkaInputDStream

Hi Dev, I added a few debug statements at the following lines and found few issues. 1. At line 254 of override def compute(validTime: Time): Option[KafkaRDD[K, V]] in DirectKafkaInputDStream.scala: System.out.print("Called commitAll at time " + validTime + " " +

Re: [DISCUSS] Migration guide on upgrading Kafka to 3.1 in Spark 3.3

I've had a small talk to the Kafka guys to find out a little bit more and the oversimplified conclusion is that if the producer version >= 3.0 and broker version < 0.11.0 with message format version V1 then either `enable.idempotence = false` needed or broker upgrade to 0.11.0+ is required to make