Hi, Divij, Another comment regarding the changes to segment.ms.
Currently, retention.ms has the following doc. "Additionally, retention.ms configuration operates independently of " segment.ms" and "segment.bytes" configurations. Moreover, it triggers the rolling of new segment if the retention.ms condition is satisfied." So, if we set a lower bound for segment.ms, should we do the same for retention.ms too? Thanks, Jun On Thu, Dec 5, 2024 at 1:26 AM Divij Vaidya <divijvaidy...@gmail.com> wrote: > Hi Ismael > > You are right. In hindsight, I should have started deprecation with one of > the earlier 3.x versions. > > But since that ship has already sailed, we can either wait for 5.0 to make > the actual change with just a deprecation notice in 4.x OR we can make the > change starting in 4.0 itself. > > I think that the latter (changing in 4.0) is desirable because the changes > are low risk. I will elaborate why I think so. > > The practice of having a deprecation period achieves two purposes: > 1\ provide users with ample time to prepare for upgrades where code changes > may be required. > 2\ allow users to measure the impact of the changes prior to moving to 4.0. > Testing on existing versions is important since in 4.0, it will be > difficult to isolate the impact of these changes due to the presence of > many other changes. > > Purpose 2 can still be achieved with the current proposal since all the > proposed changes could be tested with existing Kafka versions. There are no > new configurations which have been added in the proposal which cannot be > tested with older versions. > > For point 1, with the current proposal, 99% of users will require no code > change since we do not anticipate existing workloads to be using the now > removed lower bounds for the configurations. Here's a breakdown of risk at > a per-config level. > > 1. The changes in configuration to prevent small segments are expected to > be compatible with 99% of production workloads. This is because Kafka will > not be functional with extremely low segment size due to OS imposed limits > on file descriptors and mmaps. Hence, the risk of breaking existing > workloads by adding these new constraints is very low. As noted in the > discussion, we may impact some testing scenarios but there are quick ways > to work around them. We will also provide examples on how to write tests > where you expect a quick segment roll. > 2. The config changes which are associated with Tiered Storage (thread pool > configs) were introduced in 3.9. The customers who use the now removed, > "-1" value will have to make a change before upgrading to 4.0. I agree that > this does cause user friction during upgrade. To reduce the chance of > un-intentional error, we will add a "Pre-Upgrade Validation Tool". > 3. The change associated with the num recovery thread is an internal change > to the broker. I do not anticipate any negative side effects of changing > this default. It also does not require any code change from the customer > since no new constraint was added. > 4. `linger.ms` change is potentially the one with the largest impact to > existing production workloads. I would assume that users who > intentionally set linger.ms = 0 are expert users (since this quite an > unusual setting to get to behave correctly) and these users would pay > attention to upgrade notes (similar to how we made ack=all as default in > producers starting in 3.0). For users who un-intentionally set it to 0, I > believe that the new setting will work better for their workloads. > > I will update the KIP with the above explanation. > > Looking forward to hearing your thoughts. > > -- > Divij Vaidya > > > > On Thu, Dec 5, 2024 at 4:01 AM Ismael Juma <m...@ismaeljuma.com> wrote: > > > Hi Divij, > > > > The KIP didn't state this, but the usual practice is to have a > deprecation > > period before we make incompatible changes. Why did we reject this > option? > > We should mention that explicitly in the KIP. > > > > Ismael > > > > On Tue, Nov 19, 2024, 2:55 AM Divij Vaidya <divijvaidy...@gmail.com> > > wrote: > > > > > KT1 - That is right. We will throw a ConfigException. That is why this > > > change is considered backward incompatible. To be honest, given the > > nature > > > of suggested changes, I don't see any valid use case on why a user may > > have > > > a value which will be invalid after the new constraints. > > > > > > > > > -- > > > Divij Vaidya > > > > > > > > > > > > On Tue, Nov 19, 2024 at 2:21 AM Kirk True <k...@kirktrue.pro> wrote: > > > > > > > Hi Divij, > > > > > > > > Thanks for the KIP! > > > > > > > > My only question: > > > > > > > > KT1. In the case where we change the constraints so that a user's > > > > previously valid configuration is now invalid, do we do anything > other > > > than > > > > throw a ConfigException? > > > > > > > > Thanks, > > > > Kirk > > > > > > > > On Mon, Nov 18, 2024, at 2:13 AM, Divij Vaidya wrote: > > > > > Hey folks > > > > > > > > > > With 4.0, we have an opportunity to reset the default values and > add > > > > > constraints in the configurations based on our learnings since 3.0. > > > > > > > > > > Here's a KIP which modifies defaults for some properties and > modifies > > > the > > > > > constraints for a few others. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations > > > > > > > > > > > > > > > Looking forward for your feedback. > > > > > > > > > > (Previous discussion thread on this topic - > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp ) > > > > > > > > > > > > > > >
