Thanks David. Also, somehow I missed Rajini's email on KAFKA-13277. In any case, thanks Rajini and Colin for fixing this blocker too.
Konstantine On Wed, Sep 8, 2021 at 7:46 PM David Jacot <dja...@confluent.io.invalid> wrote: > Hi Konstantine, > > FYI, we just merged https://issues.apache.org/jira/browse/KAFKA-13266 to > trunk and to 3.0. > > Best, > David > > On Tue, Sep 7, 2021 at 11:25 PM Konstantine Karantasis < > kkaranta...@apache.org> wrote: > > > Thanks for raising this Colin. > > > > https://issues.apache.org/jira/browse/KAFKA-13277 > > > > is a one line fix with a test now. The suggestion to include it in RC2 > > makes sense to me as well. I’ll make sure it’s in. > > > > Konstantine > > > > On Tue, Sep 7, 2021 at 11:42 PM Colin McCabe <co...@cmccabe.xyz> wrote: > > > > > Hi Konstantine, > > > > > > Given that we are making a new RC, I would suggest that we merge > > > "KAFKA-13277; Fix size calculation for tagged string fields in message > > > generator" to 3.0. What do you think? > > > > > > best, > > > Colin > > > > > > > > > On Tue, Sep 7, 2021, at 13:20, Konstantine Karantasis wrote: > > > > Thanks David, > > > > > > > > Assuming we'll have the PRs you mention merged soon, I'll include > them > > in > > > > RC2 for 3.0.0 given their low risk and the fact that we need to > > generate > > > a > > > > new RC anyways. > > > > > > > > Konstantine > > > > > > > > On Tue, Sep 7, 2021 at 4:43 PM David Jacot > <dja...@confluent.io.invalid > > > > > > > wrote: > > > > > > > > > Hi Konstantine, > > > > > > > > > > I would like to propose https://github.com/apache/kafka/pull/11300 > > as > > > a > > > > > blocker > > > > > as well. The PR fixes KAFKA-13258/13259/13260. There are all very > > > small but > > > > > annoying issues. The PR is trivial. > > > > > > > > > > Regarding KAFKA-13266, the fix is quite simple, so low risk in my > > > opinion. > > > > > Jason > > > > > will review it soon. > > > > > > > > > > Best, > > > > > David > > > > > > > > > > On Tue, Sep 7, 2021 at 3:27 PM Ismael Juma <ism...@juma.me.uk> > > wrote: > > > > > > > > > > > Hi Konstantine, > > > > > > > > > > > > I will remove the final modifier for now. > > > > > > > > > > > > I added it because the removal of the deprecated `close` > overload > > > could > > > > > > lead to weird behavior if the no-args `close` was overridden (the > > > > > > implementation of the no-arg `close` delegated to the removed > > > `close`, > > > > > but > > > > > > that's no longer the case). However, I didn't realize > > `MockConsumer` > > > was > > > > > a > > > > > > public class (which is a mistake imo since we tweak its behavior > > > > > regularly > > > > > > and sometimes add methods without a KIP). > > > > > > > > > > > > Ismael > > > > > > > > > > > > On Tue, Sep 7, 2021 at 1:46 AM Konstantine Karantasis < > > > > > > kkaranta...@apache.org> wrote: > > > > > > > > > > > > > Hi Gary, > > > > > > > > > > > > > > Regarding KAFKA-13262, this might need a more detailed and > strong > > > > > > > justification to be considered as a blocker at this point in > the > > > > > release. > > > > > > > But we are already moving towards RC2 because of the blockers > > > mentioned > > > > > > > above, so I'd be in favor of a PR that would revert the change > > that > > > > > made > > > > > > > this method final if we could get one this week. I'm not sure > why > > > the > > > > > > > MockConsumer::close method was marked as final in > > > > > > > https://github.com/apache/kafka/pull/10438. The change is not > > > included > > > > > > in > > > > > > > the description. Maybe it wasn't obvious that this class is > used > > > > > outside > > > > > > > this project or maybe we thought we should discourage such use. > > > > > > > > > > > > > > Gary, is this the only change you are referring to? > > > > > > > Ismael, I see you authored the PR originally. Any thoughts on > the > > > > > > suggested > > > > > > > change? > > > > > > > > > > > > > > Konstantine > > > > > > > > > > > > > > On Tue, Sep 7, 2021 at 11:35 AM Konstantine Karantasis < > > > > > > > kkaranta...@apache.org> wrote: > > > > > > > > > > > > > > > Hi Magnus, > > > > > > > > > > > > > > > > Thanks for reporting the numbers. > > > > > > > > > > > > > > > > I agree that what you observe warrants further investigation, > > > not so > > > > > > much > > > > > > > > because these are clear performance regressions but because > > they > > > > > might > > > > > > be > > > > > > > > indications of underlying issues, as you noted. However, > from a > > > major > > > > > > > > release standpoint I'd say that such preliminary evidence > > > probably > > > > > > falls > > > > > > > > within the range of acceptable differences in the presence of > > > major > > > > > > > changes > > > > > > > > and can be optimized over time in subsequent releases. > > > > > > > > > > > > > > > > Having said that, I'd take Israel's comment one step further > > and > > > say > > > > > > that > > > > > > > > in order to be able to give more definitive answers it'd be > > good > > > to > > > > > > have > > > > > > > a > > > > > > > > description of the setup that is reproducible. Briefly, that > > > would > > > > > > > include > > > > > > > > both the benchmark setup (specs etc) as well as the benchmark > > > suite > > > > > and > > > > > > > the > > > > > > > > way to run it (links to benchmark code including > configurations > > > and > > > > > > > > description of how to run and reproduce). > > > > > > > > > > > > > > > > Small note that the attached plot is missing units on the > > x-axis > > > > > > (y-axis > > > > > > > I > > > > > > > > assume is milliseconds from your description). > > > > > > > > > > > > > > > > Konstantine > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 3, 2021 at 12:59 AM Israel Ekpo < > > > israele...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > >> Magnus, > > > > > > > >> > > > > > > > >> Please could you share the machine and network specs? > > > > > > > >> > > > > > > > >> How much CPU, RAM is available on each node? > > > > > > > >> > > > > > > > >> What JDK, JRE version are you using? > > > > > > > >> > > > > > > > >> What are your broker and client configuration values? Please > > > could > > > > > you > > > > > > > >> share this info if possible? > > > > > > > >> > > > > > > > >> Thanks. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Wed, Sep 1, 2021 at 10:25 AM Magnus Edenhill < > > > mag...@edenhill.se > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Hi Konstantine, > > > > > > > >> > > > > > > > > >> > Some findings from running 3.0.0-RC1 with the librdkafka > > test > > > > > suite: > > > > > > > >> > > > > > > > > >> > * Compaction seems to take slightly longer to kick in when > > > segment > > > > > > > sizes > > > > > > > >> > exceed their threshold. (Used to take less than 20 > > seconds, > > > now > > > > > > > takes > > > > > > > >> > 20..30 seconds.) > > > > > > > >> > > > > > > > > >> > * CreateTopic seems to take slightly longer to propagate > > > through > > > > > the > > > > > > > >> > cluster, > > > > > > > >> > e.g., before a new topic is available in metadata from > > other > > > > > > > brokers. > > > > > > > >> > > > > > > > > >> > * CreateTopics seems to take longer when the Admin request > > > timeout > > > > > > is > > > > > > > >> set, > > > > > > > >> > looks like a plateau at 10 seconds: > > > > > > > >> > https://imgur.com/a/n6y76sj > > > > > > > >> > > > > > > > > >> > (This is a 3 broker cluster with identical configs between > > > 2.8 and > > > > > > > >> 3.0.0.) > > > > > > > >> > > > > > > > > >> > Nothing critical, but could be an indication of regression > > so > > > I > > > > > > > thought > > > > > > > >> I'd > > > > > > > >> > mention it. > > > > > > > >> > > > > > > > > >> > Regards, > > > > > > > >> > Magnus > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > Den tis 31 aug. 2021 kl 17:51 skrev Konstantine > Karantasis < > > > > > > > >> > kkaranta...@apache.org>: > > > > > > > >> > > > > > > > > >> > > Small correction to my previous email. > > > > > > > >> > > The actual link for public preview of the 3.0.0 blog > post > > > draft > > > > > > is: > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache6 > > > > > > > >> > > > > > > > > > >> > > (see also the email thread with title: [DISCUSS] Please > > > review > > > > > the > > > > > > > >> 3.0.0 > > > > > > > >> > > blog post) > > > > > > > >> > > > > > > > > > >> > > Best, > > > > > > > >> > > Konstantine > > > > > > > >> > > > > > > > > > >> > > On Tue, Aug 31, 2021 at 6:34 PM Konstantine Karantasis < > > > > > > > >> > > kkaranta...@apache.org> wrote: > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > Hello Kafka users, developers and client-developers, > > > > > > > >> > > > > > > > > > > >> > > > This is the second release candidate for Apache Kafka > > > 3.0.0. > > > > > > > >> > > > It corresponds to a major release that includes many > new > > > > > > features, > > > > > > > >> > > > including: > > > > > > > >> > > > > > > > > > > >> > > > * The deprecation of support for Java 8 and Scala > 2.12. > > > > > > > >> > > > * Kafka Raft support for snapshots of the metadata > topic > > > and > > > > > > > >> > > > other improvements in the self-managed quorum. > > > > > > > >> > > > * Deprecation of message formats v0 and v1. > > > > > > > >> > > > * Stronger delivery guarantees for the Kafka producer > > > enabled > > > > > by > > > > > > > >> > default. > > > > > > > >> > > > * Optimizations in OffsetFetch and FindCoordinator > > > requests. > > > > > > > >> > > > * More flexible Mirror Maker 2 configuration and > > > deprecation > > > > > of > > > > > > > >> > > > Mirror Maker 1. > > > > > > > >> > > > * Ability to restart a connector's tasks on a single > > call > > > in > > > > > > Kafka > > > > > > > >> > > Connect. > > > > > > > >> > > > * Connector log contexts and connector client > overrides > > > are > > > > > now > > > > > > > >> enabled > > > > > > > >> > > > by default. > > > > > > > >> > > > * Enhanced semantics for timestamp synchronization in > > > Kafka > > > > > > > Streams. > > > > > > > >> > > > * Revamped public API for Stream's TaskId. > > > > > > > >> > > > * Default serde becomes null in Kafka Streams and > > several > > > > > > > >> > > > other configuration changes. > > > > > > > >> > > > > > > > > > > >> > > > You may read and review a more detailed list of > changes > > > in the > > > > > > > 3.0.0 > > > > > > > >> > blog > > > > > > > >> > > > post draft here: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache6 > > > > > > > >> > > > > > > > > > > >> > > > Release notes for the 3.0.0 release: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/RELEASE_NOTES.html > > > > > > > >> > > > > > > > > > > >> > > > *** Please download, test and vote by Wednesday, > > > September 8, > > > > > > 2021 > > > > > > > >> *** > > > > > > > >> > > > > > > > > > > >> > > > Kafka's KEYS file containing PGP keys we use to sign > the > > > > > > release: > > > > > > > >> > > > https://kafka.apache.org/KEYS > > > > > > > >> > > > > > > > > > > >> > > > * Release artifacts to be voted upon (source and > > binary): > > > > > > > >> > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/ > > > > > > > >> > > > > > > > > > > >> > > > * Maven artifacts to be voted upon: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > >> > > > > > > > > > > >> > > > * Javadoc: > > > > > > > >> > > > > > > https://home.apache.org/~kkarantasis/kafka-3.0.0-rc1/javadoc/ > > > > > > > >> > > > > > > > > > > >> > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.0 > > tag: > > > > > > > >> > > > > https://github.com/apache/kafka/releases/tag/3.0.0-rc1 > > > > > > > >> > > > > > > > > > > >> > > > * Documentation: > > > > > > > >> > > > https://kafka.apache.org/30/documentation.html > > > > > > > >> > > > > > > > > > > >> > > > * Protocol: > > > > > > > >> > > > https://kafka.apache.org/30/protocol.html > > > > > > > >> > > > > > > > > > > >> > > > * Successful Jenkins builds for the 3.0 branch: > > > > > > > >> > > > Unit/integration tests: > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/3.0/121/pipeline/ > > > > > > > >> > > > (only few flaky failures) > > > > > > > >> > > > System tests: > > > > > > > >> > > > > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.0/57/ > > > > > > > >> > > > > > > > > > > >> > > > /************************************** > > > > > > > >> > > > > > > > > > > >> > > > Thanks, > > > > > > > >> > > > Konstantine > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
