Hi Josep,
I had run tests for tests/kafkatest/tests/client/quota_test.py based on
3.8
branch, and they all passed.
*19:54:24*
================================================================================*19:54:24*
SESSION REPORT (ALL TESTS)*19:54:24* ducktape version:
0.11.4*19:54:24* session_id: 2024-07-04--001*19:54:24* run
time: 12 minutes 39.940 seconds*19:54:24* tests run:
9*19:54:24* passed: 9*19:54:24* flaky:
0*19:54:24* failed: 0*19:54:24* ignored:
0*19:54:24*
================================================================================*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.consumer_num=2*19:54:24*
status: PASS*19:54:24* run time: 3 minutes 51.280
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=True*19:54:24*
status: PASS*19:54:24* run time: 4 minutes 21.082
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=False*19:54:24*
status: PASS*19:54:24* run time: 5 minutes 14.854
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_broker_throttling_behavior=True*19:54:24*
status: PASS*19:54:24* run time: 3 minutes 0.505
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_client_throttling_behavior=True*19:54:24*
status: PASS*19:54:24* run time: 3 minutes 19.629
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=False*19:54:24*
status: PASS*19:54:24* run time: 4 minutes 11.296
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=True*19:54:24*
status: PASS*19:54:24* run time: 4 minutes 10.578
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=user.override_quota=False*19:54:24*
status: PASS*19:54:24* run time: 4 minutes 19.187
seconds*19:54:24*
--------------------------------------------------------------------------------*19:54:24*
test_id:
kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=user.override_quota=True*19:54:24*
status: PASS*19:54:24* run time: 3 minutes 13.666
seconds*19:54:24*
--------------------------------------------------------------------------------
Thanks.
Luke
On Thu, Jul 4, 2024 at 6:01 PM Josep Prat <josep.p...@aiven.io.invalid>
wrote:
Hi Luke,
Thanks for the pointer, if you have an environment where you can run the
tests I would highly appreciate it!
I managed to run this test suite locally and currently only this one
fails
consistently, the rest pass:
Module: kafkatest.tests.client.quota_test
Class: QuotaTest
Method: test_quota
Arguments:
{
"old_client_throttling_behavior": true,
"quota_type": "client-id"
}
Failure:
TimeoutError("Timed out waiting 600 seconds for service nodes to
finish. These nodes are still alive:
['ProducerPerformanceService-0-140496695824336 node 1 on worker3']")
Traceback (most recent call last):
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/jlprat/projects/kafka/tests/kafkatest/tests/client/quota_test.py",
line 157, in test_quota
producer.run()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
line 345, in run
self.wait()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/background_thread.py",
line 72, in wait
super(BackgroundThreadService, self).wait(timeout_sec)
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
line 293, in wait
raise TimeoutError("Timed out waiting %s seconds for service nodes
to finish. " % str(timeout_sec)
ducktape.errors.TimeoutError: Timed out waiting 600 seconds for
service nodes to finish. These nodes are still alive:
['ProducerPerformanceService-0-140496695824336 node 1 on worker3']
On Thu, Jul 4, 2024 at 11:57 AM Luke Chen <show...@gmail.com> wrote:
Hi Josep,
For this
- QuotaTest --> speaking with Bruno we suspect there is a problem with
the
test setup, failed with "ValueError: max() arg is an empty sequence"
It's a known issue: KAFKA-16138
<https://issues.apache.org/jira/browse/KAFKA-16138> .
It should be passed with local specific tests run.
Do you want me help verify it by running it in my environment?
Thanks.
Luke
On Thu, Jul 4, 2024 at 4:03 PM Josep Prat <josep.p...@aiven.io.invalid
wrote:
Hi all,
We have had 2[1][2] runs of the system tests since the last blocker
was
merged on 3.8. So far we have 19 tests that failed on both runs.
I've
compiled them in this list[3].
There seems to these different categories of failing tests:
- QuotaTest --> speaking with Bruno we suspect there is a problem
with
the
test setup, failed with "ValueError: max() arg is an empty sequence"
- Streams cooperative rebalance upgrade --> It fails on versions
2.3.1
or
older, failed with Timeout
- KRaft Upgrade --> from dev with Isolated and combined KRaft,
failed
with
RemoteCommandError
- Network degrade test -> failed with RemoteCommandError
- Replica verification tool test --> Timeout for KRaft, but ZK
failed
on
the first run but worked on the second
If someone has further ideas on what could be causing these
failures,
please let me know. Given holidays in the US, the possible test
setup
problem might not be able to be fixed today.
[1]:
https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-02--001.05d6b151-356a-47e5-b724-6fcd79493422--1719991984--confluentinc--3.8--49d2ee3db9/report.html
[2]:
https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-03--001.4803d99b-52df-4f6d-82c2-3f050a6207fa--1720038529--apache--3.8--2fbe32ecb9/report.html
[3]:
https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit
Best,
On Tue, Jul 2, 2024 at 7:29 PM Josep Prat <josep.p...@aiven.io>
wrote:
Hi all,
Thanks for reviewing and merging the latest blockers for 3.8.0.
Tomorrow,
I will start with the process to get the first RC out.
Best!
On Sat, Jun 29, 2024 at 9:04 PM Josep Prat <josep.p...@aiven.io>
wrote:
Hi Justine,
Marking MV 3.8-IV0 as latest
production MV is done in this PR (I did both together)
https://github.com/apache/kafka/pull/16400
Best,
------------------
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io | +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
On Sat, Jun 29, 2024, 00:52 Justine Olshan
<jols...@confluent.io.invalid
wrote:
The PR is merged. I lowered the severity of the blocker ticket
as
we
still
have the change in trunk to merge. However, the 3.8 release is
no
longer
blocked by KAFKA-17050.
I think that was the remaining blocker. The other ones are
either
already
fixed for 3.8 (KAFKA-17011) or diverted to 3.9 (KAFKA-16840)
I think there was one more needed change to mark MV 3.8-IV0 as
latest
production MV. I will follow up with that.
Justine
On Thu, Jun 27, 2024 at 2:34 PM Justine Olshan <
jols...@confluent.io
wrote:
Here is the PR: https://github.com/apache/kafka/pull/16478
Justine
On Thu, Jun 27, 2024 at 2:21 PM Justine Olshan <
jols...@confluent.io
wrote:
Hey all,
Thanks for your patience. After some discussion, we decided
to
revert
group version from 3.8 since there were too many complexities
associated
with getting it to work.
I've downgraded the severity of KAFKA-17011 to not be a
blocker
and
opened a ticket (
https://issues.apache.org/jira/browse/KAFKA-17050)
to
revert from 3.8 (and 3.9) as a blocker instead. I hope to get
the
PR
out
shortly.
This one should be less controversial and merged quickly.
Thanks again,
Justine
On Thu, Jun 27, 2024 at 1:22 AM Josep Prat
<josep.p...@aiven.io.invalid>
wrote:
Hi all,
I just wanted to ask again for your help in reviewing these
2
last
blockers
for the 3.8.0 release:
https://github.com/apache/kafka/pull/16400
https://github.com/apache/kafka/pull/16420
Thanks!
On Mon, Jun 24, 2024 at 9:27 AM Josep Prat <
josep.p...@aiven.io>
wrote:
Hi all,
We currently have a couple of blockers for the 3.8.0
release.
These are
the following:
- Reverting commit KAFKA-16154 and mark latest production
metadata
as
3.8.0: https://github.com/apache/kafka/pull/16400
- Fix some failing system tests:
https://github.com/apache/kafka/pull/16420
Can we get some eyes on these 2 PRs? Thanks!
To easily track this in feature releases, I created a new
label
called
"Blocker" the idea is to mark PRs that are solving an
Issue
marked
as
"blocker". This might increase visibility and help getting
those
reviewed
promptly. Here is the link to the PRs with this label:
https://github.com/apache/kafka/labels/Blocker
Best,
On Thu, Jun 20, 2024 at 7:09 PM Josep Prat <
josep.p...@aiven.io>
wrote:
Thanks for the heads up Justine!
On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan
<jols...@confluent.io.invalid> wrote:
Sorry to derail this conversation, but just wanted to
share
we
have a
system test blocker with
https://issues.apache.org/jira/browse/KAFKA-16990.
Hopefully we can fix this in the next day or so.
Justine
On Mon, Jun 17, 2024 at 12:19 PM David Jacot <
david.ja...@gmail.com>
wrote:
I meant it from a time perspective, not from a
branching
point
perspective.
Sorry for the confusion. As said in the other thread,
doing
it
four
months
after 3.9 is desirable for KIP-848 as I expect that we
will
need
time
to
stabilize everything after switching all the default
configs
once
3.9
is
cut.
David
Le lun. 17 juin 2024 à 19:33, Matthias J. Sax <
mj...@apache.org> a
écrit :
Why would 4.0 be based on 3.8? My understanding is,
that
it
will
be
based on 3.9.
-Matthias
On 6/14/24 11:22 PM, David Jacot wrote:
I agree that we should keep 4.0 time-based. My
question
is
based on
which
release. If I understand you, you would like to
keep
it
based
on
3.8.
This
means that 4.0 would be released in October. It
would
be
helpful
to fix
the
dates so we can plan accordingly. I will start a
separate
thread on
Monday.
David
Le sam. 15 juin 2024 à 00:44, Colin McCabe <
cmcc...@apache.org>
a
écrit
:
+1. I think it would be good to keep 4.0
time-based.
Most
of
the
refactors
we want to do are optional in some sense and can
be
descoped
if
time
runs
out. For example, we can drop support for JDK 8
without
immediately
refactoring everything that could benefit from
the
improvements in
JDK9+.
best,
Colin
On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax
wrote:
That's my understanding, and I would advocate
strongly
to
keep
the
4.0
release schedule as planed originally.
The 3.9 one should really be an additional "out
of
schedule"
release
not
impacting any other releases.
-Matthias
On 6/14/24 1:29 PM, David Jacot wrote:
The plan sounds good to me. I suppose that we
will
follow
our
regular
cadence for 4.0 and release it four months
after
3.9
(in
November?).
Is
this correct?
Best,
David
Le ven. 14 juin 2024 à 21:57, José Armando
García
Sancio
<jsan...@confluent.io.invalid> a écrit :
+1 on the proposed release plan for 3.8.
Thanks!
On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma <
m...@ismaeljuma.com
wrote:
+1 to the plan we converged on in this
thread.
Ismael
On Fri, Jun 14, 2024 at 10:46 AM Josep Prat
<josep.p...@aiven.io.invalid
wrote:
Hi all,
Thanks Colin, yes go ahead.
As we are now past code freeze I would like
to
ask
everyone
involved
in a
KIP that is not yet complete, to verify if
what
landed on
the 3.8
branch
needs to be reverted or if it can stay.
Additionally,
I'll be
pinging
KIPs
and Jira reporters asking for their status
as
some
Jiras
seem to
have
all
related GitHub PRs merged but their status
is
still
Open
or
In
Progress.
I'll be checking all the open blockers and
check
if
they
are
really a
blocker or can be pushed.
Regarding timeline, I'll attempt to generate
the
first
RC on
Wednesday
or
Thursday, so please revert any changes
<
https://www.google.com/maps/search/,+so+please+revert+any+changes+?entry=gmail&source=g
you
deem necessary by then. If
you
need more time, please ping me.
Best,
-----------------
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io | +491715557497 |
aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu
Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
On Fri, Jun 14, 2024, 19:25 Colin McCabe <
cmcc...@apache.org
wrote:
Hi all,
We have had many delays with releases this
year.
We
should
try
to
get
back
on schedule.
I agree with the idea that was proposed a
few
times
in
this
thread
of
drawing a line under 3.8 now, and doing a
short
3.9
release. I
posted a
3.9
release plan here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
I think we could start doing RCs for 3.8.0
as
early
as
next
week.
There
are a few things that need to be reverted
first
(anything
related
to
KIP-853 or KIP-966).
Josep, if you agree, I will update
KIP-1012 to
reflect
that
these
things
are landing in 3.9 rather than 3.8. And we
can
start
doing
all
the
normal
release stuff. The main blocker JIRA I'm
aware
of
is
KAFKA-16946,
which
is
a very simple fix.
best,
Colin
On Fri, Jun 14, 2024, at 03:48, Satish
Duggana
wrote:
+1 on going with 3.8 release with the
existing
plan and
targeting
the
required features in 3.9 timelines. 4.0
will
be
targeted
in the
usual
cycle(4 months) after 3.9 is released.
On Fri, 14 Jun 2024 at 15:19, Edoardo
Comar <
edoardli...@gmail.com
wrote:
Josep,
past the deadline sorry but I can't see
reasons
not to
cherry-pick
this
https://github.com/apache/kafka/pull/16326
On Wed, 12 Jun 2024 at 17:14, Josep Prat
<josep.p...@aiven.io.invalid
wrote:
Hi Edoardo,
Correct, you can still cherry-pick this
one.
I'll send an email tomorrow morning
(CEST)
asking
maintainers
to
stop
cherry picking commits unless we
discuss it
beforehand.
Best,
On Wed, Jun 12, 2024 at 6:09 PM Edoardo
Comar <
edoardli...@gmail.com>
wrote:
Hi Josep, I understand I am still in
time
to
cherry-pick on
3.8.0
https://github.com/apache/kafka/pull/16229
right?
thanks
On Wed, 12 Jun 2024 at 11:34, Ivan
Yurchenko <
i...@ivanyu.me>
wrote:
Hi,
I'll try to do all the fixes and
changes
for
KIP-899
[1]
sooner
today,
but please proceed with the release if
I
don't
manage.
Ivan
[1]
https://github.com/apache/kafka/pull/13277
On Wed, Jun 12, 2024, at 12:54, Josep
Prat
wrote:
Hi Luke,
I think Jose, also mentioned that it
won't
be
ready
for
v3.8.0
(but he
can
confirm this). My question now would
be,
given
that it
seems
we
would
need
a v3.9.0, do you think it's
important to
include
https://github.com/apache/kafka/pull/16284
in
v3.8.0?
Best,
On Wed, Jun 12, 2024 at 11:40 AM Luke
Chen <
show...@gmail.com
wrote:
Hi Josep
For KIP-966, I think Calvin had
mentioned
he
won't
complete
in
v3.8.0.
https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
For unclean leader election, all we
need
is
this
PR:
https://github.com/apache/kafka/pull/16284
For this PR, I think it needs one
more
week
to be
completed.
Thanks.
Luke
On Wed, Jun 12, 2024 at 4:51 PM
Josep
Prat
<josep.p...@aiven.io.invalid>
wrote:
Hi all,
We are now really close to the
planned
code
freeze
deadline
(today
EOD).
According to KIP-1012 [1] we
agreed to
stay
in
the
3.x
branch
until we
achieve feature parity regarding
Zookeeper
and
KRaft.
The
two main
KIPs
identified that would achieve this
are:
KIP-853
[2]
and
KIP-966
[3].
At the moment of writing this email
both
KIPs
are
not
completed. My
question to the people driving both
KIPs
would
be,
how
much
more
time do
you think it's needed to bring
these
KIPs
to
completion?
- If the time needed would be
short,
we
could
still
include
these
2 KIPs
in
the release.
- However, if the time needed
would be
on
the
scale
of
weeks, we
should
continue with the release plan for
3.8
and
after
start
working on
the 3.9
release.
What are your thoughts?
[1]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
[2]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
[3]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
On Wed, Jun 12, 2024 at 10:40 AM
Josep
Prat
<
josep.p...@aiven.io>
wrote:
Hi Rajini,
Yes, we could backport this one to
the
3.8
branch.
Would
you be
able to
do
this once you merge this PR?
Thanks
On Tue, Jun 11, 2024 at 10:53 PM
Rajini
Sivaram <
rajinisiva...@gmail.com
wrote:
Hi Josep,
The PR
https://github.com/apache/kafka/pull/13277
for
KIP-899
looks
ready
to be merged (waiting for the PR
build).The PR
changes
several
files,
but
is relatively straightforward and
not
risky.
Also
the
changes
are
under
a
config that is not enabled by
default.
Since
the
KIP
was
approved
before
KIP freeze, will it be ok to
include
in
3.8.0?
Thank you,
Rajini
On Tue, Jun 11, 2024 at 9:35 AM
Josep
Pr
<
https://www.google.com/maps/search/11,+2024+at+9:35%E2%80%AFAM+Josep+Pr?entry=gmail&source=g
at
<josep.p...@aiven.io.invalid
wrote:
Hi all,
I just want to remind everybody
that
the
code
freeze
deadline
is
approaching
<
https://www.google.com/maps/search/%3E%3E+%3E+%3E+%3E+%3E+%3E+%3E+%3E%3E+%3E+approaching?entry=gmail&source=g
.
June 12th EOD is the deadline.
Please do not autom
<
https://www.google.com/maps/search/%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E+Please+do+not+autom?entry=gmail&source=g
atically
backport any commit
to
the
3.8
branch
after
June 12th EOD. Ping me if you
think
the
commit
should
be
backported
(fixes
failures in the branch or
critical
bug
fixes).
Thanks all!
On Sat, Jun 1, 2024 at 8:43 PM
José
Armando
García
Sancio
<
https://www.google.com/maps/search/%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E+?entry=gmail&source=g
<jsan...@confluent.io.invalid> wrote:
Hi Josep,
See my comments below.
On Wed, May 29, 2024 at
10:52 AM
Josep
Prat
<josep.p...@aiven.io.invalid
wrote:
So I would propose to leave
the
deadlines
as
they
are and
manually
cherry
pick the comm
<
https://www.google.com/maps/search/%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E%3E+pick+the+comm?entry=gmail&source=g
its
related to KIP-853 and
KIP-966.
Your proposal sounds good to
me. I
suspect
that
will
be
doing
feature
development for KIP-853 past
the
feature
freeze and
code
freeze
date.
Maybe feature development will
be
finished
around
the end
of June.
I'll make sure to cherry pick
commits
for
KIP-853
to
the 3.8
branch
once we have one.
Thanks,
--
-José
--
[image: Aiven] <
https://www.aiven.io
*Josep Prat*
