Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2436

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Colt McNealy]

Hi Matthias and everyone—

Some clarification is necessary just for posterity. It turns out that, on a
fresh standby task before we start polling for records, we wouldn't be able
to get the current end offset without a network call. This leaves us three
options:

A) Make it an Optional or use a sentinel value to mark that it's not
present.
B) Perform a network call to get the endOffset when it's not there.
C) Remove it.

Option A) seems like it could be a confusing API, especially because in the
strong majority of cases, the Optional would be empty. Option B) is
undesirable because of the performance considerations—if we're going to
make a network round trip, we might as well get some records back! That
leaves us with option C), which is the least-bad of all of them.

At LittleHorse we actually do care about the endOffset in the
onUpdateStart() method, and having it would be useful to us. However, the
work-around isn't horrible, because the endOffset will be passed into the
first call to onBatchLoaded() , which normally follows onUpdateStart()
within <100ms.

Thanks,
Colt McNealy

*Founder, LittleHorse.dev*


On Thu, Nov 30, 2023 at 4:43 PM Matthias J. Sax  wrote:

> > parameter is somewhat irrelevant to our use case
>
> Sounds like a weird justification to change the KIP. Providing more
> information is usually better than less, so it seems it won't hurt to
> just keep it (seems useful in general to get the current end offset in
> this callback) -- you can always ignore it, if it's not relevant for
> your use case.
>
>
> -Matthias
>
>
> On 11/30/23 6:56 AM, Eduwer Camacaro wrote:
> > Hello everyone,
> >
> > We have come to the conclusion, during our work on this KIP's
> > implementation, that the #onUpdateStart callback's "currentEndOffset"
> > parameter is somewhat irrelevant to our use case. When this callback is
> > invoked, I think this value is usually unknown. Our choice to delete this
> > parameter from the #onUpdateStart callback requires an update to the KIP.
> >
> > Please feel free to review the PR and provide any comments you may have.
> :)
> > Thanks in advance
> >
> > Edu-
> >
> > On Thu, Oct 26, 2023 at 12:17 PM Matthias J. Sax 
> wrote:
> >
> >> Thanks. SGTM.
> >>
> >> On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:
> >>> That all sounds good to me! Thanks for the KIP
> >>>
> >>> On Wed, Oct 25, 2023 at 3:47 PM Colt McNealy 
> >> wrote:
> >>>
>  Hi Sophie, Matthias, Bruno, and Eduwer—
> 
>  Thanks for your patience as I have been scrambling to catch up after a
> >> week
>  of business travel (and a few days with no time to code). I'd like to
> >> tie
>  up some loose ends here, but in short, I don't think the KIP document
>  itself needs any changes (our internal implementation does, however).
> 
>  1. In the interest of a) not changing the KIP after it's already out
> >> for a
>  vote, and b) making sure our English grammar is "correct", let's stick
> >> with
>  'onBatchLoaded()`. It is the Store that gets updated, not the Batch.
> 
>  2. For me (and, thankfully, the community as well) adding a remote
> >> network
>  call at any point in this KIP is a non-starter. We'll ensure that
>  our implementation does not introduce one.
> 
>  3. I really don't like changing API behavior, even if it's not
> >> documented
>  in the javadoc. As such, I am strongly against modifying the behavior
> of
>  endOffsets() on the consumer as some people may implicitly depend on
> the
>  contract.
>  3a. The Consumer#currentLag() method gives us exactly what we want
> >> without
>  a network call (current lag from a cache, from which we can compute
> the
>  offset).
> 
>  4. I have no opinion about whether we should pass endOffset or
> >> currentLag
>  to the callback. Either one has the same exact information inside it.
> In
>  the interest of not changing the KIP after the vote has started, I'll
> >> leave
>  it as endOffset.
> 
>  As such, I believe the KIP doesn't need any updates, nor has it been
>  updated since the vote started.
> 
>  Would anyone else like to discuss something before the Otter Council
>  adjourns regarding this matter?
> 
>  Cheers,
>  Colt McNealy
> 
>  *Founder, LittleHorse.dev*
> 
> 
>  On Mon, Oct 23, 2023 at 10:44 PM Sophie Blee-Goldman <
>  sop...@responsive.dev>
>  wrote:
> 
> > Just want to checkpoint the current state of this KIP and make sure
> >> we're
> > on track to get it in to 3.7 (we still have a few weeks)  -- looks
> like
> > there are two remaining open questions, both relating to the
> > middle/intermediate callback:
> >
> > 1. What to name it: seems like the primary candidates are
> onBatchLoaded
>  and
> > onBatchUpdated (and maybe also onStandbyUpdated?)
> > 2. What additional information can we pass in that would strike a
> good
> > balance between 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2435

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Matthias J. Sax]

parameter is somewhat irrelevant to our use case


Sounds like a weird justification to change the KIP. Providing more 
information is usually better than less, so it seems it won't hurt to 
just keep it (seems useful in general to get the current end offset in 
this callback) -- you can always ignore it, if it's not relevant for 
your use case.



-Matthias


On 11/30/23 6:56 AM, Eduwer Camacaro wrote:

Hello everyone,

We have come to the conclusion, during our work on this KIP's
implementation, that the #onUpdateStart callback's "currentEndOffset"
parameter is somewhat irrelevant to our use case. When this callback is
invoked, I think this value is usually unknown. Our choice to delete this
parameter from the #onUpdateStart callback requires an update to the KIP.

Please feel free to review the PR and provide any comments you may have. :)
Thanks in advance

Edu-

On Thu, Oct 26, 2023 at 12:17 PM Matthias J. Sax  wrote:


Thanks. SGTM.

On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:

That all sounds good to me! Thanks for the KIP

On Wed, Oct 25, 2023 at 3:47 PM Colt McNealy 

wrote:



Hi Sophie, Matthias, Bruno, and Eduwer—

Thanks for your patience as I have been scrambling to catch up after a

week

of business travel (and a few days with no time to code). I'd like to

tie

up some loose ends here, but in short, I don't think the KIP document
itself needs any changes (our internal implementation does, however).

1. In the interest of a) not changing the KIP after it's already out

for a

vote, and b) making sure our English grammar is "correct", let's stick

with

'onBatchLoaded()`. It is the Store that gets updated, not the Batch.

2. For me (and, thankfully, the community as well) adding a remote

network

call at any point in this KIP is a non-starter. We'll ensure that
our implementation does not introduce one.

3. I really don't like changing API behavior, even if it's not

documented

in the javadoc. As such, I am strongly against modifying the behavior of
endOffsets() on the consumer as some people may implicitly depend on the
contract.
3a. The Consumer#currentLag() method gives us exactly what we want

without

a network call (current lag from a cache, from which we can compute the
offset).

4. I have no opinion about whether we should pass endOffset or

currentLag

to the callback. Either one has the same exact information inside it. In
the interest of not changing the KIP after the vote has started, I'll

leave

it as endOffset.

As such, I believe the KIP doesn't need any updates, nor has it been
updated since the vote started.

Would anyone else like to discuss something before the Otter Council
adjourns regarding this matter?

Cheers,
Colt McNealy

*Founder, LittleHorse.dev*


On Mon, Oct 23, 2023 at 10:44 PM Sophie Blee-Goldman <
sop...@responsive.dev>
wrote:


Just want to checkpoint the current state of this KIP and make sure

we're

on track to get it in to 3.7 (we still have a few weeks)  -- looks like
there are two remaining open questions, both relating to the
middle/intermediate callback:

1. What to name it: seems like the primary candidates are onBatchLoaded

and

onBatchUpdated (and maybe also onStandbyUpdated?)
2. What additional information can we pass in that would strike a good
balance between being helpful and impacting performance.

Regarding #1, I think all of the current options are reasonable enough

that

we should just let Colt decide which he prefers. I personally think
#onBatchUpdated is fine -- Bruno does make a fair point but the truth

is

that English grammar can be sticky and while it could be argued that it

is

the store which is updated, not the batch, I feel that it is perfectly
clear what is meant by "onBatchUpdated" and to me, this doesn't sound

weird

at all. That's just my two cents in case it helps, but again, whatever
makes sense to you Colt is fine

When it comes to #2 -- as much as I would love to dig into the Consumer
client lore and see if we can modify existing APIs or add new ones in

order

to get the desired offset metadata in an efficient way, I think we're
starting to go down a rabbit hole that is going to expand the scope way
beyond what Colt thought he was signing up for. I would advocate to

focus

on just the basic feature for now and drop the end-offset from the
callback. Once we have a standby listener it will be easy to expand on

with

a followup KIP if/when we find an efficient way to add additional

useful

information. I think it will also become more clear what is and isn't
useful after more people get to using it in the real world

Colt/Eduwer: how necessary is receiving the end offset during a batch
update to your own application use case?

Also, for those who really do need to check the current end offset, I
believe in theory you should be able to use the KafkaStreams#metrics

API

to

get the current lag and/or end offset for the changelog -- it's

possible

this does not represent the most up-to-date end offset (I'm 

[jira] [Resolved] (KAFKA-15527) Add reverseRange and reverseAll query over kv-store in IQv2

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15527.
-
Resolution: Fixed

> Add reverseRange and reverseAll query over kv-store in IQv2
> ---
>
> Key: KAFKA-15527
> URL: https://issues.apache.org/jira/browse/KAFKA-15527
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Hanyu Zheng
>Assignee: Hanyu Zheng
>Priority: Major
>  Labels: kip
> Fix For: 3.7.0
>
>
> Add reverseRange and reverseAll query over kv-store in IQv2
> Update an implementation of the Query interface, introduced in [KIP-796: 
> Interactive Query 
> v2|https://cwiki.apache.org/confluence/display/KAFKA/KIP-796%3A+Interactive+Query+v2]
>  , to support reverseRange and reverseAll.
> Use bounded query to achieve reverseRange and use unbounded query to achieve 
> reverseAll.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2434

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 241709 lines...]
Progress (3): 2.6/5.6 MB | 358 kB | 341/640 kB
Progress (3): 2.6/5.6 MB | 358 kB | 341/640 kB
Progress (3): 2.6/5.6 MB | 358 kB | 357/640 kB
Progress (3): 2.6/5.6 MB | 358 kB | 357/640 kB
Progress (3): 2.6/5.6 MB | 358 kB | 374/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 374/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 390/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 390/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 390/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 407/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 407/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 423/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 423/640 kB
Progress (3): 2.7/5.6 MB | 358 kB | 439/640 kB
  
Downloaded from central: 
https://repo.maven.apache.org/maven2/log4j/log4j/1.2.12/log4j-1.2.12.jar (358 
kB at 2.6 MB/s)
Progress (2): 2.7/5.6 MB | 456/640 kB
Progress (2): 2.7/5.6 MB | 472/640 kB
Progress (2): 2.7/5.6 MB | 489/640 kB
Progress (2): 2.7/5.6 MB | 505/640 kB
Progress (2): 2.7/5.6 MB | 521/640 kB
Progress (2): 2.7/5.6 MB | 521/640 kB
Progress (2): 2.7/5.6 MB | 538/640 kB
Progress (2): 2.8/5.6 MB | 538/640 kB
Progress (2): 2.8/5.6 MB | 554/640 kB
Progress (2): 2.8/5.6 MB | 554/640 kB
Progress (2): 2.8/5.6 MB | 570/640 kB
Progress (2): 2.8/5.6 MB | 570/640 kB
Progress (2): 2.8/5.6 MB | 570/640 kB
Progress (2): 2.8/5.6 MB | 587/640 kB
Progress (2): 2.8/5.6 MB | 587/640 kB
Progress (2): 2.8/5.6 MB | 603/640 kB
Progress (2): 2.8/5.6 MB | 603/640 kB
Progress (2): 2.8/5.6 MB | 620/640 kB
Progress (2): 2.8/5.6 MB | 620/640 kB
Progress (2): 2.8/5.6 MB | 636/640 kB
Progress (2): 2.8/5.6 MB | 640 kB
Progress (2): 2.9/5.6 MB | 640 kB
Progress (2): 2.9/5.6 MB | 640 kB
Progress (2): 2.9/5.6 MB | 640 kB
Progress (2): 2.9/5.6 MB | 640 kB
 
Downloaded from central: 
https://repo.maven.apache.org/maven2/com/google/collections/google-collections/1.0/google-collections-1.0.jar
 (640 kB at 4.4 MB/s)
Progress (1): 2.9/5.6 MB
Progress (1): 2.9/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.0/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.1/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.2/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.3/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.4/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.5/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.6/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.7/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.8/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 3.9/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.0/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.1/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.2/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.3/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.4/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.5/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.6/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.7/5.6 MB
Progress (1): 4.8/5.6 MB
Progress (1): 4.8/5.6 

[jira] [Created] (KAFKA-15951) MissingSourceTopicException should include topic names

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-15951:
---

 Summary: MissingSourceTopicException should include topic names
 Key: KAFKA-15951
 URL: https://issues.apache.org/jira/browse/KAFKA-15951
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


As the title say – we don't include topic names in all cases, what make it hard 
for users to identify the root cause more clearly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15629) proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15629.
-
Resolution: Fixed

> proposal to introduce IQv2 Query Types: TimestampedKeyQuery and 
> TimestampedRangeQuery
> -
>
> Key: KAFKA-15629
> URL: https://issues.apache.org/jira/browse/KAFKA-15629
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Hanyu Zheng
>Assignee: Hanyu Zheng
>Priority: Major
>  Labels: kip
> Fix For: 3.7.0
>
>
> KIP-992: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
> In the current IQv2 code, there are noticeable differences when interfacing 
> with plain-kv-store and ts-kv-store. Notably, the return type V acts as a 
> simple value for plain-kv-store but evolves into ValueAndTimestamp for 
> ts-kv-store, which presents type safety issues in the API.
> Even if IQv2 hasn't gained widespread adoption, an immediate fix might bring 
> compatibility concerns.
> This brings us to the essence of our proposal: the introduction of distinct 
> query types. One that returns a plain value, another for values accompanied 
> by timestamps.
> While querying a ts-kv-store for a plain value and then extracting it is 
> feasible, it doesn't make sense to query a plain-kv-store for a 
> ValueAndTimestamp.
> Our vision is for plain-kv-store to always return V, while ts-kv-store should 
> return ValueAndTimestamp.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13621) Resign leader on network partition

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio resolved KAFKA-13621.

Resolution: Duplicate

> Resign leader on network partition
> --
>
> Key: KAFKA-13621
> URL: https://issues.apache.org/jira/browse/KAFKA-13621
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>
> h1. Motivation
> If the current leader A at epoch X gets partition from the rest of the 
> quorum, quorum voter A will stay leader at epoch X. This happens because 
> voter A will never receive an request from the rest of the voters increasing 
> the epoch. These requests that typically increase the epoch of past leaders 
> are BeginQuorumEpoch and Vote.
> In addition if voter A (leader at epoch X) doesn't get partition from the 
> rest of the brokers (observer in the KRaft protocol) the brokers will never 
> learn about the new quorum leader. This happens because 1. observers learn 
> about the leader from the Fetch response and 2. observer send a Fetch request 
> to a random leader if the Fetch request times out.
> Neither of these two scenarios will cause the broker to send a request to a 
> different voter because the leader at epoch X will never send a different 
> leader in the response and the broker will never send a Fetch request to a 
> different voter because the Fetch request will never timeout.
> h1. Proposed Changes
> In this scenario the A, the leader at epoch X, will stop receiving Fetch 
> request from the majority of the voters. Voter A should resign as leader if 
> the Fetch request from the majority of the voters is old enough. A reasonable 
> value for "old enough" is the Fetch timeout value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2433

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 23389 lines...]
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/EndToEndLatencyTest.java 
(9496770a82299cdd8d62a8dbcf139af86ebed412)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java 
(6e38af8956ef2a979420f2680eeca2a5025367da)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/GetOffsetShellParsingTest.java 
(9980a471c37da169236e8beb6766f6dbf1003570)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/GetOffsetShellTest.java 
(417bbe711681a8a5e81f1d7fec00a8276a84ccde)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java 
(b789d75fae5b614c0146ec3bdaaf11c41125bf61)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/LeaderElectionCommandErrorTest.java 
(fef75bfd1043bdbb01ad8862e82af1ec2f88b1de)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/LeaderElectionCommandTest.java 
(0c9fa753f754de3ff55fd4ddfeed26863dafdf2c)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/LogDirsCommandTest.java 
(4dfd36896c627107494548d085702c253b463392)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandErrorTest.java 
(1cdc197d96fb15932738e2a8540a96ae6b6395ca)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandTest.java 
(4002d84b48fd5273020281ed2671f96b8ca87ada)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/ProducerPerformanceTest.java 
(f97e34dda9cbb1b844fab43674030a6ecd5f6624)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/PushHttpMetricsReporterTest.java 
(e4ed958d93b354a9712d1f9c8946fb94f5a2d69b)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/ReplicaVerificationToolTest.java 
(b5eb121ba138672b6f395a350102f580eac3dc64)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/StreamsResetterTest.java 
(84a7690bd48e781164d3f09459280912ca6c928a)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/ToolsTestUtils.java 
(ac379bcb49ecf9653f977898b1ee8d731db588c4)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/TopicCommandIntegrationTest.java 
(b934e04012cc84f6891fb95b44e0a8950f982b31)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/TopicCommandTest.java 
(6831b0bae4c31674b94ac951f52968cb74568ac3)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/TransactionsCommandTest.java 
(ef7b5b2ab9930ef6eea1733fd220d329fc42dc3c)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/reassign/ReassignPartitionsCommandArgsTest.java
 (c248796d25f078b9d4b5763e4e29d5dcc8027852)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/reassign/ReassignPartitionsIntegrationTest.java
 (b71331a6b233f9a4eb89a4a2902853938a77cccd)
error: unable to read sha1 file of 
tools/src/test/java/org/apache/kafka/tools/reassign/ReassignPartitionsUnitTest.java
 (d0e1cb4baaf691806c81aab4590fba3447cd219d)
error: unable to read sha1 file of tools/src/test/resources/log4j.properties 
(5291604d49ae535e2d3c0925cb796ebb196b55aa)
error: unable to read sha1 file of 
tools/tools-api/src/main/java/org/apache/kafka/tools/api/RecordReader.java 
(cda150064ad1a5be3d5326b89dfd061b80fd6ff2)
error: unable to read sha1 file of 
tools/tools-api/src/main/java/org/apache/kafka/tools/api/package-info.java 
(cd1129e09aa8979ab7751f664bdb282557ec5675)
error: unable to read sha1 file of 
tools/tools-api/src/test/java/org/apache/kafka/tools/api/RecordReaderTest.java 
(4c2145d7ffe2543b124f854a7476d007b2fb462e)
error: unable to read sha1 file of trogdor/.gitignore 
(ae3c1726048cd06b9a143e0376ed46dd9b9a8d53)
error: unable to read sha1 file of trogdor/README.md 
(849fc5d052d53214c4c64c1746ee38a34caffebd)
error: unable to read sha1 file of 
trogdor/src/main/java/org/apache/kafka/trogdor/agent/Agent.java 
(9a05f907b811e7ed2811d362b4fb67d84fcfb49f)
error: unable to read sha1 file of 
trogdor/src/main/java/org/apache/kafka/trogdor/agent/AgentClient.java 
(0f47e92a92e8fd9c6ccb71b9d1f9996c2cacd0b8)
error: unable to read sha1 file of 
trogdor/src/main/java/org/apache/kafka/trogdor/agent/AgentRestResource.java 
(ec3df8bf84696aa697c41d68e8bae44aa6aadac2)
error: unable to read sha1 file of 
trogdor/src/main/java/org/apache/kafka/trogdor/agent/WorkerManager.java 
(4510e1bb9221e86d8b7091eaea5eee16d280e6c9)
error: unable to read sha1 file of 
trogdor/src/main/java/org/apache/kafka/trogdor/basic/BasicNode.java 
(232a64db80b000d1ab738b832680f2964da2a90d)
error: unable to read sha1 file of 

Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery

[Hanyu (Peter) Zheng]

new KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++update+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery

On Wed, Nov 29, 2023 at 10:12 PM Hanyu (Peter) Zheng 
wrote:

> Thank you Bruno,
> 1. Thank you for the notification. I have updated the ticket link
> accordingly.
> 2. Certainly, I'll update the KIP name. Should I initiate a new discussion
> for it, because if I change the name, the link will change.
> 3. Understood, I will add that to the KIP.
> 4. I propose we accept both
> `WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2)` and
> `WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2)`,
> while also reusing the existing `withKey` method.
> 5. Following a discussion with Matthias, we've decided to defer the
> implementation of order guarantees to a future KIP.
>
> Sincerely,
> Hanyu
>
> On Wed, Nov 29, 2023 at 6:22 AM Bruno Cadonna  wrote:
>
>> Hi,
>>
>> Thanks for the updates!
>>
>>
>> 1.
>> Could you please link the correct ticket in the KIP?
>>
>> 2.
>> Could you please adapt the motivation section and the title to the
>> updated goal of the KIP? There is no fetch() or fetchAll() method in the
>> query class.
>>
>> 3.
>> Could you please add the "// newly added" comment to all parts that were
>> newly added? That is methods lowerKeyBound() and upperKeyBound().
>>
>> 4.
>> We should use a more fluent API as I proposed in my last e-mail:
>>
>> Here again
>>
>> WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2);
>> WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2);
>> WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2);
>>
>> 5.
>> We should also consider the order of the results similar as we did in
>> KIP-968. Alternatively, we do not guarantee any order and postpone the
>> order guarantees to a future KIP.
>>
>>
>> Best,
>> Bruno
>>
>>
>>
>> On 11/17/23 3:11 AM, Matthias J. Sax wrote:
>> > Thanks for the KIP.
>> >
>> > Given how `WindowRangeQuery` works right now, it's really time to
>> > improve it.
>> >
>> >
>> > 1) Agree. It's not clear what will be added right now. I think we
>> should
>> > deprecate existing `getKey()` w/o an actually replacement? For
>> > `getFromKey` and `getToKey` we should actually be `lowerKeyBound()` and
>> > `upperKeyBound()` to align to KIP-969?
>> >
>> > Also wondering if we should deprecate existing `withKey()` and
>> > `withWindowStartRange`? `withKey` only works for SessionStores and
>> > implements a single-key full-time-range query. Similarly,
>> > `withWindowStartRange` only works for WindowedStore and implements an
>> > all-key time-range query. Thus, both are rather special and it seems
>> the
>> > aim of this KIP is to generalize `WindowRangeQuery` to arbitrary
>> > key-range/time-range queries?
>> >
>> > What raises one question about time-range semantics, given that we
>> query
>> > windows with different semantics.
>> >
>> >   - The current `WindowStore` semantics used for
>> > `WindowRangeQuery#withWindowStartRange` is considering only the window
>> > start time, ie, the window-start time must fall into the query
>> > time-range to be returned.
>> >
>> >   - In contrast, `SessionStore` time ranges base on `findSession` use
>> > earliest-session-end-time and latest-session-end-time and thus
>> implement
>> > an "window-bounds / search-time-range overlap query".
>> >
>> > Is there any concern about semantic differences? I would also be
>> > possible to use the same semantics for both query types, and maybe even
>> > let the user pick with semantics they want (let users chose might
>> > actually be the best thing to do)? -- We can also do this
>> incrementally,
>> > and limit the scope of this KIP (or keep the full KIP scope but
>> > implement it incrementally only)
>> >
>> > Btw: I think we should not add any ordering at this point, and
>> > explicitly state that no ordering is guarantee whatsoever at this point.
>> >
>> >
>> >
>> > 2) Agreed. We should deprecate `getFromTime` and `getToTime` and add
>> new
>> > method `fromTime` and `toTime`.
>> >
>> >
>> >
>> > 3) About the API. If we move forward with general key-range/time-range
>> I
>> > agree that a more modular approach might be nice. Not sure right now,
>> > what the best approach would be for this? Looking into KIP-969, we
>> might
>> > want to have:
>> >
>> >   - static withKeyRange
>> >   - static withLowerKeyBound
>> >   - static withUpperKeyBound
>> >   - static withAllKeys (KIP-969 actually uses `allKeys` ?)
>> >   - fromTime
>> >   - toTime
>> >
>> > with default-time range would be "all / unbounded" ?
>> >
>> >
>> >
>> > 10: you mentioned that `WindowKeyQuery` functionality can be covered by
>> > `WindowRangeQuery`. I agree. For this case, it seems we want to
>> > deprecate `WindowKeyQuery` entirely?
>> >
>> >
>> >
>> > -Matthias
>> >
>> > On 11/16/23 1:19 AM, Bruno Cadonna wrote:
>> >> Hi Hanyu,
>> >>
>> >> Thanks for the KIP!
>> >>
>> >> 1)
>> >> Could you please mark the pieces 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Eduwer Camacaro]

Hello everyone,

We have come to the conclusion, during our work on this KIP's
implementation, that the #onUpdateStart callback's "currentEndOffset"
parameter is somewhat irrelevant to our use case. When this callback is
invoked, I think this value is usually unknown. Our choice to delete this
parameter from the #onUpdateStart callback requires an update to the KIP.

Please feel free to review the PR and provide any comments you may have. :)
Thanks in advance

Edu-

On Thu, Oct 26, 2023 at 12:17 PM Matthias J. Sax  wrote:

> Thanks. SGTM.
>
> On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:
> > That all sounds good to me! Thanks for the KIP
> >
> > On Wed, Oct 25, 2023 at 3:47 PM Colt McNealy 
> wrote:
> >
> >> Hi Sophie, Matthias, Bruno, and Eduwer—
> >>
> >> Thanks for your patience as I have been scrambling to catch up after a
> week
> >> of business travel (and a few days with no time to code). I'd like to
> tie
> >> up some loose ends here, but in short, I don't think the KIP document
> >> itself needs any changes (our internal implementation does, however).
> >>
> >> 1. In the interest of a) not changing the KIP after it's already out
> for a
> >> vote, and b) making sure our English grammar is "correct", let's stick
> with
> >> 'onBatchLoaded()`. It is the Store that gets updated, not the Batch.
> >>
> >> 2. For me (and, thankfully, the community as well) adding a remote
> network
> >> call at any point in this KIP is a non-starter. We'll ensure that
> >> our implementation does not introduce one.
> >>
> >> 3. I really don't like changing API behavior, even if it's not
> documented
> >> in the javadoc. As such, I am strongly against modifying the behavior of
> >> endOffsets() on the consumer as some people may implicitly depend on the
> >> contract.
> >> 3a. The Consumer#currentLag() method gives us exactly what we want
> without
> >> a network call (current lag from a cache, from which we can compute the
> >> offset).
> >>
> >> 4. I have no opinion about whether we should pass endOffset or
> currentLag
> >> to the callback. Either one has the same exact information inside it. In
> >> the interest of not changing the KIP after the vote has started, I'll
> leave
> >> it as endOffset.
> >>
> >> As such, I believe the KIP doesn't need any updates, nor has it been
> >> updated since the vote started.
> >>
> >> Would anyone else like to discuss something before the Otter Council
> >> adjourns regarding this matter?
> >>
> >> Cheers,
> >> Colt McNealy
> >>
> >> *Founder, LittleHorse.dev*
> >>
> >>
> >> On Mon, Oct 23, 2023 at 10:44 PM Sophie Blee-Goldman <
> >> sop...@responsive.dev>
> >> wrote:
> >>
> >>> Just want to checkpoint the current state of this KIP and make sure
> we're
> >>> on track to get it in to 3.7 (we still have a few weeks)  -- looks like
> >>> there are two remaining open questions, both relating to the
> >>> middle/intermediate callback:
> >>>
> >>> 1. What to name it: seems like the primary candidates are onBatchLoaded
> >> and
> >>> onBatchUpdated (and maybe also onStandbyUpdated?)
> >>> 2. What additional information can we pass in that would strike a good
> >>> balance between being helpful and impacting performance.
> >>>
> >>> Regarding #1, I think all of the current options are reasonable enough
> >> that
> >>> we should just let Colt decide which he prefers. I personally think
> >>> #onBatchUpdated is fine -- Bruno does make a fair point but the truth
> is
> >>> that English grammar can be sticky and while it could be argued that it
> >> is
> >>> the store which is updated, not the batch, I feel that it is perfectly
> >>> clear what is meant by "onBatchUpdated" and to me, this doesn't sound
> >> weird
> >>> at all. That's just my two cents in case it helps, but again, whatever
> >>> makes sense to you Colt is fine
> >>>
> >>> When it comes to #2 -- as much as I would love to dig into the Consumer
> >>> client lore and see if we can modify existing APIs or add new ones in
> >> order
> >>> to get the desired offset metadata in an efficient way, I think we're
> >>> starting to go down a rabbit hole that is going to expand the scope way
> >>> beyond what Colt thought he was signing up for. I would advocate to
> focus
> >>> on just the basic feature for now and drop the end-offset from the
> >>> callback. Once we have a standby listener it will be easy to expand on
> >> with
> >>> a followup KIP if/when we find an efficient way to add additional
> useful
> >>> information. I think it will also become more clear what is and isn't
> >>> useful after more people get to using it in the real world
> >>>
> >>> Colt/Eduwer: how necessary is receiving the end offset during a batch
> >>> update to your own application use case?
> >>>
> >>> Also, for those who really do need to check the current end offset, I
> >>> believe in theory you should be able to use the KafkaStreams#metrics
> API
> >> to
> >>> get the current lag and/or end offset for the changelog -- it's
> possible
> >>> this does 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2432

[Apache Jenkins Server]

See 

[REVIEW REQUEST] ConsumerGroupCommand move to tools

[Николай Ижиков]

Hello.

I prepared a PR to move `ConsumerGroupCommand` to `tools` module.
Full changes are pretty huge [1] so I split them into several PR’s.

Please, take a look at first PR in the series - 
https://github.com/apache/kafka/pull/14856

PR adds `ConsumerGroupCommandOptions` class and other command case classes to 
`tools` module.

[1] https://github.com/apache/kafka/pull/14471

[REVIEW REQUEST] ConfigType, ConfigEntityName, DynamicConfig moved to server-common

[Николай Ижиков]

Hello.

I’m working one move `ConfigCommand` to tools module.

`ConfigCommand` depends on several config classes from core.
So that classes must be moved to «server-commons» prior to command migration.

I’ve prepared several PR’s to move part of dependencies.
All PR’s seems to be very straightforward.

Please, take a look:

* Move ConfigType to «server-commons» module - 
https://github.com/apache/kafka/pull/14867
* Move ConfigEntityName to «server-commons» module - 
https://github.com/apache/kafka/pull/14868
* Move DynamicConfig to «server-commons» module - 
https://github.com/apache/kafka/pull/14871

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.6 #122

[Apache Jenkins Server]

See