Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #757

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #756

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 553830 lines...]
[2022-03-10T21:05:48.810Z] 
[2022-03-10T21:05:48.810Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] STARTED
[2022-03-10T21:05:55.752Z] 
[2022-03-10T21:05:55.752Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] PASSED
[2022-03-10T21:05:55.752Z] 
[2022-03-10T21:05:55.752Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2022-03-10T21:05:58.280Z] 
[2022-03-10T21:05:58.280Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = true] PASSED
[2022-03-10T21:05:58.280Z] 
[2022-03-10T21:05:58.280Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = false] STARTED
[2022-03-10T21:06:01.442Z] 
[2022-03-10T21:06:01.442Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2022-03-10T21:06:01.442Z] 
[2022-03-10T21:06:01.442Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] STARTED
[2022-03-10T21:06:06.754Z] 
[2022-03-10T21:06:06.754Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = false] PASSED
[2022-03-10T21:06:06.754Z] 
[2022-03-10T21:06:06.754Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] STARTED
[2022-03-10T21:06:08.384Z] 
[2022-03-10T21:06:08.384Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] PASSED
[2022-03-10T21:06:08.384Z] 
[2022-03-10T21:06:08.384Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED
[2022-03-10T21:06:13.635Z] 
[2022-03-10T21:06:13.635Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = false] PASSED
[2022-03-10T21:06:13.635Z] 
[2022-03-10T21:06:13.635Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = false] STARTED
[2022-03-10T21:06:14.074Z] 
[2022-03-10T21:06:14.074Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED
[2022-03-10T21:06:14.074Z] 
[2022-03-10T21:06:14.074Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = false] STARTED
[2022-03-10T21:06:19.809Z] 
[2022-03-10T21:06:19.809Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = false] PASSED
[2022-03-10T21:06:19.809Z] 
[2022-03-10T21:06:19.809Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = false] STARTED
[2022-03-10T21:06:20.426Z] 
[2022-03-10T21:06:20.426Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = false] PASSED
[2022-03-10T21:06:20.426Z] 
[2022-03-10T21:06:20.426Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] STARTED
[2022-03-10T21:06:26.824Z] 
[2022-03-10T21:06:26.824Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = false] PASSED
[2022-03-10T21:06:26.824Z] 
[2022-03-10T21:06:26.824Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = false] STARTED
[2022-03-10T21:06:29.322Z] 
[2022-03-10T21:06:29.322Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = false] PASSED
[2022-03-10T21:06:29.322Z] 
[2022-03-10T21:06:29.322Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED
[2022-03-10T21:06:32.578Z] 
[2022-03-10T21:06:32.578Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = false] PASSED
[2022-03-10T21:06:32.578Z] 
[2022-03-10T21:06:32.578Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = false] STARTED
[2022-03-10T21:06:36.904Z] 
[2022-03-10T21:06:36.904Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED
[2022-03-10T21:06:36.904Z] 
[2022-03-10T21:06:36.904Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = false] STARTED
[2022-03-10T21:06:38.331Z] 
[2022-03-10T21:06:38.331Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = false] PASSED
[2022-03-10T21:06:38.331Z] 

[jira] [Created] (KAFKA-13728) PushHttpMetricsReporter no longer pushes metrics when network failure is recovered.

[XiaoyiPeng (Jira)]

XiaoyiPeng created KAFKA-13728:
--

 Summary: PushHttpMetricsReporter no longer pushes metrics when 
network failure is recovered.
 Key: KAFKA-13728
 URL: https://issues.apache.org/jira/browse/KAFKA-13728
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 3.1.0
Reporter: XiaoyiPeng


The class *PushHttpMetricsReporter* no longer pushes metrics when network 
failure is recovered.

I debugged the code and found the problem here :
[https://github.com/apache/kafka/blob/dc36dedd28ff384218b669de13993646483db966/tools/src/main/java/org/apache/kafka/tools/PushHttpMetricsReporter.java#L214-L221]

 

When we submit a task to the *ScheduledThreadPoolExecutor* that needs to be 
executed periodically, if the task throws an exception and is not swallowed, 
the task will no longer be scheduled to execute.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi Jun,

32. Good point.  Do you think it's ok to defer the metrics until we run
some benchmarks so that we get a better idea of what metrics we need?

-Artem

On Thu, Mar 10, 2022 at 3:12 PM Jun Rao  wrote:

> Hi, Artem.
>
> Thanks for the reply. One more comment.
>
> 32. Do we need to add any new metric on the producer? For example, if
> partitioner.availability.timeout.ms is > 0, it might be useful to know the
> number of unavailable partitions.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 10, 2022 at 12:46 PM Artem Livshits
>  wrote:
>
> > Hi Jun,
> >
> > 30.  Clarified.
> >
> > 31. I plan to do some benchmarking once implementation is finished, I'll
> > update the KIP with the results once I have them.  The reason to make it
> > default is that it won't be used otherwise and we won't know if it's good
> > or not in practical workloads.
> >
> > -Artem
> >
> > On Thu, Mar 10, 2022 at 11:42 AM Jun Rao 
> wrote:
> >
> > > Hi, Artem,
> > >
> > > Thanks for the updated KIP. A couple of more comments.
> > >
> > > 30. For the 3 new configs, it would be useful to make it clear that
> they
> > > are only relevant when the partitioner class is null.
> > >
> > > 31. partitioner.adaptive.partitioning.enable : I am wondering whether
> it
> > > should default to true. This is a more complex behavior than "uniform
> > > sticky" and may take some time to get right. If we do want to enable it
> > by
> > > default, it would be useful to validate it with some test results.
> > >
> > > Jun
> > >
> > >
> > >
> > >
> > > On Wed, Mar 9, 2022 at 10:05 PM Artem Livshits
> > >  wrote:
> > >
> > > > Thank you for feedback, I've discussed this offline with some of the
> > > folks
> > > > and updated the KIP.  The main change is that now instead of using
> > > > DefaultPartitioner and UniformStickyPartitioners as flags, in the new
> > > > proposal the default partitioner is null, so if no custom partitioner
> > is
> > > > specified then the partitioning logic is implemented in
> KafkaProducer.
> > > > Compatibility section is updated as well.  Also the configuration
> > options
> > > > are renamed to be more consistent.
> > > >
> > > > -Artem
> > > >
> > > > On Fri, Mar 4, 2022 at 10:38 PM Luke Chen  wrote:
> > > >
> > > > > Hi Artem,
> > > > >
> > > > > Thanks for your explanation and update to the KIP.
> > > > > Some comments:
> > > > >
> > > > > 5. In the description for `enable.adaptive.partitioning`, the
> `false`
> > > > case,
> > > > > you said:
> > > > > > the producer will try to distribute messages uniformly.
> > > > > I think we should describe the possible skewing distribution.
> > > Otherwise,
> > > > > user might be confused about why adaptive partitioning is
> important.
> > > > >
> > > > > 6. In the description for `partition.availability.timeout.ms`, I
> > think
> > > > we
> > > > > should mention in the last sentence about if
> > > > `enable.adaptive.partitioning`
> > > > > is disabled this logic is also disabled.
> > > > >
> > > > > 7. Similar thoughts as Ismael, I think we should have a POC and
> test
> > to
> > > > > prove that this adaptive partitioning algorithm can have better
> > uniform
> > > > > partitioning, compared with original sticky one.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Fri, Mar 4, 2022 at 9:22 PM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Regarding `3`, we should only deprecate it if we're sure the new
> > > > approach
> > > > > > handles all cases better. Are we confident about that for both of
> > the
> > > > > > previous partitioners?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Mar 4, 2022 at 1:37 AM David Jacot
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Artem,
> > > > > > >
> > > > > > > Thanks for the KIP! I have a few comments:
> > > > > > >
> > > > > > > 1. In the preamble of the proposed change section, there is
> > still a
> > > > > > > mention of the
> > > > > > > -1 approach. My understanding is that we have moved away from
> it
> > > now.
> > > > > > >
> > > > > > > 2. I am a bit concerned by the trick suggested about the
> > > > > > > DefaultPartitioner and
> > > > > > > the UniformStickyPartitioner. I do agree that implementing the
> > > logic
> > > > in
> > > > > > the
> > > > > > > producer itself is a good thing. However, it is weird from a
> user
> > > > > > > perspective
> > > > > > > that he can set a class as partitioner that is not used in the
> > > end. I
> > > > > > > think that
> > > > > > > this will be confusing for our users. Have we considered
> changing
> > > the
> > > > > > > default
> > > > > > > value of partitioner.class to null to indicate that the new
> > > built-in
> > > > > > > partitioner
> > > > > > > must be used? By default, the built-in partitioner would be
> used
> > > > unless
> > > > > > the
> > > > > > > user explicitly specify one. The downside is that the new
> default
> > > > > > behavior
> > > > > > > would not work if the user explicitly specify the partitioner
> but
> > > we
> 

[jira] [Created] (KAFKA-13727) Edge case in cleaner can result in premature removal of ABORT marker

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13727:
---

 Summary: Edge case in cleaner can result in premature removal of 
ABORT marker
 Key: KAFKA-13727
 URL: https://issues.apache.org/jira/browse/KAFKA-13727
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


The log cleaner works by first building a map of the active keys beginning from 
the dirty offset, and then scanning forward from the beginning of the log to 
decide which records should be retained based on whether they are included in 
the map. The map of keys has a limited size. As soon as it fills up, we stop 
building it. The offset corresponding to the last record that was included in 
the map becomes the next dirty offset. Then when we are cleaning, we stop 
scanning forward at the dirty offset. Or to be more precise, we continue 
scanning until the end of the segment which includes the dirty offset, but all 
records above that offset are coped as is without checking the map of active 
keys. 

Compaction is complicated by the presence of transactions. The cleaner must 
keep track of which transactions have data remaining so that it can tell when 
it is safe to remove the respective markers. It works a bit like the consumer. 
Before scanning a segment, the cleaner consults the aborted transaction index 
to figure out which transactions have been aborted. All other transactions are 
considered committed.

The problem we have found is that the cleaner does not take into account the 
range of offsets between the dirty offset and the end offset of the segment 
containing it when querying ahead for aborted transactions. This means that 
when the cleaner is scanning forward from the dirty offset, it does not have 
the complete set of aborted transactions. The main consequence of this is that 
abort markers associated with transactions which start within this range of 
offsets become eligible for deletion even before the corresponding data has 
been removed from the log.

Here is an example. Suppose that the log contains the following entries:

offset=0, key=1

offset=1, key=2

offset=2, COMMIT

offset=3, key=3

offset=4, key=4

offset=5, COMMIT

offset=6, key=2

offset=7, ABORT

Suppose we have an offset map which can only contain 2 keys and the dirty 
offset starts at 0. The first time we scan forward, we will build a map with 
keys 1 and 2, which will allow us to move the dirty offset up to 3. Due to the 
issue documented here, we will not detect the aborted transaction starting at 
offset 6. But it will not be eligible for deletion on this round of cleaning 
because it is bound by `delete.retention.ms`. Instead, our new logic will set 
the deletion horizon for this batch based to the current time plus the 
configured `delete.retention.ms`.

offset=0, key=a

offset=1, key=b

offset=2, COMMIT

offset=3, key=c

offset=4, key=d

offset=5, COMMIT

offset=6, key=b

offset=7, ABORT (deleteHorizon: N)

Suppose that the time reaches N+1 before the next cleaning. We will begin from 
the dirty offset of 3 and collect keys c and d before stopping at offset 6. 
Again, we will not detect the aborted transaction beginning at offset 6 since 
it is out of the range. This time when we scan, the marker at offset 7 will be 
deleted because the transaction will be seen as empty and now the deletion 
horizon has passed. So we end up with this state:

offset=0, key=a

offset=1, key=b

offset=2, COMMIT

offset=3, key=c

offset=4, key=d

offset=5, COMMIT

offset=6, key=b

Effectively it becomes a hanging transaction. The interesting thing is that we 
might not even detect it. As far as the leader is concerned, it had already 
completed that transaction, so it is not expecting any additional markers. The 
transaction index would have been rewritten without the aborted transaction 
when the log was cleaned, so any consumer fetching the data would see the 
transaction as committed. On the other hand, if we did a reassignment to a new 
replica, or if we had to rebuild the full log state during recovery, then we 
would suddenly detect it.

I am not sure how likely this scenario is in practice. I think it's fair to say 
it is an extremely rare case. The cleaner has to fail to clean a full segment 
at least two times and you still need enough time to pass for the marker's 
deletion horizon to be reached. Perhaps it is possible if the cardinality of 
keys is very high and the configured memory limit for the cleaner is low.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

[Jorge Esteban Quilcate Otoya]

Thanks all!

I agree with Matthias and Jon on going forward with the new
`FixedKeyRecord` approach.
The KIP has been updated accordingly.

Feel free to add your vote or amend on the vote thread if needed.

Cheers,

On Mon, 7 Mar 2022 at 21:57, Matthias J. Sax  wrote:

> I think it's ok that we cannot prevent users from mutating a given
> read-only object. We have similar issues "all over the place" in the
> API, because it's just how Java works unfortunately (eg,
> `ValueMapperWithKey` and similar interfaces).
>
> The point being is, that the API clearly expresses that the key should
> not be changes, as `FixedKeyRecord` as not `withKey()` method, what is
> much better then having `Record.withKey()` and thus incorrectly
> indicating to user that it would be ok to set a new key.
>
> I think it's worth to add the new interfaces.
>
>
> -Matthias
>
>
> On 3/7/22 11:46 AM, Guozhang Wang wrote:
> > Thanks John! I feel a bit ashamed of just thinking loud here without
> trying
> > out prototypes myself :P
> >
> > I think the FixedKeyProcessor/Record looks very good -- like you said,
> > since we are making a new set of APIs then why don't we reconsider more
> > bolderly -- but at the same time I'd also like to make sure we agree on
> how
> > much "safety" we can achieve in runtime: even with the proposed APIs, we
> > cannot prevent users doing something like:
> >
> > ---
> > process(FixedKeyRecord inputRecord) {
> >  inputRecord.key().modifyField(...); // this is not preventable with
> > runtime key validations either since we just check the key object itself
> is
> > not replaced
> >  context.forward(inputRecord);
> > }
> >
> > ---
> >
> > I.e. in either type-safety or runtime validation, we cannot be 100% safe
> > that users would not do anything wrong. This drives me to think, how much
> > we'd like to pay to "remind" (instead of say "enforce", since we cannot
> > really do it) users the semantics of "processValue". Personally I felt
> that
> > adding the new set of APIs for that purpose only is a bit overkill, and
> > hence was leaning towards just the runtime validation. But I admit this
> is
> > purely subjective so I'm willing to yield to the group if others feel
> it's
> > worthy to do so.
> >
> >
> > Guozhang
> >
> >
> >
> > On Mon, Mar 7, 2022 at 10:32 AM Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> >> Thanks, John!
> >> This looks very promising.
> >>
> >> I will familiarize this approach and update the KIP accordingly. From
> what
> >> I can see so far, this should cover most of the open issues in this
> >> proposal.
> >>
> >> PS.
> >>
> >>> Just as a reminder, the current approach with transformers
> >>> is NOT enforced at compile time. Transformers have access to
> >>> a "forwarding disabled" processor context, which still has
> >>> the forward methods that throw a runtime exception when
> >>> invoked.
> >>
> >> Agree. I was referring to the value transformers where `readOnlyKey` is
> >> passed but not forwarded internally. Though about the "forwarding
> disabled"
> >> approach, you're totally right that is a runtime validation.
> >> Regardless, the approach proposed here will be a much better one.
> >>
> >>
> >> On Sun, 6 Mar 2022 at 18:59, John Roesler  wrote:
> >>
> >>> Hello all,
> >>>
> >>> It seems like we're making good progress on this discussion.
> >>> If I'm keeping track correctly, if we can resolve this
> >>> question about how to handle processValues(), then we should
> >>> be able to finalize the vote, right?
> >>>
> >>> I share Matthias's preference for having a type-safe API.
> >>>
> >>> Just as a reminder, the current approach with transformers
> >>> is NOT enforced at compile time. Transformers have access to
> >>> a "forwarding disabled" processor context, which still has
> >>> the forward methods that throw a runtime exception when
> >>> invoked.
> >>>
> >>> However, the spirit of the "new processor api" line of work
> >>> is to clean up a lot of the cruft around the original
> >>> processor API, so this is a good opportunity to introduce a
> >>> type-safe version if we can.
> >>>
> >>> Based on my experience adding the new processor API, I felt
> >>> like it should be possible to do what he suggests, but it
> >>> would be more involved than what he said. The biggest thing
> >>> I learned from that effort, though, is that you really have
> >>> to just try it to see what all the complications are.
> >>>
> >>> With that in mind, I went ahead and implemented the
> >>> suggestion: https://github.com/apache/kafka/pull/11854
> >>>
> >>> This is a functional prototype. It only adds processValues,
> >>> which takes a supplier of a new type, FixedKeyProcessor.
> >>> That processor only processes FixedKeyRecords, which have a
> >>> key that cannot be changed. FixedKeyProcessors have a
> >>> special context, a FixedKeyProcessorContext, which can only
> >>> forward FixedKeyRecords.
> >>>
> >>> FixedKeyRecords have "fixed keys" 

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Jun Rao]

Hi, Artem.

Thanks for the reply. One more comment.

32. Do we need to add any new metric on the producer? For example, if
partitioner.availability.timeout.ms is > 0, it might be useful to know the
number of unavailable partitions.

Thanks,

Jun

On Thu, Mar 10, 2022 at 12:46 PM Artem Livshits
 wrote:

> Hi Jun,
>
> 30.  Clarified.
>
> 31. I plan to do some benchmarking once implementation is finished, I'll
> update the KIP with the results once I have them.  The reason to make it
> default is that it won't be used otherwise and we won't know if it's good
> or not in practical workloads.
>
> -Artem
>
> On Thu, Mar 10, 2022 at 11:42 AM Jun Rao  wrote:
>
> > Hi, Artem,
> >
> > Thanks for the updated KIP. A couple of more comments.
> >
> > 30. For the 3 new configs, it would be useful to make it clear that they
> > are only relevant when the partitioner class is null.
> >
> > 31. partitioner.adaptive.partitioning.enable : I am wondering whether it
> > should default to true. This is a more complex behavior than "uniform
> > sticky" and may take some time to get right. If we do want to enable it
> by
> > default, it would be useful to validate it with some test results.
> >
> > Jun
> >
> >
> >
> >
> > On Wed, Mar 9, 2022 at 10:05 PM Artem Livshits
> >  wrote:
> >
> > > Thank you for feedback, I've discussed this offline with some of the
> > folks
> > > and updated the KIP.  The main change is that now instead of using
> > > DefaultPartitioner and UniformStickyPartitioners as flags, in the new
> > > proposal the default partitioner is null, so if no custom partitioner
> is
> > > specified then the partitioning logic is implemented in KafkaProducer.
> > > Compatibility section is updated as well.  Also the configuration
> options
> > > are renamed to be more consistent.
> > >
> > > -Artem
> > >
> > > On Fri, Mar 4, 2022 at 10:38 PM Luke Chen  wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > Thanks for your explanation and update to the KIP.
> > > > Some comments:
> > > >
> > > > 5. In the description for `enable.adaptive.partitioning`, the `false`
> > > case,
> > > > you said:
> > > > > the producer will try to distribute messages uniformly.
> > > > I think we should describe the possible skewing distribution.
> > Otherwise,
> > > > user might be confused about why adaptive partitioning is important.
> > > >
> > > > 6. In the description for `partition.availability.timeout.ms`, I
> think
> > > we
> > > > should mention in the last sentence about if
> > > `enable.adaptive.partitioning`
> > > > is disabled this logic is also disabled.
> > > >
> > > > 7. Similar thoughts as Ismael, I think we should have a POC and test
> to
> > > > prove that this adaptive partitioning algorithm can have better
> uniform
> > > > partitioning, compared with original sticky one.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Fri, Mar 4, 2022 at 9:22 PM Ismael Juma 
> wrote:
> > > >
> > > > > Regarding `3`, we should only deprecate it if we're sure the new
> > > approach
> > > > > handles all cases better. Are we confident about that for both of
> the
> > > > > previous partitioners?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Mar 4, 2022 at 1:37 AM David Jacot
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Artem,
> > > > > >
> > > > > > Thanks for the KIP! I have a few comments:
> > > > > >
> > > > > > 1. In the preamble of the proposed change section, there is
> still a
> > > > > > mention of the
> > > > > > -1 approach. My understanding is that we have moved away from it
> > now.
> > > > > >
> > > > > > 2. I am a bit concerned by the trick suggested about the
> > > > > > DefaultPartitioner and
> > > > > > the UniformStickyPartitioner. I do agree that implementing the
> > logic
> > > in
> > > > > the
> > > > > > producer itself is a good thing. However, it is weird from a user
> > > > > > perspective
> > > > > > that he can set a class as partitioner that is not used in the
> > end. I
> > > > > > think that
> > > > > > this will be confusing for our users. Have we considered changing
> > the
> > > > > > default
> > > > > > value of partitioner.class to null to indicate that the new
> > built-in
> > > > > > partitioner
> > > > > > must be used? By default, the built-in partitioner would be used
> > > unless
> > > > > the
> > > > > > user explicitly specify one. The downside is that the new default
> > > > > behavior
> > > > > > would not work if the user explicitly specify the partitioner but
> > we
> > > > > could
> > > > > > mitigate this with my next point.
> > > > > >
> > > > > > 3. Related to the previous point, I think that we could deprecate
> > > both
> > > > > the
> > > > > > DefaultPartitioner and the UniformStickyPartitioner. I would also
> > > add a
> > > > > > warning if one of them is explicitly provided by the user to
> inform
> > > > them
> > > > > > that they should switch to the new built-in one. I am pretty sure
> > > that
> > > > > most
> > > > > > of the folks use the default 

[jira] [Resolved] (KAFKA-12879) Compatibility break in Admin.listOffsets()

[Randall Hauch (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-12879.
---
  Reviewer: Randall Hauch
Resolution: Fixed

> Compatibility break in Admin.listOffsets()
> --
>
> Key: KAFKA-12879
> URL: https://issues.apache.org/jira/browse/KAFKA-12879
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.8.0, 2.7.1, 2.6.2
>Reporter: Tom Bentley
>Assignee: Philip Nee
>Priority: Major
> Fix For: 2.5.2, 2.8.2, 3.2.0, 3.1.1, 3.0.2, 2.7.3, 2.6.4
>
>
> KAFKA-12339 incompatibly changed the semantics of Admin.listOffsets(). 
> Previously it would fail with {{UnknownTopicOrPartitionException}} when a 
> topic didn't exist. Now it will (eventually) fail with {{TimeoutException}}. 
> It seems this was more or less intentional, even though it would break code 
> which was expecting and handling the {{UnknownTopicOrPartitionException}}. A 
> workaround is to use {{retries=1}} and inspect the cause of the 
> {{TimeoutException}}, but this isn't really suitable for cases where the same 
> Admin client instance is being used for other calls where retries is 
> desirable.
> Furthermore as well as the intended effect on {{listOffsets()}} it seems that 
> the change could actually affect other methods of Admin.
> More generally, the Admin client API is vague about which exceptions can 
> propagate from which methods. This means that it's not possible to say, in 
> cases like this, whether the calling code _should_ have been relying on the 
> {{UnknownTopicOrPartitionException}} or not.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka-site] guozhangwang merged pull request #399: Minor typo: "result is a" > "result in a"

[GitBox]

guozhangwang merged pull request #399:
URL: https://github.com/apache/kafka-site/pull/399


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #147

[Apache Jenkins Server]

See 


Changes:

[Randall Hauch] KAFKA-12879: Revert changes from KAFKA-12339 and instead add 
retry capability to KafkaBasedLog (#11797)

[Randall Hauch] KAFKA-12879: Addendum to reduce flakiness of tests (#11871)

[Randall Hauch] KAFKA-12879: Remove extra sleep (#11872)


--
[...truncated 4.69 MB...]
kafka.KafkaTest > testZkSslOcspEnable PASSED

kafka.KafkaTest > testConnectionsMaxReauthMsDefault STARTED

kafka.KafkaTest > testConnectionsMaxReauthMsDefault PASSED

kafka.KafkaTest > testZkSslTrustStoreLocation STARTED

kafka.KafkaTest > testZkSslTrustStoreLocation PASSED

kafka.KafkaTest > testZkSslEnabledProtocols STARTED

kafka.KafkaTest > testZkSslEnabledProtocols PASSED

kafka.KafkaTest > testKafkaSslPasswords STARTED

kafka.KafkaTest > testKafkaSslPasswords PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgs STARTED

kafka.KafkaTest > testGetKafkaConfigFromArgs PASSED

kafka.KafkaTest > testZkSslClientEnable STARTED

kafka.KafkaTest > testZkSslClientEnable PASSED

kafka.KafkaTest > testZookeeperTrustStorePassword STARTED

kafka.KafkaTest > testZookeeperTrustStorePassword PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheEnd STARTED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheEnd PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsOnly STARTED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsOnly PASSED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheBegging STARTED

kafka.KafkaTest > testGetKafkaConfigFromArgsNonArgsAtTheBegging PASSED

kafka.KafkaTest > testZkSslKeyStoreLocation STARTED

kafka.KafkaTest > testZkSslKeyStoreLocation PASSED

kafka.KafkaTest > testZkSslCrlEnable STARTED

kafka.KafkaTest > testZkSslCrlEnable PASSED

kafka.KafkaTest > testZkSslEndpointIdentificationAlgorithm STARTED

kafka.KafkaTest > testZkSslEndpointIdentificationAlgorithm PASSED

kafka.KafkaTest > testZkSslTrustStoreType STARTED

kafka.KafkaTest > testZkSslTrustStoreType PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testNonBlockingProducer STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.PlaintextProducerSendTest > testNonBlockingProducer PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > 
testSendRecordBatchWithMaxRequestSizeAndHigher STARTED

kafka.api.PlaintextProducerSendTest > 
testSendRecordBatchWithMaxRequestSizeAndHigher PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero STARTED

kafka.api.PlaintextProducerSendTest > testBatchSizeZero PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi Jun,

30.  Clarified.

31. I plan to do some benchmarking once implementation is finished, I'll
update the KIP with the results once I have them.  The reason to make it
default is that it won't be used otherwise and we won't know if it's good
or not in practical workloads.

-Artem

On Thu, Mar 10, 2022 at 11:42 AM Jun Rao  wrote:

> Hi, Artem,
>
> Thanks for the updated KIP. A couple of more comments.
>
> 30. For the 3 new configs, it would be useful to make it clear that they
> are only relevant when the partitioner class is null.
>
> 31. partitioner.adaptive.partitioning.enable : I am wondering whether it
> should default to true. This is a more complex behavior than "uniform
> sticky" and may take some time to get right. If we do want to enable it by
> default, it would be useful to validate it with some test results.
>
> Jun
>
>
>
>
> On Wed, Mar 9, 2022 at 10:05 PM Artem Livshits
>  wrote:
>
> > Thank you for feedback, I've discussed this offline with some of the
> folks
> > and updated the KIP.  The main change is that now instead of using
> > DefaultPartitioner and UniformStickyPartitioners as flags, in the new
> > proposal the default partitioner is null, so if no custom partitioner is
> > specified then the partitioning logic is implemented in KafkaProducer.
> > Compatibility section is updated as well.  Also the configuration options
> > are renamed to be more consistent.
> >
> > -Artem
> >
> > On Fri, Mar 4, 2022 at 10:38 PM Luke Chen  wrote:
> >
> > > Hi Artem,
> > >
> > > Thanks for your explanation and update to the KIP.
> > > Some comments:
> > >
> > > 5. In the description for `enable.adaptive.partitioning`, the `false`
> > case,
> > > you said:
> > > > the producer will try to distribute messages uniformly.
> > > I think we should describe the possible skewing distribution.
> Otherwise,
> > > user might be confused about why adaptive partitioning is important.
> > >
> > > 6. In the description for `partition.availability.timeout.ms`, I think
> > we
> > > should mention in the last sentence about if
> > `enable.adaptive.partitioning`
> > > is disabled this logic is also disabled.
> > >
> > > 7. Similar thoughts as Ismael, I think we should have a POC and test to
> > > prove that this adaptive partitioning algorithm can have better uniform
> > > partitioning, compared with original sticky one.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Mar 4, 2022 at 9:22 PM Ismael Juma  wrote:
> > >
> > > > Regarding `3`, we should only deprecate it if we're sure the new
> > approach
> > > > handles all cases better. Are we confident about that for both of the
> > > > previous partitioners?
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Mar 4, 2022 at 1:37 AM David Jacot
>  > >
> > > > wrote:
> > > >
> > > > > Hi Artem,
> > > > >
> > > > > Thanks for the KIP! I have a few comments:
> > > > >
> > > > > 1. In the preamble of the proposed change section, there is still a
> > > > > mention of the
> > > > > -1 approach. My understanding is that we have moved away from it
> now.
> > > > >
> > > > > 2. I am a bit concerned by the trick suggested about the
> > > > > DefaultPartitioner and
> > > > > the UniformStickyPartitioner. I do agree that implementing the
> logic
> > in
> > > > the
> > > > > producer itself is a good thing. However, it is weird from a user
> > > > > perspective
> > > > > that he can set a class as partitioner that is not used in the
> end. I
> > > > > think that
> > > > > this will be confusing for our users. Have we considered changing
> the
> > > > > default
> > > > > value of partitioner.class to null to indicate that the new
> built-in
> > > > > partitioner
> > > > > must be used? By default, the built-in partitioner would be used
> > unless
> > > > the
> > > > > user explicitly specify one. The downside is that the new default
> > > > behavior
> > > > > would not work if the user explicitly specify the partitioner but
> we
> > > > could
> > > > > mitigate this with my next point.
> > > > >
> > > > > 3. Related to the previous point, I think that we could deprecate
> > both
> > > > the
> > > > > DefaultPartitioner and the UniformStickyPartitioner. I would also
> > add a
> > > > > warning if one of them is explicitly provided by the user to inform
> > > them
> > > > > that they should switch to the new built-in one. I am pretty sure
> > that
> > > > most
> > > > > of the folks use the default configuration anyway.
> > > > >
> > > > > 4. It would be great if we could explain why the -1 way was
> rejected.
> > > At
> > > > > the moment, the rejected alternative only explain the idea but does
> > not
> > > > > say why we rejected it.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Fri, Mar 4, 2022 at 6:03 AM Artem Livshits
> > > > >  wrote:
> > > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > 2. Removed the option from the KIP.  Now the sticky partitioning
> > > > > threshold
> > > > > > is hardcoded to batch.size.
> > > > > >
> > > > > > 20. Added the 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Jun Rao]

Hi, Kirk, Sarat,

Thanks for the reply.

28. On the broker, we typically use Yammer metrics. Only for metrics that
depend on Kafka metric features (e.g., quota), we use the Kafka metric.
Yammer metrics have 4 types: gauge, meter, histogram and timer. meter
calculates a rate, but also exposes an accumulated value.

29. The Histogram class in org.apache.kafka.common.metrics.stats was never
used in the client metrics. The implementation of Histogram only provides a
fixed number of values in the domain and may not capture the quantiles very
accurately. So, we punted on using it.

Thanks,

Jun



On Thu, Mar 10, 2022 at 10:59 AM Sarat Kakarla
 wrote:

> Jun,
>
>   >>  28. For the broker metrics, could you spell out the full metric name
>   >>   including groups, tags, etc? We typically don't add the broker_id
> label for
>   >>   broker metrics. Also, brokers use Yammer metrics, which doesn't
> have type
>   >>   Sum.
>
> Sure,  I will update the KIP-714 with the above information, will remove
> the broker-id label from the metrics.
>
> Regarding the type is CumulativeSum the right type to use in the place of
> Sum?
>
> Thanks
> Sarat
>
>
> On 3/8/22, 5:48 PM, "Jun Rao"  wrote:
>
> Hi, Magnus, Sarat and Xavier,
>
> Thanks for the reply. A few more comments below.
>
> 20. It seems that we are piggybacking the plugin on the
> existing MetricsReporter. So, this seems fine.
>
> 21. That could work. Are we requiring any additional jar dependency on
> the
> client? Or, are you suggesting that we check the runtime dependency to
> pick
> the compression codec?
>
> 28. For the broker metrics, could you spell out the full metric name
> including groups, tags, etc? We typically don't add the broker_id
> label for
> broker metrics. Also, brokers use Yammer metrics, which doesn't have
> type
> Sum.
>
> 29. There are several client metrics listed as histogram. However, the
> java
> client currently doesn't support histogram type.
>
> 30. Could you show an example of the metric payload in
> PushTelemetryRequest
> to help understand how we organize metrics at different levels (per
> instance, per topic, per partition, per broker, etc)?
>
> 31. Could you add a bit more detail on which client thread sends the
> PushTelemetryRequest?
>
> Thanks,
>
> Jun
>
> On Mon, Mar 7, 2022 at 11:48 AM Magnus Edenhill 
> wrote:
>
> > Hi Jun,
> >
> > thanks for your initiated questions, see my answers below.
> > There's been a number of clarifications to the KIP.
> >
> >
> >
> > Den tors 27 jan. 2022 kl 20:08 skrev Jun Rao
> :
> >
> > > Hi, Magnus,
> > >
> > > Thanks for updating the KIP. The overall approach makes sense to
> me. A
> > few
> > > more detailed comments below.
> > >
> > > 20. ClientTelemetry: Should it be extending configurable and
> closable?
> > >
> >
> > I'll pass this question to Sarat and/or Xavier.
> >
> >
> >
> > > 21. Compression of the metrics on the client: what's the default?
> > >
> >
> > How about we specify a prioritized list: zstd, lz4, snappy, gzip?
> > But ultimately it is up to what the client supports.
> >
> >
> > 23. A client instance is considered a metric resource and the
> > > resource-level (thus client instance level) labels could include:
> > > client_software_name=confluent-kafka-python
> > > client_software_version=v2.1.3
> > > client_instance_id=B64CD139-3975-440A-91D4
> > > transactional_id=someTxnApp
> > > Are those labels added in PushTelemetryRequest? If so, are they per
> > metric
> > > or per request?
> > >
> >
> >
> > client_software* and client_instance_id are not added by the client,
> but
> > available to
> > the broker-side metrics plugin for adding as it see fits, remove
> them from
> > the KIP.
> >
> > As for transactional_id, group_id, etc, which I believe will be
> useful in
> > troubleshooting,
> > are included only once (per push) as resource-level attributes (the
> client
> > instance is a singular resource).
> >
> >
> > >
> > > 24.  "the broker will only send
> > > GetTelemetrySubscriptionsResponse.DeltaTemporality=True" :
> > > 24.1 If it's always true, does it need to be part of the protocol?
> > >
> >
> > We're anticipating that it will take a lot longer to upgrade the
> majority
> > of clients than the
> > broker/plugin side, which is why we want the client to support both
> > temporalities out-of-the-box
> > so that cumulative reporting can be turned on seamlessly in the
> future.
> >
> >
> >
> > > 24.2 Does delta only apply to Counter type?
> > >
> >
> >
> > And Histograms. More details in Xavier's OTLP link.
> >
> >
> >
> > > 24.3 In the delta representation, the first request 

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Jun Rao]

Hi, Artem,

Thanks for the updated KIP. A couple of more comments.

30. For the 3 new configs, it would be useful to make it clear that they
are only relevant when the partitioner class is null.

31. partitioner.adaptive.partitioning.enable : I am wondering whether it
should default to true. This is a more complex behavior than "uniform
sticky" and may take some time to get right. If we do want to enable it by
default, it would be useful to validate it with some test results.

Jun




On Wed, Mar 9, 2022 at 10:05 PM Artem Livshits
 wrote:

> Thank you for feedback, I've discussed this offline with some of the folks
> and updated the KIP.  The main change is that now instead of using
> DefaultPartitioner and UniformStickyPartitioners as flags, in the new
> proposal the default partitioner is null, so if no custom partitioner is
> specified then the partitioning logic is implemented in KafkaProducer.
> Compatibility section is updated as well.  Also the configuration options
> are renamed to be more consistent.
>
> -Artem
>
> On Fri, Mar 4, 2022 at 10:38 PM Luke Chen  wrote:
>
> > Hi Artem,
> >
> > Thanks for your explanation and update to the KIP.
> > Some comments:
> >
> > 5. In the description for `enable.adaptive.partitioning`, the `false`
> case,
> > you said:
> > > the producer will try to distribute messages uniformly.
> > I think we should describe the possible skewing distribution. Otherwise,
> > user might be confused about why adaptive partitioning is important.
> >
> > 6. In the description for `partition.availability.timeout.ms`, I think
> we
> > should mention in the last sentence about if
> `enable.adaptive.partitioning`
> > is disabled this logic is also disabled.
> >
> > 7. Similar thoughts as Ismael, I think we should have a POC and test to
> > prove that this adaptive partitioning algorithm can have better uniform
> > partitioning, compared with original sticky one.
> >
> > Thank you.
> > Luke
> >
> > On Fri, Mar 4, 2022 at 9:22 PM Ismael Juma  wrote:
> >
> > > Regarding `3`, we should only deprecate it if we're sure the new
> approach
> > > handles all cases better. Are we confident about that for both of the
> > > previous partitioners?
> > >
> > > Ismael
> > >
> > > On Fri, Mar 4, 2022 at 1:37 AM David Jacot  >
> > > wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > Thanks for the KIP! I have a few comments:
> > > >
> > > > 1. In the preamble of the proposed change section, there is still a
> > > > mention of the
> > > > -1 approach. My understanding is that we have moved away from it now.
> > > >
> > > > 2. I am a bit concerned by the trick suggested about the
> > > > DefaultPartitioner and
> > > > the UniformStickyPartitioner. I do agree that implementing the logic
> in
> > > the
> > > > producer itself is a good thing. However, it is weird from a user
> > > > perspective
> > > > that he can set a class as partitioner that is not used in the end. I
> > > > think that
> > > > this will be confusing for our users. Have we considered changing the
> > > > default
> > > > value of partitioner.class to null to indicate that the new built-in
> > > > partitioner
> > > > must be used? By default, the built-in partitioner would be used
> unless
> > > the
> > > > user explicitly specify one. The downside is that the new default
> > > behavior
> > > > would not work if the user explicitly specify the partitioner but we
> > > could
> > > > mitigate this with my next point.
> > > >
> > > > 3. Related to the previous point, I think that we could deprecate
> both
> > > the
> > > > DefaultPartitioner and the UniformStickyPartitioner. I would also
> add a
> > > > warning if one of them is explicitly provided by the user to inform
> > them
> > > > that they should switch to the new built-in one. I am pretty sure
> that
> > > most
> > > > of the folks use the default configuration anyway.
> > > >
> > > > 4. It would be great if we could explain why the -1 way was rejected.
> > At
> > > > the moment, the rejected alternative only explain the idea but does
> not
> > > > say why we rejected it.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Fri, Mar 4, 2022 at 6:03 AM Artem Livshits
> > > >  wrote:
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > 2. Removed the option from the KIP.  Now the sticky partitioning
> > > > threshold
> > > > > is hardcoded to batch.size.
> > > > >
> > > > > 20. Added the corresponding wording to the KIP.
> > > > >
> > > > > -Artem
> > > > >
> > > > > On Thu, Mar 3, 2022 at 10:52 AM Jun Rao 
> > > > wrote:
> > > > >
> > > > > > Hi, Artem,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 1. Sounds good.
> > > > > >
> > > > > > 2. If we don't expect users to change it, we probably could just
> > > leave
> > > > out
> > > > > > the new config. In general, it's easy to add a new config, but
> hard
> > > to
> > > > > > remove an existing config.
> > > > > >
> > > > > > 20. The two new configs enable.adaptive.partitioning and
> > > > > > 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Sarat Kakarla]

Jun,
 
  >>  28. For the broker metrics, could you spell out the full metric name
  >>   including groups, tags, etc? We typically don't add the broker_id label 
for
  >>   broker metrics. Also, brokers use Yammer metrics, which doesn't have type
  >>   Sum.

Sure,  I will update the KIP-714 with the above information, will remove the 
broker-id label from the metrics.

Regarding the type is CumulativeSum the right type to use in the place of Sum?

Thanks
Sarat


On 3/8/22, 5:48 PM, "Jun Rao"  wrote:

Hi, Magnus, Sarat and Xavier,

Thanks for the reply. A few more comments below.

20. It seems that we are piggybacking the plugin on the
existing MetricsReporter. So, this seems fine.

21. That could work. Are we requiring any additional jar dependency on the
client? Or, are you suggesting that we check the runtime dependency to pick
the compression codec?

28. For the broker metrics, could you spell out the full metric name
including groups, tags, etc? We typically don't add the broker_id label for
broker metrics. Also, brokers use Yammer metrics, which doesn't have type
Sum.

29. There are several client metrics listed as histogram. However, the java
client currently doesn't support histogram type.

30. Could you show an example of the metric payload in PushTelemetryRequest
to help understand how we organize metrics at different levels (per
instance, per topic, per partition, per broker, etc)?

31. Could you add a bit more detail on which client thread sends the
PushTelemetryRequest?

Thanks,

Jun

On Mon, Mar 7, 2022 at 11:48 AM Magnus Edenhill  wrote:

> Hi Jun,
>
> thanks for your initiated questions, see my answers below.
> There's been a number of clarifications to the KIP.
>
>
>
> Den tors 27 jan. 2022 kl 20:08 skrev Jun Rao :
>
> > Hi, Magnus,
> >
> > Thanks for updating the KIP. The overall approach makes sense to me. A
> few
> > more detailed comments below.
> >
> > 20. ClientTelemetry: Should it be extending configurable and closable?
> >
>
> I'll pass this question to Sarat and/or Xavier.
>
>
>
> > 21. Compression of the metrics on the client: what's the default?
> >
>
> How about we specify a prioritized list: zstd, lz4, snappy, gzip?
> But ultimately it is up to what the client supports.
>
>
> 23. A client instance is considered a metric resource and the
> > resource-level (thus client instance level) labels could include:
> > client_software_name=confluent-kafka-python
> > client_software_version=v2.1.3
> > client_instance_id=B64CD139-3975-440A-91D4
> > transactional_id=someTxnApp
> > Are those labels added in PushTelemetryRequest? If so, are they per
> metric
> > or per request?
> >
>
>
> client_software* and client_instance_id are not added by the client, but
> available to
> the broker-side metrics plugin for adding as it see fits, remove them from
> the KIP.
>
> As for transactional_id, group_id, etc, which I believe will be useful in
> troubleshooting,
> are included only once (per push) as resource-level attributes (the client
> instance is a singular resource).
>
>
> >
> > 24.  "the broker will only send
> > GetTelemetrySubscriptionsResponse.DeltaTemporality=True" :
> > 24.1 If it's always true, does it need to be part of the protocol?
> >
>
> We're anticipating that it will take a lot longer to upgrade the majority
> of clients than the
> broker/plugin side, which is why we want the client to support both
> temporalities out-of-the-box
> so that cumulative reporting can be turned on seamlessly in the future.
>
>
>
> > 24.2 Does delta only apply to Counter type?
> >
>
>
> And Histograms. More details in Xavier's OTLP link.
>
>
>
> > 24.3 In the delta representation, the first request needs to send the
> full
> > value, how does the broker plugin know whether a value is full or delta?
> >
>
> The client may (should) send the start time for each metric sample,
> indicating when
> the metric began to be collected.
> We've discussed whether this should be the client instance start time or
> the time when a matching
> metric subscription for that metric is received.
> For completeness we recommend using the former, the client instance start
> time.
>
>
>
> > 25. quota:
> > 25.1 Since we are fitting PushTelemetryRequest into the existing request
> > quota, it would be useful to document the impact, i.e. client metric
> > throttling causes the data from the same client to be delayed.
> > 25.2 Is PushTelemetryRequest subject to the write bandwidth quota like
> the
> > producer?
> >
>
>
> Yes, it 

[RESULTS] [VOTE] Release Kafka version 3.0.1

[Mickael Maison]

This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Tom Bentley
* David Jacot
* Bill Bejeck

Committers:
* Luke Chen

Community:
* Jakub Scholz
* Michal Tóth
* Federico Valeri

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/thread/bnkfpd25m5vbn55hyovxn41487hl2oz8

I'll continue with the release process and the release announcement
will follow in the next few days.

Mickael

Re: New contributor

[Mickael Maison]

I've added you to the wiki.

Thanks for your interest in Apache Kafka!

On Thu, Mar 10, 2022 at 6:42 PM Federico Valeri  wrote:
>
> Thanks Mickael, now you should have the Confluence account with the
> same username (fvaleri).
>
> On Thu, Mar 10, 2022 at 6:23 PM Mickael Maison  
> wrote:
> >
> > Hi Federico,
> >
> > I've added you to the contributor list on JIRA but I couldn't find
> > your user in Confluence. Can you verify the username of your
> > Confluence account?
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Thu, Mar 10, 2022 at 5:29 PM Federico Valeri  
> > wrote:
> > >
> > > Hello, could you add me to the contributor list?
> > >
> > > Jira/Wiki user: fvaleri
> > >
> > > Br
> > > Fede

Re: New contributor

[Federico Valeri]

Thanks Mickael, now you should have the Confluence account with the
same username (fvaleri).

On Thu, Mar 10, 2022 at 6:23 PM Mickael Maison  wrote:
>
> Hi Federico,
>
> I've added you to the contributor list on JIRA but I couldn't find
> your user in Confluence. Can you verify the username of your
> Confluence account?
>
> Thanks,
> Mickael
>
>
>
> On Thu, Mar 10, 2022 at 5:29 PM Federico Valeri  wrote:
> >
> > Hello, could you add me to the contributor list?
> >
> > Jira/Wiki user: fvaleri
> >
> > Br
> > Fede

Re: New contributor

[Mickael Maison]

Hi Federico,

I've added you to the contributor list on JIRA but I couldn't find
your user in Confluence. Can you verify the username of your
Confluence account?

Thanks,
Mickael



On Thu, Mar 10, 2022 at 5:29 PM Federico Valeri  wrote:
>
> Hello, could you add me to the contributor list?
>
> Jira/Wiki user: fvaleri
>
> Br
> Fede

Re: [VOTE] 3.0.1 RC0

[Mickael Maison]

Hi,

David, Bill:
Yes I haven't updated the website yet. I'll do that next.

Thanks everyone for checking the release, I'll close the vote and send
the results

On Thu, Mar 10, 2022 at 5:29 PM Federico Valeri  wrote:
>
> Hi Mickael,
>
> - Executed kafka-verify-rc with no issues
> - Used the staged Scala 2.13 binaries to run a MM2 DR scenario
> - Used staging Maven repository with my own test Java clients
>
> +1 (non-binding)
>
> Thanks
> Fede
>
>
>
> On Wed, Mar 9, 2022 at 6:01 PM Bill Bejeck  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for running the release.
> >
> > I did the following steps to validate the release:
> >
> >1. Validated signatures and checksums
> >2. Built from source and ran the unit tests
> >3. I ran the quickstart steps for ZK and Kafka Streams
> >4. Spot checked the docs and Javadocs
> >
> > I notice the same issue as David regarding referencing the 3.0.0 releases.
> > I agree with him that we don't need to block the release and update the
> > docs separately.
> >
> > +1(binding)
> >
> > Regards,
> > Bill
> >
> > On Wed, Mar 9, 2022 at 10:48 AM David Jacot 
> > wrote:
> >
> > > Thanks for running the release, Mickael.
> > >
> > > I performed the following validations:
> > > * Verified all checksums and signatures.
> > > * Built from source and ran unit tests.
> > > * Ran the first quickstart steps for both ZK and KRaft.
> > > * Spotchecked the Javadocs.
> > >
> > > However, the document still references 3.0.0 in all places. It seems
> > > that it has not been updated for 3.0.1 yet. At least, I don't see any
> > > commits for that. I would not block the release on this because we
> > > can update the documentation independently.
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > David
> > >
> > > On Wed, Mar 9, 2022 at 11:32 AM Tom Bentley  wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > In addition to the results others have posted I've also validated the
> > > > checksums and keys, built from source and executed the unit and
> > > integration
> > > > tests.
> > > >
> > > > Overall I'm +1 (binding).
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > > > On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > >  executed https://github.com/tombentley/kafka-verify-rc - with no
> > > issues.
> > > > > All checks have passed.
> > > > > * checksums, keys
> > > > > * unit tests
> > > > > * executed few system tests - all passed
> > > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Thank you
> > > > >
> > > > >
> > > > > ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Thanks for running the release!
> > > > > >
> > > > > > I did the following:
> > > > > >1. Validated the scala 2.13 checksums
> > > > > >2. Spot checked the java docs
> > > > > >3. Ran the quick start with scala 2.13 (found a minor bug
> > > KAFKA-13718
> > > > > > , won't block the
> > > > > > release)
> > > > > >
> > > > > > +1 (non-binding).
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Here is a successful Jenkins build for the 3.0 branch:
> > > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > > > > > >
> > > > > > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz 
> > > wrote:
> > > > > > > >
> > > > > > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> > > > > staging
> > > > > > > > Maven repository to run my tests. All seems to work fine, no
> > > issues
> > > > > > > found.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Jakub
> > > > > > > >
> > > > > > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison <
> > > mimai...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > >
> > > > > > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > > > > > >
> > > > > > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> > > > > fixed
> > > > > > > > > since 3.0.0.
> > > > > > > > >
> > > > > > > > > Release notes for the 3.0.1 release:
> > > > > > > > >
> > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > > > > > >
> > > > > > > > > *** Please download, test and vote by Thursday, March 10, 6pm
> > > GMT
> > > > > ***
> > > > > > > > >
> > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > release:
> > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > >
> > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > > > > > >
> > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > >
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > 

New contributor

[Federico Valeri]

Hello, could you add me to the contributor list?

Jira/Wiki user: fvaleri

Br
Fede

Re: [VOTE] 3.0.1 RC0

[Federico Valeri]

Hi Mickael,

- Executed kafka-verify-rc with no issues
- Used the staged Scala 2.13 binaries to run a MM2 DR scenario
- Used staging Maven repository with my own test Java clients

+1 (non-binding)

Thanks
Fede



On Wed, Mar 9, 2022 at 6:01 PM Bill Bejeck  wrote:
>
> Hi Mickael,
>
> Thanks for running the release.
>
> I did the following steps to validate the release:
>
>1. Validated signatures and checksums
>2. Built from source and ran the unit tests
>3. I ran the quickstart steps for ZK and Kafka Streams
>4. Spot checked the docs and Javadocs
>
> I notice the same issue as David regarding referencing the 3.0.0 releases.
> I agree with him that we don't need to block the release and update the
> docs separately.
>
> +1(binding)
>
> Regards,
> Bill
>
> On Wed, Mar 9, 2022 at 10:48 AM David Jacot 
> wrote:
>
> > Thanks for running the release, Mickael.
> >
> > I performed the following validations:
> > * Verified all checksums and signatures.
> > * Built from source and ran unit tests.
> > * Ran the first quickstart steps for both ZK and KRaft.
> > * Spotchecked the Javadocs.
> >
> > However, the document still references 3.0.0 in all places. It seems
> > that it has not been updated for 3.0.1 yet. At least, I don't see any
> > commits for that. I would not block the release on this because we
> > can update the documentation independently.
> >
> > +1 (binding)
> >
> > Best,
> > David
> >
> > On Wed, Mar 9, 2022 at 11:32 AM Tom Bentley  wrote:
> > >
> > > Hi Mickael,
> > >
> > > In addition to the results others have posted I've also validated the
> > > checksums and keys, built from source and executed the unit and
> > integration
> > > tests.
> > >
> > > Overall I'm +1 (binding).
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> > > On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:
> > >
> > > > Hello,
> > > >
> > > >  executed https://github.com/tombentley/kafka-verify-rc - with no
> > issues.
> > > > All checks have passed.
> > > > * checksums, keys
> > > > * unit tests
> > > > * executed few system tests - all passed
> > > >
> > > > +1 (non-binding).
> > > >
> > > > Thank you
> > > >
> > > >
> > > > ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for running the release!
> > > > >
> > > > > I did the following:
> > > > >1. Validated the scala 2.13 checksums
> > > > >2. Spot checked the java docs
> > > > >3. Ran the quick start with scala 2.13 (found a minor bug
> > KAFKA-13718
> > > > > , won't block the
> > > > > release)
> > > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Thank you.
> > > > >
> > > > > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Here is a successful Jenkins build for the 3.0 branch:
> > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > > > > >
> > > > > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz 
> > wrote:
> > > > > > >
> > > > > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> > > > staging
> > > > > > > Maven repository to run my tests. All seems to work fine, no
> > issues
> > > > > > found.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison <
> > mimai...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > >
> > > > > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > > > > >
> > > > > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> > > > fixed
> > > > > > > > since 3.0.0.
> > > > > > > >
> > > > > > > > Release notes for the 3.0.1 release:
> > > > > > > >
> > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > > > > >
> > > > > > > > *** Please download, test and vote by Thursday, March 10, 6pm
> > GMT
> > > > ***
> > > > > > > >
> > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > >
> > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > > > > >
> > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > >
> > > > > > > > * Javadoc:
> > > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > > > > >
> > > > > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > > > > >
> > > > > > > > * Documentation:
> > > > > > > > https://kafka.apache.org/30/documentation.html
> > > > > > > >
> > > > > > > > * Protocol:
> > > > > > > > https://kafka.apache.org/30/protocol.html
> > > > > > > >
> > > > > > > > * 

Re: [DISCUSS] Apache Kafka 3.1.1

[Mickael Maison]

+1
Thanks Tom for volunteering!

On Thu, Mar 10, 2022 at 10:34 AM Tom Bentley  wrote:
>
> I think I was supposed to publish the release plan at the same time as
> calling the vote for the release.
> Here it is:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.1
>
> Kind regards,
>
> Tom
>
> On Thu, 10 Mar 2022 at 08:49, Bruno Cadonna  wrote:
>
> > Thank you, Tom!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 09.03.22 19:55, David Jacot wrote:
> > > +1. Thanks Tom!
> > >
> > > Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a écrit :
> > >
> > >> Thanks Tom!  It's a +1 for me.
> > >>
> > >> -Bill
> > >>
> > >> On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:
> > >>
> > >>> Thanks Tom. +1
> > >>>
> > >>> Ismael
> > >>>
> > >>>
> > >>> On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley 
> > wrote:
> > >>>
> >  Hi,
> > 
> >  I'd like to volunteer to be the release manager for the 3.1.1 bugfix
> >  release.
> > 
> >  Kind regards,
> > 
> >  Tom
> > 
> > >>>
> > >>
> > >
> >
> >

[DISCUSS] Support of ARM and PowerPC

[Mickael Maison]

Hi,

We recently added ARM (aarch64) and PowerPC (ppc64le) support to the
Apache Kafka CI. So now all builds are tested (and work!) on x86_64,
aarch64 and ppc64le. Thanks again to the community members that
donated the machines.

Since we test these platforms, should we consider them as officially supported?

In our docs, we make statements about the Java versions we support. In
the Hardware and OS section [0], we also have a few statements about
the operating systems supported (which may not necessarily still be
true as it mentions Solaris being tested), but we don't actually say
anything about the hardware. Even if Kafka is a JVM based software,
some of our dependencies like rocksdb, zstd, etc rely on native code
so they may not work on all environments.

Note that on aarch64 and ppc64le we currently we only run unit tests.
We should also run integration tests.

0: https://kafka.apache.org/documentation/#hwandos

Thanks,
Mickael

Re: [DISCUSS] Apache Kafka 3.1.1

[Tom Bentley]

I think I was supposed to publish the release plan at the same time as
calling the vote for the release.
Here it is:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.1

Kind regards,

Tom

On Thu, 10 Mar 2022 at 08:49, Bruno Cadonna  wrote:

> Thank you, Tom!
>
> +1
>
> Best,
> Bruno
>
> On 09.03.22 19:55, David Jacot wrote:
> > +1. Thanks Tom!
> >
> > Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a écrit :
> >
> >> Thanks Tom!  It's a +1 for me.
> >>
> >> -Bill
> >>
> >> On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:
> >>
> >>> Thanks Tom. +1
> >>>
> >>> Ismael
> >>>
> >>>
> >>> On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley 
> wrote:
> >>>
>  Hi,
> 
>  I'd like to volunteer to be the release manager for the 3.1.1 bugfix
>  release.
> 
>  Kind regards,
> 
>  Tom
> 
> >>>
> >>
> >
>
>

Re: [DISCUSS] Apache Kafka 3.1.1

[Bruno Cadonna]

Thank you, Tom!

+1

Best,
Bruno

On 09.03.22 19:55, David Jacot wrote:

+1. Thanks Tom!

Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a écrit :


Thanks Tom!  It's a +1 for me.

-Bill

On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:


Thanks Tom. +1

Ismael


On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley  wrote:


Hi,

I'd like to volunteer to be the release manager for the 3.1.1 bugfix
release.

Kind regards,

Tom