Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

[Boyang Chen]

On Mon, Jul 27, 2020 at 4:58 AM Unmesh Joshi  wrote:

> Just checked etcd and zookeeper code, and both support leader to step down
> as a follower to make sure there are no two leaders if the leader has been
> disconnected from the majority of the followers
> For etcd this is https://github.com/etcd-io/etcd/issues/3866
> For Zookeeper its https://issues.apache.org/jira/browse/ZOOKEEPER-1699
> I was just thinking if it would be difficult to implement in the Pull based
> model, but I guess not. It is possibly the same way ISR list is managed
> currently, if leader of the controller quorum loses majority of the
> followers, it should step down and become follower, that way, telling
> client in time that it was disconnected from the quorum, and not keep on
> sending state metadata to clients.
>
> Thanks,
> Unmesh
>
>
> On Mon, Jul 27, 2020 at 9:31 AM Unmesh Joshi 
> wrote:
>
> > >>Could you clarify on this question? Which part of the raft group
> doesn't
> > >>know about leader dis-connection?
> > The leader of the controller quorum is partitioned from the controller
> > cluster, and a different leader is elected for the remaining controller
> > cluster.
>
I see your concern. For KIP-595 implementation, since there is no regular
heartbeats sent
from the leader to the followers, we decided to piggy-back on the fetch
timeout so that if the leader did not receive Fetch
requests from a majority of the quorum for that amount of time, it would
begin a new election and
start sending VoteRequest to voter nodes in the cluster to understand the
latest quorum. You could
find more details in this section

.


> > I think there are two things here,
> > 1.  The old leader will not know if it's disconnected from the rest of
> the
> > controller quorum cluster unless it receives BeginQuorumEpoch from the
> new
> > leader. So it will keep on serving stale metadata to the clients
> (Brokers,
> > Producers and Consumers)
> > 2. I assume, the Broker Leases will be managed on the controller quorum
> > leader. This partitioned leader will keep on tracking broker leases it
> has,
> > while the new leader of the quorum will also start managing broker
> leases.
> > So while the quorum leader is partitioned, there will be two membership
> > views of the kafka brokers managed on two leaders.
> > Unless broker heartbeats are also replicated as part of the Raft log,
> > there is no way to solve this?
> > I know LogCabin implementation does replicate client heartbeats. I
> suspect
> > that the same issue is there in Zookeeper, which does not replicate
> client
> > Ping requests..
> >
> > Thanks,
> > Unmesh
> >
> >
> >
> > On Mon, Jul 27, 2020 at 6:23 AM Boyang Chen 
> > wrote:
> >
> >> Thanks for the questions Unmesh!
> >>
> >> On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi 
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > In the FetchRequest Handling, how to make sure we handle scenarios
> where
> >> > the leader might have been disconnected from the cluster, but doesn't
> >> know
> >> > yet?
> >> >
> >> Could you clarify on this question? Which part of the raft group doesn't
> >> know about leader
> >> dis-connection?
> >>
> >>
> >> > As discussed in the Raft Thesis section 6.4, the linearizable
> semantics
> >> of
> >> > read requests is implemented in LogCabin by sending heartbeat to
> >> followers
> >> > and waiting till the heartbeats are successful to make sure that the
> >> leader
> >> > is still the leader.
> >> > I think for the controller quorum to make sure none of the consumers
> get
> >> > stale data, it's important to have linearizable semantics? In the pull
> >> > based model, the leader will need to wait for heartbeats from the
> >> followers
> >> > before returning each fetch request from the consumer then? Or do we
> >> need
> >> > to introduce some other request?
> >> > (Zookeeper does not have linearizable semantics for read requests, but
> >> as
> >> > of now all the kafka interactions are through writes and watches).
> >> >
> >> > This is a very good question. For our v1 implementation we are not
> >> aiming
> >> to guarantee linearizable read, which
> >> would be considered as a follow-up effort. Note that today in Kafka
> there
> >> is no guarantee on the metadata freshness either,
> >> so no regression is introduced.
> >>
> >>
> >> > Thanks,
> >> > Unmesh
> >> >
> >> > On Fri, Jul 24, 2020 at 11:36 PM Jun Rao  wrote:
> >> >
> >> > > Hi, Jason,
> >> > >
> >> > > Thanks for the reply.
> >> > >
> >> > > 101. Sounds good. Regarding clusterId, I am not sure storing it in
> the
> >> > > metadata log is enough. For example, the vote request includes
> >> clusterId.
> >> > > So, no one can vote until they know the clusterId. Also, it would be
> >> > useful
> >> > > to support the case when a voter completely loses its disk and needs
> >> to
> >> > > recover.
> >> > >
> >> > > 210. 

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Boyang Chen]

Hey there,

I'm re-opening this thread because after some initial implementations
started, we spotted some gaps in the approved KIP as well as some
inconsistencies with KIP-631 controller. There are a couple of addendums to
the existing KIP, specifically:

1. As the controller is foreseen to be only accessible to the brokers, the
new admin client would not have direct access to the controller. It is
guaranteed on the MetadataResponse level which no longer provides
`ControllerId` to client side requests.

2. The broker would forward any direct ZK path mutation requests, including
topic creation/deletion, reassignment, etc since we deprecate the direct
controller access on the client side. No more protocol version bump is
necessary for the configuration requests.

3. To make sure forwarding requests pass the authorization, broker
principal CLUSTER_ACTION would be allowed to be used as an alternative
authentication method for a variety of principal operations, including
ALTER, ALTER_CONFIG, DELETE, etc. It is because the forwarding request
needs to use the proxy broker's own principal, which is currently not
supported to be used for many configuration change authentication listed
above. The full list could be found in the KIP.

4. Add a new BROKER_AUTHORIZATION_FAILURE error code to indicate any
internal security configuration failure, when the forwarded request failed
authentication on the controller side.

Let me know what you think. With such a major refinement of the KIP, I'm
open for re-vote after discussions converge.

Boyang

On Wed, Jul 1, 2020 at 2:17 PM Boyang Chen 
wrote:

> Hey folks,
>
> I have also synced on the KIP-578 which was doing the partition limit, to
> make sure the partition limit error code would be properly propagated once
> it is done on top of KIP-590. Let me know if you have further questions or
> concerns.
>
> Boyang
>
> On Tue, Jun 23, 2020 at 5:08 PM Boyang Chen 
> wrote:
>
>> Thanks for the clarification, Colin and Ismael. Personally I also feel
>> Option A is better to prioritize fixing the gap. Just to be clear, the
>> proposed solution would be:
>>
>> 1. Bump the Metadata RPC version to return POLICY_VIOLATION. In the
>> application level, we should swap the error message with the actual failure
>> reason such as "violation of topic creation policy when attempting to auto
>> create internal topic through MetadataRequest."
>>
>> 2. For older Metadata RPC, return AUTHORIZATION_FAILED to fail fast.
>>
>> Will address our other discussed points as well in the KIP, let me know
>> if you have further questions.
>>
>> Thanks,
>> Boyang
>>
>> On Tue, Jun 23, 2020 at 10:41 AM Ismael Juma  wrote:
>>
>>> Option A is basically what I was thinking. But with a slight adjustment:
>>>
>>> New versions of MetadataResponse return POLICY_VIOLATION, old versions
>>> return AUTHORIZATION_FAILED. The latter works correctly with old Java
>>> clients (i.e. the client fails fast and propagates the error), I've
>>> tested
>>> it. Adjust new clients to treat POLICY_VIOLATION like
>>> AUTHORIZATION_FAILED,
>>> but propagate the custom error message.
>>>
>>> Ismael
>>>
>>> On Mon, Jun 22, 2020 at 11:00 PM Colin McCabe 
>>> wrote:
>>>
>>> > > > > On Fri, Jun 19, 2020 at 3:18 PM Ismael Juma 
>>> > wrote:
>>> > > > >
>>> > > > > > Hi Colin,
>>> > > > > >
>>> > > > > > The KIP states in the Compatibility section (not Future work):
>>> > > > > >
>>> > > > > > "To support the proxy of requests, we need to build a channel
>>> for
>>> > > > > > brokers to talk directly to the controller. This part of the
>>> design
>>> > > > > > is internal change only and won’t block the KIP progress."
>>> > > > > >
>>> > > > > > I am clarifying that this is not internal only due to the
>>> config.
>>> > If we
>>> > > > > > say that this KIP depends on another KIP before we can merge
>>> > > > > > it, that's fine although it feels a bit unnecessary.
>>> > > > > >
>>> >
>>> > Hi Ismael,
>>> >
>>> > I didn't realize there was still a reference to the separate controller
>>> > channel in the "Compatibility, Deprecation, and Migration Plan"
>>> section.  I
>>> > agree that it doesn't really belong there.  Given that this is creating
>>> > confusion, I would suggest that we just drop this from the KIP
>>> entirely.
>>> > It really is orthogonal to what this KIP is about-- we don't need a
>>> > separate channel to implement redirection.
>>> >
>>> > Boyang wrote:
>>> >
>>> > >
>>> > > We are only opening the doors for specific internal topics (offsets,
>>> txn
>>> > > log), which I assume the client should have no possibility to mutate
>>> the
>>> > > topic policy?
>>> > >
>>> >
>>> > Hi Boyang,
>>> >
>>> > I think you and Ismael are talking about different scenarios.  You are
>>> > describing the scenario where the broker is auto-creating the
>>> transaction
>>> > log topic or consumer offset topic.  This scenario indeed should not
>>> happen
>>> > in a properly-configured cluster.  However, Ismael is describing a
>>> scenario
>>> 

Build failed in Jenkins: kafka-trunk-jdk14 #323

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Adjust 'release.py' script to use shell when using gradlewAll 
and

[github] MINOR: Remove staticmethod tag to be able to use logger of instance

[github] MINOR: remove NewTopic#NO_PARTITIONS and NewTopic#NO_REPLICATION_FACTOR


--
[...truncated 2.81 MB...]

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceStickyAndBalancedAssignmentWhenNothingChanges[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceStickyAndBalancedAssignmentWhenNothingChanges[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithInternalTopicThatsSourceIsAnotherInternalTopic[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithInternalTopicThatsSourceIsAnotherInternalTopic[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV1V2[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV1V2[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV1V3[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV1V3[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV2V3[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnLowestAssignmentVersionForDifferentSubscriptionVersionsV2V3[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithStandbyReplicasAndStatelessTasks[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithStandbyReplicasAndStatelessTasks[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testEagerSubscription[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testEagerSubscription[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerNotConfigured[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerNotConfigured[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithInternalTopics[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testAssignWithInternalTopics[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
testCooperativeSubscription[task assignor = class 

[jira] [Created] (KAFKA-10317) Global thread should honor shutdown signal during bootstrapping

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10317:
---

 Summary: Global thread should honor shutdown signal during 
bootstrapping
 Key: KAFKA-10317
 URL: https://issues.apache.org/jira/browse/KAFKA-10317
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


During startup, the global thread will bootstrap all global stores before it 
enters its main store-update-loop. The store-update-loop exits, when 
`KafkaStreams#close()` is called.

However, during the bootstrap phase there is no check if `KafkaStreams#close()` 
was called. And thus, KafkaStreams cannot be stopped during bootstrapping (ie, 
the bootstrapping cannot be interrupted).

To make Kafka Streams more responsive, we should check if `close()` was called 
during bootstrapping, too.

Note, that `KafkaStreams#start()` actually blocks during bootstrapping atm (cf 
KAFKA-7380) and thus we should fix both issue at once if possible.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Ning Zhang]

Hello Chris,

That is an interesting KIP. I have a couple of questions:

(1) in section of pseudo-code, what if the failure happens between 4(b) and 
5(a), meaning after the producer commit the transaction, and before 
task.commitRecord().

(2) in section "source task life time",  what is the difference between "commit 
offset" and "offsets to commit"? Given that the offset storage can be a Kafka 
topic (/KafkaOffsetBackingStore.java) and producer could only produce to a 
kafka topic, are / is the topic(s) the same ? (the topic that producer writes 
offsets to and the topic task.commit() to)

(3) for JDBC source task, it relies on `context.offsetStorageReader()` 
(https://github.com/confluentinc/kafka-connect-jdbc/blob/master/src/main/java/io/confluent/connect/jdbc/source/JdbcSourceTask.java#L140)
 to retrieve the previously committed offset (if from a fresh start or resume 
from failure). so it seems that the single-source-of-truth of where to consume 
from last known / committed position stored in offset storage (e.g. kafka 
topic) managed by the periodic task.commit()?

On 2020/05/22 06:20:51, Chris Egerton  wrote: 
> Hi all,
> 
> I know it's a busy time with the upcoming 2.6 release and I don't expect
> this to get a lot of traction until that's done, but I've published a KIP
> for allowing atomic commit of offsets and records for source connectors and
> would appreciate your feedback:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Atomic+commit+of+source+connector+records+and+offsets
> 
> This feature should make it possible to implement source connectors with
> exactly-once delivery guarantees, and even allow a wide range of existing
> source connectors to provide exactly-once delivery guarantees with no
> changes required.
> 
> Cheers,
> 
> Chris
> 

Re: [DISCUSS] KIP-647: Add ability to handle late messages in streams-aggregation

[Matthias J. Sax]

Thanks for the KIP Igor.

What you propose sounds a little bit like a "dead-letter-queue" pattern.
Thus, I am wondering if we should try to do a built-in
"dead-letter-queue" feature that would be general purpose? For example,
uses can drop message in the source node if they don't have a valid
timestamp or if a deserialization error occurs and face a similar issue
for those cases (even if it might be a little simpler to handle those
cases, as custom user code is executed).

For a general purpose DLQ, the feature should be expose at the Processor
API level though, and the DSL would just use this feature (instead of
introducing it as a DSL feature).

Late records are of course only defined at the DSL level as for the PAPI
users need to define custom semantics. Also, late records are not really
corrupted. However, the pattern seems similar enough, ie, piping late
data into a topic is just a special case for a DLQ?

I am also wondering, if piping late records into a DLQ is the only way
to handle them? For example, I could imagine that a user just wants to
trigger a side-effect (similar to what you mention in rejected
alternatives)? Or maybe a user might even want to somehow process those
record and feed them back into the actually processing pipeline.

Last, a DLQ is only useful if somebody consumes from the topic and does
something with the data. Can you elaborate on the use-case how a user
would use the preserved late records?



-Matthias

On 7/27/20 1:45 AM, Igor Piddubnyi wrote:
> Hi everybody,
> I would like to start off the discussion for KIP-647:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-647%3A+Add+ability+to+handle+late+messages+in+streams-aggregation
>  
> 
> 
> 
> This KIP proposes a minor adjustment in the kafka-streams
> aggregation-api, adding an ability for processing late messages.
> [WIP] PR here:https://github.com/apache/kafka/pull/9017
> 
> Please check.
> Regards, Igor.
> 
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

[John Roesler]

Thanks for the reply, Sophie.

Yes, I'd neglected to specify that Windows will implement maxSize()
by delegating to size(). It's updated now. I'd also neglected to say that
I plan to alter both windowBy methods to use the new interface now.
Because Windows will implement the new interface, all implementations
will continue to work with windowBy.
So, yes, there are public methods changed, but no compatibility issues
arise. Existing implementations will only get a deprecation warning.

The Window type is interesting. It actually seems to serve as just a data
container. It almost doesn't need to be subclassed at all, since all
implementations would just need to store the start and end bounds.
As far as I can tell, the only purpose to subclass it is to implement
"overlap(Window other)", to tell if the window is both the same type
as and overlaps with the other window. But weirdly, this is unused
in the codebase.

So one way we could go is to try and migrate it to just a final class,
effectively a tuple of `(start, end)`.

However, another opportunity is to let it be a witness of the actual type
of the window, so that you wouldn't be able to join a TimeWindow
with a SessionWindow, for example.

However, because of covariance, it's more painful to change Window
than Windows, so it might not be worth it right now. If anything, it
would be more feasible to migrate toward the "simple data container"
approach. What are your thoughts?

Thanks,
-John


On Mon, Jul 27, 2020, at 18:19, Sophie Blee-Goldman wrote:
> Thanks for taking the time to really fill in the background details for
> this KIP.
> The Motivation section is very informative. Hopefully this will also serve
> as a
> warning against making similar such mistakes in the future :P
> 
> I notice that the `Window` class that
> parametrizes EnumerableWindowDefinition
> is also abstract. Did you consider migrating that to an interface as well?
> 
> Also, pretty awesome that we can solve the issue with varying fixed sized
> windows
> (eg calendar months) on the side. For users who may have already extended
> Windows,
> do you plan to just have Windows implement the new #maxSize method and
> return the existing
> size until Windows gets removed?
> 
> One final note: this seems to be implicitly implied throughout the KIP but
> just to be clear,
> you will be replacing any DSL methods that accept Windows with identical
> DSL methods
> that take the new EnumerableWindowDefinition as argument. So there is
> nothing deprecated
> and nothing added, but there are public methods changed. Is that right?
> 
> On Sun, Jul 26, 2020 at 1:23 PM John Roesler  wrote:
> 
> > Thanks Sophie and Boyang for asking for specifics.
> >
> > As far as I can tell, if we were to _remove_ all the non-public-method
> > members from Windows, including any constructors, we are left with
> > effectively an interface. I think this was my plan before. I don't think
> > I realized at the time that it's possible to replace the entire class with
> > an interface. Now I realize it is possible, hence the motivation to do it.
> >
> > We can choose either to maintain forever the tech debt of having to
> > enforce that Windows look, sound, and act just like an interface, or we
> > can just replace it with an interface and be done with it. I.e., the
> > motivation is less maintenence burden for us and for users.
> >
> > Coincidentally, I had an interesting conversation with Matthias about
> > this interface, and he made me realize that "fixed size" isn't the
> > essential
> > nature of this entity. Instead being enumerable is. Reframing the interface
> > in this way will enable us to implement variable sized windows like
> > MonthlyWindows.
> >
> > So, now there are two good reasons to vote for this KIP :)
> >
> > Anyway, I've updated the proposed interface and the motivation.
> >
> > To Sophie latter question, all of the necessary deprecation is listed
> > in the KIP. We do not have to deprecate any windowBy methods.
> >
> > Thanks,
> > -John
> >
> > On Sat, Jul 25, 2020, at 00:52, Boyang Chen wrote:
> > > Thanks for the KIP John. I share a similar concern with the motivation,
> > it
> > > would be good if you could cast light on the actual downside of using a
> > > base class vs the interface, is it making the code fragile, or requiring
> > > redundant implementation, etc.
> > >
> > > Boyang
> > >
> > > On Tue, Jul 21, 2020 at 2:19 PM Sophie Blee-Goldman  > >
> > > wrote:
> > >
> > > > Hey John,
> > > >
> > > > Thanks for the KIP. I know this has been bugging you :)
> > > >
> > > > That said, I think the KIP is missing some elaboration in the
> > Motivation
> > > > section.
> > > > You mention a number of problems we've had and lived with in the past
> > --
> > > > could
> > > > you give an example of one, and how it would be solved by your
> > proposal?
> > > >
> > > > By the way, I assume we would also need to deprecate all APIs that
> > accept a
> > > > Windows
> > > > parameter in favor of new ones 

Build failed in Jenkins: kafka-2.3-jdk8 #220

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Remove staticmethod tag to be able to use logger of instance


--
[...truncated 2.70 MB...]

kafka.log.LogCleanerTest > testSegmentWithOffsetOverflow PASSED

kafka.log.LogCleanerTest > testPartialSegmentClean STARTED

kafka.log.LogCleanerTest > testPartialSegmentClean PASSED

kafka.log.LogCleanerTest > testCommitMarkerRemoval STARTED

kafka.log.LogCleanerTest > testCommitMarkerRemoval PASSED

kafka.log.LogCleanerTest > testCleanSegmentsWithConcurrentSegmentDeletion 
STARTED

kafka.log.LogCleanerTest > testCleanSegmentsWithConcurrentSegmentDeletion PASSED

kafka.log.LogValidatorTest > testRecompressedBatchWithoutRecordsNotAllowed 
STARTED

kafka.log.LogValidatorTest > testRecompressedBatchWithoutRecordsNotAllowed 
PASSED

kafka.log.LogValidatorTest > testCompressedV1 STARTED

kafka.log.LogValidatorTest > testCompressedV1 PASSED

kafka.log.LogValidatorTest > testCompressedV2 STARTED

kafka.log.LogValidatorTest > testCompressedV2 PASSED

kafka.log.LogValidatorTest > testDownConversionOfIdempotentRecordsNotPermitted 
STARTED

kafka.log.LogValidatorTest > testDownConversionOfIdempotentRecordsNotPermitted 
PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV2NonCompressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV2NonCompressed PASSED

kafka.log.LogValidatorTest > testAbsoluteOffsetAssignmentCompressed STARTED

kafka.log.LogValidatorTest > testAbsoluteOffsetAssignmentCompressed PASSED

kafka.log.LogValidatorTest > testLogAppendTimeWithRecompressionV1 STARTED

kafka.log.LogValidatorTest > testLogAppendTimeWithRecompressionV1 PASSED

kafka.log.LogValidatorTest > testLogAppendTimeWithRecompressionV2 STARTED

kafka.log.LogValidatorTest > testLogAppendTimeWithRecompressionV2 PASSED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV0ToV1 STARTED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV0ToV1 PASSED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV0ToV2 STARTED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV0ToV2 PASSED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV1ToV2 STARTED

kafka.log.LogValidatorTest > testCreateTimeUpConversionV1ToV2 PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV0Compressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV0Compressed PASSED

kafka.log.LogValidatorTest > testZStdCompressedWithUnavailableIBPVersion STARTED

kafka.log.LogValidatorTest > testZStdCompressedWithUnavailableIBPVersion PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV1ToV2Compressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV1ToV2Compressed PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV1NonCompressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV1NonCompressed PASSED

kafka.log.LogValidatorTest > 
testDownConversionOfTransactionalRecordsNotPermitted STARTED

kafka.log.LogValidatorTest > 
testDownConversionOfTransactionalRecordsNotPermitted PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV1Compressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterUpConversionV0ToV1Compressed PASSED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentNonCompressedV1 STARTED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentNonCompressedV1 PASSED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentNonCompressedV2 STARTED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentNonCompressedV2 PASSED

kafka.log.LogValidatorTest > testControlRecordsNotAllowedFromClients STARTED

kafka.log.LogValidatorTest > testControlRecordsNotAllowedFromClients PASSED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentCompressedV1 STARTED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentCompressedV1 PASSED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentCompressedV2 STARTED

kafka.log.LogValidatorTest > testRelativeOffsetAssignmentCompressedV2 PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV1NonCompressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV1NonCompressed PASSED

kafka.log.LogValidatorTest > testLogAppendTimeNonCompressedV1 STARTED

kafka.log.LogValidatorTest > testLogAppendTimeNonCompressedV1 PASSED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV0NonCompressed STARTED

kafka.log.LogValidatorTest > 
testOffsetAssignmentAfterDownConversionV2ToV0NonCompressed PASSED

kafka.log.LogValidatorTest > testControlRecordsNotCompressed STARTED

kafka.log.LogValidatorTest > testControlRecordsNotCompressed PASSED

kafka.log.LogValidatorTest > testInvalidCreateTimeNonCompressedV1 STARTED

[jira] [Created] (KAFKA-10316) Consider renaming getter method for Interactive Queries

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10316:
---

 Summary: Consider renaming getter method for Interactive Queries
 Key: KAFKA-10316
 URL: https://issues.apache.org/jira/browse/KAFKA-10316
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


In the 2.5 release, we introduce new classes for Interactive Queries via 
KIP-535 (cf https://issues.apache.org/jira/browse/KAFKA-6144). The KIP did not 
specify the names for getter methods of `KeyQueryMetadata` explicitly and they 
were added in the PR as `getActiveHost()`, `getStandbyHosts()`, and 
`getPartition()`.

However, in Kafka it is custom to not use the `get` prefix for getters and thus 
the methods should have been added as `activeHost()`, `standbyHosts()`, and 
`partition()`, respectively.

We should consider renaming the methods accordingly, by deprecating the 
existing ones and adding the new ones in parallel.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Matthias J. Sax]

+1 (binding)

On 7/27/20 4:55 PM, Guozhang Wang wrote:
> +1. Thanks Jorge for bringing in this KIP!
> 
> Guozhang
> 
> On Mon, Jul 27, 2020 at 10:07 AM Leah Thomas  wrote:
> 
>> Hi Jorge,
>>
>> Looks great. +1 (non-binding)
>>
>> Best,
>> Leah
>>
>> On Thu, Jul 16, 2020 at 6:39 PM Sophie Blee-Goldman 
>> wrote:
>>
>>> Hey Jorge,
>>>
>>> Thanks for the reminder -- +1 (non-binding)
>>>
>>> Cheers,
>>> Sophie
>>>
>>> On Thu, Jul 16, 2020 at 4:06 PM Jorge Esteban Quilcate Otoya <
>>> quilcate.jo...@gmail.com> wrote:
>>>
 Bumping this vote thread to check if there's any feedback.

 Cheers,
 Jorge.

 On Sat, Jul 4, 2020 at 6:20 PM John Roesler 
>> wrote:

> Thanks Jorge,
>
> I’m +1 (binding)
>
> -John
>
> On Fri, Jul 3, 2020, at 10:26, Jorge Esteban Quilcate Otoya wrote:
>> Hola everyone,
>>
>> I'd like to start a new thread to vote for KIP-617 as there have
>> been
>> significant changes since the previous vote started.
>>
>> KIP wiki page:
>>
>

>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards
>>
>> Many thanks!
>>
>> Jorge.
>>
>

>>>
>>
> 
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-2.5-jdk8 #172

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Remove staticmethod tag to be able to use logger of instance


--
[...truncated 2.93 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 

Build failed in Jenkins: kafka-trunk-jdk11 #1674

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Remove staticmethod tag to be able to use logger of instance

[github] MINOR: remove NewTopic#NO_PARTITIONS and NewTopic#NO_REPLICATION_FACTOR


--
[...truncated 3.21 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Guozhang Wang]

+1. Thanks Jorge for bringing in this KIP!

Guozhang

On Mon, Jul 27, 2020 at 10:07 AM Leah Thomas  wrote:

> Hi Jorge,
>
> Looks great. +1 (non-binding)
>
> Best,
> Leah
>
> On Thu, Jul 16, 2020 at 6:39 PM Sophie Blee-Goldman 
> wrote:
>
> > Hey Jorge,
> >
> > Thanks for the reminder -- +1 (non-binding)
> >
> > Cheers,
> > Sophie
> >
> > On Thu, Jul 16, 2020 at 4:06 PM Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > Bumping this vote thread to check if there's any feedback.
> > >
> > > Cheers,
> > > Jorge.
> > >
> > > On Sat, Jul 4, 2020 at 6:20 PM John Roesler 
> wrote:
> > >
> > > > Thanks Jorge,
> > > >
> > > > I’m +1 (binding)
> > > >
> > > > -John
> > > >
> > > > On Fri, Jul 3, 2020, at 10:26, Jorge Esteban Quilcate Otoya wrote:
> > > > > Hola everyone,
> > > > >
> > > > > I'd like to start a new thread to vote for KIP-617 as there have
> been
> > > > > significant changes since the previous vote started.
> > > > >
> > > > > KIP wiki page:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards
> > > > >
> > > > > Many thanks!
> > > > >
> > > > > Jorge.
> > > > >
> > > >
> > >
> >
>


-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk11 #1673

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Adjust 'release.py' script to use shell when using gradlewAll 
and


--
[...truncated 5.03 MB...]
kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldCompleteAppendToLogOnEndTxnWhenSendMarkersSucceed PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldGenerateRequestPerPartitionPerBroker PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldRemoveMarkersForTxnPartitionWhenPartitionEmigrated PASSED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound STARTED

kafka.coordinator.transaction.TransactionMarkerChannelManagerTest > 
shouldSkipSendMarkersWhenLeaderNotFound PASSED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
STARTED

kafka.coordinator.transaction.TransactionLogTest > shouldReadWriteMessages 
PASSED

kafka.coordinator.transaction.TransactionLogTest > 
testTransactionMetadataParsing STARTED

kafka.coordinator.transaction.TransactionLogTest > 
testTransactionMetadataParsing PASSED

kafka.coordinator.transaction.TransactionLogTest > 
testTransactionMetadataTombstoneParsing STARTED

kafka.coordinator.transaction.TransactionLogTest > 
testTransactionMetadataTombstoneParsing PASSED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn STARTED

kafka.coordinator.transaction.TransactionLogTest > 
shouldThrowExceptionWriteInvalidTxn PASSED

kafka.network.ConnectionQuotasTest > testFailDecrementForUnknownIp STARTED

kafka.network.ConnectionQuotasTest > testFailDecrementForUnknownIp PASSED

kafka.network.ConnectionQuotasTest > testMaxConnectionsPerIp STARTED

kafka.network.ConnectionQuotasTest > testMaxConnectionsPerIp PASSED

kafka.network.ConnectionQuotasTest > testMaxBrokerWideConnectionLimit STARTED

kafka.network.ConnectionQuotasTest > testMaxBrokerWideConnectionLimit PASSED

kafka.network.ConnectionQuotasTest > testNoConnectionLimitsByDefault STARTED

kafka.network.ConnectionQuotasTest > testNoConnectionLimitsByDefault PASSED

kafka.network.ConnectionQuotasTest > testMaxListenerConnectionLimits STARTED

kafka.network.ConnectionQuotasTest > testMaxListenerConnectionLimits PASSED

kafka.network.ConnectionQuotasTest > testFailWhenNoListeners STARTED

kafka.network.ConnectionQuotasTest > testFailWhenNoListeners PASSED

kafka.network.RequestChannelTest > testNonAlterRequestsNotTransformed STARTED

kafka.network.RequestChannelTest > testNonAlterRequestsNotTransformed PASSED

kafka.network.RequestChannelTest > testAlterRequests STARTED

kafka.network.RequestChannelTest > testAlterRequests PASSED

kafka.network.RequestChannelTest > testIncrementalAlterRequests STARTED

kafka.network.RequestChannelTest > testIncrementalAlterRequests PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionFullReplicaAssignmentReturnsEmptyAssignmentIfTopicOrPartitionDoesNotExist
 STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionFullReplicaAssignmentReturnsEmptyAssignmentIfTopicOrPartitionDoesNotExist
 PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsEmptyMapIfTopicDoesNotExist STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsEmptyMapIfTopicDoesNotExist PASSED

unit.kafka.controller.ControllerContextTest > 
testPreferredReplicaImbalanceMetric STARTED

unit.kafka.controller.ControllerContextTest > 
testPreferredReplicaImbalanceMetric PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsExpectedReplicaAssignments STARTED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentForTopicReturnsExpectedReplicaAssignments PASSED

unit.kafka.controller.ControllerContextTest > testReassignTo STARTED

unit.kafka.controller.ControllerContextTest > testReassignTo PASSED

unit.kafka.controller.ControllerContextTest > testPartitionReplicaAssignment 
STARTED

unit.kafka.controller.ControllerContextTest > testPartitionReplicaAssignment 
PASSED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionFullReplicaAssignmentUpdatesReplicaAssignment STARTED

unit.kafka.controller.ControllerContextTest > 
testUpdatePartitionFullReplicaAssignmentUpdatesReplicaAssignment PASSED

unit.kafka.controller.ControllerContextTest > 
testPartitionReplicaAssignmentReturnsEmptySeqIfTopicOrPartitionDoesNotExist 
STARTED

Re: [DISCUSS] KIP-645: Replace abstract class Windows with a proper interface

[Sophie Blee-Goldman]

Thanks for taking the time to really fill in the background details for
this KIP.
The Motivation section is very informative. Hopefully this will also serve
as a
warning against making similar such mistakes in the future :P

I notice that the `Window` class that
parametrizes EnumerableWindowDefinition
is also abstract. Did you consider migrating that to an interface as well?

Also, pretty awesome that we can solve the issue with varying fixed sized
windows
(eg calendar months) on the side. For users who may have already extended
Windows,
do you plan to just have Windows implement the new #maxSize method and
return the existing
size until Windows gets removed?

One final note: this seems to be implicitly implied throughout the KIP but
just to be clear,
you will be replacing any DSL methods that accept Windows with identical
DSL methods
that take the new EnumerableWindowDefinition as argument. So there is
nothing deprecated
and nothing added, but there are public methods changed. Is that right?

On Sun, Jul 26, 2020 at 1:23 PM John Roesler  wrote:

> Thanks Sophie and Boyang for asking for specifics.
>
> As far as I can tell, if we were to _remove_ all the non-public-method
> members from Windows, including any constructors, we are left with
> effectively an interface. I think this was my plan before. I don't think
> I realized at the time that it's possible to replace the entire class with
> an interface. Now I realize it is possible, hence the motivation to do it.
>
> We can choose either to maintain forever the tech debt of having to
> enforce that Windows look, sound, and act just like an interface, or we
> can just replace it with an interface and be done with it. I.e., the
> motivation is less maintenence burden for us and for users.
>
> Coincidentally, I had an interesting conversation with Matthias about
> this interface, and he made me realize that "fixed size" isn't the
> essential
> nature of this entity. Instead being enumerable is. Reframing the interface
> in this way will enable us to implement variable sized windows like
> MonthlyWindows.
>
> So, now there are two good reasons to vote for this KIP :)
>
> Anyway, I've updated the proposed interface and the motivation.
>
> To Sophie latter question, all of the necessary deprecation is listed
> in the KIP. We do not have to deprecate any windowBy methods.
>
> Thanks,
> -John
>
> On Sat, Jul 25, 2020, at 00:52, Boyang Chen wrote:
> > Thanks for the KIP John. I share a similar concern with the motivation,
> it
> > would be good if you could cast light on the actual downside of using a
> > base class vs the interface, is it making the code fragile, or requiring
> > redundant implementation, etc.
> >
> > Boyang
> >
> > On Tue, Jul 21, 2020 at 2:19 PM Sophie Blee-Goldman  >
> > wrote:
> >
> > > Hey John,
> > >
> > > Thanks for the KIP. I know this has been bugging you :)
> > >
> > > That said, I think the KIP is missing some elaboration in the
> Motivation
> > > section.
> > > You mention a number of problems we've had and lived with in the past
> --
> > > could
> > > you give an example of one, and how it would be solved by your
> proposal?
> > >
> > > By the way, I assume we would also need to deprecate all APIs that
> accept a
> > > Windows
> > > parameter in favor of new ones that accept a
> FixedSizeWindowDefinition? Off
> > > the
> > > top of my head that would be the windowedBy methods in KGroupedStream
> and
> > > CogroupedKStream
> > >
> > > On Tue, Jul 21, 2020 at 1:46 PM John Roesler 
> wrote:
> > >
> > > > Hello all,
> > > >
> > > > I'd like to propose KIP-645, to correct a small API mistake in
> Streams.
> > > > Fixing this now allows us to avoid perpetuating the mistake in new
> work.
> > > > For example, it will allow us to implement KIP-450 cleanly.
> > > >
> > > > The change itself should be seamless for users.
> > > >
> > > > Please see https://cwiki.apache.org/confluence/x/6SN4CQ for details.
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-10315) Consider to throw exception for failed fetch requests

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10315:
---

 Summary: Consider to throw exception for failed fetch requests
 Key: KAFKA-10315
 URL: https://issues.apache.org/jira/browse/KAFKA-10315
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Matthias J. Sax


The current `Consumer#poll(Duration)` method is designed to block until data is 
available or the provided poll timeout expires. This implies, that if fetch 
requests fail the consumer retries them internally and eventually returns an 
empty set of records. – Thus, from a user point of view, returning an empty set 
of records can mean that no data is available broker side or that the broker 
cannot be reached.

For Kafka Streams, this behavior is problematic as its runtime would like to 
distinguish both cases, to apply its own timeouts (cf 
https://issues.apache.org/jira/browse/KAFKA-9274).

One idea to address this issue is to add a new method `Consumer#pollOnce()` 
that would throw an exception if a fetch request fails instead of retrying 
internally.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-450: Sliding Windows

[Leah Thomas]

A small tweak - to make it more clear to users that grace is required, as
well as cleaning up some of the confusing grammar semantics of windows, the
main builder method for *slidingWindows* will be *withSizeAndGrace* instead
of *of*.  This looks like it'll be the last change (for now) on the public
API. If anyone has any comments or thoughts let me know. Otherwise, I'll
take this to vote shortly.

Best,
Leah

On Fri, Jul 24, 2020 at 3:45 PM Leah Thomas  wrote:

> To accommodate the change to a final class, I've added another
> *windowedBy()* function in *CogroupedKStream.java *to handle
> SlidingWindows.
>
> As far as the discussion goes, I think this is the last change we've
> talked about. If anyone has other comments or concerns, please let me know!
>
> Cheers,
> Leah
>
> On Thu, Jul 23, 2020 at 7:34 PM Leah Thomas  wrote:
>
>> Thanks for the discussion about extending TimeWindows. I agree that
>> making it future proof is important, and the implementation of
>> SlidingWindows is unique enough that it seems logical to make it its own
>> final class.
>>
>> On that note, I've updated the KIP to make SlidingWindows a stand alone
>> final class, and added the *windowedBy() *API in *KGroupedStream *to
>> handle SlidingWindows. It seems that SlidingWindows would still be able to
>> leverage *TimeWindowedKStream by* creating a SlidingWindows version of
>> *TimeWindowedKStreamImpl* that implements *TimeWindowedKStream. *If
>> anyone sees issues with this implementation, please let me know.
>>
>> Best,
>> Leah
>>
>> On Wed, Jul 22, 2020 at 10:47 PM John Roesler 
>> wrote:
>>
>>> Thanks for the reply, Sophie.
>>>
>>> That all sounds about right to me.
>>>
>>> The Windows “interface”/algorithm is quite flexible, so it makes sense
>>> for it to be extensible. Different implementations can (and do) enumerate
>>> different windows to suit different use cases.
>>>
>>> On the other hand, I can’t think of any way to extend SessionWindows to
>>> do something different using the same algorithm, so it makes sense for it
>>> to stay final.
>>>
>>> If we think SlidingWindows is similarly not usefully extensible, then we
>>> can make it final. It’s easy to remove final later, but adding it is not
>>> possible. Or we could go the other route and just make it an interface, on
>>> general principle. Both of these choices are safe API design.
>>>
>>> Thanks again,
>>> John
>>>
>>> On Wed, Jul 22, 2020, at 21:54, Sophie Blee-Goldman wrote:
>>> > >
>>> > > Users could pass in a custom `SessionWindows` as
>>> > > long as the session algorithm works correctly for it.
>>> >
>>> >
>>> > Well not really, SessionWindows is a final class. TimeWindows is also a
>>> > final class, so neither of these can be extended/customized. For a
>>> given
>>> > Windows then there would only be three (non-overlapping) possibilities:
>>> > either it's TimeWindows, SlidingWindows, or a custom  Windows. I don't
>>> > think there's any problem with determining what the user wants in this
>>> case
>>> > --
>>> > we would just check if it's a SlidingWindows and connect the new
>>> processor,
>>> > or else connect the existing hopping/tumbling window processor.
>>> >
>>> > I'll admit that last sentence does leave a bad taste in my mouth. Part
>>> of
>>> > that
>>> > is probably the "leaking" API Matthias pointed out; we just assume the
>>> > hopping/tumbling window implementation fits all custom windows. But I
>>> guess
>>> > if you really needed to customize the algorithm any further you may as
>>> well
>>> > stick in a transformer and do it all yourself.
>>> >
>>> > Anyways, given what we have, it does seem weird to apply one algorithm
>>> > for most Windows types and then swap in a different one for one
>>> specific
>>> > extension of Windows. So adding a new #windowedBy(SlidingWindows)
>>> > sounds reasonable to me.
>>> >
>>> > I'm still not convinced that we need a whole new TimeWindowedKStream
>>> > equivalent class for sliding windows though. It seems like the
>>> > hopping/tumbling
>>> > window implementation could benefit just as much from a subtractor as
>>> the
>>> > sliding windows so the only future-proofing we get is the ability to be
>>> > lazy and
>>> > add the subtractor to one but not the other. Of course it would only
>>> be an
>>> > optimization so we could just not apply it to one type and nothing
>>> would
>>> > break.
>>> > It does make me nervous to go against the "future-proof" direction,
>>> though.
>>> > Are there any other examples of things we might want to add to one
>>> window
>>> > type but not to another?
>>> >
>>> > On another note, this actually brings up a new question: should
>>> > SlidingWindows
>>> > also be final? My take is "yes" since there's no reasonable
>>> customization of
>>> > sliding windows, at least not that I can think of. Thoughts?
>>> >
>>> >
>>> > On Wed, Jul 22, 2020 at 7:15 PM John Roesler 
>>> wrote:
>>> >
>>> > > Thanks, all,
>>> > >
>>> > > I can see how my conclusion was kind of a leap.
>>> > >
>>> 

Build failed in Jenkins: kafka-trunk-jdk14 #322

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update dependencies for Kafka 2.7 (part 1) (#9082)


--
[...truncated 1.91 MB...]

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.integration.UncleanLeaderElectionTest > 
testTopicUncleanLeaderElectionEnable STARTED

kafka.integration.UncleanLeaderElectionTest > 
testTopicUncleanLeaderElectionEnable PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionDisabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionDisabledByTopicOverride PASSED

kafka.cluster.PartitionTest > 
testMakeLeaderDoesNotUpdateEpochCacheForOldFormats STARTED

kafka.cluster.PartitionTest > 
testMakeLeaderDoesNotUpdateEpochCacheForOldFormats PASSED

kafka.cluster.PartitionTest > testIsrExpansion STARTED

kafka.cluster.PartitionTest > testIsrExpansion PASSED

kafka.cluster.PartitionTest > testReadRecordEpochValidationForLeader STARTED

kafka.cluster.PartitionTest > testReadRecordEpochValidationForLeader PASSED

kafka.cluster.PartitionTest > testIsrNotShrunkIfUpdateFails STARTED

kafka.cluster.PartitionTest > testIsrNotShrunkIfUpdateFails PASSED

kafka.cluster.PartitionTest > 
testFetchOffsetForTimestampEpochValidationForFollower STARTED

kafka.cluster.PartitionTest > 
testFetchOffsetForTimestampEpochValidationForFollower PASSED

kafka.cluster.PartitionTest > testIsrNotExpandedIfUpdateFails STARTED

kafka.cluster.PartitionTest > testIsrNotExpandedIfUpdateFails PASSED

kafka.cluster.PartitionTest > testLogConfigDirtyAsBrokerUpdated STARTED

kafka.cluster.PartitionTest > testLogConfigDirtyAsBrokerUpdated PASSED

kafka.cluster.PartitionTest > testAddAndRemoveMetrics STARTED

kafka.cluster.PartitionTest > testAddAndRemoveMetrics PASSED

kafka.cluster.PartitionTest > testListOffsetIsolationLevels STARTED

kafka.cluster.PartitionTest > testListOffsetIsolationLevels PASSED

kafka.cluster.PartitionTest > testAppendRecordsAsFollowerBelowLogStartOffset 
STARTED

kafka.cluster.PartitionTest > testAppendRecordsAsFollowerBelowLogStartOffset 
PASSED

kafka.cluster.PartitionTest > testFetchLatestOffsetIncludesLeaderEpoch STARTED

kafka.cluster.PartitionTest > testFetchLatestOffsetIncludesLeaderEpoch PASSED

kafka.cluster.PartitionTest > testUnderReplicatedPartitionsCorrectSemantics 
STARTED

kafka.cluster.PartitionTest > testUnderReplicatedPartitionsCorrectSemantics 
PASSED

kafka.cluster.PartitionTest > testFetchOffsetSnapshotEpochValidationForFollower 
STARTED

kafka.cluster.PartitionTest > testFetchOffsetSnapshotEpochValidationForFollower 
PASSED

kafka.cluster.PartitionTest > testMaybeShrinkIsr STARTED

kafka.cluster.PartitionTest > testMaybeShrinkIsr PASSED

kafka.cluster.PartitionTest > testLogConfigNotDirty STARTED

kafka.cluster.PartitionTest > testLogConfigNotDirty PASSED

kafka.cluster.PartitionTest > testMonotonicOffsetsAfterLeaderChange STARTED

kafka.cluster.PartitionTest > testMonotonicOffsetsAfterLeaderChange PASSED

kafka.cluster.PartitionTest > testUpdateAssignmentAndIsr STARTED

kafka.cluster.PartitionTest > testUpdateAssignmentAndIsr PASSED

kafka.cluster.PartitionTest > testMakeFollowerWithNoLeaderIdChange STARTED

kafka.cluster.PartitionTest > testMakeFollowerWithNoLeaderIdChange PASSED

kafka.cluster.PartitionTest > 
testAppendRecordsToFollowerWithNoReplicaThrowsException STARTED

kafka.cluster.PartitionTest > 
testAppendRecordsToFollowerWithNoReplicaThrowsException PASSED

kafka.cluster.PartitionTest > 
testFollowerDoesNotJoinISRUntilCaughtUpToOffsetWithinCurrentLeaderEpoch STARTED

kafka.cluster.PartitionTest > 
testFollowerDoesNotJoinISRUntilCaughtUpToOffsetWithinCurrentLeaderEpoch PASSED

kafka.cluster.PartitionTest > testShouldNotShrinkIsrIfFollowerCaughtUpToLogEnd 
STARTED

kafka.cluster.PartitionTest > testShouldNotShrinkIsrIfFollowerCaughtUpToLogEnd 
PASSED

kafka.cluster.PartitionTest > testFetchOffsetSnapshotEpochValidationForLeader 
STARTED

kafka.cluster.PartitionTest > testFetchOffsetSnapshotEpochValidationForLeader 
PASSED

kafka.cluster.PartitionTest > testOffsetForLeaderEpochValidationForLeader 
STARTED

Re: [VOTE] KIP-572: Improve timeouts and retires in Kafka Streams

[Jason Gustafson]

Hi Matthias,

Sorry for jumping in so late here. I am trying to understand why it was
necessary to deprecate `retries` in the producer. One of the use cases that
I see in practice is setting `retries` to 0. This allows applications to
control the retry semantics themselves. For example, I have seen this in
flume. As far as I can tell, once `retries` is gone, we will not have a way
to do the same thing. The best we can suggest to users is to enable
idempotence so that any retries will not cause duplicates. My concern is
that this is going to slow client upgrades with little benefit in return.

Thanks,
Jason



On Mon, Jul 20, 2020 at 2:40 PM Matthias J. Sax  wrote:

> While working on the PR, we realized that the command line tool
>
>   bin/kafka-console-producer.sh
>
> has a flag `--message-send-max-retries` to set the producer's `retries`
> config. We also need to deprecate this flag.
>
> I updated the KIP accordingly. Please let us know if there are any
> concerns to this minor change to the KIP.
>
> Thanks.
>
>
> -Matthias
>
> On 6/10/20 11:16 AM, Matthias J. Sax wrote:
> > Thanks!
> >
> > +1 (binding) from myself.
> >
> >
> > I am closing the vote as accepted with 3 binding and 3 non-binding votes.
> >
> > binding:
> >  - John
> >  - Guozhang
> >  - Matthias
> >
> > non-binding:
> >  - Sophie
> >  - Boyang
> >  - Bruno
> >
> >
> >
> > -Matthias
> >
> > On 6/4/20 5:26 PM, Matthias J. Sax wrote:
> >> Guozhang,
> >>
> >> what you propose makes sense, but this seems to get into implementation
> >> detail territory already?
> >>
> >> Thus, if there are nor further change requests to the KIP wiki page
> >> itself, I would like to proceed with the VOTE.
> >>
> >>
> >> -Matthias
> >>
> >> On 5/20/20 12:30 PM, Guozhang Wang wrote:
> >>> Thanks Matthias,
> >>>
> >>> I agree with you on all the bullet points above. Regarding the
> admin-client
> >>> outer-loop retries inside partition assignor, I think we should treat
> error
> >>> codes differently from those two blocking calls:
> >>>
> >>> Describe-topic:
> >>> * unknown-topic (3): add this topic to the to-be-created topic list.
> >>> * leader-not-available (5): do not try to create, retry in the outer
> loop.
> >>> * request-timeout: break the current loop and retry in the outer loop.
> >>> * others: fatal error.
> >>>
> >>> Create-topic:
> >>> * topic-already-exists: retry in the outer loop to validate the
> >>> num.partitions match expectation.
> >>> * request-timeout: break the current loop and retry in the outer loop.
> >>> * others: fatal error.
> >>>
> >>> And in the outer-loop, I think we can have a global timer for the whole
> >>> "assign()" function, not only for the internal-topic-manager, and the
> timer
> >>> can be hard-coded with, e.g. half of the rebalance.timeout to get rid
> of
> >>> the `retries`; if we cannot complete the assignment before the timeout
> runs
> >>> out, we can return just the partial assignment (e.g. if there are two
> >>> tasks, but we can only get the topic metadata for one of them, then
> just do
> >>> the assignment for that one only) while encoding in the error-code
> field to
> >>> request for another rebalance.
> >>>
> >>> Guozhang
> >>>
> >>>
> >>>
> >>> On Mon, May 18, 2020 at 7:26 PM Matthias J. Sax 
> wrote:
> >>>
>  No worries Guozhang, any feedback is always very welcome! My reply is
>  going to be a little longer... Sorry.
> 
> 
> 
> > 1) There are some inconsistent statements in the proposal regarding
> what
>  to
> > deprecated:
> 
>  The proposal of the KIP is to deprecate `retries` for producer, admin,
>  and Streams. Maybe the confusion is about the dependency of those
>  settings within Streams and that we handle the deprecation somewhat
>  different for plain clients vs Streams:
> 
>  For plain producer/admin the default `retries` is set to MAX_VALUE.
> The
>  config will be deprecated but still be respected.
> 
>  For Streams, the default `retries` is set to zero, however, this
> default
>  retry does _not_ affect the embedded producer/admin clients -- both
>  clients stay on their own default of MAX_VALUES.
> 
>  Currently, this introduces the issue, that if a user wants to increase
>  Streams retries, she might by accident reduce the embedded client
>  retries, too. To avoid this issue, she would need to set
> 
>  retries=100
>  producer.retires=MAX_VALUE
>  admin.retries=MAX_VALUE
> 
>  This KIP will fix this issue only in the long term though, ie, when
>  `retries` is finally removed. Short term, using `retries` in
>  StreamsConfig would still affect the embedded clients, but Streams, as
>  well as both client would log a WARN message. This preserves backward
>  compatibility.
> 
>  Withing Streams `retries` is ignored and the new `task.timeout.ms` is
>  used instead. This increase the default resilience of Kafka Streams
>  itself. We could also 

Build failed in Jenkins: kafka-trunk-jdk8 #4748

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Remove staticmethod tag to be able to use logger of instance

[github] MINOR: remove NewTopic#NO_PARTITIONS and NewTopic#NO_REPLICATION_FACTOR


--
[...truncated 3.18 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Ron Dagostino]

Hi Jose.  Thanks for the KIP.  Here are some questions and some nit corrections.

<<< In KIP-500 the Kafka Controller, which is the quorum leader from
KIP-595, will materialize the entries in the metadata log into memory.
Technically I think the quorum leader is referred to as the Active
Controller in KIP-500.  Maybe replace "Kafka Controller" with "Active
Controller"?  I think the term "Kafka Controller" is fine as used
throughout the rest of the KIP to refer to the entire thing, but when
referring specifically to the leader I think "Active Controller" is
the term that is defined in KIP-500.


<<< Each broker in KIP-500, which will be a replica of the metadata
log, will materialize the entries in the log into a metadata cache
This wording confused me because I assumed that "replica" was a formal
term and only (non-Active) Controllers are formally "replicas" of the
metadata log -- Kafka brokers would be clients that read the log and
then use the data for their own purpose as opposed to formally being
replicas with this understanding of the term "replica".  Is that
correct, and if so, maybe replace "replica" with "client"?


<<< The type of in-memory state machines what we plan to implement
>>> The type of in-memory state machines that we plan to implement
nit


<<< doesn't map very well to an key and offset based clean up policy.
>>> doesn't map very well to a key and offset based clean up policy.
nit


<<< When starting a broker either because it is a new broker, a broker
was upgraded or a failed broker is restarting. Loading the state
represented by the __cluster_metadata topic partition is required
before the broker is available
>>> When starting a broker either because it is a new broker or it is 
>>> restarting, loading the state represented by the __cluster_metadata topic 
>>> partition is required before the broker is available.
Reword for simplicity and clarity?


<<< With snapshot based of the in-memory state Kafka can be much more aggressive
>>> By taking and transmitting a snapshot of the in-memory state as described 
>>> below Kafka can be much more aggressive
Tough to refer to the concept of snapshot here without having
described what it is, so refer to "as described below" to help orient
the reader?


<<< In the future this mechanism will also be useful for
high-throughput topic partitions like the Group Coordinator and
Transaction Coordinator.
>>> In the future this mechanism may also be useful for high-throughput topic 
>>> partitions like the Group Coordinator and Transaction Coordinator.
Tough to say "will" when that is an assumption that would depend on a KIP?


<<>> Follower and observer replicas fetch a snapshot from the leader when they 
>>> attempt to fetch an offset from the leader and the leader doesn’t have that 
>>> offset in the log
nit


>>> Generating and loading the snapshot will be delegated to the Kafka 
>>> Controller.
>>> The Kafka Controller will notify the Kafka Raft client when it has 
>>> generated a snapshot and up to which offset is included in the snapshot.
>>> The Kafka Raft client will notify the Kafka Controller when a new snapshot 
>>> has been fetched from the leader.
This paragraph confuses me.  What is the "Kafka Raft client" -- is
this the broker? Or is it some other subsystem (or all other
subsystems aside from log replication) within the Controller?  Has
this been defined somewhere?  If so it would be good to refer to that
definition.  (Actually, now that I've read further down, I think you
refer to this as "Kafka Raft" later in the KIP; a reference to these
later sections or naming it Kafka Raft Client later on would have
helped me avoid confusion -- I searched the doc for raft client rather
than kafka raft, so I missed this when I searched.)


<<< The Kafka Controller will notify the Kafka Raft client when it has
finished generating a new snapshots.
Same comment about "Kafka Raft client".


<<< It is safe for the log to truncate a prefix of the log up to the
latest snapshot.
"log to truncate a prefix of the log" -- That first mention of "log"
needs to be something else I assume -- LogManager maybe?


<<< In the example above, if the Kafka topic partition leader receives
a fetch request with an offset and epoch greater than or equal to the
log begin offset (x, a)
<<< In the example above, offset=x, epoch=a does not appear in the
diagram because it is before the log begin offset (x+1, b)   If the
Kafka topic partition leader receives a fetch request with an offset
and epoch greater than or equal to the LBO
Maybe add an explicit comment that offset=x, epoch=a does not appear
in the diagram because it is before the LBO of (x+1, b)?  Also need to
fix LBO reference (currently incorrectly stated as (x, a).


<<< LBO can be increase/set to an offset X if the following is true:
<<< 2. One of the following is true:
Do the pair of conditions in (2) only apply to the leader/Active Controller?


<<< The broker will delete any snapshot with a latest offset and epoch

Jenkins build is back to normal : kafka-2.6-jdk8 #99

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Guozhang Wang]

+1 (binding).

Thanks.

On Mon, Jul 27, 2020 at 9:24 AM Bill Bejeck  wrote:

> Thanks for the KIP Bruno.
>
> +1 (binding)
>
> -Bill
>
> On Mon, Jul 27, 2020 at 4:26 AM Navinder Brar
>  wrote:
>
> > +1 (non-binding). Thanks for the KIP, Bruno.
> >
> > ~Navinder
> >
> > On Friday, 24 July, 2020, 08:41:03 pm IST, John Roesler <
> > vvcep...@apache.org> wrote:
> >
> >  Thanks, Bruno!
> >
> > I'm +1 (binding)
> >
> > -John
> >
> > On Fri, Jul 24, 2020, at 07:04, Bruno Cadonna wrote:
> > > Hi,
> > >
> > > After re-opening the discussion about
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB
> > >
> > > I would like to re-open the voting for this KIP.
> > >
> > > The discussion thread can be found here:
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> > >
> > > Best,
> > > Bruno
> > >
> > > On 19.05.20 10:00, Bruno Cadonna wrote:
> > > > Thank you for voting!
> > > >
> > > > This KIP passes with:
> > > > 4 binding +1
> > > > 1 non-binding +1
> > > > 0 -1
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On Fri, May 15, 2020 at 11:34 PM Matthias J. Sax 
> > wrote:
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> -Matthias
> > > >>
> > > >> On 5/15/20 11:48 AM, John Roesler wrote:
> > > >>> Thanks, Bruno!
> > > >>>
> > > >>> I’m +1 (binding)
> > > >>>
> > > >>> -John
> > > >>>
> > > >>> On Fri, May 15, 2020, at 11:32, Sophie Blee-Goldman wrote:
> > >  Thanks Bruno! +1 (non-binding)
> > > 
> > >  Sophie
> > > 
> > >  On Fri, May 15, 2020 at 8:15 AM Bill Bejeck 
> > wrote:
> > > 
> > > > Thanks for the KIP!
> > > >
> > > > +1 (binding)
> > > >
> > > > -Bill
> > > >
> > > > On Fri, May 15, 2020 at 11:12 AM Guozhang Wang <
> wangg...@gmail.com>
> > wrote:
> > > >
> > > >> +1.
> > > >>
> > > >> Thanks！
> > > >>
> > > >> On Fri, May 15, 2020 at 1:36 AM Bruno Cadonna <
> br...@confluent.io
> > >
> > > > wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> I'd like to call for votes on
> > > >>>
> > > >>> KIP-607: Add Metrics to Record the Memory Used by RocksDB to
> > Kafka
> > > >> Streams
> > > >>>
> > > >>> The KIP can be found here
> > > >>>
> > > >>>
> > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Record+the+Memory+Used+by+RocksDB
> > > >>>
> > > >>> The discussion thread can be found here:
> > > >>>
> > > >>>
> > > >>
> > > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> > > >>>
> > > >>> Best,
> > > >>> Bruno
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > > 
> > > >>
> > >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Guozhang Wang]

Thanks Bruno,

That is clear. I think my misunderstanding is that I thought compaction is
done by loading sst files into block cache (not OS), do the merge-sort and
then write back to sst. But on a second thought I agree that it is not a
good use case fitted for caching anyways. Using non-cached OS memory is
more reasonable.


Guozhang


On Mon, Jul 27, 2020 at 2:49 AM Bruno Cadonna  wrote:

> Hi Guozhang,
>
> Do you mean compression or compaction?
>
> Regarding compression, I agree with you except for the merge-sorting
> part. The docs describing what is stored in the block cache can be found
> under https://github.com/facebook/rocksdb/wiki/Block-Cache.
>
> Regarding compaction, my statement in my previous e-mail about
> compaction not using block cache was a guess. To get to bottom of it, I
> asked somebody from RockDB and compaction does indeed not use block
> cache. Compaction uses the OS to read in the data to compact. But it
> also uses fadvise to tell the kernel to not cache the data in the OS
> buffer cache.
>
> Hope that clears up the conflicts! ;-)
>
> Best,
> Bruno
>
> On 24.07.20 19:37, Guozhang Wang wrote:
> > Ack, thanks for the clarification folks! Yeah I agree from JVM's point
> all
> > rocksDB memory are off-heap basically (which makes operations harder,
> > sigh..)
> >
> > Regarding the block cache, my understanding is that by default compressed
> > blocks are in the OS page cache, uncompressed blocks are in the RocksDB
> > block cache. In Streams, we do not use compression by default, so these
> > data blocks would be read into the block cache for merge-sorting while
> > index / bloom filter / other compressed dictionary blocks are read into
> OS
> > cache by default. Obviously that conflicts from yours, maybe you can
> point
> > me to the related docs?
> >
> > Guozhang
> >
> > On Fri, Jul 24, 2020 at 2:15 AM Bruno Cadonna 
> wrote:
> >
> >> Hi Guozhang and Sophie,
> >>
> >> 1)
> >> My understanding is also that the memtables are off-heap (as almost
> >> every data structure in RocksDB).
> >>
> >> According to the docs, if after a write the size of the memtable exceeds
> >> option write_buffer_size the memtable is flushed. I would not call it
> >> hard bounded since it seems the memtable can exceed this size.
> >>
> >> Guozhang's other statements about memtables seem correct to me.
> >>
> >> 2)
> >> According to the docs, the block cache caches data in memory for reads.
> >> I do not think RocksDB uses the block cache for compaction, because that
> >> would mean each compaction would interfere with the used cache
> >> replacement policy (LRU is the default in Streams). I suppose RocksDB
> >> uses the OS cache during compactions. So block cache usage contains data
> >> blocks for reading and it can also contain index blocks and filter block
> >> if configured accordingly (e.g. by using the BoundedMemoryRocksDBConfig
> >> described under
> >>
> >>
> https://kafka.apache.org/25/documentation/streams/developer-guide/memory-mgmt.html
> ).
> >>
> >> The pinned usage are blocks pinned by table readers like iterators.
> >> The block cache can be soft or hard bounded. However, there is currently
> >> an open bug in RocksDB regarding hard-bounded block caches.
> >>
> >> 3)
> >> The statements seem correct.
> >>
> >> The total memory usage seems also correct.
> >>
> >> Best,
> >> Bruno
> >>
> >> On 23.07.20 20:46, Sophie Blee-Goldman wrote:
> >>> Guozhang,
> >>>
> >>> Just to clarify, the "heap" for all these objects is actually the C++
> >> heap,
> >>> not the JVM heap. So in the words of a Java application these would
> >>>all be considered "off-heap", right?
> >>>
> >>> (Of course there are some pointers in the Java heap to the off-heap
> >>> objects but that size is trivial compared to the actual objects)
> >>>
> >>> Sorry for being pedantic. I just happen to know this is a question that
> >>> gets frequently asked so it's probably good to be as clear  as possible
> >>> in the KIP/metrics description.
> >>>
> >>> Also, can you clarify a bit what you mean by hard bounded vs soft
> >> bounded?
> >>> For example, my impression is that the memtables are actually not hard
> >>> bounded at all, while the block cache is soft bounded by default but
> can
> >>> be configured to be hard bounded. And obviously the OS cache is not
> >>> exactly bounded but it shouldn't cause you to run out of usable memory
> >>> (like the memtables for example might, and have). But I think maybe
> >> you're
> >>> using a different definition of hard/soft bounded than I'm thinking of
> >>>
> >>> On Thu, Jul 23, 2020 at 8:07 AM Guozhang Wang 
> >> wrote:
> >>>
>  Thanks Bruno, they made sense to me.
> 
>  Regarding the last comment, my main reasoning is that it's better to
>  explain to users the rocksDB memory usage and link the metrics to
> >> different
>  categories.
> 
>  Just to kick off this (and also asking for correction of my own
>  understanding :) here's what I read from the 

Re: [VOTE] 2.6.0 RC1

[Randall Hauch]

Thanks, John. Looks like we're still trying to get a green build for
https://github.com/apache/kafka/pull/9066.

On Fri, Jul 24, 2020 at 3:46 PM John Roesler  wrote:

> Hi Randall,
>
> I'm sorry to say we have also identified that this flaky test
> failure turned out to be a real blocker bug:
> https://issues.apache.org/jira/browse/KAFKA-10287
>
> There is a PR in progress.
>
> Thanks,
> -John
>
> On Fri, Jul 24, 2020, at 12:26, Matthias J. Sax wrote:
> > We found a regression bug that seems to be a blocker:
> > https://issues.apache.org/jira/browse/KAFKA-10306
> >
> > Will work on a PR today.
> >
> >
> > -Matthias
> >
> > On 7/22/20 9:40 AM, Randall Hauch wrote:
> > > Any thoughts, Rajini?
> > >
> > > On Mon, Jul 20, 2020 at 9:55 PM Randall Hauch 
> wrote:
> > >
> > >>
> > >> When I was checking the documentation for RC1 after the tag was
> pushed, I
> > >> noticed that the fix Rajini mentioned in the RC0 vote thread (
> > >> https://github.com/apache/kafka/pull/8979
> > >> <
> https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21
> >)
> > >> and merged to the `2.6` branch includes the following comment about
> being
> > >> deprecated in 2.7:
> > >>
> https://github.com/apache/kafka/pull/8979/files#diff-369f0debebfcda6709beeaf11612b34bR20-R21
> > >> .
> > >>
> > >> Rajini, can you please check the commits merged to the `2.6` do not
> have
> > >> the reference to 2.7? Since these are JavaDocs, I'm assuming that
> we'll
> > >> need to cut RC2.
> > >>
> > >> But it'd be good for everyone else to double check this release.
> > >>
> > >> Best regards,
> > >>
> > >> Randall Hauch
> > >>
> > >> On Mon, Jul 20, 2020 at 9:50 PM Randall Hauch 
> wrote:
> > >>
> > >>> Hello Kafka users, developers and client-developers,
> > >>>
> > >>> This is the second candidate for release of Apache Kafka 2.6.0. This
> is a
> > >>> major release that includes many new features, including:
> > >>>
> > >>> * TLSv1.3 has been enabled by default for Java 11 or newer.
> > >>> * Smooth scaling out of Kafka Streams applications
> > >>> * Kafka Streams support for emit on change
> > >>> * New metrics for better operational insight
> > >>> * Kafka Connect can automatically create topics for source connectors
> > >>> * Improved error reporting options for sink connectors in Kafka
> Connect
> > >>> * New Filter and conditional SMTs in Kafka Connect
> > >>> * The default value for the `client.dns.lookup` configuration is
> > >>> now `use_all_dns_ips`
> > >>> * Upgrade Zookeeper to 3.5.8
> > >>>
> > >>> This release also includes a few other features, 76 improvements,
> and 165
> > >>> bug fixes.
> > >>>
> > >>> Release notes for the 2.6.0 release:
> > >>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/RELEASE_NOTES.html
> > >>>
> > >>> *** Please download, test and vote by Monday, July 20, 9am PT
> > >>>
> > >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> > >>> https://kafka.apache.org/KEYS
> > >>>
> > >>> * Release artifacts to be voted upon (source and binary):
> > >>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/
> > >>>
> > >>> * Maven artifacts to be voted upon:
> > >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>>
> > >>> * Javadoc:
> > >>> https://home.apache.org/~rhauch/kafka-2.6.0-rc1/javadoc/
> > >>>
> > >>> * Tag to be voted upon (off 2.6 branch) is the 2.6.0 tag:
> > >>> https://github.com/apache/kafka/releases/tag/2.6.0-rc1
> > >>>
> > >>> * Documentation:
> > >>> https://kafka.apache.org/26/documentation.html
> > >>>
> > >>> * Protocol:
> > >>> https://kafka.apache.org/26/protocol.html
> > >>>
> > >>> * Successful Jenkins builds for the 2.6 branch:
> > >>> Unit/integration tests:
> https://builds.apache.org/job/kafka-2.6-jdk8/91/ (one
> > >>> flaky test)
> > >>> System tests: (link to follow)
> > >>>
> > >>> Thanks,
> > >>> Randall Hauch
> > >>>
> > >>
> > >
> >
> >
> > Attachments:
> > * signature.asc
>

Build failed in Jenkins: kafka-trunk-jdk8 #4747

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update dependencies for Kafka 2.7 (part 1) (#9082)

[github] MINOR: Adjust 'release.py' script to use shell when using gradlewAll 
and


--
[...truncated 3.18 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled 

Re: [VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Leah Thomas]

Hi Jorge,

Looks great. +1 (non-binding)

Best,
Leah

On Thu, Jul 16, 2020 at 6:39 PM Sophie Blee-Goldman 
wrote:

> Hey Jorge,
>
> Thanks for the reminder -- +1 (non-binding)
>
> Cheers,
> Sophie
>
> On Thu, Jul 16, 2020 at 4:06 PM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Bumping this vote thread to check if there's any feedback.
> >
> > Cheers,
> > Jorge.
> >
> > On Sat, Jul 4, 2020 at 6:20 PM John Roesler  wrote:
> >
> > > Thanks Jorge,
> > >
> > > I’m +1 (binding)
> > >
> > > -John
> > >
> > > On Fri, Jul 3, 2020, at 10:26, Jorge Esteban Quilcate Otoya wrote:
> > > > Hola everyone,
> > > >
> > > > I'd like to start a new thread to vote for KIP-617 as there have been
> > > > significant changes since the previous vote started.
> > > >
> > > > KIP wiki page:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards
> > > >
> > > > Many thanks!
> > > >
> > > > Jorge.
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Jun Rao]

Hi, Colin,

Thanks for the KIP. A few comments below.

10. Some of the choices in this KIP are not consistent with KIP-595. It
would be useful to make consistent choices between the two KIPs.
10.1 KIP-595 doesn't use a separate Heartbeat request and heartbeat is
piggybacked through the Fetch request.
10.2 The discussion in KIP-595 still assumes a separate controlled shutdown
request instead of heartbeat.
10.3 My understanding is that the controller is just the leader in the Raft
quorum. If so, do we need process.roles and controller.connect  in this KIP
given quorum.voters in KIP-595?

11. Fencing: It would be useful to clarify whether the fencing is 1-way or
2-way. In ZK, the fencing is 1-way. ZK server determines if a ZK session is
expired or not. An expired ZK client doesn't know it's fenced until it can
connect to ZK server. It seems in this KIP, the proposal is for the fencing
to work in both ways, i.e., the controller can fence a broker and a broker
can fence itself based on heartbeat independently. There are some tradeoffs
between these two approaches. It would be useful to document the benefits
and the limitations of the proposed approach. For example, I wonder what
happens if the controller and the broker make inconsistent fencing
decisions in the new approach.

12. BrokerRecord:
12.1 Currently, BrokerEpoch is the ZK session id. How is BrokerEpoch
generated without ZK?
12.2 KIP-584 is in progress. So, we need to include the features field.

13. PartitionRecord/IsrChange. IsrChange seems to be representing an
incremental change to ISR in PartitionRecord. For consistency, should we
have a separate record for representing incremental change to replicas?
Currently RemovingReplicas/AddingReplicas are included with many other
fields in PartitionRecord?

14. "When the active controller decides that a standby controller should
start a snapshot, it will communicate that information in its response to
the periodic heartbeat sent by that node.  When the active controller
decides that it itself should create a snapshot, it will first try to give
up the leadership of the Raft quorum in order to avoid unnecessary delays
while writing the snapshot." Is it truly necessary to only do snapshotting
in the follower? It seems it's simpler to just let every replica do
snapshotting in a background thread.

15. Currently, we store SCRAM hashes and delegation tokens in ZooKeeper.
Should we add records to account for those?

16. The description of leaderEpoch says "An epoch that gets incremented
each time we change the ISR." Currently, we only increment leaderEpoch when
the leader changes.

17. Metrics
17.1 "kafka.controller:type=KafkaController,name=MetadataSnapshotLag The
offset delta between the latest stable offset of the metadata topic and the
offset of the last snapshot (or 0 if there are no snapshots)". 0 could be a
valid lag. So using that to represent no snapshots can cause confusion.
17.2 kafka.controller:type=KafkaController,name=ControllerRequestsRate: We
already have a rateAndTIme metric per ControllerState. Do we need this new
metric?

18. Do we need a separate DeletePartition record? This could be useful to
represent the successful deletion of a single partition.

19. Do we need brokerEpoch in DeleteBroker?

20. controller.id: I had the same feeling as Jason. Requiring the user to
configure a separate controller id for each broker seems to add more
complexity. So, we need a good reason to do that. So far, it seems that's
just for having a unique id when creating the NetworkClient for the
controller. That's internal and there could be other ways to achieve this.
Thanks,

Jun


On Thu, Jul 23, 2020 at 11:02 PM Boyang Chen 
wrote:

> Hey Colin,
>
> some more questions I have about the proposal:
>
> 1. We mentioned in the networking section that "The only time when clients
> should contact a controller node directly is when they are debugging system
> issues". But later we didn't talk about how to enable this debug mode,
> could you consider getting a section about that?
>
> 2. “When the active controller decides that a standby controller should
> start a snapshot, it will communicate that information in its response to
> the periodic heartbeat sent by that node.“ In the KIP-595, we provide an
> RPC called `EndQuorumEpoch` which would transfer the leadership role to a
> dedicated successor, do you think we could reuse that method instead of
> piggy-backing on the heartbeat RPC?
>
> 3. The `DeleteBroker` record is listed but not mentioned in details for the
> KIP. Are we going to support removing a broker in runtime, or this record
> is just for the sake of removing an obsolete broker due to heartbeat
> failure?
>
> 4. In the rejected alternatives, we mentioned we don't want to combine
> heartbeats and fetch and listed out the reason was due to extra complexity.
> However, we should also mention some cons caused by this model, for example
> we are doing 2X round trips to maintain a liveness, where as a regular
> 

Re: [VOTE] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Bill Bejeck]

Thanks for the KIP Bruno.

+1 (binding)

-Bill

On Mon, Jul 27, 2020 at 4:26 AM Navinder Brar
 wrote:

> +1 (non-binding). Thanks for the KIP, Bruno.
>
> ~Navinder
>
> On Friday, 24 July, 2020, 08:41:03 pm IST, John Roesler <
> vvcep...@apache.org> wrote:
>
>  Thanks, Bruno!
>
> I'm +1 (binding)
>
> -John
>
> On Fri, Jul 24, 2020, at 07:04, Bruno Cadonna wrote:
> > Hi,
> >
> > After re-opening the discussion about
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB
> >
> > I would like to re-open the voting for this KIP.
> >
> > The discussion thread can be found here:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> >
> > Best,
> > Bruno
> >
> > On 19.05.20 10:00, Bruno Cadonna wrote:
> > > Thank you for voting!
> > >
> > > This KIP passes with:
> > > 4 binding +1
> > > 1 non-binding +1
> > > 0 -1
> > >
> > > Best,
> > > Bruno
> > >
> > > On Fri, May 15, 2020 at 11:34 PM Matthias J. Sax 
> wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >> -Matthias
> > >>
> > >> On 5/15/20 11:48 AM, John Roesler wrote:
> > >>> Thanks, Bruno!
> > >>>
> > >>> I’m +1 (binding)
> > >>>
> > >>> -John
> > >>>
> > >>> On Fri, May 15, 2020, at 11:32, Sophie Blee-Goldman wrote:
> >  Thanks Bruno! +1 (non-binding)
> > 
> >  Sophie
> > 
> >  On Fri, May 15, 2020 at 8:15 AM Bill Bejeck 
> wrote:
> > 
> > > Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > -Bill
> > >
> > > On Fri, May 15, 2020 at 11:12 AM Guozhang Wang 
> wrote:
> > >
> > >> +1.
> > >>
> > >> Thanks！
> > >>
> > >> On Fri, May 15, 2020 at 1:36 AM Bruno Cadonna  >
> > > wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> I'd like to call for votes on
> > >>>
> > >>> KIP-607: Add Metrics to Record the Memory Used by RocksDB to
> Kafka
> > >> Streams
> > >>>
> > >>> The KIP can be found here
> > >>>
> > >>>
> > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Record+the+Memory+Used+by+RocksDB
> > >>>
> > >>> The discussion thread can be found here:
> > >>>
> > >>>
> > >>
> > >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > 
> > >>
> >

[jira] [Created] (KAFKA-10314) KafkaStorageException on reassignment when offline log directories exist

[Noa Resare (Jira)]

Noa Resare created KAFKA-10314:
--

 Summary: KafkaStorageException on reassignment when offline log 
directories exist
 Key: KAFKA-10314
 URL: https://issues.apache.org/jira/browse/KAFKA-10314
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.5.0
Reporter: Noa Resare


If a reassignment of a partition is triggered to a broker with an offline 
directory, the new broker will fail to follow, instead raising a 
KafkaStorageException which causes the reassignment to stall indefinitely. The 
error message we see is the following:

{{[2020-07-23 13:11:08,727] ERROR [Broker id=1] Skipped the become-follower 
state change with correlation id 14 from controller 1 epoch 1 for partition 
t2-0 (last update controller epoch 1) with leader 2 since the replica for the 
partition is offline due to disk error 
org.apache.kafka.common.errors.KafkaStorageException: Can not create log for 
t2-0 because log directories /tmp/kafka/d1 are offline (state.change.logger)}}

It seems to me that unless the partition in question already existed on the 
offline log partition, a better behaviour would simply be to assign the 
partition to one of the available log directories.

The conditional in 
[LogManager.scala:769|https://github.com/apache/kafka/blob/11f75691b87fcecc8b29bfd25c7067e054e408ea/core/src/main/scala/kafka/log/LogManager.scala#L769]
 was introduced to prevent the issue in 
[KAFKA-4763|https://issues.apache.org/jira/browse/KAFKA-4763] where partitions 
in offline logdirs would be re-created in an online directory as soon as a 
LeaderAndISR message gets processed. However, the semantics of isNew seems 
different in LogManager (the replica is new on this broker) compared to when 
isNew is set in 
[KafkaController.scala|https://github.com/apache/kafka/blob/11f75691b87fcecc8b29bfd25c7067e054e408ea/core/src/main/scala/kafka/controller/KafkaController.scala#L879]
 (where it seems to refer to whether the topic partition in itself is new, all 
followers gets {{isNew=false}})



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #1672

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #4746

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Recommend Java 11 (#9080)

[github] MINOR: INFO log4j when request re-join (#9068)


--
[...truncated 2.77 MB...]
org.apache.kafka.connect.json.JsonConverterTest > stringHeaderToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > longToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > longToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
structWithOptionalFieldToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
structWithOptionalFieldToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
highPrecisionNumericDecimalToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
highPrecisionNumericDecimalToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToNumericJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > decimalToNumericJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > numericDecimalToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > numericDecimalToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToJsonWithoutSchema 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > decimalToJsonWithoutSchema 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson PASSED

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

[Unmesh Joshi]

Just checked etcd and zookeeper code, and both support leader to step down
as a follower to make sure there are no two leaders if the leader has been
disconnected from the majority of the followers
For etcd this is https://github.com/etcd-io/etcd/issues/3866
For Zookeeper its https://issues.apache.org/jira/browse/ZOOKEEPER-1699
I was just thinking if it would be difficult to implement in the Pull based
model, but I guess not. It is possibly the same way ISR list is managed
currently, if leader of the controller quorum loses majority of the
followers, it should step down and become follower, that way, telling
client in time that it was disconnected from the quorum, and not keep on
sending state metadata to clients.

Thanks,
Unmesh


On Mon, Jul 27, 2020 at 9:31 AM Unmesh Joshi  wrote:

> >>Could you clarify on this question? Which part of the raft group doesn't
> >>know about leader dis-connection?
> The leader of the controller quorum is partitioned from the controller
> cluster, and a different leader is elected for the remaining controller
> cluster.
> I think there are two things here,
> 1.  The old leader will not know if it's disconnected from the rest of the
> controller quorum cluster unless it receives BeginQuorumEpoch from the new
> leader. So it will keep on serving stale metadata to the clients (Brokers,
> Producers and Consumers)
> 2. I assume, the Broker Leases will be managed on the controller quorum
> leader. This partitioned leader will keep on tracking broker leases it has,
> while the new leader of the quorum will also start managing broker leases.
> So while the quorum leader is partitioned, there will be two membership
> views of the kafka brokers managed on two leaders.
> Unless broker heartbeats are also replicated as part of the Raft log,
> there is no way to solve this?
> I know LogCabin implementation does replicate client heartbeats. I suspect
> that the same issue is there in Zookeeper, which does not replicate client
> Ping requests..
>
> Thanks,
> Unmesh
>
>
>
> On Mon, Jul 27, 2020 at 6:23 AM Boyang Chen 
> wrote:
>
>> Thanks for the questions Unmesh!
>>
>> On Sun, Jul 26, 2020 at 6:18 AM Unmesh Joshi 
>> wrote:
>>
>> > Hi,
>> >
>> > In the FetchRequest Handling, how to make sure we handle scenarios where
>> > the leader might have been disconnected from the cluster, but doesn't
>> know
>> > yet?
>> >
>> Could you clarify on this question? Which part of the raft group doesn't
>> know about leader
>> dis-connection?
>>
>>
>> > As discussed in the Raft Thesis section 6.4, the linearizable semantics
>> of
>> > read requests is implemented in LogCabin by sending heartbeat to
>> followers
>> > and waiting till the heartbeats are successful to make sure that the
>> leader
>> > is still the leader.
>> > I think for the controller quorum to make sure none of the consumers get
>> > stale data, it's important to have linearizable semantics? In the pull
>> > based model, the leader will need to wait for heartbeats from the
>> followers
>> > before returning each fetch request from the consumer then? Or do we
>> need
>> > to introduce some other request?
>> > (Zookeeper does not have linearizable semantics for read requests, but
>> as
>> > of now all the kafka interactions are through writes and watches).
>> >
>> > This is a very good question. For our v1 implementation we are not
>> aiming
>> to guarantee linearizable read, which
>> would be considered as a follow-up effort. Note that today in Kafka there
>> is no guarantee on the metadata freshness either,
>> so no regression is introduced.
>>
>>
>> > Thanks,
>> > Unmesh
>> >
>> > On Fri, Jul 24, 2020 at 11:36 PM Jun Rao  wrote:
>> >
>> > > Hi, Jason,
>> > >
>> > > Thanks for the reply.
>> > >
>> > > 101. Sounds good. Regarding clusterId, I am not sure storing it in the
>> > > metadata log is enough. For example, the vote request includes
>> clusterId.
>> > > So, no one can vote until they know the clusterId. Also, it would be
>> > useful
>> > > to support the case when a voter completely loses its disk and needs
>> to
>> > > recover.
>> > >
>> > > 210. There is no longer a FindQuorum request. When a follower
>> restarts,
>> > how
>> > > does it discover the leader? Is that based on DescribeQuorum? It
>> would be
>> > > useful to document this.
>> > >
>> > > Jun
>> > >
>> > > On Fri, Jul 17, 2020 at 2:15 PM Jason Gustafson 
>> > > wrote:
>> > >
>> > > > Hi Jun,
>> > > >
>> > > > Thanks for the questions.
>> > > >
>> > > > 101. I am treating some of the bootstrapping problems as out of the
>> > scope
>> > > > of this KIP. I am working on a separate proposal which addresses
>> > > > bootstrapping security credentials specifically. Here is a rough
>> sketch
>> > > of
>> > > > how I am seeing it:
>> > > >
>> > > > 1. Dynamic broker configurations including encrypted passwords will
>> be
>> > > > persisted in the metadata log and cached in the broker's
>> > > `meta.properties`
>> > > > file.
>> > > > 2. We will provide a tool which allows 

[jira] [Created] (KAFKA-10313) Out of range offset errors leading to offset reset

[Varsha Abhinandan (Jira)]

Varsha Abhinandan created KAFKA-10313:
-

 Summary: Out of range offset errors leading to offset reset
 Key: KAFKA-10313
 URL: https://issues.apache.org/jira/browse/KAFKA-10313
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 2.2.2
Reporter: Varsha Abhinandan


Hi,
 
We have been occasionally noticing offset resets happening on the Kafka 
consumer because of offset out of range error. However, I don't see any errors 
in the broker logs. No logs related to leader-election, replica lag, Kafka 
broker pod restarts or anything. (just info logs were enabled in the prod 
environment).
 
It appeared from the logs that the out of range error was because of the fetch 
offset being larger than the offset range on the broker. Noticed this happening 
multiple times on different consumers, stream apps in the prod environment. So, 
it doesn't seem like an application bug and more like a bug in the 
KafkaConsumer. Would like to understand the cause for such errors.
 
Also, none of the offset reset options are desirable. Choosing "earliest" 
creates a sudden huge lag (we have a retention of 24hours) and choosing 
"latest" leads to data loss (the records produced between the out of range 
error and when offset reset happens on the consumer). So, wondering if it is 
better for the Kafka client to separate out 'auto.offset.reset' config for just 
offset not found. For, out of range error maybe the Kafka client can 
automatically reset the offset to latest if the fetch offset is higher to 
prevent data loss. Also, automatically reset it to earliest if the fetch offset 
is lesser than the start offset. 
 
 
Following are the logs on the consumer side :
 
[2020-07-17T08:46:00,322Z] [INFO ] [pipeline-thread-12 
([prd453-19-event-upsert]-bo-pipeline-12)] [o.a.k.c.consumer.internals.Fetcher] 
[Consumer 
clientId=bo-indexer-group-prd453-19-on-c19-bo-indexer-upsert-blue-5d665bcbb7-dnvkh-pid-1-kafka-message-source-id-544,
 groupId=bo-indexer-group-prd453-19] Fetch offset 476383711 is out of range for 
partition prd453-19-event-upsert-32, resetting offset[2020-07-17T08:46:00,330Z] 
[INFO ] [pipeline-thread-12 ([prd453-19-event-upsert]-bo-pipeline-12)] 
[o.a.k.c.consumer.internals.Fetcher] [Consumer 
clientId=bo-indexer-group-prd453-19-on-c19-bo-indexer-upsert-blue-5d665bcbb7-dnvkh-pid-1-kafka-message-source-id-544,
 groupId=bo-indexer-group-prd453-19] Resetting offset for partition 
prd453-19-event-upsert-32 to offset 453223789.
 
Broker logs for the partition :
_[2020-07-17T07:40:12,082Z]  [INFO ]  [kafka-scheduler-4]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Found deletable 
segments with base offsets [452091893] due to retention time 8640ms breach_
_[2020-07-17T07:40:12,082Z]  [INFO ]  [kafka-scheduler-4]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Scheduling log 
segment [baseOffset 452091893, size 1073741693] for deletion._
_[2020-07-17T07:40:12,083Z]  [INFO ]  [kafka-scheduler-4]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Incrementing log 
start offset to 453223789_
_[2020-07-17T07:41:12,083Z]  [INFO ]  [kafka-scheduler-7]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Deleting segment 
452091893_
_[2020-07-17T07:41:12,114Z]  [INFO ]  [kafka-scheduler-7]  
[kafka.log.LogSegment]  Deleted log 
/data/kafka/prd453-19-event-upsert-32/000452091893.log.deleted._
_[2020-07-17T07:41:12,114Z]  [INFO ]  [kafka-scheduler-7]  
[kafka.log.LogSegment]  Deleted offset index 
/data/kafka/prd453-19-event-upsert-32/000452091893.index.deleted._
_[2020-07-17T07:41:12,114Z]  [INFO ]  [kafka-scheduler-7]  
[kafka.log.LogSegment]  Deleted time index 
/data/kafka/prd453-19-event-upsert-32/000452091893.timeindex.deleted._
_[2020-07-17T07:52:31,836Z]  [INFO ]  [data-plane-kafka-request-handler-3]  
[kafka.log.ProducerStateManager]  [ProducerStateManager 
partition=prd453-19-event-upsert-32] Writing producer snapshot at offset 
475609786_
_[2020-07-17T07:52:31,836Z]  [INFO ]  [data-plane-kafka-request-handler-3]  
[kafka.log.Log]  [Log partition=prd453-19-event-upsert-32, dir=/data/kafka] 
Rolled new log segment at offset 475609786 in 1 ms._

_[2020-07-17T09:05:12,075Z]  [INFO ]  [kafka-scheduler-2]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Found deletable 
segments with base offsets [453223789] due to retention time 8640ms breach_
_[2020-07-17T09:05:12,075Z]  [INFO ]  [kafka-scheduler-2]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Scheduling log 
segment [baseOffset 453223789, size 1073741355] for deletion._
_[2020-07-17T09:05:12,075Z]  [INFO ]  [kafka-scheduler-2]  [kafka.log.Log]  
[Log partition=prd453-19-event-upsert-32, dir=/data/kafka] Incrementing log 
start 

Build failed in Jenkins: kafka-trunk-jdk14 #321

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: INFO log4j when request re-join (#9068)


--
[...truncated 2.80 MB...]
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetTime[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigPortIsNotAnInteger[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldNotFailOnBranchedMultiLevelRepartitionConnectedTopology[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldNotFailOnBranchedMultiLevelRepartitionConnectedTopology[task assignor = 
class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfVersionProbingFlagConfigIsNotAtomicInteger[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfVersionProbingFlagConfigIsNotAtomicInteger[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerConfigIsNotTaskManagerInstance[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowKafkaExceptionIfTaskMangerConfigIsNotTaskManagerInstance[task 
assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldTriggerImmediateRebalanceOnHostInfoChange[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldTriggerImmediateRebalanceOnHostInfoChange[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceMaximallyStickyAssignmentWhenMemberLeaves[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldProduceMaximallyStickyAssignmentWhenMemberLeaves[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetAssignmentConfigs[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldGetAssignmentConfigs[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowIfV2SubscriptionAndFutureSubscriptionIsMixed[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldThrowIfV2SubscriptionAndFutureSubscriptionIsMixed[task assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnInterleavedAssignmentWithUnrevokedPartitionsRemovedWhenNewConsumerJoins[task
 assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 STARTED

org.apache.kafka.streams.processor.internals.StreamsPartitionAssignorTest > 
shouldReturnInterleavedAssignmentWithUnrevokedPartitionsRemovedWhenNewConsumerJoins[task
 assignor = class 
org.apache.kafka.streams.processor.internals.assignment.FallbackPriorTaskAssignor]
 PASSED

Re: [DISCUSS] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Bruno Cadonna]

Hi Guozhang,

Do you mean compression or compaction?

Regarding compression, I agree with you except for the merge-sorting 
part. The docs describing what is stored in the block cache can be found 
under https://github.com/facebook/rocksdb/wiki/Block-Cache.


Regarding compaction, my statement in my previous e-mail about 
compaction not using block cache was a guess. To get to bottom of it, I 
asked somebody from RockDB and compaction does indeed not use block 
cache. Compaction uses the OS to read in the data to compact. But it 
also uses fadvise to tell the kernel to not cache the data in the OS 
buffer cache.


Hope that clears up the conflicts! ;-)

Best,
Bruno

On 24.07.20 19:37, Guozhang Wang wrote:

Ack, thanks for the clarification folks! Yeah I agree from JVM's point all
rocksDB memory are off-heap basically (which makes operations harder,
sigh..)

Regarding the block cache, my understanding is that by default compressed
blocks are in the OS page cache, uncompressed blocks are in the RocksDB
block cache. In Streams, we do not use compression by default, so these
data blocks would be read into the block cache for merge-sorting while
index / bloom filter / other compressed dictionary blocks are read into OS
cache by default. Obviously that conflicts from yours, maybe you can point
me to the related docs?

Guozhang

On Fri, Jul 24, 2020 at 2:15 AM Bruno Cadonna  wrote:


Hi Guozhang and Sophie,

1)
My understanding is also that the memtables are off-heap (as almost
every data structure in RocksDB).

According to the docs, if after a write the size of the memtable exceeds
option write_buffer_size the memtable is flushed. I would not call it
hard bounded since it seems the memtable can exceed this size.

Guozhang's other statements about memtables seem correct to me.

2)
According to the docs, the block cache caches data in memory for reads.
I do not think RocksDB uses the block cache for compaction, because that
would mean each compaction would interfere with the used cache
replacement policy (LRU is the default in Streams). I suppose RocksDB
uses the OS cache during compactions. So block cache usage contains data
blocks for reading and it can also contain index blocks and filter block
if configured accordingly (e.g. by using the BoundedMemoryRocksDBConfig
described under

https://kafka.apache.org/25/documentation/streams/developer-guide/memory-mgmt.html).

The pinned usage are blocks pinned by table readers like iterators.
The block cache can be soft or hard bounded. However, there is currently
an open bug in RocksDB regarding hard-bounded block caches.

3)
The statements seem correct.

The total memory usage seems also correct.

Best,
Bruno

On 23.07.20 20:46, Sophie Blee-Goldman wrote:

Guozhang,

Just to clarify, the "heap" for all these objects is actually the C++

heap,

not the JVM heap. So in the words of a Java application these would
   all be considered "off-heap", right?

(Of course there are some pointers in the Java heap to the off-heap
objects but that size is trivial compared to the actual objects)

Sorry for being pedantic. I just happen to know this is a question that
gets frequently asked so it's probably good to be as clear  as possible
in the KIP/metrics description.

Also, can you clarify a bit what you mean by hard bounded vs soft

bounded?

For example, my impression is that the memtables are actually not hard
bounded at all, while the block cache is soft bounded by default but can
be configured to be hard bounded. And obviously the OS cache is not
exactly bounded but it shouldn't cause you to run out of usable memory
(like the memtables for example might, and have). But I think maybe

you're

using a different definition of hard/soft bounded than I'm thinking of

On Thu, Jul 23, 2020 at 8:07 AM Guozhang Wang 

wrote:



Thanks Bruno, they made sense to me.

Regarding the last comment, my main reasoning is that it's better to
explain to users the rocksDB memory usage and link the metrics to

different

categories.

Just to kick off this (and also asking for correction of my own
understanding :) here's what I read from the metrics:

1. Memtables (aka writer buffers, AND read buffers for iterators which
would pin the immutable memtables from flushing). It is allocated

on-heap

and hard-bounded (via memtable_size * max_num_memtables).

 - cur-size-active-mem-table: active
 - cur-size-all-mem-tables: active + unflushed write
 - size-all-mem-tables: active + unflushed write + pinned read

2. Block cache (used for merging / compaction, reads). Allocated on-heap
and soft-bounded.

 - block-cache-usage: compaction + read
 - block-cache-pinned-usage: read

3. OS cache (read buffer), which is the memory usage for filters /

indices

that are outside block cache. Allocated off-heap and not bounded at all.

 - estimate-table-readers-mem


The total memory usage (on-heap and off-heap) is "size-all-mem-tables" +
"block-cache-usage" + "estimate-table-readers-mem".

[DISCUSS] KIP-647: Add ability to handle late messages in streams-aggregation

[Igor Piddubnyi]

Hi everybody,
I would like to start off the discussion for KIP-647:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-647%3A+Add+ability+to+handle+late+messages+in+streams-aggregation
  


This KIP proposes a minor adjustment in the kafka-streams aggregation-api, 
adding an ability for processing late messages.
[WIP] PR here:https://github.com/apache/kafka/pull/9017

Please check.
Regards, Igor.

Re: [VOTE] KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka Streams

[Navinder Brar]

+1 (non-binding). Thanks for the KIP, Bruno.

~Navinder 

On Friday, 24 July, 2020, 08:41:03 pm IST, John Roesler 
 wrote:  
 
 Thanks, Bruno!

I'm +1 (binding)

-John

On Fri, Jul 24, 2020, at 07:04, Bruno Cadonna wrote:
> Hi,
> 
> After re-opening the discussion about
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Report+Properties+of+RocksDB
> 
> I would like to re-open the voting for this KIP.
> 
> The discussion thread can be found here:
>  
> http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> 
> Best,
> Bruno
> 
> On 19.05.20 10:00, Bruno Cadonna wrote:
> > Thank you for voting!
> > 
> > This KIP passes with:
> > 4 binding +1
> > 1 non-binding +1
> > 0 -1
> > 
> > Best,
> > Bruno
> > 
> > On Fri, May 15, 2020 at 11:34 PM Matthias J. Sax  wrote:
> >>
> >> +1 (binding)
> >>
> >> -Matthias
> >>
> >> On 5/15/20 11:48 AM, John Roesler wrote:
> >>> Thanks, Bruno!
> >>>
> >>> I’m +1 (binding)
> >>>
> >>> -John
> >>>
> >>> On Fri, May 15, 2020, at 11:32, Sophie Blee-Goldman wrote:
>  Thanks Bruno! +1 (non-binding)
> 
>  Sophie
> 
>  On Fri, May 15, 2020 at 8:15 AM Bill Bejeck  wrote:
> 
> > Thanks for the KIP!
> >
> > +1 (binding)
> >
> > -Bill
> >
> > On Fri, May 15, 2020 at 11:12 AM Guozhang Wang  
> > wrote:
> >
> >> +1.
> >>
> >> Thanks！
> >>
> >> On Fri, May 15, 2020 at 1:36 AM Bruno Cadonna 
> > wrote:
> >>
> >>> Hi all,
> >>>
> >>> I'd like to call for votes on
> >>>
> >>> KIP-607: Add Metrics to Record the Memory Used by RocksDB to Kafka
> >> Streams
> >>>
> >>> The KIP can be found here
> >>>
> >>>
> >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-607%3A+Add+Metrics+to+Kafka+Streams+to+Record+the+Memory+Used+by+RocksDB
> >>>
> >>> The discussion thread can be found here:
> >>>
> >>>
> >>
> > http://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCADR0NwzJBJa6WihnpmGj0R%2BYPVrojq4Kg_hOArNEytHAG-tZAQ%40mail.gmail.com%3E
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> 
> >>
>  

[VOTE] KIP-635: GetOffsetShell: support for multiple topics and consumer configuration override

[Dániel Urbán]

Hello everyone,

I'd like to start a vote on KIP-635. The KIP enhances the GetOffsetShell
tool by enabling querying multiple topic-partitions, adding new filtering
options, and adding a config override option.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-635%3A+GetOffsetShell%3A+support+for+multiple+topics+and+consumer+configuration+override

The original discussion thread was named "[DISCUSS] KIP-308:
GetOffsetShell: new KafkaConsumer API, support for multiple topics,
minimize the number of requests to server". The id had to be changed as
there was a collision, and the KIP also had to be renamed, as some of its
motivations were outdated.

Thanks,
Daniel

for help about kafka

[琴台之声]

hello,apache kafka:
 I've already subscribed to Kafka® development.
 I am a user of kafka，the version I am using is 
kafka_2.12-2.1.1
 now ，I want to update my kafka broker to a new version 
kafka_2.12-2.4.1，cause I know the new version has a lot of optimizations, such 
as improvement about kafka consumer rebalance. so ,I want to know 
how I can verify this by experiment，this is very important for me,cause I need 
test results to prove the necessity of the upgrade.



ps: you can also send me the result of the test from kafka inside if there is a 
such result and this result is allowed to be published


Looking forward to your reply as soon as possible！！！

  

[jira] [Resolved] (KAFKA-10301) Partition#remoteReplicasMap can be empty in certain race conditions

[Stanislav Kozlovski (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stanislav Kozlovski resolved KAFKA-10301.
-
Resolution: Fixed

> Partition#remoteReplicasMap can be empty in certain race conditions
> ---
>
> Key: KAFKA-10301
> URL: https://issues.apache.org/jira/browse/KAFKA-10301
> Project: Kafka
>  Issue Type: Bug
>Reporter: Stanislav Kozlovski
>Assignee: Stanislav Kozlovski
>Priority: Blocker
>
> In Partition#updateAssignmentAndIsr, we would previously update the 
> `partition#remoteReplicasMap` by adding the new replicas to the map and then 
> removing the old ones 
> ([source]([https://github.com/apache/kafka/blob/7f9187fe399f3f6b041ca302bede2b3e780491e7/core/src/main/scala/kafka/cluster/Partition.scala#L657)]
> During a recent refactoring, we changed it to first clear the map and then 
> add all the replicas to it 
> ([source]([https://github.com/apache/kafka/blob/2.6/core/src/main/scala/kafka/cluster/Partition.scala#L663]))
> While this is done in a write lock (`inWriteLock(leaderIsrUpdateLock)`), not 
> all callers that access the map structure use a lock. Some examples:
>  - Partition#updateFollowerFetchState
>  - DelayedDeleteRecords#tryComplete
>  - Partition#getReplicaOrException - called in 
> `checkEnoughReplicasReachOffset` without a lock, which itself is called by 
> DelayedProduce. I think this can fail a  `ReplicaManager#appendRecords` call.
> While we want to polish the code to ensure these sort of race conditions 
> become harder (or impossible) to introduce, it sounds safest to revert to the 
> previous behavior given the timelines regarding the 2.6 release. Jira 
> https://issues.apache.org/jira/browse/KAFKA-10302 tracks further 
> modifications to the code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)