Re: Stream config in StreamPartitionAssignor

[Boyang Chen]

Hey Guozhang,


thanks for the reply. Actually I'm not trying to pass config into 
StreamPartitionAssignor. My debugging was about some other configs that wasn't 
transferred properly (completely irrelevant), but the effort was delayed 
because the log would print "StreamsConfig values" for StreamPartitionAssignor, 
where it will fallback all the config to defaults. So it would give me as an 
end user the impression that my config passed in was "not successfully" 
conveyed to each Stream Thread.


I'm thinking whether we could disable stream partitioner to print out its 
config in the log, or have a more explicit way saying this is not Stream Thread 
config? Hope I made myself clear to you.


Also yes, I could put some comment in the docs to let user use Consumer config 
prefix if that could help.


Best,

Boyang


From: Guozhang Wang 
Sent: Tuesday, July 10, 2018 5:35 AM
To: dev
Subject: Re: Stream config in StreamPartitionAssignor

Hello Boyang,

Thanks for brining this up. Currently since the StreamsPartitionAssingor
can only be initiated within the Consumer instance, I think letting users
to pass in the config values with prefix is the preferrable way, i.e. we
can improve our docs to educate users about that. BTW I'm curious what
configs you want to pass into the StreamsPartitionAssignor? Currently since
there is no user configurable part inside that class, I do not know what
configs can take effects.


Guozhang

On Mon, Jul 9, 2018 at 2:06 PM, Boyang Chen  wrote:

> Hey there,
>
>
> over the weekend I was debugging the streams configuration not passed
> within threads. I noticed that one of the code path from KafkaConsumer
> (L743) was to initialize the StreamPartitionAssignor:
>
> this.assignors = config.getConfiguredInstances(
> ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> PartitionAssignor.class);
>
> However, it was using the ConsumerConfig instance (that config is passed
> in), so if I want to make some configuration change in the assignor, I need
> to put consumer prefix. To make the debugging even harder, there was an
> logAll() function in AbstractConfig which will print "StreamsConfig values"
> at the beginning, since it is indeed a stream config:
>
> @Override
> public void configure(final Map configs) {
> final StreamsConfig streamsConfig = new StreamsConfig(configs);
>
> (L190 in StreamPartitionAssignor)
>
>
> This would further confuse developer as they see two different sets of
> StreamsConfig: one from top level, one from this derived level per thread.
>
>
> My point is that we could either: 1. let developer be aware that they need
> to add consumer prefix to pass in configs to StreamPartitionAssignor 2. we
> found a way to pass in original StreamsConfig.
>
> I know this is a little bit lengthy description, let me know if you feel
> unclear about my proposal, or this is not a concern since most people
> already know the trick here, thank you!
>
>


--
-- Guozhang

[jira] [Created] (KAFKA-7144) Kafka Streams doesn't properly balance partition assignment

[James Cheng (JIRA)]

James Cheng created KAFKA-7144:
--

 Summary: Kafka Streams doesn't properly balance partition 
assignment
 Key: KAFKA-7144
 URL: https://issues.apache.org/jira/browse/KAFKA-7144
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: James Cheng
 Attachments: OneThenTwelve.java

Kafka Streams doesn't always spread the tasks across all available 
instances/threads

I have a topology which consumes a single partition topic and goes .through() a 
12 partition topic. The makes 13 partitions.

 

I then started 2 instances of the application. I would have expected the 13 
partitions to be split across the 2 instances roughly evenly (7 partitions on 
one, 6 partitions on the other).

Instead, one instance gets 12 partitions, and the other instance gets 1 
partition.

 

Repro case attached. I ran it a couple times, and it was fairly repeatable.

Setup for the repro:
{code:java}
$ ./bin/kafka-topics.sh --zookeeper localhost --create --topic one --partitions 
1 --replication-factor 1 
$ ./bin/kafka-topics.sh --zookeeper localhost --create --topic twelve 
--partitions 12 --replication-factor 1
$ echo foo | kafkacat -P -b 127.0.0.1 -t one
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7143) Cannot use KafkaConsumer with Kotlin coroutines due to Thread id check

[Raman Gupta (JIRA)]

Raman Gupta created KAFKA-7143:
--

 Summary: Cannot use KafkaConsumer with Kotlin coroutines due to 
Thread id check
 Key: KAFKA-7143
 URL: https://issues.apache.org/jira/browse/KAFKA-7143
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 1.1.0
Reporter: Raman Gupta


I am using a new KafkaConsumer via Kotlin. My application makes use of Kotlin 
[coroutines|https://kotlinlang.org/docs/reference/coroutines.html], which 
supports a style of async programming that avoids the need for callbacks (and 
existing callback-based API's such as Kafka's can easily be adapted to this). 
With coroutines, methods with callbacks are suspended, and resumed once the 
call is complete. With this approach, while access to the KafkaConsumer is done 
in a thread-safe way, it does NOT happen from a single thread -- a different 
underlying thread may actually execute the code after the suspension point.

However, the KafkaConsumer includes additional checks to verify not only the 
thread safety of the client, but that the *same thread* is being used -- if the 
same thread (by id) is not being used the consumer throws an exception like:

{code}
Exception in thread "ForkJoinPool.commonPool-worker-25" 
java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
multi-threaded access
at 
org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1824)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1808)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1321)
{code}

I understand this check is present to protect people from themselves, but I'd 
like the ability to disable this check so that this code can be used 
effectively by libraries such as Kotlin coroutines.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk10 #281

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Add Scalafmt to Streams Scala API (#4965)

--
[...truncated 1.53 MB...]
kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId PASSED

kafka.coordinator.transaction.TransactionStateManagerTest > 
testAppendTransactionToLogWhileProducerFenced STARTED

kafka.coordinator.transaction.TransactionStateManagerTest > 

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Anna Povzner]

Hi Dong,


Thanks for considering my suggestions.


Based on your comments, I realized that my suggestion was not complete with
regard to KafkaConsumer API vs. consumer-broker protocol. While I propose
to keep KafkaConsumer#seek() unchanged and take offset only, the underlying
consumer will send the next FetchRequest() to broker with offset and
leaderEpoch if it is known (based on leader epoch cache in consumer) — note
that this is different from the current KIP, which suggests to always send
unknown leader epoch after seek(). This way, if the consumer and a broker
agreed on the point of non-divergence, which is some {offset, leaderEpoch}
pair, the new leader which causes another truncation (even further back)
will be able to detect new divergence and restart the process of finding
the new point of non-divergence. So, to answer your question, If the
truncation happens just after the user calls
KafkaConsumer#findOffsets(offset, leaderEpoch) followed by seek(offset),
the user will not seek to the wrong position without knowing that
truncation has happened, because the consumer will get another truncation
error, and seek again.


I am afraid, I did not understand your second question. Let me summarize my
suggestions again, and then give an example to hopefully make my
suggestions more clear. Also, the last part of my example shows how the
use-case in your first question will work. If it does not answer your
second question, would you mind clarifying? I am also focusing on the case
of a consumer having enough entries in the cache. The case of restarting
from committed offset either stored externally or internally will probably
need to be discussed more.


Let me summarize my suggestion again:

1) KafkaConsumer#seek() and KafkaConsumer#position() remains unchanged

2) New KafkaConsumer#findOffsets() takes {offset, leaderEpoch} pair per
topic partition and returns offset per topic partition.

3) FetchRequest() to broker after KafkaConsumer#seek() will contain the
offset set by seek and leaderEpoch that corresponds to the offset based on
leader epoch cache in the consumer.


The rest of this e-mail is a long and contrived example with several log
truncations and unclean leader elections to illustrate the API and your
first use-case. Suppose we have three brokers. Initially, Broker A, B, and
C has one message at offset 0 with leader epoch 0. Then, Broker A goes down
for some time. Broker B becomes a leader with epoch 1, and writes messages
to offsets 1 and 2. Broker C fetches offset 1, but before fetching offset
2, becomes a leader with leader epoch 2 and writes a message at offset 2.
Here is the state of brokers at this point:

> Broker A:
> offset 0, epoch 0 <— leader
> goes down…


> Broker B:
> offset 0, epoch 0
> offset 1, epoch 1  <- leader
> offset 2, epoch 1



Broker C:
> offset 0, epoch 0
> offset 1, epoch 1
> offset 2, epoch 2 <— leader


Before Broker C becomes a leader with leader epoch 2, the consumer consumed
the following messages from broker A and broker B:

{offset=0, leaderEpoch=0}, {offset=1, leaderEpoch=1}, {offset=2,
leaderEpoch=1}.

Consumer’s leader epoch cache at this point contains the following entries:

(leaderEpoch=0, startOffset=0)

(leaderEpoch=1, startOffset=1)

endOffset = 3


Then, broker B becomes the follower of broker C, truncates and starts
fetching from offset 2.

Consumer sends fetchRequest(offset=3, leaderEpoch=1) and gets LOG_TRUNCATION
error from broker C.

In response, the client calls KafkaConsumer#findOffsets(offset=3,
leaderEpoch=1). The underlying consumer sends
OffsetsForLeaderEpoch(leaderEpoch=1), broker C responds with
{leaderEpoch=1, endOffset=2}. So, KafkaConsumer#findOffsets(offset=3,
leaderEpoch=1) returns offset=2.

In response, consumer calls KafkaConsumer@seek(offset=2) followed by
poll(), which results in FetchRequest(offset=2, leaderEpoch=1) to broker C.


I will continue with this example with the goal to answer your first
question about truncation just after findOffsets() followed by seek():

Suppose, brokers B and C go down, and broker A comes up and becomes a
leader with leader epoch 3, and writes a message to offset 1. Suppose, this
happens before the consumer gets response from broker C to the previous
fetch request:  FetchRequest(offset=2, leaderEpoch=1).

Consumer re-sends FetchRequest(offset=2, leaderEpoch=1) to broker A, which
returns LOG_TRUNCATION error, because broker A has leader epoch 3 >  leader
epoch in FetchRequest with starting offset = 1 < offset 2 in FetchRequest().

In response, the user calls KafkaConsumer#findOffsets(offset=2,
leaderEpoch=1). The underlying consumer sends
OffsetsForLeaderEpoch(leaderEpoch=1), broker A responds with
{leaderEpoch=0, endOffset=1}; the underlying consumer finds leaderEpoch = 0
in its cache with end offset == 1, which results in
KafkaConsumer#findOffsets(offset=2, leaderEpoch=1) returning offset = 1.

In response, the user calls KafkaConsumer@seek(offset=1) followed by
poll(), which results in 

Re: [VOTE] 1.1.1 RC3

[Satish Duggana]

+1 (non-binding)

- Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag with jdk-8.
- Ran through quickstart of core/streams on builds.
- Ran few internal apps targeting to topics on 3 node cluster.

Thanks,
Satish.


On Mon, Jul 9, 2018 at 11:39 PM, Harsha  wrote:

> +1.
>
> * Ran unit tests
> * Installed in a cluster and ran simple tests
>
> Thanks,
> Harsha
>
> On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu  wrote:
>
> >
> >
> >
> > +1
> >
> > Ran test suite.
> >
> > Checked signatures.
> >
> >
> >
> > On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon...@gmail.com > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the fourth candidate for release of Apache Kafka 1.1.1.
> > >
> > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> > first
> > > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> > since
> > > that release. A few of the more significant fixes include:
> > >
> > > KAFKA-6925 < https://issues.apache.org/jira/browse/KAFKA-6925> - Fix
> > memory
> > > leak in StreamsMetricsThreadImpl
> > > KAFKA-6937 < https://issues.apache.org/jira/browse/KAFKA-6937> -
> In-sync
> >
> > > replica delayed during fetch if replica throttle is exceeded
> > > KAFKA-6917 < https://issues.apache.org/jira/browse/KAFKA-6917> -
> Process
> >
> > > txn
> > > completion asynchronously to avoid deadlock
> > > KAFKA-6893 < https://issues.apache.org/jira/browse/KAFKA-6893> -
> Create
> > > processors before starting acceptor to avoid ArithmeticException
> > > KAFKA-6870 < https://issues.apache.org/jira/browse/KAFKA-6870> -
> > > Fix ConcurrentModificationException in SampledStat
> > > KAFKA-6878 < https://issues.apache.org/jira/browse/KAFKA-6878> - Fix
> > > NullPointerException when querying global state store
> > > KAFKA-6879 < https://issues.apache.org/jira/browse/KAFKA-6879> -
> Invoke
> > > session init callbacks outside lock to avoid Controller deadlock
> > > KAFKA-6857 < https://issues.apache.org/jira/browse/KAFKA-6857> -
> Prevent
> >
> > > follower from truncating to the wrong offset if undefined leader epoch
> > is
> > > requested
> > > KAFKA-6854 < https://issues.apache.org/jira/browse/KAFKA-6854> - Log
> > > cleaner
> > > fails with transaction markers that are deleted during clean
> > > KAFKA-6747 < https://issues.apache.org/jira/browse/KAFKA-6747> - Check
> > > whether there is in-flight transaction before aborting transaction
> > > KAFKA-6748 < https://issues.apache.org/jira/browse/KAFKA-6748> -
> Double
> > > check before scheduling a new task after the punctuate call
> > > KAFKA-6739 < https://issues.apache.org/jira/browse/KAFKA-6739> -
> > > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> > > KAFKA-6728 < https://issues.apache.org/jira/browse/KAFKA-6728> -
> > > Fix NullPointerException when instantiating the HeaderConverter
> > >
> > > Kafka 1.1.1 release plan:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> > >
> > > Release notes for the 1.1.1 release:
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Thursday, July 12, 12pm PT ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
> > >
> > > * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> > > https://github.com/apache/kafka/tree/1.1.1-rc3
> > >
> > > * Documentation:
> > > http://kafka.apache.org/11/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/11/protocol.html
> > >
> > > * Successful Jenkins builds for the 1.1 branch:
> > > Unit/integration tests: * https://builds.apache.org/job/
> kafka-1.1-jdk7/162
> >
> > > < https://builds.apache.org/job/kafka-1.1-jdk7/162>*
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
> > >
> > > Please test and verify the release artifacts and submit a vote for this
> > RC,
> > > or report any issues so we can fix them and get a new RC out ASAP.
> > Although
> > > this release vote requires PMC votes to pass, testing, votes, and bug
> > > reports are valuable and appreciated from everyone.
> > >
> > >
> > > Regards,
> > > Dong
> > >
> >
> >
> >
> >
> >
> >
>

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

[Lucas Wang]

Hi Dong,

The simulation of disk being slow is merely for me to easily construct a
testing scenario
with a backlog of produce requests. In production, other than the disk
being slow, a backlog of
produce requests may also be caused by high produce QPS.
In that case, we may not want to kill the broker and that's when this KIP
can be useful, both for JBOD
and non-JBOD setup.

Going back to your previous question about each ProduceRequest covering 20
partitions that are randomly
distributed, let's say a LeaderAndIsr request is enqueued that tries to
switch the current broker, say broker0, from leader to follower
*for one of the partitions*, say *test-0*. For the sake of argument, let's
also assume the other brokers, say broker1, have *stopped* fetching from
the current broker, i.e. broker0.
1. If the enqueued produce requests have acks =  -1 (ALL)
  1.1 without this KIP, the ProduceRequests ahead of LeaderAndISR will be
put into the purgatory,
and since they'll never be replicated to other brokers (because of
the assumption made above), they will
be completed either when the LeaderAndISR request is processed or
when the timeout happens.
  1.2 With this KIP, broker0 will immediately transition the partition
test-0 to become a follower,
after the current broker sees the replication of the remaining 19
partitions, it can send a response indicating that
it's no longer the leader for the "test-0".
  To see the latency difference between 1.1 and 1.2, let's say there are
24K produce requests ahead of the LeaderAndISR, and there are 8 io threads,
  so each io thread will process approximately 3000 produce requests. Now
let's investigate the io thread that finally processed the LeaderAndISR.
  For the 3000 produce requests, if we model the time when their remaining
19 partitions catch up as t0, t1, ...t2999, and the LeaderAndISR request is
processed at time t3000.
  Without this KIP, the 1st produce request would have waited an extra
t3000 - t0 time in the purgatory, the 2nd an extra time of t3000 - t1, etc.
  Roughly speaking, the latency difference is bigger for the earlier
produce requests than for the later ones. For the same reason, the more
ProduceRequests queued
  before the LeaderAndISR, the bigger benefit we get (capped by the produce
timeout).
2. If the enqueued produce requests have acks=0 or acks=1
  There will be no latency differences in this case, but
  2.1 without this KIP, the records of partition test-0 in the
ProduceRequests ahead of the LeaderAndISR will be appended to the local log,
and eventually be truncated after processing the LeaderAndISR. This
is what's referred to as
"some unofficial definition of data loss in terms of messages
beyond the high watermark".
  2.2 with this KIP, we can mitigate the effect since if the LeaderAndISR
is immediately processed, the response to producers will have
the NotLeaderForPartition error, causing producers to retry

This explanation above is the benefit for reducing the latency of a broker
becoming the follower,
closely related is reducing the latency of a broker becoming the leader.
In this case, the benefit is even more obvious, if other brokers have
resigned leadership, and the
current broker should take leadership. Any delay in processing the
LeaderAndISR will be perceived
by clients as unavailability. In extreme cases, this can cause failed
produce requests if the retries are
exhausted.

Another two types of controller requests are UpdateMetadata and
StopReplica, which I'll briefly discuss as follows:
For UpdateMetadata requests, delayed processing means clients receiving
stale metadata, e.g. with the wrong leadership info
for certain partitions, and the effect is more retries or even fatal
failure if the retries are exhausted.

For StopReplica requests, a long queuing time may degrade the performance
of topic deletion.

Regarding your last question of the delay for DescribeLogDirsRequest, you
are right
that this KIP cannot help with the latency in getting the log dirs info,
and it's only relevant
when controller requests are involved.

Regards,
Lucas


On Tue, Jul 3, 2018 at 5:11 PM, Dong Lin  wrote:

> Hey Jun,
>
> Thanks much for the comments. It is good point. So the feature may be
> useful for JBOD use-case. I have one question below.
>
> Hey Lucas,
>
> Do you think this feature is also useful for non-JBOD setup or it is only
> useful for the JBOD setup? It may be useful to understand this.
>
> When the broker is setup using JBOD, in order to move leaders on the failed
> disk to other disks, the system operator first needs to get the list of
> partitions on the failed disk. This is currently achieved using
> AdminClient.describeLogDirs(), which sends DescribeLogDirsRequest to the
> broker. If we only prioritize the controller requests, then the
> DescribeLogDirsRequest
> may still take a long time to be processed by the broker. So the overall
> time to move leaders away from the failed disk 

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[Guozhang Wang]

Let me clarify a bit on what I meant about moving `retentionPeriod` to
WindowStoreBuilder:

In another discussion we had around KIP-319 / 330, that the "retention
period" should not really be a window spec, but only a window store spec,
as it only affects how long to retain each window to be queryable along
with the storage cost.

More specifically, today the "maintainMs" returned from Windows is used in
three places:

1) for windowed aggregations, they are passed in directly into
`Stores.persistentWindows()` as the retention period parameters. For this
use case we should just let the WindowStoreBuilder to specify this value
itself.

NOTE: It is also returned in the KStreamWindowAggregate processor, to
determine if a received record should be dropped due to its lateness. We
may need to think of another way to get this value inside the processor

2) for windowed stream-stream join, it is used as the join range parameter
but only to check that "windowSizeMs <= retentionPeriodMs". We can do this
check at the store builder lever instead of at the processor level.


If we can remove its usage in both 1) and 2), then we should be able to
safely remove this from the `Windows` spec.


Guozhang


On Mon, Jul 9, 2018 at 3:53 PM, John Roesler  wrote:

> Thanks for the reply, Guozhang,
>
> Good! I agree, that is also a good reason, and I actually made use of that
> in my tests. I'll update the KIP.
>
> By the way, I chose "allowedLateness" as I was trying to pick a better name
> than "close", but I think it's actually the wrong name. We don't want to
> bound the lateness of events in general, only with respect to the end of
> their window.
>
> If we have a window [0,10), with "allowedLateness" of 5, then if we get an
> event with timestamp 3 at time 9, the name implies we'd reject it, which
> seems silly. Really, we'd only want to start rejecting that event at stream
> time 15.
>
> What I meant was more like "allowedLatenessAfterWindowEnd", but that's too
> verbose. I think that "close" + some documentation about what it means will
> be better.
>
> 1: "Close" would be measured from the end of the window, so a reasonable
> default would be "0". Recall that "close" really only needs to be specified
> for final results, and a default of 0 would produce the most intuitive
> results. If folks later discover that they are missing some late events,
> they can adjust the parameter accordingly. IMHO, any other value would just
> be a guess on our part.
>
> 2a:
> I think you're saying to re-use "until" instead of adding "close" to the
> window.
>
> The downside here would be that the semantic change could be more confusing
> than deprecating "until" and introducing window "close" and a
> "retentionTime" on the store builder. The deprecation is a good, controlled
> way for us to make sure people are getting the semantics they think they're
> getting, as well as giving us an opportunity to link people to the API they
> should use instead.
>
> I didn't fully understand the second part, but it sounds like you're
> suggesting to add a new "retentionTime" setter to Windows to bridge the gap
> until we add it to the store builder? That seems kind of roundabout to me,
> if that's what you meant. We could just immediately add it to the store
> builders in the same PR.
>
> 2b: Sounds good to me!
>
> Thanks again,
> -John
>
>
> On Mon, Jul 9, 2018 at 4:55 PM Guozhang Wang  wrote:
>
> > John,
> >
> > Thanks for your replies. As for the two options of the API, I think I'm
> > slightly inclined to the first option as well. My motivation is a bit
> > different, as I think of the first one maybe more flexible, for example:
> >
> > KTable> table = ... count();
> >
> > table.toStream().peek(..);   // want to peek at the changelog stream, do
> > not care about final results.
> >
> > table.suppress().toStream().to("topic");// sending to a topic, want
> to
> > only send the final results.
> >
> > --
> >
> > Besides that, I have a few more minor questions:
> >
> > 1. For "allowedLateness", what should be the default value? I.e. if user
> do
> > not specify "allowedLateness" in TimeWindows, what value should we set?
> >
> > 2. For API names, some personal suggestions here:
> >
> > 2.a) "allowedLateness"  -> "until" (semantics changed, and also value is
> > defined as delta on top of window length), where "until" ->
> > "retentionPeriod", and the latter will be removed from `Windows` to `
> > WindowStoreBuilder` in the future.
> >
> > 2.b) "BufferConfig" -> "Buffered" ?
> >
> >
> >
> > Guozhang
> >
> >
> > On Mon, Jul 9, 2018 at 2:09 PM, John Roesler  wrote:
> >
> > > Hey Matthias and Guozhang,
> > >
> > > Sorry for the slow reply. I was mulling about your feedback and
> weighing
> > > some ideas in a sketchbook PR: https://github.com/apache/
> kafka/pull/5337
> > .
> > >
> > > Your thought about keeping suppression independent of business logic
> is a
> > > very good one. I agree that it would make more sense to add some kind
> of
> 

[jira] [Created] (KAFKA-7142) Rebalancing large consumer group can block the coordinator broker for several seconds

[Ying Zheng (JIRA)]

Ying Zheng created KAFKA-7142:
-

 Summary: Rebalancing large consumer group can block the 
coordinator broker for several seconds
 Key: KAFKA-7142
 URL: https://issues.apache.org/jira/browse/KAFKA-7142
 Project: Kafka
  Issue Type: Improvement
Reporter: Ying Zheng






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[John Roesler]

Thanks for the reply, Guozhang,

Good! I agree, that is also a good reason, and I actually made use of that
in my tests. I'll update the KIP.

By the way, I chose "allowedLateness" as I was trying to pick a better name
than "close", but I think it's actually the wrong name. We don't want to
bound the lateness of events in general, only with respect to the end of
their window.

If we have a window [0,10), with "allowedLateness" of 5, then if we get an
event with timestamp 3 at time 9, the name implies we'd reject it, which
seems silly. Really, we'd only want to start rejecting that event at stream
time 15.

What I meant was more like "allowedLatenessAfterWindowEnd", but that's too
verbose. I think that "close" + some documentation about what it means will
be better.

1: "Close" would be measured from the end of the window, so a reasonable
default would be "0". Recall that "close" really only needs to be specified
for final results, and a default of 0 would produce the most intuitive
results. If folks later discover that they are missing some late events,
they can adjust the parameter accordingly. IMHO, any other value would just
be a guess on our part.

2a:
I think you're saying to re-use "until" instead of adding "close" to the
window.

The downside here would be that the semantic change could be more confusing
than deprecating "until" and introducing window "close" and a
"retentionTime" on the store builder. The deprecation is a good, controlled
way for us to make sure people are getting the semantics they think they're
getting, as well as giving us an opportunity to link people to the API they
should use instead.

I didn't fully understand the second part, but it sounds like you're
suggesting to add a new "retentionTime" setter to Windows to bridge the gap
until we add it to the store builder? That seems kind of roundabout to me,
if that's what you meant. We could just immediately add it to the store
builders in the same PR.

2b: Sounds good to me!

Thanks again,
-John


On Mon, Jul 9, 2018 at 4:55 PM Guozhang Wang  wrote:

> John,
>
> Thanks for your replies. As for the two options of the API, I think I'm
> slightly inclined to the first option as well. My motivation is a bit
> different, as I think of the first one maybe more flexible, for example:
>
> KTable> table = ... count();
>
> table.toStream().peek(..);   // want to peek at the changelog stream, do
> not care about final results.
>
> table.suppress().toStream().to("topic");// sending to a topic, want to
> only send the final results.
>
> --
>
> Besides that, I have a few more minor questions:
>
> 1. For "allowedLateness", what should be the default value? I.e. if user do
> not specify "allowedLateness" in TimeWindows, what value should we set?
>
> 2. For API names, some personal suggestions here:
>
> 2.a) "allowedLateness"  -> "until" (semantics changed, and also value is
> defined as delta on top of window length), where "until" ->
> "retentionPeriod", and the latter will be removed from `Windows` to `
> WindowStoreBuilder` in the future.
>
> 2.b) "BufferConfig" -> "Buffered" ?
>
>
>
> Guozhang
>
>
> On Mon, Jul 9, 2018 at 2:09 PM, John Roesler  wrote:
>
> > Hey Matthias and Guozhang,
> >
> > Sorry for the slow reply. I was mulling about your feedback and weighing
> > some ideas in a sketchbook PR: https://github.com/apache/kafka/pull/5337
> .
> >
> > Your thought about keeping suppression independent of business logic is a
> > very good one. I agree that it would make more sense to add some kind of
> > "window close" concept to the window definition.
> >
> > In fact, doing that immediately solves the inconsistency problem Guozhang
> > brought up. There's no need to add a "final results" or "emission" option
> > to the windowed aggregation.
> >
> > What do you think about an API more like this:
> >
> > final StreamsBuilder builder = new StreamsBuilder();
> >
> > builder
> >   .stream("input", Consumed.with(STRING_SERDE, STRING_SERDE))
> >   .groupBy(
> > (String k1, String v1) -> k1,
> > Serialized.with(STRING_SERDE, STRING_SERDE)
> >   )
> >   .windowedBy(TimeWindows
> > .of(scaledTime(2L))
> > .until(scaledTime(3L))
> > .allowedLateness(scaledTime(1L))
> >   )
> >   .count(Materialized.as("counts"))
> >   .suppress(
> > emitFinalResultsOnly(
> >   BufferConfig.withBufferKeys(10_000L).bufferFullStrategy(SHUT_DOWN)
> > )
> >   )
> >   .toStream()
> >   .to("output-suppressed", Produced.with(STRING_SERDE, LONG_SERDE));
> >
> > Note that:
> >  * "emitFinalResultsOnly" is available *only* on windowed tables
> (enforced
> > by the type system at compile time), and it determines the time to wait
> by
> > looking at "allowedLateness" on the TimeWindows config.
> >  * querying "counts" will produce results (eventually) consistent with
> > what's observable in "output-suppressed".
> >  * in all cases, "suppress" has no effect on business logic, just on
> event
> > suppression.
> >
> > Is this API 

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

[Ismael Juma]

Late +1 (binding) from me too.

Ismael

On Mon, Jul 9, 2018 at 3:18 PM Vahid S Hashemian 
wrote:

> KIP-231 is approved with 3 "+1" binding votes by Ewen, Jason, and Harsha.
>
> Thanks to everyone who provided feedback and/or voted.
>
> Regards.
> --Vahid
>
>
>
>
> From:   Vahid S Hashemian/Silicon Valley/IBM
> To: dev 
> Date:   12/19/2017 11:30 AM
> Subject:[VOTE] KIP-231: Improve the Required ACL of ListGroups API
>
>
> I believe the concerns on this KIP have been addressed so far.
> Therefore, I'd like to start a vote.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API
>
> Thanks.
> --Vahid
>
>
>
>

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

[Vahid S Hashemian]

KIP-231 is approved with 3 "+1" binding votes by Ewen, Jason, and Harsha.

Thanks to everyone who provided feedback and/or voted.

Regards.
--Vahid




From:   Vahid S Hashemian/Silicon Valley/IBM
To: dev 
Date:   12/19/2017 11:30 AM
Subject:[VOTE] KIP-231: Improve the Required ACL of ListGroups API


I believe the concerns on this KIP have been addressed so far.
Therefore, I'd like to start a vote.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API

Thanks.
--Vahid

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[Guozhang Wang]

John,

Thanks for your replies. As for the two options of the API, I think I'm
slightly inclined to the first option as well. My motivation is a bit
different, as I think of the first one maybe more flexible, for example:

KTable> table = ... count();

table.toStream().peek(..);   // want to peek at the changelog stream, do
not care about final results.

table.suppress().toStream().to("topic");// sending to a topic, want to
only send the final results.

--

Besides that, I have a few more minor questions:

1. For "allowedLateness", what should be the default value? I.e. if user do
not specify "allowedLateness" in TimeWindows, what value should we set?

2. For API names, some personal suggestions here:

2.a) "allowedLateness"  -> "until" (semantics changed, and also value is
defined as delta on top of window length), where "until" ->
"retentionPeriod", and the latter will be removed from `Windows` to `
WindowStoreBuilder` in the future.

2.b) "BufferConfig" -> "Buffered" ?



Guozhang


On Mon, Jul 9, 2018 at 2:09 PM, John Roesler  wrote:

> Hey Matthias and Guozhang,
>
> Sorry for the slow reply. I was mulling about your feedback and weighing
> some ideas in a sketchbook PR: https://github.com/apache/kafka/pull/5337.
>
> Your thought about keeping suppression independent of business logic is a
> very good one. I agree that it would make more sense to add some kind of
> "window close" concept to the window definition.
>
> In fact, doing that immediately solves the inconsistency problem Guozhang
> brought up. There's no need to add a "final results" or "emission" option
> to the windowed aggregation.
>
> What do you think about an API more like this:
>
> final StreamsBuilder builder = new StreamsBuilder();
>
> builder
>   .stream("input", Consumed.with(STRING_SERDE, STRING_SERDE))
>   .groupBy(
> (String k1, String v1) -> k1,
> Serialized.with(STRING_SERDE, STRING_SERDE)
>   )
>   .windowedBy(TimeWindows
> .of(scaledTime(2L))
> .until(scaledTime(3L))
> .allowedLateness(scaledTime(1L))
>   )
>   .count(Materialized.as("counts"))
>   .suppress(
> emitFinalResultsOnly(
>   BufferConfig.withBufferKeys(10_000L).bufferFullStrategy(SHUT_DOWN)
> )
>   )
>   .toStream()
>   .to("output-suppressed", Produced.with(STRING_SERDE, LONG_SERDE));
>
> Note that:
>  * "emitFinalResultsOnly" is available *only* on windowed tables (enforced
> by the type system at compile time), and it determines the time to wait by
> looking at "allowedLateness" on the TimeWindows config.
>  * querying "counts" will produce results (eventually) consistent with
> what's observable in "output-suppressed".
>  * in all cases, "suppress" has no effect on business logic, just on event
> suppression.
>
> Is this API straightforward? Or do you still prefer the version that both
> proposed:
>
>   ...
>   .windowedBy(TimeWindows
> .of(scaledTime(2L))
> .until(scaledTime(3L))
> .allowedLateness(scaledTime(1L))
>   )
>   .count(
> Materialized.as("counts"),
> emitFinalResultsOnly(
>   BufferConfig.withBufferKeys(10_000L).bufferFullStrategy(SHUT_DOWN)
> )
>   )
>   ...
>
> To me, these two are practically identical, and I still vaguely prefer the
> first one.
>
> The prototype has made clearer to me that users of "final results for
> windows" and users of "suppression for table events" both need to configure
> the suppression buffer.
>
> This buffer configuration consists of:
> 1. how many keys or bytes to keep in memory
> 2. what to do if memory runs out (shut down, start using disk, ...)
>
> So it's not as simple as setting a "final results" flag. We'll either have
> an "Emit" config object on the windowed aggregators that takes the same
> BufferConfig that the "Suppress" config on the suppression operator, or we
> just use the suppression operator for both.
>
> Perhaps it would sweeten the deal a little to point out that we have 2
> overloads already for each windowed aggregator (with and without
> Materialized). Adding "Emitted" or something would mean that we'd add a new
> overload for each one, taking us up to 4 overloads each for "count",
> "aggregate" and "reduce". Using "suppress" means that we don't add any new
> overloads.
>
> Thanks again for helping to hash this out,
> -John
>
> On Fri, Jul 6, 2018 at 6:20 PM Guozhang Wang  wrote:
>
> > I think I agree with Matthias for having dedicated APIs for windowed
> > operation final output scenario, PLUS separating the window close which
> the
> > "final output" would rely on, from the window retention time itself
> > (admittedly it would make this KIP effort larger, but if we believe we
> need
> > to do this separation anyways we could just do it now).
> >
> > And then we can have the `KTable#suppress()` for intermediate-suppression
> > only, not for late-record-suppression, until we've seen that becomes a
> > common feature request because our current design still allows to be
> > extended for that purpose.
> >
> >
> > 

Re: Stream config in StreamPartitionAssignor

[Guozhang Wang]

Hello Boyang,

Thanks for brining this up. Currently since the StreamsPartitionAssingor
can only be initiated within the Consumer instance, I think letting users
to pass in the config values with prefix is the preferrable way, i.e. we
can improve our docs to educate users about that. BTW I'm curious what
configs you want to pass into the StreamsPartitionAssignor? Currently since
there is no user configurable part inside that class, I do not know what
configs can take effects.


Guozhang

On Mon, Jul 9, 2018 at 2:06 PM, Boyang Chen  wrote:

> Hey there,
>
>
> over the weekend I was debugging the streams configuration not passed
> within threads. I noticed that one of the code path from KafkaConsumer
> (L743) was to initialize the StreamPartitionAssignor:
>
> this.assignors = config.getConfiguredInstances(
> ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
> PartitionAssignor.class);
>
> However, it was using the ConsumerConfig instance (that config is passed
> in), so if I want to make some configuration change in the assignor, I need
> to put consumer prefix. To make the debugging even harder, there was an
> logAll() function in AbstractConfig which will print "StreamsConfig values"
> at the beginning, since it is indeed a stream config:
>
> @Override
> public void configure(final Map configs) {
> final StreamsConfig streamsConfig = new StreamsConfig(configs);
>
> (L190 in StreamPartitionAssignor)
>
>
> This would further confuse developer as they see two different sets of
> StreamsConfig: one from top level, one from this derived level per thread.
>
>
> My point is that we could either: 1. let developer be aware that they need
> to add consumer prefix to pass in configs to StreamPartitionAssignor 2. we
> found a way to pass in original StreamsConfig.
>
> I know this is a little bit lengthy description, let me know if you feel
> unclear about my proposal, or this is not a concern since most people
> already know the trick here, thank you!
>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

[John Roesler]

Hey Matthias and Guozhang,

Sorry for the slow reply. I was mulling about your feedback and weighing
some ideas in a sketchbook PR: https://github.com/apache/kafka/pull/5337.

Your thought about keeping suppression independent of business logic is a
very good one. I agree that it would make more sense to add some kind of
"window close" concept to the window definition.

In fact, doing that immediately solves the inconsistency problem Guozhang
brought up. There's no need to add a "final results" or "emission" option
to the windowed aggregation.

What do you think about an API more like this:

final StreamsBuilder builder = new StreamsBuilder();

builder
  .stream("input", Consumed.with(STRING_SERDE, STRING_SERDE))
  .groupBy(
(String k1, String v1) -> k1,
Serialized.with(STRING_SERDE, STRING_SERDE)
  )
  .windowedBy(TimeWindows
.of(scaledTime(2L))
.until(scaledTime(3L))
.allowedLateness(scaledTime(1L))
  )
  .count(Materialized.as("counts"))
  .suppress(
emitFinalResultsOnly(
  BufferConfig.withBufferKeys(10_000L).bufferFullStrategy(SHUT_DOWN)
)
  )
  .toStream()
  .to("output-suppressed", Produced.with(STRING_SERDE, LONG_SERDE));

Note that:
 * "emitFinalResultsOnly" is available *only* on windowed tables (enforced
by the type system at compile time), and it determines the time to wait by
looking at "allowedLateness" on the TimeWindows config.
 * querying "counts" will produce results (eventually) consistent with
what's observable in "output-suppressed".
 * in all cases, "suppress" has no effect on business logic, just on event
suppression.

Is this API straightforward? Or do you still prefer the version that both
proposed:

  ...
  .windowedBy(TimeWindows
.of(scaledTime(2L))
.until(scaledTime(3L))
.allowedLateness(scaledTime(1L))
  )
  .count(
Materialized.as("counts"),
emitFinalResultsOnly(
  BufferConfig.withBufferKeys(10_000L).bufferFullStrategy(SHUT_DOWN)
)
  )
  ...

To me, these two are practically identical, and I still vaguely prefer the
first one.

The prototype has made clearer to me that users of "final results for
windows" and users of "suppression for table events" both need to configure
the suppression buffer.

This buffer configuration consists of:
1. how many keys or bytes to keep in memory
2. what to do if memory runs out (shut down, start using disk, ...)

So it's not as simple as setting a "final results" flag. We'll either have
an "Emit" config object on the windowed aggregators that takes the same
BufferConfig that the "Suppress" config on the suppression operator, or we
just use the suppression operator for both.

Perhaps it would sweeten the deal a little to point out that we have 2
overloads already for each windowed aggregator (with and without
Materialized). Adding "Emitted" or something would mean that we'd add a new
overload for each one, taking us up to 4 overloads each for "count",
"aggregate" and "reduce". Using "suppress" means that we don't add any new
overloads.

Thanks again for helping to hash this out,
-John

On Fri, Jul 6, 2018 at 6:20 PM Guozhang Wang  wrote:

> I think I agree with Matthias for having dedicated APIs for windowed
> operation final output scenario, PLUS separating the window close which the
> "final output" would rely on, from the window retention time itself
> (admittedly it would make this KIP effort larger, but if we believe we need
> to do this separation anyways we could just do it now).
>
> And then we can have the `KTable#suppress()` for intermediate-suppression
> only, not for late-record-suppression, until we've seen that becomes a
> common feature request because our current design still allows to be
> extended for that purpose.
>
>
> Guozhang
>
> On Wed, Jul 4, 2018 at 12:53 PM, Matthias J. Sax 
> wrote:
>
> > Thanks for the discussion. I am just catching up.
> >
> > In general, I think we have different uses cases and non-windowed and
> > windowed is quite different. For the non-windowed case, suppress() has
> > no (useful) close or retention time, no final semantics, and also no
> > business logic impact.
> >
> > On the other hand, for windowed aggregations, close time and final
> > result do have a meaning. IMHO, `close()` is part of business logic
> > while retention time is not. Also, suppression of intermediate result is
> > not a business rule and there might be use case for which either "early
> > intermediate" (before window end time) are suppressed only, or all
> > intermediates are suppressed (maybe also something in the middle, ie,
> > just reduce the load of intermediate updates). Thus, window-suppression
> > is much richer.
> >
> > IMHO, a generic `suppress()` operator that can be inserted into the data
> > flow at any point is useful. Maybe we should keep is as generic as
> > possible. However, it might be difficult to use with regard to
> > windowing, as the mental effort to use it is high.
> >
> > With regard to Guozhang's comment:
> >
> > > we will actually
> > > 

Stream config in StreamPartitionAssignor

[Boyang Chen]

Hey there,


over the weekend I was debugging the streams configuration not passed within 
threads. I noticed that one of the code path from KafkaConsumer (L743) was to 
initialize the StreamPartitionAssignor:

this.assignors = config.getConfiguredInstances(
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
PartitionAssignor.class);

However, it was using the ConsumerConfig instance (that config is passed in), 
so if I want to make some configuration change in the assignor, I need to put 
consumer prefix. To make the debugging even harder, there was an logAll() 
function in AbstractConfig which will print "StreamsConfig values" at the 
beginning, since it is indeed a stream config:

@Override
public void configure(final Map configs) {
final StreamsConfig streamsConfig = new StreamsConfig(configs);

(L190 in StreamPartitionAssignor)


This would further confuse developer as they see two different sets of 
StreamsConfig: one from top level, one from this derived level per thread.


My point is that we could either: 1. let developer be aware that they need to 
add consumer prefix to pass in configs to StreamPartitionAssignor 2. we found a 
way to pass in original StreamsConfig.

I know this is a little bit lengthy description, let me know if you feel 
unclear about my proposal, or this is not a concern since most people already 
know the trick here, thank you!

Recovering partition leadership outside ISR

[Jack Foy]

Hi all,

We had an outage last week that I think we could have prevented, and I'd
like to get some feedback on the idea.

tl;dr:

When a partition leader writes an updated ISR, it should also record its current
log-end-offset. On leader election, if there are no live replicas in
the ISR, then a
replica with this same log-end-offset should be preferred before considering
unclean leader election.

Details and use case:

We have a 5-node Kafka 1.0.0 cluster (since upgraded to 1.1.0) with unclean
leader election disabled. Well-configured topics have replication factor 3 and
min.insync.replicas 2, with producers setting acks=all.

On Monday our cloud provider suffered hardware failure, causing a partial
outage on network connectivity to disk storage. Broker 5's storage was on the
orphaned side of the network partition.

At the very start of the incident, broker 5 dropped all followers on brokers 1
and 4 out of the ISR for partitions it was leading. Its connections to brokers
2 and 3 and to Zookeeper stayed up, including to the controller on broker 3.
Broker 5 went offline entirely a few moments later, and stayed down with disk
state inaccessible for several hours.

We had configured multiple partitions with broker 5 as their leader and
followers on brokers 1 and 4. Before the incident those partitions had ISR
{5,1,4}, which shrank to {5} before broker 5 disappeared - leaving us with no
eligible replicas to become leader.

The only ways to bring these partitions back up were to either recover broker
5's up-to-date disk state, or to enable unclean leader election. Had we lost
one follower, then the other, and then the leader, enabling unclean leader
election would have carried 50% risk of message loss.

In the end, we decided that the lowest-risk option was to enable unclean leader
election on the affected topics, force a controller election, watch the
partitions come back up, and disable unclean election.

I think there's a safer recovery path that Kafka could support:

The leader should also record its current log-end-offset when it writes an
updated ISR. If the controller determines that it can't promote a replica from
the ISR, it should next look for a replica that has that same log-end-offset.
Only if that step also fails should it then consider unclean leader election.

For our failure case, at least, this would have allowed a clean and automatic
recovery. Has this idea been considered before? Does it have fatal flaws?

Thanks,

--
Jack Foy 

Re: [VOTE] 1.1.1 RC3

[Harsha]

+1.

* Ran unit tests
* Installed in a cluster and ran simple tests

Thanks,
Harsha

On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu  wrote:

> 
> 
> 
> +1
> 
> Ran test suite.
> 
> Checked signatures.
> 
> 
> 
> On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon...@gmail.com > wrote:
> 
> > Hello Kafka users, developers and client-developers,
> >
> > This is the fourth candidate for release of Apache Kafka 1.1.1.
> >
> > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> first
> > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> since
> > that release. A few of the more significant fixes include:
> >
> > KAFKA-6925 < https://issues.apache.org/jira/browse/KAFKA-6925> - Fix
> memory
> > leak in StreamsMetricsThreadImpl
> > KAFKA-6937 < https://issues.apache.org/jira/browse/KAFKA-6937> - In-sync
> 
> > replica delayed during fetch if replica throttle is exceeded
> > KAFKA-6917 < https://issues.apache.org/jira/browse/KAFKA-6917> - Process
> 
> > txn
> > completion asynchronously to avoid deadlock
> > KAFKA-6893 < https://issues.apache.org/jira/browse/KAFKA-6893> - Create
> > processors before starting acceptor to avoid ArithmeticException
> > KAFKA-6870 < https://issues.apache.org/jira/browse/KAFKA-6870> -
> > Fix ConcurrentModificationException in SampledStat
> > KAFKA-6878 < https://issues.apache.org/jira/browse/KAFKA-6878> - Fix
> > NullPointerException when querying global state store
> > KAFKA-6879 < https://issues.apache.org/jira/browse/KAFKA-6879> - Invoke
> > session init callbacks outside lock to avoid Controller deadlock
> > KAFKA-6857 < https://issues.apache.org/jira/browse/KAFKA-6857> - Prevent
> 
> > follower from truncating to the wrong offset if undefined leader epoch
> is
> > requested
> > KAFKA-6854 < https://issues.apache.org/jira/browse/KAFKA-6854> - Log
> > cleaner
> > fails with transaction markers that are deleted during clean
> > KAFKA-6747 < https://issues.apache.org/jira/browse/KAFKA-6747> - Check
> > whether there is in-flight transaction before aborting transaction
> > KAFKA-6748 < https://issues.apache.org/jira/browse/KAFKA-6748> - Double
> > check before scheduling a new task after the punctuate call
> > KAFKA-6739 < https://issues.apache.org/jira/browse/KAFKA-6739> -
> > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> > KAFKA-6728 < https://issues.apache.org/jira/browse/KAFKA-6728> -
> > Fix NullPointerException when instantiating the HeaderConverter
> >
> > Kafka 1.1.1 release plan:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> >
> > Release notes for the 1.1.1 release:
> > http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday, July 12, 12pm PT ***
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/
> >
> > * Javadoc:
> > http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
> >
> > * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> > https://github.com/apache/kafka/tree/1.1.1-rc3
> >
> > * Documentation:
> > http://kafka.apache.org/11/documentation.html
> >
> > * Protocol:
> > http://kafka.apache.org/11/protocol.html
> >
> > * Successful Jenkins builds for the 1.1 branch:
> > Unit/integration tests: * https://builds.apache.org/job/kafka-1.1-jdk7/162
> 
> > < https://builds.apache.org/job/kafka-1.1-jdk7/162>*
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
> >
> > Please test and verify the release artifacts and submit a vote for this
> RC,
> > or report any issues so we can fix them and get a new RC out ASAP.
> Although
> > this release vote requires PMC votes to pass, testing, votes, and bug
> > reports are valuable and appreciated from everyone.
> >
> >
> > Regards,
> > Dong
> >
> 
> 
> 
> 
> 
>

[jira] [Created] (KAFKA-7141) kafka-consumer-group doesn't describe existing group

[Bohdana Panchenko (JIRA)]

Bohdana Panchenko created KAFKA-7141:


 Summary: kafka-consumer-group doesn't describe existing group
 Key: KAFKA-7141
 URL: https://issues.apache.org/jira/browse/KAFKA-7141
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 1.0.1, 0.11.0.0
Reporter: Bohdana Panchenko


I am running two consumers: akka-stream-kafka consumer with standard config 
section as described in the 
[https://doc.akka.io/docs/akka-stream-kafka/current/consumer.html] and  
kafka-console-consumer.
I am able to see the both groups with the command

*kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list*
{color:#205081}_Note: This will not show information about old Zookeeper-based 
consumers._{color}

{color:#205081}_myakkastreamkafka-1_{color}
{color:#205081}_console-consumer-57171_
{color}

{color:#33}I am able to view details about the console consumer group{color}

*kafka-consumer-groups --describe --bootstrap-server 127.0.0.1:9092 --group 
console-consumer-57171*
_{color:#205081}Note: This will not show information about old Zookeeper-based 
consumers.{color}_


_{color:#205081}TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID 
HOST CLIENT-ID{color}_
_{color:#205081}STREAM-TEST 0 0 0 0 
consumer-1-6b928e07-196a-4322-9928-068681617878 /172.19.0.4 consumer-1{color}_

{color:#33}But the command to describe my akka stream consumer gives me 
empty output:{color}

*kafka-consumer-groups --describe --bootstrap-server 127.0.0.1:9092 --group 
myakkastreamkafka-1*
{color:#205081}_Note: This will not show information about old Zookeeper-based 
consumers._{color}


{color:#205081}_TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID 
HOST CLIENT-ID_{color}

 

{color:#33}That is strange. Can you please check the issue?{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Register now for ApacheCon and save $250

[Rich Bowen]

Greetings, Apache software enthusiasts!

(You’re getting this because you’re on one or more dev@ or users@ lists 
for some Apache Software Foundation project.)


ApacheCon North America, in Montreal, is now just 80 days away, and 
early bird prices end in just two weeks - on July 21. Prices will be 
going up from $550 to $800 so register NOW to save $250, at 
http://apachecon.com/acna18


And don’t forget to reserve your hotel room. We have negotiated a 
special rate and the room block closes August 24. 
http://www.apachecon.com/acna18/venue.html


Our schedule includes over 100 talks and we’ll be featuring talks from 
dozens of ASF projects.,  We have inspiring keynotes from some of the 
brilliant members of our community and the wider tech space, including:


 * Myrle Krantz, PMC chair for Apache Fineract, and leader in the open 
source financing space
 * Cliff Schmidt, founder of Literacy Bridge (now Amplio) and creator 
of the Talking Book project

 * Bridget Kromhout, principal cloud developer advocate at Microsoft
 * Euan McLeod, Comcast engineer, and pioneer in streaming video

We’ll also be featuring tracks for Geospatial science, Tomcat, 
Cloudstack, and Big Data, as well as numerous other fields where Apache 
software is leading the way. See the full schedule at 
http://apachecon.com/acna18/schedule.html


As usual we’ll be running our Apache BarCamp, the traditional ApacheCon 
Hackathon, and the Wednesday evening Lighting Talks, too, so you’ll want 
to be there.


Register today at http://apachecon.com/acna18 and we’ll see you in Montreal!

--
Rich Bowen
VP, Conferences, The Apache Software Foundation
h...@apachecon.com
@ApacheCon

[jira] [Created] (KAFKA-7140) Remove deprecated poll usages

[Viktor Somogyi (JIRA)]

Viktor Somogyi created KAFKA-7140:
-

 Summary: Remove deprecated poll usages
 Key: KAFKA-7140
 URL: https://issues.apache.org/jira/browse/KAFKA-7140
 Project: Kafka
  Issue Type: Improvement
Reporter: Viktor Somogyi
Assignee: Viktor Somogyi


There are a couple of poll(long) usages of the consumer in test and non-test 
code. This jira would aim to remove the non-test usages of the method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 1.1.1 RC3

[Ted Yu]

+1

Ran test suite.

Checked signatures.

On Sun, Jul 8, 2018 at 3:36 PM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> *
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
>
> Regards,
> Dong
>

Re: [VOTE] KIP-280: Enhanced log compaction

[Andras Beni]

Hi Luís,

Can you please clarify how the header value has to be encoded in case log
compaction strategy is 'header'. As I see current PR reads varLong in
CleanerCache.extractVersion and read String and uses toLong in
Cleaner.extractVersion while the KIP says no more than 'the header value
(which must be of type "long")'.

Otherwise +1 for the KIP

As for current implementation: it seems in Cleaner class header key
"version" is hardwired.

Andras



On Fri, Jul 6, 2018 at 10:36 PM Jun Rao  wrote:

> Hi, Guozhang,
>
> For #4, what you suggested could make sense for timestamp based de-dup, but
> not sure how general it is since the KIP also supports de-dup based on
> header.
>
> Thanks,
>
> Jun
>
> On Fri, Jul 6, 2018 at 1:12 PM, Guozhang Wang  wrote:
>
> > Hello Jun,
> > Thanks for your feedbacks. I'd agree on #3 that it's worth adding a
> special
> > check to not delete the last message, since although unlikely, it is
> still
> > possible that a new active segment gets rolled out but contains no data
> > yet, and hence the actual last message in this case would be in a
> > "compact-able" segment.
> >
> > For the second part of #4 you raised, maybe we could educate users to
> set "
> > message.timestamp.difference.max.ms" to be no larger than "
> > log.cleaner.delete.retention.ms" (its default value is Long.MAX_VALUE)?
> A
> > more aggressive approach would be changing the default value of the
> former
> > to be the value of the latter if:
> >
> > 1. cleanup.policy = compact OR compact,delete
> > 2. log.cleaner.compaction.strategy != offset
> >
> > Because in this case I think it makes sense to really allow users send
> any
> > data longer than "log.cleaner.delete.retention.ms", WDYT?
> >
> >
> > Guozhang
> >
> >
> > On Fri, Jul 6, 2018 at 11:51 AM, Jun Rao  wrote:
> >
> > > Hi, Luis,
> > >
> > > 1. The cleaning policy is configurable at both global and topic level.
> > The
> > > global one has the name log.cleanup.policy and the topic level has the
> > name
> > > cleanup.policy by just stripping the log prefix. We can probably do the
> > > same for the new configs.
> > >
> > > 2. Since this KIP may require an admin to configure a larger dedup
> buffer
> > > size, it would be useful to document this impact in the wiki and the
> > > release notes.
> > >
> > > 3. Yes, it's unlikely for the last message to be removed in the current
> > > implementation since we never clean the active segment. However, in
> > theory,
> > > this can happen. So it would be useful to guard this explicitly.
> > >
> > > 4. Just thought about another issue. We probably want to be a bit
> careful
> > > with key deletion. Currently, one can delete a key by sending a message
> > > with a delete tombstone (a null payload). To prevent a reader from
> > missing
> > > a deletion if it's removed too quickly, we depend on a configuration
> > > log.cleaner.delete.retention.ms (defaults to 1 day). The delete
> > tombstone
> > > will only be physically removed from the log after that amount of time.
> > The
> > > expectation is that a reader should finish reading to the end of the
> log
> > > after consuming a message within that configured time. With the new
> > > strategy, we have similar, but slightly different problems. The first
> > > problem is that the delete tombstone may be delivered earlier than an
> > > outdated record in offset order to a consumer. In order for the
> consumer
> > > not to take the outdated record, the consumer should cache the deletion
> > > tombstone for some configured amount of time. We ca probably piggyback
> > this
> > > on log.cleaner.delete.retention.ms, but we need to document this. The
> > > second problem is that once the delete tombstone is physically removed
> > from
> > > the log, how can we prevent outdated records to be added (otherwise,
> they
> > > will never be garbage collected)? Not sure what's the best way to do
> > this.
> > > One possible way is to push this back to the application and require
> the
> > > user not to publish outdated records after
> log.cleaner.delete.retention.
> > ms
> > > .
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jul 4, 2018 at 11:11 AM, Luís Cabral
> >  > > >
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > -:  1. I guess both new configurations will be at the topic level?
> > > >
> > > > They will exist in the global configuration, at the very least.
> > > > I would like to have them on the topic level as well, but there is an
> > > > inconsistency between the cleanup/compaction properties that exist
> > “only
> > > > globally” vs “globally + per topic”.
> > > > I haven’t gotten around to investigating why, and if that reason
> would
> > > > then also impact the properties I’m suggesting. At first glance they
> > seem
> > > > to belong with the properties that are "only globally” configured,
> but
> > > > Guozhang has written earlier with a suggestion of a compaction
> property
> > > > that works for both (though I haven’t had the time to look into it
> 

Re: [DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma]

Thanks for the KIP. It would be helpful to understand the user experience
for the case where the implementor uses the headers. It seems like it would
require overriding two methods?

Ismael

On Mon, Jul 9, 2018 at 1:50 AM Viktor Somogyi 
wrote:

> Hi folks,
>
> I've published KIP-336 which is about consolidating the
> Serializer/Deserializer interfaces.
>
> Basically the story here is when ExtendedSerializer and
> ExtendedDeserializer were added we still supported Java 7 and therefore had
> to use compatible constructs which now seem unnecessary since we dropped
> support for Java 7. Now in this KIP I propose a way to deprecate those
> patterns:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242
>
> I'd be happy to receive some feedback about the KIP I published.
>
> Cheers,
> Viktor
>

[jira] [Created] (KAFKA-7139) Support to exclude the internal topics in kafka-topics.sh command

[Chia-Ping Tsai (JIRA)]

Chia-Ping Tsai created KAFKA-7139:
-

 Summary: Support to exclude the internal topics in kafka-topics.sh 
command
 Key: KAFKA-7139
 URL: https://issues.apache.org/jira/browse/KAFKA-7139
 Project: Kafka
  Issue Type: Task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Usually we have no interesting in internal topics. However, the kafka-topics.sh 
can't exclude the internal topics simply. We have to write the regular to 
exclude the internal topics. This issue tries to add a flag (perhaps 
"exclude-internal") to kafka-topics.sh. If user set the flag, all internal 
topics will be excluded by the following commands - "list", "describe", 
"delete" and "alter"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSSION] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Viktor Somogyi]

Hi folks,

I've published KIP-336 which is about consolidating the
Serializer/Deserializer interfaces.

Basically the story here is when ExtendedSerializer and
ExtendedDeserializer were added we still supported Java 7 and therefore had
to use compatible constructs which now seem unnecessary since we dropped
support for Java 7. Now in this KIP I propose a way to deprecate those
patterns:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242

I'd be happy to receive some feedback about the KIP I published.

Cheers,
Viktor

Re: Old deprecated producer

[jna]

Thx, Ismael. I mean "user batch".
Not a background thread batching. So i guess that transactions allow me 
to do that.



On 07/07/2018 17:34, Ismael Juma wrote:

The old Scala producers were removed in 2.0.0. The Java producer supports
batching.

Ismael


On Sat, 7 Jul 2018, 05:38 jna,  wrote:


Hello,

I'm using the old producer API, and i saw since a long time (few
versions) that this API is deprecated. When will you decide to remove
this old API? If you won't remove it, perhaps you could remove the
deprecated. Will you provide a new way to produce a user batch of
records or the transactions replace already this batch producer?

Thanks.

Build failed in Jenkins: kafka-trunk-jdk8 #2795

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Upgrade rocksdb to 5.14.2 (#5343)

--
[...truncated 435.12 KB...]
kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered STARTED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered PASSED

kafka.utils.LoggingTest > testLogName STARTED

kafka.utils.LoggingTest > testLogName PASSED

kafka.utils.LoggingTest > testLogNameOverride STARTED

kafka.utils.LoggingTest > testLogNameOverride PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.CoreUtilsTest > testAbs STARTED

kafka.utils.CoreUtilsTest > testAbs PASSED

kafka.utils.CoreUtilsTest > testReplaceSuffix STARTED

kafka.utils.CoreUtilsTest > testReplaceSuffix PASSED

kafka.utils.CoreUtilsTest > testCircularIterator STARTED

kafka.utils.CoreUtilsTest > testCircularIterator PASSED

kafka.utils.CoreUtilsTest > testReadBytes STARTED

kafka.utils.CoreUtilsTest > testReadBytes PASSED

kafka.utils.CoreUtilsTest > testCsvList STARTED

kafka.utils.CoreUtilsTest > testCsvList PASSED

kafka.utils.CoreUtilsTest > testReadInt STARTED

kafka.utils.CoreUtilsTest > testReadInt PASSED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate STARTED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

Build failed in Jenkins: kafka-trunk-jdk10 #280

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Upgrade rocksdb to 5.14.2 (#5343)

--
[...truncated 1.98 MB...]
org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetInternalLeaveGroupOnCloseConfigToFalseInConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldForwardCustomConfigsWithNoPrefixToAllClients PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfRestoreConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyCorrectValueSerdeClassOnError PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfGlobalConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerIsolationLevelIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedAdminConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigCommitIntervalMsIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportMultipleBootstrapServers PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyNoOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionWhenOptimizationConfigNotValueInRange PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfKeySerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportNonPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetRestoreConsumerConfigs 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfBootstrapServersIsNotSet PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigMustUseAdminClientConfigForRetries PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetMainConsumerConfigsWithMainConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedRestoreConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedRestoreConsumerConfigs PASSED