Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Rajini and Jun. I'll update the shouldBeRebuilt() docs on what
happens to existing SSL Connections.

Thanks everybody who participated in the discussion thread and voting
thread and spending the valuable time reviewing the KIP/PR. Could not have
done it without your support. Really appreciate it.

Now we can close on the voting phase since we had it open for > 72 hrs but
since this is my first KIP, can you please guide what happens now? Can I
move the KIP to accepted state myself?

Thanks
Maulin

On Tue, Mar 31, 2020 at 10:27 AM Jun Rao  wrote:

> Hi, Rajini, Maulin,
>
> 1. Ok. Then we can keep the package name as it is.
>
> 2. Thanks for updating the javadoc for shouldBeRebuilt(). Could you also
> clarify after the SslEngine is rebuilt, what happens to existing SSL
> connections?
>
> Thanks,
>
> Jun
>
> On Tue, Mar 31, 2020 at 2:07 AM Rajini Sivaram 
> wrote:
>
> > Hi Jun, Maulin,
> >
> >  org.apache.kafka.common.security.ssl contains internal classes like
> > SslFactory.  org.apache.kafka.common.security.auth is a public package
> > which contains all our current authentication-related classes. If we want
> > to move the new interface into an SSL-specific package, we should perhaps
> > create a new public package rather than use an existing internal one?
> >
> > On Tue, Mar 31, 2020 at 7:56 AM Manikumar 
> > wrote:
> >
> > > +1 (binding).
> > > Thanks for the KIP.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > So far we got 3 Binding votes. I am planning to keep the voting phase
> > > open
> > > > until Tuesday 10 PM Pacific Time which will be more than 72 hours
> from
> > > the
> > > > first binding vote on Thursday 12:36 PM Pacific Time.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I updated the Javadoc in the KIP details and the actual
> > > SslEngineFactory
> > > > > interface for shouldBeRebuilt(). For the first comment, probably
> I'll
> > > try
> > > > > to address it tomorrow.
> > > > >
> > > > > Thanks
> > > > > Maulin
> > > > >
> > > > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> > > > maulin.vasav...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Thanks Jun Rao for your vote and comments.
> > > > >>
> > > > >> For 1) Earlier it was the security.ssl package but after a review
> I
> > > > >> changed it to .auth since there are some public interfaces in that
> > > > package.
> > > > >> I am open to move it under .ssl package.
> > > > >>
> > > > >> For 2) Sure. Will document in Javadocs for the method.
> > > > >>
> > > > >> Thanks
> > > > >> Maulin
> > > > >>
> > > > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> > > > >>
> > > > >>> Hi, Maulin,
> > > > >>>
> > > > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
> > > below.
> > > > >>>
> > > > >>> 1. Should the package name of the new
> > > > >>> interface SslEngineFactory be
> org.apache.kafka.common.security.ssl
> > > > >>> instead
> > > > >>> of org.apache.kafka.common.security.auth?
> > > > >>> 2. Could you document when shouldBeRebuilt() will be called?
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> > > > >>> maulin.vasav...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1
> Binding
> > > vote
> > > > >>> so
> > > > >>> > far, can you please vote so that we can try to move forward
> with
> > > > >>> changes?
> > > > >>> >
> > > > >>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas
> > > >  > > > >>> >
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > +1 (non-binding)
> > > > >>> > >
> > > > >>> > > Regards,
> > > > >>> > > Thomas
> > > > >>> > >
> > > > >>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" <
> > rajinisiva...@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>> > >
> > > > >>> > > +1 (binding)
> > > > >>> > > Thanks for the KIP, Maulin!
> > > > >>> > >
> > > > >>> > > Regards,
> > > > >>> > >
> > > > >>> > > Rajini
> > > > >>> > >
> > > > >>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> > > > >>> > > maulin.vasav...@gmail.com>
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > > > FYI - we have updated the KIP documentation also with
> > > > >>> appropriate
> > > > >>> > > code
> > > > >>> > > > samples for interfaces and few important changes.
> > > > >>> > > >
> > > > >>> > > > Thanks
> > > > >>> > > > Maulin
> > > > >>> > > >
> > > > >>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > > > >>> > > > maulin.vasav...@gmail.com>
> > > > >>> > > > wrote:
> > > > >>> > > >
> > > > >>> > > > > bump
> > > > >>> > > > >
> > > > >>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> 

[jira] [Created] (KAFKA-9793) Stream HandleAssignment should guarantee task close

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9793:
--

 Summary: Stream HandleAssignment should guarantee task close
 Key: KAFKA-9793
 URL: https://issues.apache.org/jira/browse/KAFKA-9793
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen
Assignee: Boyang Chen


When triggering the `handleAssignment` call, if task preCommit throws, the 
doom-to-fail task shall not be closed, thus causing a RocksDB metrics recorder 
re-addition, which is fatal:

 

 

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) [2020-03-31 
23:50:42,668] INFO 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] 
stream-thread 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] Handle 
new assignment with:

        New active tasks: [1_0, 0_1, 2_0]

        New standby tasks: []

        Existing active tasks: [0_1, 1_0, 2_0, 3_1]

        Existing standby tasks: [] 
(org.apache.kafka.streams.processor.internals.TaskManager)

 

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) [2020-03-31 
23:50:42,671] INFO 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] 
stream-thread 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] task 
[3_1] Prepared clean close 
(org.apache.kafka.streams.processor.internals.StreamTask)

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) [2020-03-31 
23:50:42,671] INFO 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] 
stream-thread 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] task 
[0_1] Prepared task for committing 
(org.apache.kafka.streams.processor.internals.StreamTask)

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) [2020-03-31 
23:50:42,682] ERROR 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] 
stream-thread 
[stream-soak-test-714fba71-3f5c-4418-8613-22d7b085949c-StreamThread-3] task 
[1_0] Failed to flush state store logData10MinuteFinalCount-store:  
(org.apache.kafka.streams.processor.internals.ProcessorStateManager)

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) 
org.apache.kafka.streams.errors.TaskMigratedException: Error encountered 
sending record to topic windowed-node-counts for task 1_0 due to:

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) 
org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an 
operation with an old epoch. Either there is a newer producer with the same 
transactionalId, or the producer's transaction has been expired by the broker.

[2020-03-31T16:50:43-07:00] 
(streams-soak-trunk-eos_soak_i-022f109d75764a250_streamslog) Written offsets 
would not be recorded and no more records would be sent since the producer is 
fenced, indicating the task may be migrated out; it means all tasks belonging 
to this thread should be migrated.

        at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:202)

        at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:185)

        at 
org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1352)

        at 
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:231)

        at 
org.apache.kafka.clients.producer.internals.ProducerBatch.abort(ProducerBatch.java:159)

        at 
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:768)

        at 
org.apache.kafka.clients.producer.internals.Sender.maybeAbortBatches(Sender.java:485)

        at 
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:304)

        at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)

        at java.lang.Thread.run(Thread.java:748)

 

The correct solution is to wrap the whole code block by try-catch to avoid 
unexpected close failure.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

[James Cheng]

I agree that we should prevent the creation of such a topic configuration. That 
would mean catching it at topic-creation time, as well as catching it on any 
topic-configuration changes that might make min.isr > replication factor.

Not sure how we would detect things if someone changed the broker-default 
configuration. That could be tricky.

Btw, I was the person who filed the original JIRA and as Mickael guessed, it 
was done by mistake.

-James

> On Mar 31, 2020, at 9:30 AM, Ismael Juma  wrote:
> 
> Hi Paolo,
> 
> Thanks for the KIP. Why would one want to set min.isr to be higher than
> replication factor even in that case? Mickael's suggestion seems better to
> me.
> 
> Ismael
> 
> On Fri, Mar 13, 2020 at 10:28 AM Paolo Moriello 
> wrote:
> 
>> Hi Mickael,
>> 
>> Thanks for your interest in this. The main motivation to NOT make topic
>> creation fail when this mismatch happens is because at the moment it is
>> possible to produce/consume on topics if acks is not set to all. I'm not
>> sure we want to disable this behavior (as we would by failing at topic
>> creation). That's why I decided to go for a softer approach, which at least
>> gives some more clarity to the users and avoids other issues mentioned in
>> the KIP.
>> 
>> Let's see what others think!
>> 
>> On Fri, 13 Mar 2020 at 17:16, Mickael Maison 
>> wrote:
>> 
>>> Hi Paolo,
>>> 
>>> Thanks for looking at this issue. This can indeed be a source of
>> confusion.
>>> 
>>> I'm wondering if we should prevent the creation of topics with
>>> min.insync.replicas > replication.factor?
>>> You listed that as a rejected alternative because it requires more
>>> changes. However, I can't think of any scenarios where a user would
>>> want to create such a topic. I'm guessing it's probably always by
>>> mistake.
>>> 
>>> Let's see what other people think but I think it's worth checking what
>>> needs to be done if we wanted to prevent topics with bogus configs
>>> 
>>> On Fri, Mar 13, 2020 at 3:28 PM Paolo Moriello
>>>  wrote:
 
 Hi,
 
 Following this Jira ticket (
>>> https://issues.apache.org/jira/browse/KAFKA-4680),
 I've created a proposal (
 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
>>> )
 to add a new exception/error to be used on min.insync.replicas >
 replication.factor.
 
 The proposal aims to introduce a new exception specific for the
 configuration mismatch above to be used when producers requires acks =
>>> all.
 At the moment we are using NotEnoughReplicaException, which is a
>>> retriable
 exception and is used to fail on insync replicas < min isr. Plan is to
>>> have
 a new, non-retriable exception, to separate the two cases.
 
 I've also submitted a PR for the change mentioned above:
 https://github.com/apache/kafka/pull/8225
 
 Please have a look and let me know what you think.
 
 Thanks,
 Paolo
>>> 
>> 

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Boyang Chen]

Thanks Kowshik, the answers are making sense. Some follow-ups:

On Tue, Mar 31, 2020 at 6:51 PM Jun Rao  wrote:

> Hi, Kowshik,
>
> Thanks for the KIP. Looks good overall. A few comments below.
>
> 100. UpdateFeaturesRequest/UpdateFeaturesResponse
> 100.1 Since this request waits for responses from brokers, should we add a
> timeout in the request (like createTopicRequest)?
> 100.2 The response schema is a bit weird. Typically, the response just
> shows an error code and an error message, instead of echoing the request.
> 100.3 Should we add a separate request to list/describe the existing
> features?
> 100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
> DELETE, the version field doesn't make sense. So, I guess the broker just
> ignores this? An alternative way is to have a separate
> DeleteFeaturesRequest
> 100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
> version of the metadata for finalized features." I am wondering why the
> ordering is important?
> 100.6 Could you specify the required ACL for this new request?
>
> 101. For the broker registration ZK node, should we bump up the version in
> the json?
>
> 102. For the /features ZK node, not sure if we need the epoch field. Each
> ZK node has an internal version field that is incremented on every update.
>
> 103. "Enabling the actual semantics of a feature version cluster-wide is
> left to the discretion of the logic implementing the feature (ex: can be
> done via dynamic broker config)." Does that mean the broker registration ZK
> node will be updated dynamically when this happens?
>
> 104. UpdateMetadataRequest
> 104.1 It would be useful to describe when the feature metadata is included
> in the request. My understanding is that it's only included if (1) there is
> a change to the finalized feature; (2) broker restart; (3) controller
> failover.
> 104.2 The new fields have the following versions. Why are the versions 3+
> when the top version is bumped to 6?
>   "fields":  [
> {"name": "Name", "type":  "string", "versions":  "3+",
>   "about": "The name of the feature."},
> {"name":  "Version", "type":  "int64", "versions":  "3+",
>   "about": "The finalized version for the feature."}
>   ]
>
> 105. kafka-features.sh: Instead of using update/delete, perhaps it's better
> to use enable/disable?
>
> Jun
>
> On Tue, Mar 31, 2020 at 5:29 PM Kowshik Prakasam 
> wrote:
>
> > Hey Boyang,
> >
> > Thanks for the great feedback! I have updated the KIP based on your
> > feedback.
> > Please find my response below for your comments, look for sentences
> > starting
> > with "(Kowshik)" below.
> >
> >
> > > 1. "When is it safe for the brokers to begin handling EOS traffic"
> could
> > be
> > > converted as "When is it safe for the brokers to start serving new
> > > Exactly-Once(EOS) semantics" since EOS is not explained earlier in the
> > > context.
> >
> > (Kowshik): Great point! Done.
> >

> > 2. In the *Explanation *section, the metadata version number part seems
> a
> > > bit blurred. Could you point a reference to later section that we going
> > to
> > > store it in Zookeeper and update it every time when there is a feature
> > > change?
> >
> > (Kowshik): Great point! Done. I've added a reference in the KIP.
> >
> >
> > > 3. For the feature downgrade, although it's a Non-goal of the KIP, for
> > > features such as group coordinator semantics, there is no legal
> scenario
> > to
> > > perform a downgrade at all. So having downgrade door open is pretty
> > > error-prone as human faults happen all the time. I'm assuming as new
> > > features are implemented, it's not very hard to add a flag during
> feature
> > > creation to indicate whether this feature is "downgradable". Could you
> > > explain a bit more on the extra engineering effort for shipping this
> KIP
> > > with downgrade protection in place?
> >
> > (Kowshik): Great point! I'd agree and disagree here. While I agree that
> > accidental
> > downgrades can cause problems, I also think sometimes downgrades should
> > be allowed for emergency reasons (not all downgrades cause issues).
> > It is just subjective to the feature being downgraded.
> >
> > To be more strict about feature version downgrades, I have modified the
> KIP
> > proposing that we mandate a `--force-downgrade` flag be used in the
> > UPDATE_FEATURES api
> > and the tooling, whenever the human is downgrading a finalized feature
> > version.
> > Hopefully this should cover the requirement, until we find the need for
> > advanced downgrade support.
> >
>
+1 for adding this flag.

> > > 4. "Each broker’s supported dictionary of feature versions will be
> > defined
> > > in the broker code." So this means in order to restrict a certain
> > feature,
> > > we need to start the broker first and then send a feature gating
> request
> > > immediately, which introduces a time gap and the intended-to-close
> > feature
> > > could actually serve request during this 

Build failed in Jenkins: kafka-2.5-jdk8 #85

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: more logs for empty assignment (#8397)


--
[...truncated 2.90 MB...]
org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task 

[jira] [Created] (KAFKA-9792) Improve sticky task assignment for previous active tasks

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-9792:
--

 Summary: Improve sticky task assignment for previous active tasks
 Key: KAFKA-9792
 URL: https://issues.apache.org/jira/browse/KAFKA-9792
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Sophie Blee-Goldman
Assignee: Sophie Blee-Goldman


Due to the way the client quota is computed, we may end up passing over a 
client that is not yet full. This can cause us to “miss” the client while 
attempting to assign active tasks to their previous owners, and ultimately 
forcing an active task _away_ from its previous client.

It’s not quite a bug, but it’s definitely sub-optimal behavior and can cause 
unnecessary task shuffling. It also makes the assignment harder to predict/debug

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1308

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9753: Add active tasks process ratio (#8370)

[github] MINOR: more logs for empty assignment (#8397)


--
[...truncated 2.97 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task 

Build failed in Jenkins: kafka-trunk-jdk8 #4388

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9753: Add active tasks process ratio (#8370)

[github] MINOR: more logs for empty assignment (#8397)


--
[...truncated 2.95 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task 

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Jun Rao]

Hi, Kowshik,

Thanks for the KIP. Looks good overall. A few comments below.

100. UpdateFeaturesRequest/UpdateFeaturesResponse
100.1 Since this request waits for responses from brokers, should we add a
timeout in the request (like createTopicRequest)?
100.2 The response schema is a bit weird. Typically, the response just
shows an error code and an error message, instead of echoing the request.
100.3 Should we add a separate request to list/describe the existing
features?
100.4 We are mixing ADD_OR_UPDATE and DELETE in a single request. For
DELETE, the version field doesn't make sense. So, I guess the broker just
ignores this? An alternative way is to have a separate DeleteFeaturesRequest
100.5 In UpdateFeaturesResponse, we have "The monotonically increasing
version of the metadata for finalized features." I am wondering why the
ordering is important?
100.6 Could you specify the required ACL for this new request?

101. For the broker registration ZK node, should we bump up the version in
the json?

102. For the /features ZK node, not sure if we need the epoch field. Each
ZK node has an internal version field that is incremented on every update.

103. "Enabling the actual semantics of a feature version cluster-wide is
left to the discretion of the logic implementing the feature (ex: can be
done via dynamic broker config)." Does that mean the broker registration ZK
node will be updated dynamically when this happens?

104. UpdateMetadataRequest
104.1 It would be useful to describe when the feature metadata is included
in the request. My understanding is that it's only included if (1) there is
a change to the finalized feature; (2) broker restart; (3) controller
failover.
104.2 The new fields have the following versions. Why are the versions 3+
when the top version is bumped to 6?
  "fields":  [
{"name": "Name", "type":  "string", "versions":  "3+",
  "about": "The name of the feature."},
{"name":  "Version", "type":  "int64", "versions":  "3+",
  "about": "The finalized version for the feature."}
  ]

105. kafka-features.sh: Instead of using update/delete, perhaps it's better
to use enable/disable?

Jun

On Tue, Mar 31, 2020 at 5:29 PM Kowshik Prakasam 
wrote:

> Hey Boyang,
>
> Thanks for the great feedback! I have updated the KIP based on your
> feedback.
> Please find my response below for your comments, look for sentences
> starting
> with "(Kowshik)" below.
>
>
> > 1. "When is it safe for the brokers to begin handling EOS traffic" could
> be
> > converted as "When is it safe for the brokers to start serving new
> > Exactly-Once(EOS) semantics" since EOS is not explained earlier in the
> > context.
>
> (Kowshik): Great point! Done.
>
> > 2. In the *Explanation *section, the metadata version number part seems a
> > bit blurred. Could you point a reference to later section that we going
> to
> > store it in Zookeeper and update it every time when there is a feature
> > change?
>
> (Kowshik): Great point! Done. I've added a reference in the KIP.
>
>
> > 3. For the feature downgrade, although it's a Non-goal of the KIP, for
> > features such as group coordinator semantics, there is no legal scenario
> to
> > perform a downgrade at all. So having downgrade door open is pretty
> > error-prone as human faults happen all the time. I'm assuming as new
> > features are implemented, it's not very hard to add a flag during feature
> > creation to indicate whether this feature is "downgradable". Could you
> > explain a bit more on the extra engineering effort for shipping this KIP
> > with downgrade protection in place?
>
> (Kowshik): Great point! I'd agree and disagree here. While I agree that
> accidental
> downgrades can cause problems, I also think sometimes downgrades should
> be allowed for emergency reasons (not all downgrades cause issues).
> It is just subjective to the feature being downgraded.
>
> To be more strict about feature version downgrades, I have modified the KIP
> proposing that we mandate a `--force-downgrade` flag be used in the
> UPDATE_FEATURES api
> and the tooling, whenever the human is downgrading a finalized feature
> version.
> Hopefully this should cover the requirement, until we find the need for
> advanced downgrade support.
>
> > 4. "Each broker’s supported dictionary of feature versions will be
> defined
> > in the broker code." So this means in order to restrict a certain
> feature,
> > we need to start the broker first and then send a feature gating request
> > immediately, which introduces a time gap and the intended-to-close
> feature
> > could actually serve request during this phase. Do you think we should
> also
> > support configurations as well so that admin user could freely roll up a
> > cluster with all nodes complying the same feature gating, without
> worrying
> > about the turnaround time to propagate the message only after the cluster
> > starts up?
>
> (Kowshik): This is a great point/question. One of the expectations out of
> 

Re: Working on a contribution for suppressing exceptions from KafkaConnect

[John Roesler]

Hi Connor,

Unfortunately, the wiki has a separate user list from the Jira project. You’ll 
have to create a user in the wiki and then let us know the I’d so we can give 
you edit permission (so you can create the KIP). 

Thanks!
-John


On Tue, Mar 31, 2020, at 13:38, Connor Penhale wrote:
> Hi Chris,
> 
> No problem! My customer understands they need to push their Kafka 
> release up to current ASAP, but they need to know which release they'd 
> be likely targeting to have this kind of feature. I'll go through the 
> KIP process and submit that. Should I use this as an opportunity now to 
> ask for permission to "Create KIP" in JIRA? My user ID is the email I 
> submitted the Improvement with.
> 
> Thanks again!
> Connor
> 
> On 3/31/20, 12:30 PM, "Christopher Egerton"  wrote:
> 
> Hi Connor,
> 
> Thanks for the contribution! It looks like the feature you've 
> implemented
> changes public interface, which means that a KIP would be required 
> in order
> to merge them into Kafka. You can find more information about what 
> kinds of
> changes require KIPs, what a KIP should consist of, and the process 
> for
> creating one at
> 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> .
> 
> Additionally, if the changes constitute a feature addition (which it
> appears yours do), they're generally not applicable for backporting, and
> should therefore be targeted against the latest branch of Kafka (i.e.,
> trunk) and will only be viable for inclusion in future releases. The
> earliest possible release that such changes could be included in would be
> 2.6, since the KIP deadline has already passed for the upcoming 2.5 
> release.
> 
> Hope this helps!
> 
> Cheers,
> 
> Chris
> 
> On Tue, Mar 31, 2020 at 11:12 AM Connor Penhale 
> wrote:
> 
> > Hi Everyone!
> >
> > I’ve submitted a contribution for suppressing detailed exception 
> messages
> > from Kafka Connect here: 
> https://issues.apache.org/jira/browse/KAFKA-9766,
> > with a pull request here: https://github.com/apache/kafka/pull/8355. I
> > was hoping to get some feedback on the code, and see if there’s 
> anything I
> > can do to help this feature become part of Kafka! We have a customer 
> that
> > really wants this feature, because their security team for PCI-DSS is
> > unhappy with how Kafka Connect handles things like malformed JSON, as an
> > example. Eager to work with everyone!
> >
> > Thanks!
> > Connor
> >
> > Connor Penhale | Enterprise Architect, OpenLogic 
> (https://openlogic.com/)
> > Perforce (https://www.perforce.com/)
> > Support: +1 866.399.6736
> >
> >
> >
> > This e-mail may contain information that is privileged or confidential. 
> If
> > you are not the intended recipient, please delete the e-mail and any
> > attachments and notify us immediately.
> >
> >
> 
> 
> CAUTION: This email originated from outside of the organization. Do 
> not click on links or open attachments unless you recognize the sender 
> and know the content is safe.
> 
> 
> 
> This e-mail may contain information that is privileged or confidential. 
> If you are not the intended recipient, please delete the e-mail and any 
> attachments and notify us immediately.
> 
>

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hey Boyang,

Thanks for the great feedback! I have updated the KIP based on your
feedback.
Please find my response below for your comments, look for sentences starting
with "(Kowshik)" below.


> 1. "When is it safe for the brokers to begin handling EOS traffic" could
be
> converted as "When is it safe for the brokers to start serving new
> Exactly-Once(EOS) semantics" since EOS is not explained earlier in the
> context.

(Kowshik): Great point! Done.

> 2. In the *Explanation *section, the metadata version number part seems a
> bit blurred. Could you point a reference to later section that we going to
> store it in Zookeeper and update it every time when there is a feature
> change?

(Kowshik): Great point! Done. I've added a reference in the KIP.


> 3. For the feature downgrade, although it's a Non-goal of the KIP, for
> features such as group coordinator semantics, there is no legal scenario
to
> perform a downgrade at all. So having downgrade door open is pretty
> error-prone as human faults happen all the time. I'm assuming as new
> features are implemented, it's not very hard to add a flag during feature
> creation to indicate whether this feature is "downgradable". Could you
> explain a bit more on the extra engineering effort for shipping this KIP
> with downgrade protection in place?

(Kowshik): Great point! I'd agree and disagree here. While I agree that
accidental
downgrades can cause problems, I also think sometimes downgrades should
be allowed for emergency reasons (not all downgrades cause issues).
It is just subjective to the feature being downgraded.

To be more strict about feature version downgrades, I have modified the KIP
proposing that we mandate a `--force-downgrade` flag be used in the
UPDATE_FEATURES api
and the tooling, whenever the human is downgrading a finalized feature
version.
Hopefully this should cover the requirement, until we find the need for
advanced downgrade support.

> 4. "Each broker’s supported dictionary of feature versions will be defined
> in the broker code." So this means in order to restrict a certain feature,
> we need to start the broker first and then send a feature gating request
> immediately, which introduces a time gap and the intended-to-close feature
> could actually serve request during this phase. Do you think we should
also
> support configurations as well so that admin user could freely roll up a
> cluster with all nodes complying the same feature gating, without worrying
> about the turnaround time to propagate the message only after the cluster
> starts up?

(Kowshik): This is a great point/question. One of the expectations out of
this KIP, which is
already followed in the broker, is the following.
 - Imagine at time T1 the broker starts up and registers it’s presence in
ZK,
   along with advertising it’s supported features.
 - Imagine at a future time T2 the broker receives the UpdateMetadataRequest
   from the controller, which contains the latest finalized features as
seen by
   the controller. The broker validates this data against it’s supported
features to
   make sure there is no mismatch (it will shutdown if there is an
incompatibility).

It is expected that during the time between the 2 events T1 and T2, the
broker is
almost a silent entity in the cluster. It does not add any value to the
cluster, or carry
out any important broker activities. By “important”, I mean it is not doing
mutations
on it’s persistence, not mutating critical in-memory state, won’t be serving
produce/fetch requests. Note it doesn’t even know it’s assigned partitions
until
it receives UpdateMetadataRequest from controller. Anything the broker is
doing up
until this point is not damaging/useful.

I’ve clarified the above in the KIP, see this new section:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioningschemeforfeatures-Incompatiblebrokerlifetime
.

> 5. "adding a new Feature, updating or deleting an existing Feature", may
be
> I misunderstood something, I thought the features are defined in broker
> code, so admin could not really create a new feature?

(Kowshik): Great point! You understood this right. Here adding a feature
means we are
adding a cluster-wide finalized *max* version for a feature that was
previously never finalized.
I have clarified this in the KIP now.

> 6. I think we need a separate error code like FEATURE_UPDATE_IN_PROGRESS
to
> reject a concurrent feature update request.

(Kowshik): Great point! I have modified the KIP adding the above (see
'Tooling support -> Admin API changes').

> 7. I think we haven't discussed the alternative solution to pass the
> feature information through Zookeeper. Is that mentioned in the KIP to
> justify why using UpdateMetadata is more favorable?

(Kowshik): Nice question! The broker reads finalized feature info stored in
ZK,
only during startup when it does a validation. When serving
`ApiVersionsRequest`, the
broker does not read this info from ZK 

[jira] [Resolved] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9783.
--
Fix Version/s: 2.6.0
   Resolution: Fixed

I think https://github.com/apache/kafka/pull/8370 should have fixed it.

> Flaky Test QueryableStateIntegrationTest#concurrentAccesses
> ---
>
> Key: KAFKA-9783
> URL: https://issues.apache.org/jira/browse/KAFKA-9783
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Affects Versions: 2.6.0
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>Priority: Critical
>  Labels: flaky-test
> Fix For: 2.6.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
> {quote}*02:17:54* 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
> concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not 
> receive all 48 records from topic output-concurrent-2 within 12 
> ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54* 
>  but: <0> was less than <48>*02:17:54* at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
>  at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
>  at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
>  at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-5842) QueryableStateIntegrationTest may fail with JDK 7

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5842.
--
Resolution: Fixed

Hopefully it has fixed via https://github.com/apache/kafka/pull/8370

> QueryableStateIntegrationTest may fail with JDK 7
> -
>
> Key: KAFKA-5842
> URL: https://issues.apache.org/jira/browse/KAFKA-5842
> Project: Kafka
>  Issue Type: Test
>  Components: unit tests
>Reporter: Ted Yu
>Priority: Minor
>
> Found the following when running test suite for 0.11.0.1 RC0 :
> {code}
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
> concurrentAccesses FAILED
> java.lang.AssertionError: Key not found one
> at org.junit.Assert.fail(Assert.java:88)
> at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.verifyGreaterOrEqual(QueryableStateIntegrationTest.java:893)
> at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:399)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9753) Add task-level active-process-ratio to Streams metrics

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9753.
--
Fix Version/s: 2.6.0
   Resolution: Fixed

> Add task-level active-process-ratio to Streams metrics
> --
>
> Key: KAFKA-9753
> URL: https://issues.apache.org/jira/browse/KAFKA-9753
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 2.6.0
>
>
> This is described as part of KIP-444 (which is mostly done in 2.4 / 2.5).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Kowshik Prakasam]

Hi Colin,

Thanks for the suggestions. It is a good idea to refer to the
'__data_version__' as just 'epoch', to avoid any confusions. However note
that this is not the same as broker epoch. The main distinction is that
this epoch is bumped by the controller whenever a modification made to the
finalized feature versions is persisted into ZK.

I have updated the KIP to use the new schema for the ‘/features’ ZK node:

   - We use 2 separate fields ‘epoch’ and ‘version’. The latter describing
   changes to the overall schema of the data that is written to ZooKeeper in
   the '/features' node.
   - We don’t have a header and a data section separately, I have clubbed
   these so that we have just 1 dictionary containing both.


Here is a link to the updated section:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-584

%3A+Versioning+scheme+for+features#KIP-584
:Versioningschemeforfeatures-Persistenceoffinalizedfeatureversions
.

Please feel free to let me know if you have any questions or concerns.


Cheers,
Kowshik


On Mon, Mar 30, 2020 at 4:53 PM Colin McCabe  wrote:

> On Thu, Mar 26, 2020, at 19:24, Kowshik Prakasam wrote:
> > Hi Colin,
> >
> > Thanks for the feedback! I've changed the KIP to address your
> > suggestions.
> > Please find below my explanation. Here is a link to KIP 584:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features
> > .
> >
> > 1. '__data_version__' is the version of the finalized feature metadata
> > (i.e. actual ZK node contents), while the '__schema_version__' is the
> > version of the schema of the data persisted in ZK. These serve different
> > purposes. '__data_version__' is is useful mainly to clients during reads,
> > to differentiate between the 2 versions of eventually consistent
> 'finalized
> > features' metadata (i.e. larger metadata version is more recent).
> > '__schema_version__' provides an additional degree of flexibility, where
> if
> > we decide to change the schema for '/features' node in ZK (in the
> future),
> > then we can manage broker roll outs suitably (i.e.
> > serialization/deserialization of the ZK data can be handled safely).
>
> Hi Kowshik,
>
> If you're talking about a number that lets you know if data is more or
> less recent, we would typically call that an epoch, and not a version.  For
> the ZK data structures, the word "version" is typically reserved for
> describing changes to the overall schema of the data that is written to
> ZooKeeper.  We don't even really change the "version" of those schemas that
> much, since most changes are backwards-compatible.  But we do include that
> version field just in case.
>
> I don't think we really need an epoch here, though, since we can just look
> at the broker epoch.  Whenever the broker registers, its epoch will be
> greater than the previous broker epoch.  And the newly registered data will
> take priority.  This will be a lot simpler than adding a separate epoch
> system, I think.
>
> >
> > 2. Regarding admin client needing min and max information - you are
> right!
> > I've changed the KIP such that the Admin API also allows the user to read
> > 'supported features' from a specific broker. Please look at the section
> > "Admin API changes".
>
> Thanks.
>
> >
> > 3. Regarding the use of `long` vs `Long` - it was not deliberate. I've
> > improved the KIP to just use `long` at all places.
>
> Sounds good.
>
> >
> > 4. Regarding kafka.admin.FeatureCommand tool - you are right! I've
> updated
> > the KIP sketching the functionality provided by this tool, with some
> > examples. Please look at the section "Tooling support examples".
> >
> > Thank you!
>
>
> Thanks, Kowshik.
>
> cheers,
> Colin
>
> >
> >
> > Cheers,
> > Kowshik
> >
> > On Wed, Mar 25, 2020 at 11:31 PM Colin McCabe 
> wrote:
> >
> > > Thanks, Kowshik, this looks good.
> > >
> > > In the "Schema" section, do we really need both __schema_version__ and
> > > __data_version__?  Can we just have a single version field here?
> > >
> > > Shouldn't the Admin(Client) function have some way to get the min and
> max
> > > information that we're exposing as well?  I guess we could have min,
> max,
> > > and current.  Unrelated: is the use of Long rather than long deliberate
> > > here?
> > >
> > > It would be good to describe how the command line tool
> > > kafka.admin.FeatureCommand will work.  For example the flags that it
> will
> > > take and the output that it will generate to STDOUT.
> > >
> > > cheers,
> > > Colin
> > >
> > >
> > > On Tue, Mar 24, 2020, at 17:08, Kowshik Prakasam wrote:
> > > > Hi all,
> > > >
> > > > I've opened KIP-584 
> 
> > > > which
> > > > is intended to provide a versioning scheme for features. I'd like to
> use
> > > > this thread to discuss the same. I'd appreciate any feedback on 

[jira] [Created] (KAFKA-9791) Streams should log the effective RocksDB configuration when opening a database

[John Roesler (Jira)]

John Roesler created KAFKA-9791:
---

 Summary: Streams should log the effective RocksDB configuration 
when opening a database
 Key: KAFKA-9791
 URL: https://issues.apache.org/jira/browse/KAFKA-9791
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: John Roesler


RocksDB has quite a few configuration options. Streams sets some and offers an 
API for users to set more, so it can be difficult to determine in an 
operational scenario what the used configuration actually is. It would save 
quite a bit of time if the RocksDB stores would actually log the configuration 
they use when initializing the database, although logging at INFO level might 
be too verbose, as there can be large numbers of such databases in a topology.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Request permission to "Create KIP" for user: showuon

[Matthias J. Sax]

Done.

On 3/31/20 7:08 AM, Luke Chen wrote:
> Hi Devs,
> I’d like to create a KIP and need the access to the wiki page. Please help.
> My wiki ID: showuon.
> 
> Thank you.
> Luke
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-584: Versioning scheme for features

[Boyang Chen]

Hey Kowshik,

thanks for the revised KIP. Got a couple of questions:

1. "When is it safe for the brokers to begin handling EOS traffic" could be
converted as "When is it safe for the brokers to start serving new
Exactly-Once(EOS) semantics" since EOS is not explained earlier in the
context.

2. In the *Explanation *section, the metadata version number part seems a
bit blurred. Could you point a reference to later section that we going to
store it in Zookeeper and update it every time when there is a feature
change?

3. For the feature downgrade, although it's a Non-goal of the KIP, for
features such as group coordinator semantics, there is no legal scenario to
perform a downgrade at all. So having downgrade door open is pretty
error-prone as human faults happen all the time. I'm assuming as new
features are implemented, it's not very hard to add a flag during feature
creation to indicate whether this feature is "downgradable". Could you
explain a bit more on the extra engineering effort for shipping this KIP
with downgrade protection in place?

4. "Each broker’s supported dictionary of feature versions will be defined
in the broker code." So this means in order to restrict a certain feature,
we need to start the broker first and then send a feature gating request
immediately, which introduces a time gap and the intended-to-close feature
could actually serve request during this phase. Do you think we should also
support configurations as well so that admin user could freely roll up a
cluster with all nodes complying the same feature gating, without worrying
about the turnaround time to propagate the message only after the cluster
starts up?

5. "adding a new Feature, updating or deleting an existing Feature", may be
I misunderstood something, I thought the features are defined in broker
code, so admin could not really create a new feature?

6. I think we need a separate error code like FEATURE_UPDATE_IN_PROGRESS to
reject a concurrent feature update request.

7. I think we haven't discussed the alternative solution to pass the
feature information through Zookeeper. Is that mentioned in the KIP to
justify why using UpdateMetadata is more favorable?

8. I was under the impression that user could configure a range of
supported versions, what's the trade-off for allowing single finalized
version only?

9. One minor syntax fix: Note that here the "client" here may be a producer

Boyang

On Mon, Mar 30, 2020 at 4:53 PM Colin McCabe  wrote:

> On Thu, Mar 26, 2020, at 19:24, Kowshik Prakasam wrote:
> > Hi Colin,
> >
> > Thanks for the feedback! I've changed the KIP to address your
> > suggestions.
> > Please find below my explanation. Here is a link to KIP 584:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features
> > .
> >
> > 1. '__data_version__' is the version of the finalized feature metadata
> > (i.e. actual ZK node contents), while the '__schema_version__' is the
> > version of the schema of the data persisted in ZK. These serve different
> > purposes. '__data_version__' is is useful mainly to clients during reads,
> > to differentiate between the 2 versions of eventually consistent
> 'finalized
> > features' metadata (i.e. larger metadata version is more recent).
> > '__schema_version__' provides an additional degree of flexibility, where
> if
> > we decide to change the schema for '/features' node in ZK (in the
> future),
> > then we can manage broker roll outs suitably (i.e.
> > serialization/deserialization of the ZK data can be handled safely).
>
> Hi Kowshik,
>
> If you're talking about a number that lets you know if data is more or
> less recent, we would typically call that an epoch, and not a version.  For
> the ZK data structures, the word "version" is typically reserved for
> describing changes to the overall schema of the data that is written to
> ZooKeeper.  We don't even really change the "version" of those schemas that
> much, since most changes are backwards-compatible.  But we do include that
> version field just in case.
>
> I don't think we really need an epoch here, though, since we can just look
> at the broker epoch.  Whenever the broker registers, its epoch will be
> greater than the previous broker epoch.  And the newly registered data will
> take priority.  This will be a lot simpler than adding a separate epoch
> system, I think.
>
> >
> > 2. Regarding admin client needing min and max information - you are
> right!
> > I've changed the KIP such that the Admin API also allows the user to read
> > 'supported features' from a specific broker. Please look at the section
> > "Admin API changes".
>
> Thanks.
>
> >
> > 3. Regarding the use of `long` vs `Long` - it was not deliberate. I've
> > improved the KIP to just use `long` at all places.
>
> Sounds good.
>
> >
> > 4. Regarding kafka.admin.FeatureCommand tool - you are right! I've
> updated
> > the KIP sketching the functionality provided by this tool, with some
> > examples. Please look 

Re: Working on a contribution for suppressing exceptions from KafkaConnect

[Connor Penhale]

Hi Chris,

No problem! My customer understands they need to push their Kafka release up to 
current ASAP, but they need to know which release they'd be likely targeting to 
have this kind of feature. I'll go through the KIP process and submit that. 
Should I use this as an opportunity now to ask for permission to "Create KIP" 
in JIRA? My user ID is the email I submitted the Improvement with.

Thanks again!
Connor

On 3/31/20, 12:30 PM, "Christopher Egerton"  wrote:

Hi Connor,

Thanks for the contribution! It looks like the feature you've implemented
changes public interface, which means that a KIP would be required in order
to merge them into Kafka. You can find more information about what kinds of
changes require KIPs, what a KIP should consist of, and the process for
creating one at

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
.

Additionally, if the changes constitute a feature addition (which it
appears yours do), they're generally not applicable for backporting, and
should therefore be targeted against the latest branch of Kafka (i.e.,
trunk) and will only be viable for inclusion in future releases. The
earliest possible release that such changes could be included in would be
2.6, since the KIP deadline has already passed for the upcoming 2.5 release.

Hope this helps!

Cheers,

Chris

On Tue, Mar 31, 2020 at 11:12 AM Connor Penhale 
wrote:

> Hi Everyone!
>
> I’ve submitted a contribution for suppressing detailed exception messages
> from Kafka Connect here: https://issues.apache.org/jira/browse/KAFKA-9766,
> with a pull request here: https://github.com/apache/kafka/pull/8355. I
> was hoping to get some feedback on the code, and see if there’s anything I
> can do to help this feature become part of Kafka! We have a customer that
> really wants this feature, because their security team for PCI-DSS is
> unhappy with how Kafka Connect handles things like malformed JSON, as an
> example. Eager to work with everyone!
>
> Thanks!
> Connor
>
> Connor Penhale | Enterprise Architect, OpenLogic (https://openlogic.com/)
> Perforce (https://www.perforce.com/)
> Support: +1 866.399.6736
>
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>


CAUTION: This email originated from outside of the organization. Do not 
click on links or open attachments unless you recognize the sender and know the 
content is safe.



This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Build failed in Jenkins: kafka-trunk-jdk11 #1307

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix MockAdminClient to not throw IndexOutOfBoundsException when


--
[...truncated 2.97 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

Re: Working on a contribution for suppressing exceptions from KafkaConnect

[Christopher Egerton]

Hi Connor,

Thanks for the contribution! It looks like the feature you've implemented
changes public interface, which means that a KIP would be required in order
to merge them into Kafka. You can find more information about what kinds of
changes require KIPs, what a KIP should consist of, and the process for
creating one at
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
.

Additionally, if the changes constitute a feature addition (which it
appears yours do), they're generally not applicable for backporting, and
should therefore be targeted against the latest branch of Kafka (i.e.,
trunk) and will only be viable for inclusion in future releases. The
earliest possible release that such changes could be included in would be
2.6, since the KIP deadline has already passed for the upcoming 2.5 release.

Hope this helps!

Cheers,

Chris

On Tue, Mar 31, 2020 at 11:12 AM Connor Penhale 
wrote:

> Hi Everyone!
>
> I’ve submitted a contribution for suppressing detailed exception messages
> from Kafka Connect here: https://issues.apache.org/jira/browse/KAFKA-9766,
> with a pull request here: https://github.com/apache/kafka/pull/8355. I
> was hoping to get some feedback on the code, and see if there’s anything I
> can do to help this feature become part of Kafka! We have a customer that
> really wants this feature, because their security team for PCI-DSS is
> unhappy with how Kafka Connect handles things like malformed JSON, as an
> example. Eager to work with everyone!
>
> Thanks!
> Connor
>
> Connor Penhale | Enterprise Architect, OpenLogic (https://openlogic.com/)
> Perforce (https://www.perforce.com/)
> Support: +1 866.399.6736
>
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>

Working on a contribution for suppressing exceptions from KafkaConnect

[Connor Penhale]

Hi Everyone!

I’ve submitted a contribution for suppressing detailed exception messages from 
Kafka Connect here: https://issues.apache.org/jira/browse/KAFKA-9766, with a 
pull request here: https://github.com/apache/kafka/pull/8355. I was hoping to 
get some feedback on the code, and see if there’s anything I can do to help 
this feature become part of Kafka! We have a customer that really wants this 
feature, because their security team for PCI-DSS is unhappy with how Kafka 
Connect handles things like malformed JSON, as an example. Eager to work with 
everyone!

Thanks!
Connor

Connor Penhale | Enterprise Architect, OpenLogic (https://openlogic.com/)
Perforce (https://www.perforce.com/)
Support: +1 866.399.6736



This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Build failed in Jenkins: kafka-trunk-jdk8 #4387

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix MockAdminClient to not throw IndexOutOfBoundsException when


--
[...truncated 2.95 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain 

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Jun Rao]

Hi, Rajini, Maulin,

1. Ok. Then we can keep the package name as it is.

2. Thanks for updating the javadoc for shouldBeRebuilt(). Could you also
clarify after the SslEngine is rebuilt, what happens to existing SSL
connections?

Thanks,

Jun

On Tue, Mar 31, 2020 at 2:07 AM Rajini Sivaram 
wrote:

> Hi Jun, Maulin,
>
>  org.apache.kafka.common.security.ssl contains internal classes like
> SslFactory.  org.apache.kafka.common.security.auth is a public package
> which contains all our current authentication-related classes. If we want
> to move the new interface into an SSL-specific package, we should perhaps
> create a new public package rather than use an existing internal one?
>
> On Tue, Mar 31, 2020 at 7:56 AM Manikumar 
> wrote:
>
> > +1 (binding).
> > Thanks for the KIP.
> >
> > Thanks,
> > Manikumar
> >
> > On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > So far we got 3 Binding votes. I am planning to keep the voting phase
> > open
> > > until Tuesday 10 PM Pacific Time which will be more than 72 hours from
> > the
> > > first binding vote on Thursday 12:36 PM Pacific Time.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I updated the Javadoc in the KIP details and the actual
> > SslEngineFactory
> > > > interface for shouldBeRebuilt(). For the first comment, probably I'll
> > try
> > > > to address it tomorrow.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > >> Thanks Jun Rao for your vote and comments.
> > > >>
> > > >> For 1) Earlier it was the security.ssl package but after a review I
> > > >> changed it to .auth since there are some public interfaces in that
> > > package.
> > > >> I am open to move it under .ssl package.
> > > >>
> > > >> For 2) Sure. Will document in Javadocs for the method.
> > > >>
> > > >> Thanks
> > > >> Maulin
> > > >>
> > > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> > > >>
> > > >>> Hi, Maulin,
> > > >>>
> > > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
> > below.
> > > >>>
> > > >>> 1. Should the package name of the new
> > > >>> interface SslEngineFactory be org.apache.kafka.common.security.ssl
> > > >>> instead
> > > >>> of org.apache.kafka.common.security.auth?
> > > >>> 2. Could you document when shouldBeRebuilt() will be called?
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> > > >>> maulin.vasav...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding
> > vote
> > > >>> so
> > > >>> > far, can you please vote so that we can try to move forward with
> > > >>> changes?
> > > >>> >
> > > >>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas
> > >  > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > +1 (non-binding)
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > > Thomas
> > > >>> > >
> > > >>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" <
> rajinisiva...@gmail.com
> > >
> > > >>> wrote:
> > > >>> > >
> > > >>> > > +1 (binding)
> > > >>> > > Thanks for the KIP, Maulin!
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > >
> > > >>> > > Rajini
> > > >>> > >
> > > >>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> > > >>> > > maulin.vasav...@gmail.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > > > FYI - we have updated the KIP documentation also with
> > > >>> appropriate
> > > >>> > > code
> > > >>> > > > samples for interfaces and few important changes.
> > > >>> > > >
> > > >>> > > > Thanks
> > > >>> > > > Maulin
> > > >>> > > >
> > > >>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > > >>> > > > maulin.vasav...@gmail.com>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > bump
> > > >>> > > > >
> > > >>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> > > >>> > > > > maulin.vasav...@gmail.com> wrote:
> > > >>> > > > >
> > > >>> > > > >> Hi all
> > > >>> > > > >>
> > > >>> > > > >> After much await on the approach conclusion we have a
> PR
> > > >>> > > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> > > >>> > > .
> > > >>> > > > >>
> > > >>> > > > >> Can you please provide your vote so that we can more
> > this
> > > >>> > forward?
> > > >>> > > > >>
> > > >>> > > > >> Thanks
> > > >>> > > > >> Maulin
> > > >>> > > > >>
> > > >>> > > > >> On Sun, Jan 26, 

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Renuka M]

+1 (non-binding)

Thanks
Renuka M

On Tue, Mar 31, 2020 at 2:07 AM Rajini Sivaram 
wrote:

> Hi Jun, Maulin,
>
>  org.apache.kafka.common.security.ssl contains internal classes like
> SslFactory.  org.apache.kafka.common.security.auth is a public package
> which contains all our current authentication-related classes. If we want
> to move the new interface into an SSL-specific package, we should perhaps
> create a new public package rather than use an existing internal one?
>
> On Tue, Mar 31, 2020 at 7:56 AM Manikumar 
> wrote:
>
> > +1 (binding).
> > Thanks for the KIP.
> >
> > Thanks,
> > Manikumar
> >
> > On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > So far we got 3 Binding votes. I am planning to keep the voting phase
> > open
> > > until Tuesday 10 PM Pacific Time which will be more than 72 hours from
> > the
> > > first binding vote on Thursday 12:36 PM Pacific Time.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I updated the Javadoc in the KIP details and the actual
> > SslEngineFactory
> > > > interface for shouldBeRebuilt(). For the first comment, probably I'll
> > try
> > > > to address it tomorrow.
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > > wrote:
> > > >
> > > >> Thanks Jun Rao for your vote and comments.
> > > >>
> > > >> For 1) Earlier it was the security.ssl package but after a review I
> > > >> changed it to .auth since there are some public interfaces in that
> > > package.
> > > >> I am open to move it under .ssl package.
> > > >>
> > > >> For 2) Sure. Will document in Javadocs for the method.
> > > >>
> > > >> Thanks
> > > >> Maulin
> > > >>
> > > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> > > >>
> > > >>> Hi, Maulin,
> > > >>>
> > > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
> > below.
> > > >>>
> > > >>> 1. Should the package name of the new
> > > >>> interface SslEngineFactory be org.apache.kafka.common.security.ssl
> > > >>> instead
> > > >>> of org.apache.kafka.common.security.auth?
> > > >>> 2. Could you document when shouldBeRebuilt() will be called?
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> > > >>> maulin.vasav...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding
> > vote
> > > >>> so
> > > >>> > far, can you please vote so that we can try to move forward with
> > > >>> changes?
> > > >>> >
> > > >>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas
> > >  > > >>> >
> > > >>> > wrote:
> > > >>> >
> > > >>> > > +1 (non-binding)
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > > Thomas
> > > >>> > >
> > > >>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" <
> rajinisiva...@gmail.com
> > >
> > > >>> wrote:
> > > >>> > >
> > > >>> > > +1 (binding)
> > > >>> > > Thanks for the KIP, Maulin!
> > > >>> > >
> > > >>> > > Regards,
> > > >>> > >
> > > >>> > > Rajini
> > > >>> > >
> > > >>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> > > >>> > > maulin.vasav...@gmail.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > > > FYI - we have updated the KIP documentation also with
> > > >>> appropriate
> > > >>> > > code
> > > >>> > > > samples for interfaces and few important changes.
> > > >>> > > >
> > > >>> > > > Thanks
> > > >>> > > > Maulin
> > > >>> > > >
> > > >>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > > >>> > > > maulin.vasav...@gmail.com>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > bump
> > > >>> > > > >
> > > >>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> > > >>> > > > > maulin.vasav...@gmail.com> wrote:
> > > >>> > > > >
> > > >>> > > > >> Hi all
> > > >>> > > > >>
> > > >>> > > > >> After much await on the approach conclusion we have a
> PR
> > > >>> > > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> > > >>> > > .
> > > >>> > > > >>
> > > >>> > > > >> Can you please provide your vote so that we can more
> > this
> > > >>> > forward?
> > > >>> > > > >>
> > > >>> > > > >> Thanks
> > > >>> > > > >> Maulin
> > > >>> > > > >>
> > > >>> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> > > >>> > > > >> maulin.vasav...@gmail.com> wrote:
> > > >>> > > > >>
> > > >>> > > > >>> Hi all
> > > >>> > > > >>>
> > > >>> > > > >>> After a good 

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[Ismael Juma]

I commented in the discussion thread. I think the rejected alternative of
validating during topic creation seems better.

On Tue, Mar 31, 2020 at 6:17 AM Paolo Moriello 
wrote:

> Hello,
>
> Thanks to everybody who has given feedback. I've incorporated the
> suggestions and think that this is now ready for a vote.
>
> KIP 579:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
>
> PR:
> https://github.com/apache/kafka/pull/8225
>
> Thanks,
> Paolo
>

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

[Ismael Juma]

Hi Paolo,

Thanks for the KIP. Why would one want to set min.isr to be higher than
replication factor even in that case? Mickael's suggestion seems better to
me.

Ismael

On Fri, Mar 13, 2020 at 10:28 AM Paolo Moriello 
wrote:

> Hi Mickael,
>
> Thanks for your interest in this. The main motivation to NOT make topic
> creation fail when this mismatch happens is because at the moment it is
> possible to produce/consume on topics if acks is not set to all. I'm not
> sure we want to disable this behavior (as we would by failing at topic
> creation). That's why I decided to go for a softer approach, which at least
> gives some more clarity to the users and avoids other issues mentioned in
> the KIP.
>
> Let's see what others think!
>
> On Fri, 13 Mar 2020 at 17:16, Mickael Maison 
> wrote:
>
> > Hi Paolo,
> >
> > Thanks for looking at this issue. This can indeed be a source of
> confusion.
> >
> > I'm wondering if we should prevent the creation of topics with
> > min.insync.replicas > replication.factor?
> > You listed that as a rejected alternative because it requires more
> > changes. However, I can't think of any scenarios where a user would
> > want to create such a topic. I'm guessing it's probably always by
> > mistake.
> >
> > Let's see what other people think but I think it's worth checking what
> > needs to be done if we wanted to prevent topics with bogus configs
> >
> > On Fri, Mar 13, 2020 at 3:28 PM Paolo Moriello
> >  wrote:
> > >
> > > Hi,
> > >
> > > Following this Jira ticket (
> > https://issues.apache.org/jira/browse/KAFKA-4680),
> > > I've created a proposal (
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
> > )
> > > to add a new exception/error to be used on min.insync.replicas >
> > > replication.factor.
> > >
> > > The proposal aims to introduce a new exception specific for the
> > > configuration mismatch above to be used when producers requires acks =
> > all.
> > > At the moment we are using NotEnoughReplicaException, which is a
> > retriable
> > > exception and is used to fail on insync replicas < min isr. Plan is to
> > have
> > > a new, non-retriable exception, to separate the two cases.
> > >
> > > I've also submitted a PR for the change mentioned above:
> > > https://github.com/apache/kafka/pull/8225
> > >
> > > Please have a look and let me know what you think.
> > >
> > > Thanks,
> > > Paolo
> >
>

Request permission to "Create KIP" for user: showuon

[Luke Chen]

Hi Devs,
I’d like to create a KIP and need the access to the wiki page. Please help.
My wiki ID: showuon.

Thank you.
Luke

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[Gokul Ramanan Subramanian]

+1 (non binding).

Thanks.

On Tue, Mar 31, 2020 at 2:43 PM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> +1 (non binding).
>
> Thanks.
>
> Le mar. 31 mars 2020 à 14:24, M. Manna  a écrit :
> >
> > +1 (binding).
> >
> > Thanks for the KIP.
> >
> > On Tue, 31 Mar 2020 at 14:17, Paolo Moriello 
> > wrote:
> >
> > > Hello,
> > >
> > > Thanks to everybody who has given feedback. I've incorporated the
> > > suggestions and think that this is now ready for a vote.
> > >
> > > KIP 579:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
> > >
> > > PR:
> > > https://github.com/apache/kafka/pull/8225
> > >
> > > Thanks,
> > > Paolo
> > >
>

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[Alexandre Dupriez]

+1 (non binding).

Thanks.

Le mar. 31 mars 2020 à 14:24, M. Manna  a écrit :
>
> +1 (binding).
>
> Thanks for the KIP.
>
> On Tue, 31 Mar 2020 at 14:17, Paolo Moriello 
> wrote:
>
> > Hello,
> >
> > Thanks to everybody who has given feedback. I've incorporated the
> > suggestions and think that this is now ready for a vote.
> >
> > KIP 579:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
> >
> > PR:
> > https://github.com/apache/kafka/pull/8225
> >
> > Thanks,
> > Paolo
> >

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[M. Manna]

+1 (binding).

Thanks for the KIP.

On Tue, 31 Mar 2020 at 14:17, Paolo Moriello 
wrote:

> Hello,
>
> Thanks to everybody who has given feedback. I've incorporated the
> suggestions and think that this is now ready for a vote.
>
> KIP 579:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
>
> PR:
> https://github.com/apache/kafka/pull/8225
>
> Thanks,
> Paolo
>

[VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[Paolo Moriello]

Hello,

Thanks to everybody who has given feedback. I've incorporated the
suggestions and think that this is now ready for a vote.

KIP 579:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor

PR:
https://github.com/apache/kafka/pull/8225

Thanks,
Paolo

[jira] [Created] (KAFKA-9790) Wrong metric bean name for connect worker metrics

[Alexandre YANG (Jira)]

Alexandre YANG created KAFKA-9790:
-

 Summary: Wrong metric bean name for connect worker metrics
 Key: KAFKA-9790
 URL: https://issues.apache.org/jira/browse/KAFKA-9790
 Project: Kafka
  Issue Type: Bug
 Environment: Tested with this version of connect:  
cnfldemos/cp-server-connect-datagen:0.2.0-5.4.0
Reporter: Alexandre YANG
 Attachments: image-2020-03-31-15-11-51-031.png

The connect worker metrics implemented here: 
 * [https://github.com/apache/kafka/pull/6843/files]
 * 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metrics+to+Measure+Number+of+Tasks+on+a+Connector]

should have this bean name:

 
{code:java}
kafka.connect:type=connect-worker-metrics,connector="{connector}"
{code}
but it's currently (see screenshot)

 
{code:java}
kafka.connect:type=connector-metrics,connector=datagen2
{code}
 

 

 

!image-2020-03-31-15-11-51-031.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9789) if one machine has four broker, i want three is cluster ,how do ?

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-9789.
-
Resolution: Not A Bug

Hi [~startjava], 

this jira is to track bugs related to Kafka, for questions on how to use it, 
please post to the [Kafka Users mailing 
list|https://lists.apache.org/list.html?us...@kafka.apache.org]

> if one machine has four broker, i want three is cluster ,how do ?
> -
>
> Key: KAFKA-9789
> URL: https://issues.apache.org/jira/browse/KAFKA-9789
> Project: Kafka
>  Issue Type: Test
>Reporter: startjava
>Priority: Major
>
> if  me has one machine has four broker, i want three broker is cluster ,how 
> do ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Rajini Sivaram]

Hi Jun, Maulin,

 org.apache.kafka.common.security.ssl contains internal classes like
SslFactory.  org.apache.kafka.common.security.auth is a public package
which contains all our current authentication-related classes. If we want
to move the new interface into an SSL-specific package, we should perhaps
create a new public package rather than use an existing internal one?

On Tue, Mar 31, 2020 at 7:56 AM Manikumar  wrote:

> +1 (binding).
> Thanks for the KIP.
>
> Thanks,
> Manikumar
>
> On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > So far we got 3 Binding votes. I am planning to keep the voting phase
> open
> > until Tuesday 10 PM Pacific Time which will be more than 72 hours from
> the
> > first binding vote on Thursday 12:36 PM Pacific Time.
> >
> > Thanks
> > Maulin
> >
> > On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I updated the Javadoc in the KIP details and the actual
> SslEngineFactory
> > > interface for shouldBeRebuilt(). For the first comment, probably I'll
> try
> > > to address it tomorrow.
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > >> Thanks Jun Rao for your vote and comments.
> > >>
> > >> For 1) Earlier it was the security.ssl package but after a review I
> > >> changed it to .auth since there are some public interfaces in that
> > package.
> > >> I am open to move it under .ssl package.
> > >>
> > >> For 2) Sure. Will document in Javadocs for the method.
> > >>
> > >> Thanks
> > >> Maulin
> > >>
> > >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> > >>
> > >>> Hi, Maulin,
> > >>>
> > >>> Thanks for the KIP. +1 from me. Just a couple of minor comments
> below.
> > >>>
> > >>> 1. Should the package name of the new
> > >>> interface SslEngineFactory be org.apache.kafka.common.security.ssl
> > >>> instead
> > >>> of org.apache.kafka.common.security.auth?
> > >>> 2. Could you document when shouldBeRebuilt() will be called?
> > >>>
> > >>> Jun
> > >>>
> > >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> > >>> maulin.vasav...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding
> vote
> > >>> so
> > >>> > far, can you please vote so that we can try to move forward with
> > >>> changes?
> > >>> >
> > >>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas
> >  > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > +1 (non-binding)
> > >>> > >
> > >>> > > Regards,
> > >>> > > Thomas
> > >>> > >
> > >>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram"  >
> > >>> wrote:
> > >>> > >
> > >>> > > +1 (binding)
> > >>> > > Thanks for the KIP, Maulin!
> > >>> > >
> > >>> > > Regards,
> > >>> > >
> > >>> > > Rajini
> > >>> > >
> > >>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> > >>> > > maulin.vasav...@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > FYI - we have updated the KIP documentation also with
> > >>> appropriate
> > >>> > > code
> > >>> > > > samples for interfaces and few important changes.
> > >>> > > >
> > >>> > > > Thanks
> > >>> > > > Maulin
> > >>> > > >
> > >>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> > >>> > > > maulin.vasav...@gmail.com>
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > bump
> > >>> > > > >
> > >>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> > >>> > > > > maulin.vasav...@gmail.com> wrote:
> > >>> > > > >
> > >>> > > > >> Hi all
> > >>> > > > >>
> > >>> > > > >> After much await on the approach conclusion we have a PR
> > >>> > > > >>
> > >>> > >
> > >>> >
> > >>>
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> > >>> > > .
> > >>> > > > >>
> > >>> > > > >> Can you please provide your vote so that we can more
> this
> > >>> > forward?
> > >>> > > > >>
> > >>> > > > >> Thanks
> > >>> > > > >> Maulin
> > >>> > > > >>
> > >>> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> > >>> > > > >> maulin.vasav...@gmail.com> wrote:
> > >>> > > > >>
> > >>> > > > >>> Hi all
> > >>> > > > >>>
> > >>> > > > >>> After a good discussion on the KIP at
> > >>> > > > >>>
> > >>> > >
> > >>> >
> > >>>
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
> > >>> > > I
> > >>> > > > >>> think we are 

Build failed in Jenkins: kafka-2.5-jdk8 #84

[Apache Jenkins Server]

See 


Changes:

[konstantine] KAFKA-9706: Handle null in keys or values when Flatten 
transformation is


--
[...truncated 5.87 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = 

Re: [VOTE] KIP-519: Make SSL context/engine configuration extensible

[Manikumar]

+1 (binding).
Thanks for the KIP.

Thanks,
Manikumar

On Tue, Mar 31, 2020 at 11:24 AM Maulin Vasavada 
wrote:

> Hi all,
>
> So far we got 3 Binding votes. I am planning to keep the voting phase open
> until Tuesday 10 PM Pacific Time which will be more than 72 hours from the
> first binding vote on Thursday 12:36 PM Pacific Time.
>
> Thanks
> Maulin
>
> On Mon, Mar 30, 2020 at 10:32 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I updated the Javadoc in the KIP details and the actual SslEngineFactory
> > interface for shouldBeRebuilt(). For the first comment, probably I'll try
> > to address it tomorrow.
> >
> > Thanks
> > Maulin
> >
> > On Mon, Mar 30, 2020 at 7:44 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> >> Thanks Jun Rao for your vote and comments.
> >>
> >> For 1) Earlier it was the security.ssl package but after a review I
> >> changed it to .auth since there are some public interfaces in that
> package.
> >> I am open to move it under .ssl package.
> >>
> >> For 2) Sure. Will document in Javadocs for the method.
> >>
> >> Thanks
> >> Maulin
> >>
> >> On Mon, Mar 30, 2020 at 5:46 PM Jun Rao  wrote:
> >>
> >>> Hi, Maulin,
> >>>
> >>> Thanks for the KIP. +1 from me. Just a couple of minor comments below.
> >>>
> >>> 1. Should the package name of the new
> >>> interface SslEngineFactory be org.apache.kafka.common.security.ssl
> >>> instead
> >>> of org.apache.kafka.common.security.auth?
> >>> 2. Could you document when shouldBeRebuilt() will be called?
> >>>
> >>> Jun
> >>>
> >>> On Mon, Mar 30, 2020 at 5:07 PM Maulin Vasavada <
> >>> maulin.vasav...@gmail.com>
> >>> wrote:
> >>>
> >>> > ^^^  bump ^^^ The vote is open for 2-3 days and gotten 1 Binding vote
> >>> so
> >>> > far, can you please vote so that we can try to move forward with
> >>> changes?
> >>> >
> >>> > On Thu, Mar 26, 2020 at 4:11 PM Zhou, Thomas
>  >>> >
> >>> > wrote:
> >>> >
> >>> > > +1 (non-binding)
> >>> > >
> >>> > > Regards,
> >>> > > Thomas
> >>> > >
> >>> > > On 3/26/20, 12:36 PM, "Rajini Sivaram" 
> >>> wrote:
> >>> > >
> >>> > > +1 (binding)
> >>> > > Thanks for the KIP, Maulin!
> >>> > >
> >>> > > Regards,
> >>> > >
> >>> > > Rajini
> >>> > >
> >>> > > On Thu, Mar 26, 2020 at 4:14 PM Maulin Vasavada <
> >>> > > maulin.vasav...@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > FYI - we have updated the KIP documentation also with
> >>> appropriate
> >>> > > code
> >>> > > > samples for interfaces and few important changes.
> >>> > > >
> >>> > > > Thanks
> >>> > > > Maulin
> >>> > > >
> >>> > > > On Wed, Mar 25, 2020 at 10:21 AM Maulin Vasavada <
> >>> > > > maulin.vasav...@gmail.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > bump
> >>> > > > >
> >>> > > > > On Wed, Mar 25, 2020 at 10:20 AM Maulin Vasavada <
> >>> > > > > maulin.vasav...@gmail.com> wrote:
> >>> > > > >
> >>> > > > >> Hi all
> >>> > > > >>
> >>> > > > >> After much await on the approach conclusion we have a PR
> >>> > > > >>
> >>> > >
> >>> >
> >>>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fpull%2F8338data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=1ydk0OMaucb8QhTyyQ8Ua3ereGzcS4usRlavU1RixkE%3Dreserved=0
> >>> > > .
> >>> > > > >>
> >>> > > > >> Can you please provide your vote so that we can more this
> >>> > forward?
> >>> > > > >>
> >>> > > > >> Thanks
> >>> > > > >> Maulin
> >>> > > > >>
> >>> > > > >> On Sun, Jan 26, 2020 at 11:03 PM Maulin Vasavada <
> >>> > > > >> maulin.vasav...@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi all
> >>> > > > >>>
> >>> > > > >>> After a good discussion on the KIP at
> >>> > > > >>>
> >>> > >
> >>> >
> >>>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fdev%40kafka.apache.org%2Fmsg101011.htmldata=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=qsvbqkoxL6NSPDV6rm9B9xqZG5xvYaZkj0cYrTM6bPw%3Dreserved=0
> >>> > > I
> >>> > > > >>> think we are ready to start voting.
> >>> > > > >>>
> >>> > > > >>> KIP:
> >>> > > > >>>
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D128650952data=01%7C01%7Cthzhou%40paypal.com%7C4520b56f3b1f44cceddb08d7d1bd052a%7Cfb00791460204374977e21bac5f3f4c8%7C1sdata=rcqWc2inIbrWlMj2jssHPKcMlHuDuLvicmYHHDYWrF8%3Dreserved=0
> >>> > > > >>>
> >>> > > > >>> The KIP proposes - Making SSLEngine creation pluggable to
> >>> > support
> >>> > > > >>> customization of various security related aspects.
> >>> > > > >>>
> >>> > > > >>> Thanks
> >>> > > > >>> Maulin
> >>> > > > >>>
> >>> > > > >>
> >>> > > 

[jira] [Created] (KAFKA-9789) if one machine has four broker, i want three is cluster ,how do ?

[startjava (Jira)]

startjava created KAFKA-9789:


 Summary: if one machine has four broker, i want three is cluster 
,how do ?
 Key: KAFKA-9789
 URL: https://issues.apache.org/jira/browse/KAFKA-9789
 Project: Kafka
  Issue Type: Test
Reporter: startjava


if  me has one machine has four broker, i want three broker is cluster ,how do ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4386

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9777; Remove txn purgatory to fix race condition on txn 
completion


--
[...truncated 2.95 MB...]

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task 

Build failed in Jenkins: kafka-2.4-jdk8 #181

[Apache Jenkins Server]

See 


Changes:

[konstantine] KAFKA-9706: Handle null in keys or values when Flatten 
transformation is


--
[...truncated 5.58 MB...]
org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata STARTED

org.apache.kafka.streams.MockProcessorContextTest >