Re: [jira] [Created] (KAFKA-9413) Add audit of Kafka cluster

[Владимир Беруненко]

On 2020/01/13 11:15:00, "Igor Martemyanov (Jira)"  wrote: 
> Igor Martemyanov created KAFKA-9413:
> ---
> 
>  Summary: Add audit of Kafka cluster
>  Key: KAFKA-9413
>  URL: https://issues.apache.org/jira/browse/KAFKA-9413
>  Project: Kafka
>   Issue Type: New Feature
>   Components: core, logging
> Affects Versions: 2.4.0
> Reporter: Igor Martemyanov
> 
> 
> Sometimes it could be necessary to know who add/edit/delete some cluster 
> resource. E.g. create topic, add ACL or delete its. I suggest adding default 
> implementation of this functionality which send audit records into separate 
> log file. And other users could implement another logic(by implementing 
> common API in plugins), e.g. send audit events to another cluster, or 
> database or anywhere else by changing some property's default value.
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
> 

Hi Igor!

It seems to be a valuable feature for enterprises in the financial sector. 
Ready to help design and implement this functionality.

Best wishes, Vladimir

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Matthias J. Sax]

Thanks. SGTM.

-Matthias

On 1/14/20 4:52 PM, John Roesler wrote:
> Hey Matthias,
> 
> Thanks for taking a look! I felt a little uneasy about it, but didn’t think 
> about the case you pointed out. Throwing an exception would indeed be safer.
> 
> Given a choice between throwing in the default method or defining a new 
> interface and throwing if the wrong interface is implemented, it seems nicer 
> for everyone to go the default method route. Since we’re not referencing the 
> other method anymore, I should probably deprecate it, too. 
> 
> Thanks again for your help. I really appreciate it.
> 
> -John
> 
> On Tue, Jan 14, 2020, at 18:15, Matthias J. Sax wrote:
>> Thanks for the PR. That helps a lot.
>>
>> I actually do have a concern: the proposed default method, would disable
>> the new feature to allow querying an active task during restore
>> automatically. Hence, if a user has an existing custom store type, and
>> would use the new
>>
>> KafkaStreams.store(..., true)
>>
>> method to enable querying during restore it would not work, and it would
>> be unclear why. It would even be worth if there are two developers and
>> one provide the store type while the other one just uses it.
>>
>> Hence, the default implementation should maybe throw an exception by
>> default? Or maybe, we would introduce a new interface that extends
>> `QueryableStoreType` and add this new method. For this case, we could
>> check within
>>
>> KafkaStreams.store(..., true)
>>
>> if the StoreType implements the new interface and if not, throw an
>> exception there.
>>
>> Those exceptions would be more descriptive (ie, state store does not
>> support querying during restore) and give the user a chance to figure
>> out what's wrong.
>>
>> Not sure if overwriting a default method or a new interface is the
>> better choice to let people opt-into the feature.
>>
>>
>>
>> -Matthias
>>
>> On 1/14/20 3:22 PM, John Roesler wrote:
>>> Hi again all,
>>>
>>> I've sent a PR including this new option, and while implementing it, I 
>>> realized we also have to make a (source-compatible) addition to the 
>>> QueryableStoreType interface, so that the IQ store wrapper can pass the
>>> flag down to the composite store provider.
>>>
>>> See https://github.com/apache/kafka/pull/7962
>>> In particular: 
>>> https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
>>>
>>> If there are no objections to these additions, I'll update the KIP tomorrow.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
 Thanks for calling this out, Matthias. You're correct that this looks like 
 a
 harmful behavioral change. I'm fine with adding the new overload you
 mentioned, just a simple boolean flag to enable the new behavior.

 I'd actually propose that we call this flag "queryStaleData", with a 
 default
 of "false". The meaning of this would be to preserve exactly the existing
 semantics. I.e., that the store must be both active and running to be
 included.

 It seems less severe to just suddenly start returning queries from 
 standbys,
 but in the interest of safety, the easiest thing is just flag the whole 
 feature.

 If you, Navinder, and Vinoth agree, we can just update the KIP. It seems 
 like
 a pretty uncontroversial amendment to avoid breaking query consistency
 semantics.

 Thanks,
 -John


 On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> During the discussion of KIP-216
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
> we encountered that KIP-535 introduces a behavior change that is not
> backward compatible, hence, I would like to request a small change.
>
> KIP-535 suggests, that active tasks can be queried during recovery and
> no exception would be thrown and longer. This is a change in behavior
> and in fact introduces a race condition for users that only want to
> query consistent state. Querying inconsistent state should be an opt-in,
> and for StandbyTasks, user can opt-in by querying them or opt-out by not
> querying them. However, for active task, if we don't throw an exception
> during recovery, users cannot opt-out of querying potentially
> inconsistent state, because they would need to check the state (ie, ==
> RUNNING) before they would query the active task -- however, the state
> might change at any point in between, and there is a race.
>
> Hence, I would suggest to actually not change the default behavior of
> existing methods and we should throw an exception during restore if an
> active task is queried. To allow user to opt-in to query an active task
> during restore, we would add an overload
>
>> KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)
>
> (with an hopefully 

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Matthias J. Sax]

I was just checking the existing code, and we currently log at WARN
level if the handler returns CONTINUE -- we did not have any complaints
about it, hence, I don't see an issue with WARN -- and we should keep it
consistent.

I think the explicit mentioning of `ClassCastException` is based on the
current code that catches this exception to rethrow it -- this was a
minor improvement to help people to more easily detect miss-configure
serdes.

In think, we can just catch all exception and the handler can decide
what to do. Thinking about this once more, it might actually be better
if we could _exclude_ `ClassCastException` as it may indicate a miss
configured Serde?


-Matthias

On 1/14/20 4:15 PM, Bill Bejeck wrote:
> Hi Anna,
> 
> Thanks for getting this KIP going again.
> 
> I agree with pushing forward on option 0 as well.  I a couple of questions
> about the KIP as written.
> 
> The KIP states that any {{ClassCastException}} thrown plus any other
> unchecked exceptions will result in a log statement and not stop processing
> if the handler returns CONTINUE.
> 
>1. I'm wondering if DEBUG is the correct level vs. a WARN, although, at
>WARN, we could end up spamming the log file.
>2. Are allowing all unchecked exceptions to proceed to permissive?  I
>could be overly cautious here, but I'm afraid of masking a serious
>problem.
> 
> Overall I'm in favor of this KIP and if you feel it's good as is, I
> wouldn't block on these questions  I just wanted to throw in my 2 cents.
> 
> Thanks,
> Bill
> 
> On Sat, Jan 11, 2020 at 7:02 PM Mitchell  wrote:
> 
>> I'm happy to have the serialization handler now.  I've hit this issue
>> a number of times in the past.
>>
>> I think the other options are also large enough they probably deserve
>> their own KIPs to properly document the changes.
>> -mitch
>>
>> On Fri, Jan 10, 2020 at 7:33 PM am  wrote:
>>>
>>> Hello,
>>>
>>> I would like to re-restart the discussion of KIP-399
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
>>>
>>> The last conversation centered on if this KIP should address the issues
>>> around state store/change log divergence with Matthias presenting three
>>> options:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *To move this KIP forward, maybe we can just (0) add the handler
>>> forserialization exceptions when writing into any topic and consider it
>>> anincremental improvement. Ie, (1) we keep the door open to let state
>>> andchangelog topic diverge (current status) (2) we allow people to
>>> violateEOS (current state) (3) and we don't improve the handling of DSL
>>> statestore serialization exceptions.We could address (1), (2), and/or (3)
>>> in follow up KIPs.Thoughts? Let us know if you only want to address (0),
>> or
>>> extend thecurrent KIP to include any of (1-3).*
>>>
>>>
>>> I would like to propose we go with option 0 and treat this as an
>>> incremental improvement that applies to any topic and address the issue
>> of
>>> divergence in future KIP(s).
>>>
>>> Feedback, thoughts and musings appreciated,
>>>
>>> anna
>>
> 



signature.asc
Description: OpenPGP digital signature

CompletableFuture?

[radai]

Hi

With kip-118 (Drop Support for Java 7) officially done, is there a
timeline replacing usage of "plain" Futures with java 8
CompletableFutures?

kafka 2.0 was mentioned at some point as a possible target for this ...

[jira] [Resolved] (KAFKA-9365) Add consumer group information to txn commit

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-9365.

Resolution: Fixed

> Add consumer group information to txn commit 
> -
>
> Key: KAFKA-9365
> URL: https://issues.apache.org/jira/browse/KAFKA-9365
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>
> This effort adds consumer group information to the txn commit protocol on the 
> broker side.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1082

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9294: Add tests for Named parameter (#7927)

[mumrah] KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning

[wangguoz] KAFKA-9365: Add server side change  to include consumer group


--
[...truncated 2.84 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 

Build failed in Jenkins: kafka-trunk-jdk8 #4160

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9294: Add tests for Named parameter (#7927)

[mumrah] KAFKA-8764: LogCleanerManager endless loop while compacting/cleaning

[wangguoz] KAFKA-9365: Add server side change  to include consumer group


--
[...truncated 2.82 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task 

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[John Roesler]

Hey Matthias,

Thanks for taking a look! I felt a little uneasy about it, but didn’t think 
about the case you pointed out. Throwing an exception would indeed be safer.

Given a choice between throwing in the default method or defining a new 
interface and throwing if the wrong interface is implemented, it seems nicer 
for everyone to go the default method route. Since we’re not referencing the 
other method anymore, I should probably deprecate it, too. 

Thanks again for your help. I really appreciate it.

-John

On Tue, Jan 14, 2020, at 18:15, Matthias J. Sax wrote:
> Thanks for the PR. That helps a lot.
> 
> I actually do have a concern: the proposed default method, would disable
> the new feature to allow querying an active task during restore
> automatically. Hence, if a user has an existing custom store type, and
> would use the new
> 
> KafkaStreams.store(..., true)
> 
> method to enable querying during restore it would not work, and it would
> be unclear why. It would even be worth if there are two developers and
> one provide the store type while the other one just uses it.
> 
> Hence, the default implementation should maybe throw an exception by
> default? Or maybe, we would introduce a new interface that extends
> `QueryableStoreType` and add this new method. For this case, we could
> check within
> 
> KafkaStreams.store(..., true)
> 
> if the StoreType implements the new interface and if not, throw an
> exception there.
> 
> Those exceptions would be more descriptive (ie, state store does not
> support querying during restore) and give the user a chance to figure
> out what's wrong.
> 
> Not sure if overwriting a default method or a new interface is the
> better choice to let people opt-into the feature.
> 
> 
> 
> -Matthias
> 
> On 1/14/20 3:22 PM, John Roesler wrote:
> > Hi again all,
> > 
> > I've sent a PR including this new option, and while implementing it, I 
> > realized we also have to make a (source-compatible) addition to the 
> > QueryableStoreType interface, so that the IQ store wrapper can pass the
> > flag down to the composite store provider.
> > 
> > See https://github.com/apache/kafka/pull/7962
> > In particular: 
> > https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
> > 
> > If there are no objections to these additions, I'll update the KIP tomorrow.
> > 
> > Thanks,
> > -John
> > 
> > On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
> >> Thanks for calling this out, Matthias. You're correct that this looks like 
> >> a
> >> harmful behavioral change. I'm fine with adding the new overload you
> >> mentioned, just a simple boolean flag to enable the new behavior.
> >>
> >> I'd actually propose that we call this flag "queryStaleData", with a 
> >> default
> >> of "false". The meaning of this would be to preserve exactly the existing
> >> semantics. I.e., that the store must be both active and running to be
> >> included.
> >>
> >> It seems less severe to just suddenly start returning queries from 
> >> standbys,
> >> but in the interest of safety, the easiest thing is just flag the whole 
> >> feature.
> >>
> >> If you, Navinder, and Vinoth agree, we can just update the KIP. It seems 
> >> like
> >> a pretty uncontroversial amendment to avoid breaking query consistency
> >> semantics.
> >>
> >> Thanks,
> >> -John
> >>
> >>
> >> On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> >>> During the discussion of KIP-216
> >>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
> >>> we encountered that KIP-535 introduces a behavior change that is not
> >>> backward compatible, hence, I would like to request a small change.
> >>>
> >>> KIP-535 suggests, that active tasks can be queried during recovery and
> >>> no exception would be thrown and longer. This is a change in behavior
> >>> and in fact introduces a race condition for users that only want to
> >>> query consistent state. Querying inconsistent state should be an opt-in,
> >>> and for StandbyTasks, user can opt-in by querying them or opt-out by not
> >>> querying them. However, for active task, if we don't throw an exception
> >>> during recovery, users cannot opt-out of querying potentially
> >>> inconsistent state, because they would need to check the state (ie, ==
> >>> RUNNING) before they would query the active task -- however, the state
> >>> might change at any point in between, and there is a race.
> >>>
> >>> Hence, I would suggest to actually not change the default behavior of
> >>> existing methods and we should throw an exception during restore if an
> >>> active task is queried. To allow user to opt-in to query an active task
> >>> during restore, we would add an overload
> >>>
>  KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)
> >>>
> >>> (with an hopefully better/short variable name). Developers would use
> >>> this new method to opt-into querying active tasks during restore.
> 

Build failed in Jenkins: kafka-trunk-jdk11 #1081

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9420; Add flexible version support for converted protocols 
(#7931)


--
[...truncated 5.72 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Matthias J. Sax]

Thanks for the PR. That helps a lot.

I actually do have a concern: the proposed default method, would disable
the new feature to allow querying an active task during restore
automatically. Hence, if a user has an existing custom store type, and
would use the new

KafkaStreams.store(..., true)

method to enable querying during restore it would not work, and it would
be unclear why. It would even be worth if there are two developers and
one provide the store type while the other one just uses it.

Hence, the default implementation should maybe throw an exception by
default? Or maybe, we would introduce a new interface that extends
`QueryableStoreType` and add this new method. For this case, we could
check within

KafkaStreams.store(..., true)

if the StoreType implements the new interface and if not, throw an
exception there.

Those exceptions would be more descriptive (ie, state store does not
support querying during restore) and give the user a chance to figure
out what's wrong.

Not sure if overwriting a default method or a new interface is the
better choice to let people opt-into the feature.



-Matthias

On 1/14/20 3:22 PM, John Roesler wrote:
> Hi again all,
> 
> I've sent a PR including this new option, and while implementing it, I 
> realized we also have to make a (source-compatible) addition to the 
> QueryableStoreType interface, so that the IQ store wrapper can pass the
> flag down to the composite store provider.
> 
> See https://github.com/apache/kafka/pull/7962
> In particular: 
> https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
> 
> If there are no objections to these additions, I'll update the KIP tomorrow.
> 
> Thanks,
> -John
> 
> On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
>> Thanks for calling this out, Matthias. You're correct that this looks like a
>> harmful behavioral change. I'm fine with adding the new overload you
>> mentioned, just a simple boolean flag to enable the new behavior.
>>
>> I'd actually propose that we call this flag "queryStaleData", with a default
>> of "false". The meaning of this would be to preserve exactly the existing
>> semantics. I.e., that the store must be both active and running to be
>> included.
>>
>> It seems less severe to just suddenly start returning queries from standbys,
>> but in the interest of safety, the easiest thing is just flag the whole 
>> feature.
>>
>> If you, Navinder, and Vinoth agree, we can just update the KIP. It seems like
>> a pretty uncontroversial amendment to avoid breaking query consistency
>> semantics.
>>
>> Thanks,
>> -John
>>
>>
>> On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
>>> During the discussion of KIP-216
>>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
>>> we encountered that KIP-535 introduces a behavior change that is not
>>> backward compatible, hence, I would like to request a small change.
>>>
>>> KIP-535 suggests, that active tasks can be queried during recovery and
>>> no exception would be thrown and longer. This is a change in behavior
>>> and in fact introduces a race condition for users that only want to
>>> query consistent state. Querying inconsistent state should be an opt-in,
>>> and for StandbyTasks, user can opt-in by querying them or opt-out by not
>>> querying them. However, for active task, if we don't throw an exception
>>> during recovery, users cannot opt-out of querying potentially
>>> inconsistent state, because they would need to check the state (ie, ==
>>> RUNNING) before they would query the active task -- however, the state
>>> might change at any point in between, and there is a race.
>>>
>>> Hence, I would suggest to actually not change the default behavior of
>>> existing methods and we should throw an exception during restore if an
>>> active task is queried. To allow user to opt-in to query an active task
>>> during restore, we would add an overload
>>>
 KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)
>>>
>>> (with an hopefully better/short variable name). Developers would use
>>> this new method to opt-into querying active tasks during restore.
>>>
>>> Thoughts?
>>>
>>>
>>> -Matthias
>>>
>>> On 11/18/19 10:45 AM, Vinoth Chandar wrote:
 Thanks, everyone involved!

 On Mon, Nov 18, 2019 at 7:51 AM John Roesler  wrote:

> Thanks to you, also, Navinder!
>
> Looking forward to getting this feature in.
> -John
>
> On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
>  wrote:
>>
>>  Hello all,
>>
>> With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck,
>> and John Roesler, the vote passes.
>> Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy
> discussions and Vinoth for all the help on this KIP.
>> Best,
>> Navinder
>>
>> On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler <
> j...@confluent.io> wrote:

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Bill Bejeck]

Hi Anna,

Thanks for getting this KIP going again.

I agree with pushing forward on option 0 as well.  I a couple of questions
about the KIP as written.

The KIP states that any {{ClassCastException}} thrown plus any other
unchecked exceptions will result in a log statement and not stop processing
if the handler returns CONTINUE.

   1. I'm wondering if DEBUG is the correct level vs. a WARN, although, at
   WARN, we could end up spamming the log file.
   2. Are allowing all unchecked exceptions to proceed to permissive?  I
   could be overly cautious here, but I'm afraid of masking a serious
   problem.

Overall I'm in favor of this KIP and if you feel it's good as is, I
wouldn't block on these questions  I just wanted to throw in my 2 cents.

Thanks,
Bill

On Sat, Jan 11, 2020 at 7:02 PM Mitchell  wrote:

> I'm happy to have the serialization handler now.  I've hit this issue
> a number of times in the past.
>
> I think the other options are also large enough they probably deserve
> their own KIPs to properly document the changes.
> -mitch
>
> On Fri, Jan 10, 2020 at 7:33 PM am  wrote:
> >
> > Hello,
> >
> > I would like to re-restart the discussion of KIP-399
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> >
> > The last conversation centered on if this KIP should address the issues
> > around state store/change log divergence with Matthias presenting three
> > options:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *To move this KIP forward, maybe we can just (0) add the handler
> > forserialization exceptions when writing into any topic and consider it
> > anincremental improvement. Ie, (1) we keep the door open to let state
> > andchangelog topic diverge (current status) (2) we allow people to
> > violateEOS (current state) (3) and we don't improve the handling of DSL
> > statestore serialization exceptions.We could address (1), (2), and/or (3)
> > in follow up KIPs.Thoughts? Let us know if you only want to address (0),
> or
> > extend thecurrent KIP to include any of (1-3).*
> >
> >
> > I would like to propose we go with option 0 and treat this as an
> > incremental improvement that applies to any topic and address the issue
> of
> > divergence in future KIP(s).
> >
> > Feedback, thoughts and musings appreciated,
> >
> > anna
>

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

Oh, one more thing:

9. There's no mention of how the status topic is partitioned, or how
partitioning will be used by the new topic records. The KIP should probably
outline this for clarity and completeness.

Best regards,

Randall

On Tue, Jan 14, 2020 at 5:25 PM Randall Hauch  wrote:

> Thanks, Konstantine. Overall, this KIP looks interesting and really
> useful, and for the most part is spot on. I do have a number of
> questions/comments about specifics:
>
>1. The topic records have a value that includes the connector name,
>task number that last reported the topic is used, and the topic name.
>There's no mention of record timestamps, but I wonder if it'd be useful to
>record this. One challenge might be that a connector does not write to a
>topic for a while or the task remains running for long periods of time and
>therefore the worker doesn't record that this topic has been newly written
>to since it the task was restarted. IOW, the semantics of the timestamp may
>be a bit murky. Have you thought about recording the timestamp, and if so
>what are the pros and cons?
>- The "Recording active topics" section says the following:
>   "As soon as a worker detects the addition of a topic to a
>   connector's set of active topics, all the connector's tasks that inspect
>   source or sink records will cease to post update messages to the
>   status.storage.topic."
>   This probably means the timestamp won't be very useful.
>2. The KIP says "the Kafka record value stores the ID of the task that
>succeeded to store a topic status record last." However, this is a bit
>unclear: is it really storing the last task that successfully wrote to that
>topic (as this would require very frequent writes to this topic), or is it
>more that this is the task that was last *recorded* as having written
>to the topic? (Here, "recorded" could be a bit of a gray area, since this
>would depend on the how the worker periodically records this information.)
>Any kind of clarity here might be helpful.
>3. In the "Recording active topics" section (and the surrounding
>sections), the "task" is used ambiguously. For example, "when its tasks
>start processing their first records ... these tasks will start inspecting
>which is the Kafka topic of each of these records". IIUC, the first "task"
>mentioned is the connector's task, and the second is the worker's task. Do
>we need to distinguish this more clearly?
>4. Maybe I missed it, but does this KIP explicitly say that the
>Connector API is unchanged? It's probably worth pointing out to help
>assuage any concerns that connector implementations have to change to make
>use of this feature.
>5. In the "Resetting a connector's set of active topics" section the
>behavior is not exactly clear. Consider a user running connector "A", the
>connector has been fully started and is processing records, and the worker
>has recorded topic usage records. Then the user resets the active topics
>for connector A while the connector is still running? If the connector
>writes to no new topics, before the tasks are rebalanced then is it correct
>that Connect would report no active topics? And after the tasks are
>rebalance, will the worker record any topics used by connector A?
>6. In the "Restaring" (misspelled) section: "Reconfiguring a source
>connector has also no altering effect for a source connector. However, when
>reconfiguring a sink connector if the new configuration no longer includes
>any of the previously tracked topics, these topics will be removed from the
>set of active topics for this sink connector by appending tombstone
>messages appropriately after the reconfiguration of the connector." Would
>it be better to not automatically reset connector's active topics when a
>sink connector is restarted? Isn't that more consistent with the
>"Resetting" behavior and the goals at the top of the KIP: "it'd be useful
>for users, operators and applications to know which are the topics that a
>connector has used since it was first created"?
>7. The `PUT /connectors/{name}/topics/reset` endpoint "this request
>can be reapplied after the deletion of the connector". IOW, even though
>connector with that name doesn't exist, we can still make this request? How
>does this compare with other methods such as "status"?
>8. What are the security implications of this proposal?
>
> As you can see, most of these can probably be addressed without much work.
>
> Best regards,
>
> Randall
>
> On Mon, Jan 13, 2020 at 11:05 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
>> Hi all.
>>
>> I just posted KIP-558: Track the set of actively used topics by connectors
>> in Kafka Connect
>>
>> Wiki link here:
>>
>> 

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

[Bill Bejeck]

Thanks for the KIP, Konstantine.  The KIP is very clear and it seems like
it will have a very positive impact.

+1(binding)

-Bill

On Tue, Jan 14, 2020 at 2:02 PM Tom Bentley  wrote:

> +1 (non-binding). Thanks for the KIP!
>
> On Tue, Jan 14, 2020 at 6:45 PM Randall Hauch  wrote:
>
> > Thanks for the updated KIP, Konstantine. I have a few minor nits, but all
> > are around the implementation details.
> >
> > +1 (binding)
> >
> > Best regards,
> >
> > Randall
> >
> > On Mon, Jan 13, 2020 at 10:16 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Hi everyone.
> > >
> > > I hope y'all had a nice break. The discussion on KIP-158 seems to have
> > > wrapped up since last year, so I'd like to open the vote on this KIP.
> > >
> > > A reminder that this is an updated KIP-158 (that had also been approved
> > in
> > > its earlier version) and it seems to be a highly anticipated feature
> for
> > > many of us. I hope we can get this in for the upcoming release.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics
> > >
> > > Best,
> > > Konstantine
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk8 #4159

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

Thanks, Konstantine. Overall, this KIP looks interesting and really useful,
and for the most part is spot on. I do have a number of questions/comments
about specifics:

   1. The topic records have a value that includes the connector name, task
   number that last reported the topic is used, and the topic name. There's no
   mention of record timestamps, but I wonder if it'd be useful to record
   this. One challenge might be that a connector does not write to a topic for
   a while or the task remains running for long periods of time and therefore
   the worker doesn't record that this topic has been newly written to since
   it the task was restarted. IOW, the semantics of the timestamp may be a bit
   murky. Have you thought about recording the timestamp, and if so what are
   the pros and cons?
   - The "Recording active topics" section says the following:
  "As soon as a worker detects the addition of a topic to a connector's
  set of active topics, all the connector's tasks that inspect
source or sink
  records will cease to post update messages to the status.storage.topic."
  This probably means the timestamp won't be very useful.
   2. The KIP says "the Kafka record value stores the ID of the task that
   succeeded to store a topic status record last." However, this is a bit
   unclear: is it really storing the last task that successfully wrote to that
   topic (as this would require very frequent writes to this topic), or is it
   more that this is the task that was last *recorded* as having written to
   the topic? (Here, "recorded" could be a bit of a gray area, since this
   would depend on the how the worker periodically records this information.)
   Any kind of clarity here might be helpful.
   3. In the "Recording active topics" section (and the surrounding
   sections), the "task" is used ambiguously. For example, "when its tasks
   start processing their first records ... these tasks will start inspecting
   which is the Kafka topic of each of these records". IIUC, the first "task"
   mentioned is the connector's task, and the second is the worker's task. Do
   we need to distinguish this more clearly?
   4. Maybe I missed it, but does this KIP explicitly say that the
   Connector API is unchanged? It's probably worth pointing out to help
   assuage any concerns that connector implementations have to change to make
   use of this feature.
   5. In the "Resetting a connector's set of active topics" section the
   behavior is not exactly clear. Consider a user running connector "A", the
   connector has been fully started and is processing records, and the worker
   has recorded topic usage records. Then the user resets the active topics
   for connector A while the connector is still running? If the connector
   writes to no new topics, before the tasks are rebalanced then is it correct
   that Connect would report no active topics? And after the tasks are
   rebalance, will the worker record any topics used by connector A?
   6. In the "Restaring" (misspelled) section: "Reconfiguring a source
   connector has also no altering effect for a source connector. However, when
   reconfiguring a sink connector if the new configuration no longer includes
   any of the previously tracked topics, these topics will be removed from the
   set of active topics for this sink connector by appending tombstone
   messages appropriately after the reconfiguration of the connector." Would
   it be better to not automatically reset connector's active topics when a
   sink connector is restarted? Isn't that more consistent with the
   "Resetting" behavior and the goals at the top of the KIP: "it'd be useful
   for users, operators and applications to know which are the topics that a
   connector has used since it was first created"?
   7. The `PUT /connectors/{name}/topics/reset` endpoint "this request can
   be reapplied after the deletion of the connector". IOW, even though
   connector with that name doesn't exist, we can still make this request? How
   does this compare with other methods such as "status"?
   8. What are the security implications of this proposal?

As you can see, most of these can probably be addressed without much work.

Best regards,

Randall

On Mon, Jan 13, 2020 at 11:05 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Hi all.
>
> I just posted KIP-558: Track the set of actively used topics by connectors
> in Kafka Connect
>
> Wiki link here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
>
> I think it's a nice extension to follow up on KIP-158 and a useful feature
> to the ever increasing number of applications that are built around Kafka
> Connect.
> Would love to hear what you think.
>
> Best,
> Konstantine
>

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[John Roesler]

Hi again all,

I've sent a PR including this new option, and while implementing it, I 
realized we also have to make a (source-compatible) addition to the 
QueryableStoreType interface, so that the IQ store wrapper can pass the
flag down to the composite store provider.

See https://github.com/apache/kafka/pull/7962
In particular: 
https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41

If there are no objections to these additions, I'll update the KIP tomorrow.

Thanks,
-John

On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
> Thanks for calling this out, Matthias. You're correct that this looks like a
> harmful behavioral change. I'm fine with adding the new overload you
> mentioned, just a simple boolean flag to enable the new behavior.
> 
> I'd actually propose that we call this flag "queryStaleData", with a default
> of "false". The meaning of this would be to preserve exactly the existing
> semantics. I.e., that the store must be both active and running to be
> included.
> 
> It seems less severe to just suddenly start returning queries from standbys,
> but in the interest of safety, the easiest thing is just flag the whole 
> feature.
> 
> If you, Navinder, and Vinoth agree, we can just update the KIP. It seems like
> a pretty uncontroversial amendment to avoid breaking query consistency
> semantics.
> 
> Thanks,
> -John
> 
> 
> On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> > During the discussion of KIP-216
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
> > we encountered that KIP-535 introduces a behavior change that is not
> > backward compatible, hence, I would like to request a small change.
> > 
> > KIP-535 suggests, that active tasks can be queried during recovery and
> > no exception would be thrown and longer. This is a change in behavior
> > and in fact introduces a race condition for users that only want to
> > query consistent state. Querying inconsistent state should be an opt-in,
> > and for StandbyTasks, user can opt-in by querying them or opt-out by not
> > querying them. However, for active task, if we don't throw an exception
> > during recovery, users cannot opt-out of querying potentially
> > inconsistent state, because they would need to check the state (ie, ==
> > RUNNING) before they would query the active task -- however, the state
> > might change at any point in between, and there is a race.
> > 
> > Hence, I would suggest to actually not change the default behavior of
> > existing methods and we should throw an exception during restore if an
> > active task is queried. To allow user to opt-in to query an active task
> > during restore, we would add an overload
> > 
> > > KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)
> > 
> > (with an hopefully better/short variable name). Developers would use
> > this new method to opt-into querying active tasks during restore.
> > 
> > Thoughts?
> > 
> > 
> > -Matthias
> > 
> > On 11/18/19 10:45 AM, Vinoth Chandar wrote:
> > > Thanks, everyone involved!
> > > 
> > > On Mon, Nov 18, 2019 at 7:51 AM John Roesler  wrote:
> > > 
> > >> Thanks to you, also, Navinder!
> > >>
> > >> Looking forward to getting this feature in.
> > >> -John
> > >>
> > >> On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
> > >>  wrote:
> > >>>
> > >>>  Hello all,
> > >>>
> > >>> With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill 
> > >>> Bejeck,
> > >>> and John Roesler, the vote passes.
> > >>> Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy
> > >> discussions and Vinoth for all the help on this KIP.
> > >>> Best,
> > >>> Navinder
> > >>>
> > >>> On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler <
> > >> j...@confluent.io> wrote:
> > >>>
> > >>>  I'm +1 (binding) as well.
> > >>>
> > >>> Thanks,
> > >>> -John
> > >>>
> > >>> On Fri, Nov 15, 2019 at 6:20 AM Bill Bejeck  wrote:
> > 
> >  +1 (binding)
> > 
> >  On Fri, Nov 15, 2019 at 1:11 AM Matthias J. Sax  > >>>
> >  wrote:
> > 
> > > +1 (binding)
> > >
> > >
> > > On 11/14/19 3:48 PM, Guozhang Wang wrote:
> > >> +1 (binding), thanks for the KIP!
> > >>
> > >> Guozhang
> > >>
> > >> On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar
> > >>  wrote:
> > >>
> > >>> Hello all,
> > >>>
> > >>> I'd like to propose a vote for serving interactive queries during
> > >>> Rebalancing, as it is a big deal for applications looking for high
> > >>> availability. With this change, users will have control over the
> > > tradeoff
> > >>> between consistency and availability during serving.
> > >>> The full KIP is provided here:
> > >>>
> > >>>
> > >
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance
> > >>>
> > >>>
> > >>> Thanks,
> > >>> Navinder
> > >>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[James Cheng]

Congrats Colin, Vahid, and Manikumar!

-James

> On Jan 14, 2020, at 10:59 AM, Tom Bentley  wrote:
> 
> Congratulations!
> 
> On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram 
> wrote:
> 
>> Congratulations Colin, Vahid and Manikumar!
>> 
>> Regards,
>> Rajini
>> 
>> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison 
>> wrote:
>> 
>>> Congrats Colin, Vahid and Manikumar!
>>> 
>>> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
 
 Congratulations Colin, Vahid and Manikumar!
 
 Ismael
 
 On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
>> wrote:
 
> Hi everyone,
> 
> I'm happy to announce that Colin McCabe, Vahid Hashemian and
>> Manikumar
> Reddy are now members of Apache Kafka PMC.
> 
> Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> 2019. They all contributed many patches, code reviews and
>> participated
> in many KIP discussions. We appreciate their contributions and are
> looking forward to many more to come.
> 
> Congrats Colin, Vahid and Manikumar!
> 
> Gwen, on behalf of Apache Kafka PMC
> 
>>> 
>> 

[jira] [Reopened] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reopened KAFKA-8677:


Reopening this ticket. Test failed again: 
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4226/testReport/junit/kafka.api/GroupEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/]
{code:java}
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
'responses': Error reading array of size 131085, only 28 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110)
at 
org.apache.kafka.common.protocol.ApiKeys.parseResponse(ApiKeys.java:313)
at 
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:719)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:833)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:556)
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1306)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1246)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1214)
at 
kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:795)
at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1351)
at 
kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:537)
at 
kafka.api.EndToEndAuthorizationTest.consumeRecordsIgnoreOneAuthorizationException(EndToEndAuthorizationTest.scala:556)
at 
kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:376)
{code}

> Flakey test 
> GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> 
>
> Key: KAFKA-8677
> URL: https://issues.apache.org/jira/browse/KAFKA-8677
> Project: Kafka
>  Issue Type: Bug
>  Components: core, security, unit tests
>Affects Versions: 2.4.0
>Reporter: Boyang Chen
>Assignee: Guozhang Wang
>Priority: Blocker
>  Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console]
>  
> *18:43:39* kafka.api.GroupEndToEndAuthorizationTest > 
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00* 
> kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
>  failed, log available in 
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00*
>  *18:44:00* kafka.api.GroupEndToEndAuthorizationTest > 
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00* 
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before 
> timeout instead of the expected 1 records
> ---
> I found this flaky test is actually exposing a real bug in consumer: within 
> {{KafkaConsumer.poll}}, we have an optimization to try to send the next fetch 
> request before returning the data in order to pipelining the fetch requests:
> {code}
> if (!records.isEmpty()) {
> // before returning the fetched records, we can send off 
> the next round of fetches
> // and avoid block waiting for their responses to enable 
> pipelining while the user
> // is handling the fetched records.
> //
> // NOTE: since the consumed position has already been 
> updated, we must not allow
> // wakeups or any other errors to be triggered prior to 
> returning the fetched records.
> if (fetcher.sendFetches() > 0 || 
> client.hasPendingRequests()) {
> client.pollNoWakeup();
> }
> return this.interceptors.onConsume(new 
> ConsumerRecords<>(records));
> }
> {code}
> As the NOTE mentioned, this pollNoWakeup should NOT throw any exceptions, 
> since at this point the fetch position has been updated. If an exception is 
> thrown here, and the callers decides to capture and continue, those records 
> would never be returned again, causing data loss.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

[Brian Byrne]

Hi Colin,

Your feedback is appreciated, thank you.

On Tue, Jan 14, 2020 at 11:34 AM Colin McCabe  wrote:

> This is probably a nitpick, but it would be nice to specify that this list
> is in order of highest priority to lowest.
>

Done.


> Hmm.  Maybe --show-overridden or --include-overridden is a better flag
> name?
>

Done (--show-overridden).


> I think it would be nice to avoid using enums for QuotaEntity#Type,
> QuotaKey#Type, and QuotaFilter#Rule.  With enums, we have to worry about
> forwards and backwards compatibility problems.  For example, what do you do
> when you're querying a broker that has a new value for one of these, that
> is not in your enum?  In the  past, we've created an UNKNOWN value for enum
> types to solve this conundrum, but I'm not sure the extra complexity is
> worth it here.  We can jut make them strings and avoid worrying about the
> compatibility issues.
>

Makes sense. Done.


> Is QuotaKey#Units really needed?  It seems like perhaps QuotaKey#Type
> could imply the units used.
>

Possibly, maybe. It depends on how much structure is useful, which
influences the implementation in the broker. For example, for the existing
(global) bytes-per-second types (e.g. consumer byte rate), it may be useful
to define them on a per-broker BPS basis, and in some cases, in terms of
shares. The question becomes whether it'd be better to have a module in the
broker that is capable of deducing the effective quota automatically among
different units for the same quota type, or whether each should be handled
individually.

Given we don't expect many units, and some units may be incompatible with
others, perhaps it is best to have the unit implicit in the type string, to
be handled by the broker appropriately.

I've updated the KIP to reflect this change, which factors out the
QuotaKey. Let me know your thoughts.


> How common is the prefix matching use-case?  I haven't heard about people
> setting up principal names with a common prefix or anything like that-- is
> that commonly done?
>

It was, in part, for exposition, but would handle a case where principals
could be prefixed by organization/team name, numbered, or the like. If you
prefer I remove the rules and just accept a pattern, that's also an option.



> I sort of feel like maybe we could have a simpler API for describeQuotas
> where it takes a map of quota entity type to value, and we do a logical AND
> On that.  I'm not sure if there's really a reason why it needs to be a
> collection rather than a set, in other words...
>

For clarification, are you suggesting an interface where the user might
provide {type=user, name=x} and it would return all entities that match,
with their resulting quota values? Should I scrap pattern matching for now,
since it's a simple extension that can be done at a future time?

Thanks,
Brian



> On Wed, Dec 11, 2019, at 15:30, Brian Byrne wrote:
> > Hello all,
> >
> > I'm reviving the discussion for adding a quotas API to the admin client
> by
> > submitting a new proposal. There are some notable changes from previous
> > attempts, namely a way to deduce the effective quota for a client
> (entity),
> > a way to query for configured quotas, and the concept of "units" on
> quotas,
> > among other minor updates.
> >
> > Please take a look, and I'll be happy to make any clarifications and
> > modifications in regards to feedback.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
> >
> > Thanks,
> > Brian
> >
>

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Gwen Shapira]

+1 (binding, re-vote)

On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe  wrote:
>
> Hi all,
>
> I'm reposting this since I've been informed that gmail mashed the original 
> VOTE thread into a different email thread.  Hopefully the different thread 
> title will prevent it doing that in this case.
>
> I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
> Kafka Administrative Tools.
>
> KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
>
> Discussion thread: 
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
>
> cheers,
> Colin

[jira] [Resolved] (KAFKA-9346) Consumer fetch offset back-off with pending transactions

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-9346.

Resolution: Fixed

> Consumer fetch offset back-off with pending transactions
> 
>
> Key: KAFKA-9346
> URL: https://issues.apache.org/jira/browse/KAFKA-9346
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[John Roesler]

Thanks for calling this out, Matthias. You're correct that this looks like a
harmful behavioral change. I'm fine with adding the new overload you
mentioned, just a simple boolean flag to enable the new behavior.

I'd actually propose that we call this flag "queryStaleData", with a default
of "false". The meaning of this would be to preserve exactly the existing
semantics. I.e., that the store must be both active and running to be
included.

It seems less severe to just suddenly start returning queries from standbys,
but in the interest of safety, the easiest thing is just flag the whole feature.

If you, Navinder, and Vinoth agree, we can just update the KIP. It seems like
a pretty uncontroversial amendment to avoid breaking query consistency
semantics.

Thanks,
-John


On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> During the discussion of KIP-216
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
> we encountered that KIP-535 introduces a behavior change that is not
> backward compatible, hence, I would like to request a small change.
> 
> KIP-535 suggests, that active tasks can be queried during recovery and
> no exception would be thrown and longer. This is a change in behavior
> and in fact introduces a race condition for users that only want to
> query consistent state. Querying inconsistent state should be an opt-in,
> and for StandbyTasks, user can opt-in by querying them or opt-out by not
> querying them. However, for active task, if we don't throw an exception
> during recovery, users cannot opt-out of querying potentially
> inconsistent state, because they would need to check the state (ie, ==
> RUNNING) before they would query the active task -- however, the state
> might change at any point in between, and there is a race.
> 
> Hence, I would suggest to actually not change the default behavior of
> existing methods and we should throw an exception during restore if an
> active task is queried. To allow user to opt-in to query an active task
> during restore, we would add an overload
> 
> > KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)
> 
> (with an hopefully better/short variable name). Developers would use
> this new method to opt-into querying active tasks during restore.
> 
> Thoughts?
> 
> 
> -Matthias
> 
> On 11/18/19 10:45 AM, Vinoth Chandar wrote:
> > Thanks, everyone involved!
> > 
> > On Mon, Nov 18, 2019 at 7:51 AM John Roesler  wrote:
> > 
> >> Thanks to you, also, Navinder!
> >>
> >> Looking forward to getting this feature in.
> >> -John
> >>
> >> On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
> >>  wrote:
> >>>
> >>>  Hello all,
> >>>
> >>> With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck,
> >>> and John Roesler, the vote passes.
> >>> Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy
> >> discussions and Vinoth for all the help on this KIP.
> >>> Best,
> >>> Navinder
> >>>
> >>> On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler <
> >> j...@confluent.io> wrote:
> >>>
> >>>  I'm +1 (binding) as well.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Fri, Nov 15, 2019 at 6:20 AM Bill Bejeck  wrote:
> 
>  +1 (binding)
> 
>  On Fri, Nov 15, 2019 at 1:11 AM Matthias J. Sax  >>>
>  wrote:
> 
> > +1 (binding)
> >
> >
> > On 11/14/19 3:48 PM, Guozhang Wang wrote:
> >> +1 (binding), thanks for the KIP!
> >>
> >> Guozhang
> >>
> >> On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar
> >>  wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to propose a vote for serving interactive queries during
> >>> Rebalancing, as it is a big deal for applications looking for high
> >>> availability. With this change, users will have control over the
> > tradeoff
> >>> between consistency and availability during serving.
> >>> The full KIP is provided here:
> >>>
> >>>
> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance
> >>>
> >>>
> >>> Thanks,
> >>> Navinder
> >>
> >>
> >>
> >
> >
> >>>
> >>
> > 
> 
> 
> Attachments:
> * signature.asc

Jenkins build is back to normal : kafka-trunk-jdk11 #1080

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Rajini Sivaram]

Hi Ron,

Thanks for the detailed explanation, sounds good to me.

A few more questions:

1) At the moment, all sensitive broker configs including
keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
starting up brokers. We are now adding SSL keystore/truststore passwords
for ZooKeeper client that cannot be stored in ZooKeeper since you need
these to connect to ZK. We should perhaps document that these configs can
be encrypted using KIP-421.

2) We can dynamically update keystores and trust stores used by brokers
without restarting the broker. Can we support this easily for ZK clients
created by the broker, for example by adding our own
`zookeeper.ssl.context.supplier.class`?

3) Looks like we are using config names that directly map to ZK configs.
Have we considered using equivalent Kafka config names with prefixes,
perhaps with inheritance from the non-prefixed names? Not sure if this is a
good idea, but perhaps worth documenting in Rejected Alternatives at least.


On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:

> Hi Ron,
>
> Thanks for the explanation.  I guess thinking about it a little bit more,
> we should just add --zk-tls-config-file to all of these commands.
>
> We will be removing this option (plus ZK in general) from these commands
> in the next major release, but ZK is still supported in 2.5, so we should
> just do the logical thing.  (The exception is ZkSecurityMigrator, which
> will stay).
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > Hi Colin.
> >
> > <<< It seems like this [--zk-tls-config-file information] could just
> appear
> > in a configuration file, which all of these tools already accept (I
> think)
> >
> > ZkSecurityMigrator has no such property file facility; adding a
> > "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > (the same code is simply reused; it ends up being just a few extra
> lines).
> > I do not see any parameter in the other two commands to adjust the ZK
> > connection; ConfigCommand accepts a "--command-config" flag, but
> according
> > to the code "This is used only with --bootstrap-server option for
> > describing and altering broker configs."
> >
> > I do agree there would be no need to add "--zk-tls-config-file" to
> > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> time
> > for the next release.
> >
> > ConfigCommand supports the "--bootstrap-server" option and will have its
> > direct ZooKeeper access formally deprecated as per KIP-555, but the
> special
> > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > prior to starting Kafka will still exist, so it feels like while
> > "--zk-tls-config-file" would never be used except for this use case it
> > could probably still be added for this particular situation.
> >
> > Ron
> >
> > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > sent) did not go through.  I am trying to get emails through now to move
> > this discussion forward.
> >
> > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe  wrote:
> >
> > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > Hi everyone.  I would like to make the following changes to the KIP.
> > > >
> > > > MOTIVATION:
> > > > Include a statement that it will be difficult in the short term to
> > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > (which
> > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> cluster
> > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > case;
> > > > therefore it is in scope to be able to securely configure the CLI
> tools
> > > > that still leverage non-deprecated direct Zookeeper communication
> for TLS
> > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > zookeeper-security-migration.sh).
> > >
> > > Hi Ron,
> > >
> > > Thanks for the KIP.
> > >
> > > About deprecations:
> > >
> > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > > would not make sense, since it would defeat the whole point of the
> tool :)
> > >
> > > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > > implementation has dragged a bit, but this should get done soon.
> Certainly
> > > before the next release.
> > >
> > > * kafka-configs: I think ZK access should be deprecated here as well.
> I
> > > agree there is a super-special bootstrapping case here, but that should
> > > have its own tool, not use kafka-configs.
> > >
> > > I will post a separate KIP for this, though.
> > >
> > > >
> > > > GOALS:
> > > > Support the secure configuration of TLS-encrypted communication
> between
> > > > Zookeeper and:
> > > >   a) Kafka brokers
> > > >   b) The three CLI tools mentioned above that still support direct,
> > > > non-deprecated communication to Zookeeper
> > > > It is 

[jira] [Resolved] (KAFKA-9420) Bump APIs to enable flexible versions

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9420.

Fix Version/s: 2.5.0
   Resolution: Fixed

> Bump APIs to enable flexible versions
> -
>
> Key: KAFKA-9420
> URL: https://issues.apache.org/jira/browse/KAFKA-9420
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.5.0
>
>
> As part of KIP-482 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields),
>  we need to bump the versions of all protocols which have been converted to 
> use the generated protocol in order to enable flexible version support . This 
> ticket covers the following APIs which now support the generated protocol:
> - SaslAuthenticate
> - SaslHandshake
> - CreatePartitions
> - DescribeDelegationToken
> - ExpireDelegationToken
> - RenewDelegationToken



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

[Colin McCabe]

Hi Brian,

Thanks for the KIP.

The KIP says:

 > As represented by the current ZK node structure, the order in which quotas 
 > are matched are as follows. Note  is a specified user principal, 
 >  is a specified client ID, and  is a special default 
 > user/client ID that matches to all users or clients IDs.

This is probably a nitpick, but it would be nice to specify that this list is 
in order of highest priority to lowest.

> --include-overrides: Whether to include overridden config entries.

Hmm.  Maybe --show-overridden or --include-overridden is a better flag name?

I think it would be nice to avoid using enums for QuotaEntity#Type, 
QuotaKey#Type, and QuotaFilter#Rule.  With enums, we have to worry about 
forwards and backwards compatibility problems.  For example, what do you do 
when you're querying a broker that has a new value for one of these, that is 
not in your enum?  In the  past, we've created an UNKNOWN value for enum types 
to solve this conundrum, but I'm not sure the extra complexity is worth it 
here.  We can jut make them strings and avoid worrying about the compatibility 
issues.

Is QuotaKey#Units really needed?  It seems like perhaps QuotaKey#Type could 
imply the units used.

>public DescribeQuotasResult(KafkaFuture Long>>> entities);

How common is the prefix matching use-case?  I haven't heard about people 
setting up principal names with a common prefix or anything like that-- is that 
commonly done?

I sort of feel like maybe we could have a simpler API for describeQuotas where 
it takes a map of quota entity type to value, and we do a logical AND On that.  
I'm not sure if there's really a reason why it needs to be a collection rather 
than a set, in other words...

cheers,
Colin


On Wed, Dec 11, 2019, at 15:30, Brian Byrne wrote:
> Hello all,
> 
> I'm reviving the discussion for adding a quotas API to the admin client by
> submitting a new proposal. There are some notable changes from previous
> attempts, namely a way to deduce the effective quota for a client (entity),
> a way to query for configured quotas, and the concept of "units" on quotas,
> among other minor updates.
> 
> Please take a look, and I'll be happy to make any clarifications and
> modifications in regards to feedback.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
> 
> Thanks,
> Brian
>

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Ron Dagostino]

Hi Colin. +1 from me (non-binding). (re-voting on this thread)

Ron

On Tue, Jan 14, 2020 at 2:23 PM Colin McCabe  wrote:
>
> Hi all,
>
> I'm reposting this since I've been informed that gmail mashed the original 
> VOTE thread into a different email thread.  Hopefully the different thread 
> title will prevent it doing that in this case.
>
> I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
> Kafka Administrative Tools.
>
> KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
>
> Discussion thread: 
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
>
> cheers,
> Colin

Re: [VOTE] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

[Ron Dagostino]

Hi Colin.  +1 from me (non-binding).

Ron


On Tue, Jan 14, 2020 at 12:40 PM Colin McCabe  wrote:
>
> Hi all,
>
> I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
> Kafka Administrative Tools.
>
> KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
>
> Discussion thread: 
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
>
> cheers,
> Colin

[VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Colin McCabe]

Hi all,

I'm reposting this since I've been informed that gmail mashed the original VOTE 
thread into a different email thread.  Hopefully the different thread title 
will prevent it doing that in this case.

I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
Kafka Administrative Tools.

KIP:  https://cwiki.apache.org/confluence/x/Wg6dC

Discussion thread: 
https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E

cheers,
Colin

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Matthias J. Sax]

During the discussion of KIP-216
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
we encountered that KIP-535 introduces a behavior change that is not
backward compatible, hence, I would like to request a small change.

KIP-535 suggests, that active tasks can be queried during recovery and
no exception would be thrown and longer. This is a change in behavior
and in fact introduces a race condition for users that only want to
query consistent state. Querying inconsistent state should be an opt-in,
and for StandbyTasks, user can opt-in by querying them or opt-out by not
querying them. However, for active task, if we don't throw an exception
during recovery, users cannot opt-out of querying potentially
inconsistent state, because they would need to check the state (ie, ==
RUNNING) before they would query the active task -- however, the state
might change at any point in between, and there is a race.

Hence, I would suggest to actually not change the default behavior of
existing methods and we should throw an exception during restore if an
active task is queried. To allow user to opt-in to query an active task
during restore, we would add an overload

> KafkaStream#store(..., boolean allowQueryWhileStateIsRestoring)

(with an hopefully better/short variable name). Developers would use
this new method to opt-into querying active tasks during restore.

Thoughts?


-Matthias

On 11/18/19 10:45 AM, Vinoth Chandar wrote:
> Thanks, everyone involved!
> 
> On Mon, Nov 18, 2019 at 7:51 AM John Roesler  wrote:
> 
>> Thanks to you, also, Navinder!
>>
>> Looking forward to getting this feature in.
>> -John
>>
>> On Sun, Nov 17, 2019 at 11:34 PM Navinder Brar
>>  wrote:
>>>
>>>  Hello all,
>>>
>>> With 4 binding +1 votes from Guozhang Wang, Matthias J. Sax, Bill Bejeck,
>>> and John Roesler, the vote passes.
>>> Thanks Guozhang, Matthias, Bill, John, Sophie for the healthy
>> discussions and Vinoth for all the help on this KIP.
>>> Best,
>>> Navinder
>>>
>>> On Friday, 15 November, 2019, 11:32:31 pm IST, John Roesler <
>> j...@confluent.io> wrote:
>>>
>>>  I'm +1 (binding) as well.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Fri, Nov 15, 2019 at 6:20 AM Bill Bejeck  wrote:

 +1 (binding)

 On Fri, Nov 15, 2019 at 1:11 AM Matthias J. Sax >>
 wrote:

> +1 (binding)
>
>
> On 11/14/19 3:48 PM, Guozhang Wang wrote:
>> +1 (binding), thanks for the KIP!
>>
>> Guozhang
>>
>> On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar
>>  wrote:
>>
>>> Hello all,
>>>
>>> I'd like to propose a vote for serving interactive queries during
>>> Rebalancing, as it is a big deal for applications looking for high
>>> availability. With this change, users will have control over the
> tradeoff
>>> between consistency and availability during serving.
>>> The full KIP is provided here:
>>>
>>>
>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance
>>>
>>>
>>> Thanks,
>>> Navinder
>>
>>
>>
>
>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Matthias J. Sax]

Vito,

It's still unclear to me what the advantage is, to have both
`StreamsNotRunningException` and `StateStoreNotAvailableException`?

For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
and thus, for a user point of view, why does it matter if the store is
closed on not? I don't understand why/how this information would be
useful? Do you have a concrete example in mind how a user would react
differently to both exceptions?


@Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
actually do this on a per-query basis, ie, have an overload
`KafkaStreams#store(...)` that takes a boolean flag that allow to
_disable_ the exception and opt-in to query a active store during
recovery. However, as KIP-535 actually introduces this change in
behavior, I think KIP-216 should not cover this, but KIP-535 should be
updated. I'll follow up on the other KIP thread to raise this point.


-Matthias

On 1/11/20 12:26 AM, Vito Jeng wrote:
> Hi, Matthias & Vinoth,
> 
> Thanks for the feedback.
> 
>> What is still unclear to me is, what we gain by having both
>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>> state store is closed on not -- I can't query it anyway? Maybe I miss
>> something thought?
> 
> Yes, both `StreamsNotRunningException` and
> `StateStoreNotAvailableException` are fatal exception.
> But `StateStoreNotAvailableException` is fatal exception about state store
> related.
> I think it would be helpful that if user need to distinguish these two
> different case to handle it.
> 
> I'm not very sure, does that make sense?
> 
> 
> ---
> Vito
> 
> 
> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar  wrote:
> 
>> +1 on merging `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
>> its best to have these exceptions be about the state store (and not streams
>> state), to easier understanding.
>>
>> Additionally, KIP-535 allows for querying of state stores in rebalancing
>> state. So do we need the StreamsRebalancingException?
>>
>>
>> On 2020/01/09 03:38:11, "Matthias J. Sax"  wrote:
>>> Sorry that I dropped the ball on this...
>>>
>>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
>>> thread.
>>>
>>> What is still unclear to me is, what we gain by having both
>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>>> state store is closed on not -- I can't query it anyway? Maybe I miss
>>> something thought?
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
 Sorry for the late reply, thanks for the review.


> About `StateStoreMigratedException`:
>
> Why is it only thrown if the state is REBALANCING? A store might be
> migrated during a rebalance, and Kafka Streams might resume back to
> RUNNING state and afterward somebody tries to use an old store handle.
> Also, if state is REBALANCING, should we throw
> `StreamThreadRebalancingException`? Hence, I think
> `StateStoreMigratedException` does only make sense during `RUNNING`
>> state.
>

 Thank you point this, already updated.


 Why do we need to distinguish between `KafkaStreamsNotRunningException`
> and `StateStoreNotAvailableException`?
>

 `KafkaStreamsNotRunningException` may be caused by various reasons, I
>> think
 it would be helpful that the
 user can distinguish whether it is caused by the state store closed.
 (Maybe I am wrong...)


 Last, why do we distinguish between `KafkaStreams` instance and
> `StreamsThread`? To me, it seems we should always refer to the
>> instance,
> because that is the level of granularity in which we enable/disable
>> IQ atm.
>

 Totally agree. Do you mean the naming of state store exceptions?
 I don't have special reason to distinguish these two.
 Your suggestion look more reasonable for the exception naming.


 Last, for `StateStoreMigratedException`, I would add that a user need
>> to
> rediscover the store and cannot blindly retry as the store handle is
> invalid and a new store handle must be retrieved. That is a difference
> to `StreamThreadRebalancingException` that allows for "blind" retries
> that either resolve (if the store is still on the same instance after
> rebalancing finishes, or changes to `StateStoreMigratedException` if
>> the
> store was migrated away during rebalancing).
>

 Nice, it's great! Thank you.


 The KIP already updated, please take a look. :)



 On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax >>
 wrote:

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

[Tom Bentley]

+1 (non-binding). Thanks for the KIP!

On Tue, Jan 14, 2020 at 6:45 PM Randall Hauch  wrote:

> Thanks for the updated KIP, Konstantine. I have a few minor nits, but all
> are around the implementation details.
>
> +1 (binding)
>
> Best regards,
>
> Randall
>
> On Mon, Jan 13, 2020 at 10:16 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Hi everyone.
> >
> > I hope y'all had a nice break. The discussion on KIP-158 seems to have
> > wrapped up since last year, so I'd like to open the vote on this KIP.
> >
> > A reminder that this is an updated KIP-158 (that had also been approved
> in
> > its earlier version) and it seems to be a highly anticipated feature for
> > many of us. I hope we can get this in for the upcoming release.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics
> >
> > Best,
> > Konstantine
> >
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Tom Bentley]

Congratulations!

On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram 
wrote:

> Congratulations Colin, Vahid and Manikumar!
>
> Regards,
> Rajini
>
> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison 
> wrote:
>
> > Congrats Colin, Vahid and Manikumar!
> >
> > On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
> > >
> > > Congratulations Colin, Vahid and Manikumar!
> > >
> > > Ismael
> > >
> > > On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm happy to announce that Colin McCabe, Vahid Hashemian and
> Manikumar
> > > > Reddy are now members of Apache Kafka PMC.
> > > >
> > > > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> > > > 2019. They all contributed many patches, code reviews and
> participated
> > > > in many KIP discussions. We appreciate their contributions and are
> > > > looking forward to many more to come.
> > > >
> > > > Congrats Colin, Vahid and Manikumar!
> > > >
> > > > Gwen, on behalf of Apache Kafka PMC
> > > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Matthias J. Sax]

Congrats!

On 1/14/20 10:49 AM, Rajini Sivaram wrote:
> Congratulations Colin, Vahid and Manikumar!
> 
> Regards,
> Rajini
> 
> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison 
> wrote:
> 
>> Congrats Colin, Vahid and Manikumar!
>>
>> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
>>>
>>> Congratulations Colin, Vahid and Manikumar!
>>>
>>> Ismael
>>>
>>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira  wrote:
>>>
 Hi everyone,

 I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
 Reddy are now members of Apache Kafka PMC.

 Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
 2019. They all contributed many patches, code reviews and participated
 in many KIP discussions. We appreciate their contributions and are
 looking forward to many more to come.

 Congrats Colin, Vahid and Manikumar!

 Gwen, on behalf of Apache Kafka PMC

>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-551: Expose disk read and write metrics

[David Arthur]

+1 binding

This will be very nice to have. Thanks for the KIP, Colin.

-David

On Tue, Jan 14, 2020 at 11:39 AM Sönke Liebau
 wrote:

> +1 (non-binding)
>
> Thanks for creating this!
>
> On Tue, 14 Jan 2020 at 17:36, Mitchell  wrote:
>
> > +1 (non-binding)!
> > Very useful kip.
> > -mitch
> >
> > On Tue, Jan 14, 2020 at 10:26 AM Manikumar 
> > wrote:
> > >
> > > +1 (binding).
> > >
> > > Thanks for the KIP.
> > >
> > >
> > >
> > > On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet 
> > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
> > > >
> > > > > Hey Colin,
> > > > >
> > > > > +1 - Really useful for folks managing a cluster by themselves.
> > > > >
> > > > > M. MAnna
> > > > >
> > > > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio <
> > jsan...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > +1, LGTM.
> > > > > >
> > > > > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> > > > wrote:
> > > > > >
> > > > > > > +1, thanks for driving this
> > > > > > >
> > > > > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe <
> cmcc...@apache.org
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to start the vote on KIP-551: Expose disk read and
> > write
> > > > > > > metrics.
> > > > > > > >
> > > > > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > > > > >
> > > > > > > > Discussion thread:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > > > > >
> > > > > > > > cheers,
> > > > > > > > Colin
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -Jose
> > > > > >
> > > > >
> > > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>


-- 
David Arthur

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Rajini Sivaram]

Congratulations Colin, Vahid and Manikumar!

Regards,
Rajini

On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison 
wrote:

> Congrats Colin, Vahid and Manikumar!
>
> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
> >
> > Congratulations Colin, Vahid and Manikumar!
> >
> > Ismael
> >
> > On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira  wrote:
> >
> > > Hi everyone,
> > >
> > > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
> > > Reddy are now members of Apache Kafka PMC.
> > >
> > > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> > > 2019. They all contributed many patches, code reviews and participated
> > > in many KIP discussions. We appreciate their contributions and are
> > > looking forward to many more to come.
> > >
> > > Congrats Colin, Vahid and Manikumar!
> > >
> > > Gwen, on behalf of Apache Kafka PMC
> > >
>

Build failed in Jenkins: kafka-trunk-jdk8 #4158

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-7538: Reduce lock contention for Partition ISR lock (#5866)


--
[...truncated 2.82 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldStoreAndReturnStateStores PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureOutputRecordsUsingTo PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCaptureOutputRecords 
PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
fullConstructorShouldSetAllExpectedAttributes PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureCommitsAndAllowReset PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildName PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldThrowIfForwardedWithDeprecatedChildIndex PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureApplicationAndRecordMetadata PASSED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName STARTED

org.apache.kafka.streams.MockProcessorContextTest > 
shouldCaptureRecordsOutputToChildByName PASSED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
STARTED

org.apache.kafka.streams.MockProcessorContextTest > shouldCapturePunctuator 
PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Brian Byrne]

Hello all,

After further offline discussion, I've removed any efforts to control
metadata RPC sizes. There are now only two items proposed in this KIP:

(1) When encountering a new topic, only issue a metadata request for that
particular topic. For all other cases, continue as it does today with a
full working set refresh.

(2) Introduces client configuration flag "metadata.eviction.period.ms" to
control cache eviction duration. I've reset the default back to the current
(hard-coded) value of 5 minutes since we can identify cases where changing
it would cause surprises.

The votes have been cleared. My apologies for continually interrupting and
making changes to the KIP, but hopefully this is an agreeable minimum
solution to move forward.

Thanks,
Brian

On Mon, Jan 6, 2020 at 5:23 PM Colin McCabe  wrote:

> On Mon, Jan 6, 2020, at 14:40, Brian Byrne wrote:
> > So the performance of a metadata RPC that occurs every once every 10
> > seconds should not be measured strictly in CPU cost, but rather the
> effect
> > on the 95-99%. The larger the request is, the more opportunity there is
> to
> > put a burst stress on the producer and broker, and the larger the
> response
> > payload to push through the control plane socket. Maybe that's not at 5k
> > topics, but there are groups that are 10k+ topics and pushing further.
>
> KAFKA-7019 made reading the metadata lock-free.  There is no a priori
> reason to prefer lots of small requests to a few big requests (within
> reason!)  In fact, it's quite the opposite: when we make lots of small
> requests, it uses more network bandwidth than when we make a few big ones.
> There are a few reasons for this: the request and response headers have a
> fixed overhead, one big array takes less space when serialized than several
> small ones, etc.  There is also TCP and IP overhead, etc.
>
> The broker can only push a few tens of thousands of metadata requests a
> second, due to the overhead of message processing.  This is why almost all
> of the admin commands support batching.  So if you need to create 1,000
> topics, you make one request, not 1,000 requests, for example.
>
> It's definitely reasonable to limit the number of topics made per metadata
> request.  But the reason for this is not improving performance, but
> preventing certain bad corner cases that happen when RPCs get too big.  For
> example, one problem that can happen when a metadata response gets too big
> is that the client could time out before it finishes reading the response.
> Or if the response got way too big, it could even exceed the maximum
> response size.
>
> So I think the limit should be pretty high here.  We might also consider
> putting the limit in terms of number of partitions rather than number of
> topics, since that's what really matters here (this is harder to implement,
> I realize...)  If I had to put a rough number on it, I'd say we don't want
> more than like 50 MB of response data.  This is vaguely in line with how we
> do fetch responses as well (although I think the limit there is higher).
>
> We should also keep in mind that anyone with a wildcard subscription is
> making full metadata requests, which will return back information about
> every topic in the system.
>
> >
> > There's definitely weight to the metadata RPCs. Looking at a previous
> > local, non-loaded test I ran, I calculate about 2 microseconds per
> > partition latency to the producer. At 10,000 topics with 100 partitions
> > each, that's a full 2-second bubble in the best case. I can rerun a more
> > targeted performance test, but I feel that's missing the point.
> >
>
> If the metadata is fetched in the background, there should be no impact on
> producer latency, right?
>
> It would be good to talk more about the importance of background metadata
> fetching in the KIP.  The fact that we don't do this is actually a big
> problem with the current implementation.  As I understand it, when the
> metadata gets too old, we slam on the brakes and wait for a metadata fetch
> to complete, rather than starting the metadata fetch BEFORE we need it.
> It's just bad scheduling.
>
> best,
> Colin
>
> >
> > Brian
> >
> > On Mon, Jan 6, 2020 at 1:31 PM Colin McCabe  wrote:
> >
> > > On Mon, Jan 6, 2020, at 13:07, Brian Byrne wrote:
> > > > Hi Colin,
> > > >
> > > > Thanks again for the feedback!
> > > >
> > > > On Mon, Jan 6, 2020 at 12:07 PM Colin McCabe 
> wrote:
> > > >
> > > > > Metadata requests don't (always) go to the controller, right?  We
> > > should
> > > > > fix the wording in this section.
> > > > >
> > > >
> > > > You're correct, s/controller/broker(s)/.
> > > >
> > > > I feel like "Proposed Changes" should come before "Public Interfaces"
> > > > > here.  The new configuration won't make sense to the reader until
> he
> > > or she
> > > > > has read the "changes" section.  Also, it's not clear from the name
> > > that
> > > > > "metadata evict" refers to a span of time.  What do you think
> about "
> > > > > 

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

[Randall Hauch]

Thanks for the updated KIP, Konstantine. I have a few minor nits, but all
are around the implementation details.

+1 (binding)

Best regards,

Randall

On Mon, Jan 13, 2020 at 10:16 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Hi everyone.
>
> I hope y'all had a nice break. The discussion on KIP-158 seems to have
> wrapped up since last year, so I'd like to open the vote on this KIP.
>
> A reminder that this is an updated KIP-158 (that had also been approved in
> its earlier version) and it seems to be a highly anticipated feature for
> many of us. I hope we can get this in for the upcoming release.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics
>
> Best,
> Konstantine
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Mickael Maison]

Congrats Colin, Vahid and Manikumar!

On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
>
> Congratulations Colin, Vahid and Manikumar!
>
> Ismael
>
> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira  wrote:
>
> > Hi everyone,
> >
> > I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
> > Reddy are now members of Apache Kafka PMC.
> >
> > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> > 2019. They all contributed many patches, code reviews and participated
> > in many KIP discussions. We appreciate their contributions and are
> > looking forward to many more to come.
> >
> > Congrats Colin, Vahid and Manikumar!
> >
> > Gwen, on behalf of Apache Kafka PMC
> >

[jira] [Resolved] (KAFKA-9395) Improve Kafka scheduler's periodic maybeShrinkIsr()

[Brian Byrne (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Byrne resolved KAFKA-9395.

  Assignee: Rajini Sivaram  (was: Brian Byrne)
Resolution: Done

> Improve Kafka scheduler's periodic maybeShrinkIsr()
> ---
>
> Key: KAFKA-9395
> URL: https://issues.apache.org/jira/browse/KAFKA-9395
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Brian Byrne
>Assignee: Rajini Sivaram
>Priority: Major
>
> The ReplicaManager schedules a periodic call to maybeShrinkIsr() with the 
> KafkaScheduler for a period of replica.lag.time.max.ms / 2. While 
> replica.lag.time.max.ms defaults to 30s, my setup was 45s, which means 
> maybeShrinkIsr() was being called every 22.5 seconds. Normally this is not a 
> problem.
> Fetch/produce requests hold a partition's leaderIsrUpdateLock in reader mode 
> while they are running. When a partition is requested to check whether it 
> should shrink its ISR, it acquires a write lock. So there's potential for 
> contention here, and if the fetch/produce requests are long running, they may 
> block maybeShrinkIsr() for hundreds of ms.
> This becomes a problem due to the way the scheduler runnable is set up: it 
> calls maybeShrinkIsr() for partition per single scheduler invocation. If 
> there's a lot of partitions, this could take many seconds, even minutes. 
> However, the runnable is scheduled via 
> ScheduledThreadPoolExecutor#scheduleAtFixedRate, which means if it exceeds 
> its period, it's immediately scheduled to run again. So it backs up enough 
> that the scheduler is always executing this function.
> This may cause partitions to periodically check their ISR a lot less 
> frequently than intended. This also contributes a huge source of contention 
> for cases where the produce/fetch requests are long-running.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Ismael Juma]

Congratulations Colin, Vahid and Manikumar!

Ismael

On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira  wrote:

> Hi everyone,
>
> I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
> Reddy are now members of Apache Kafka PMC.
>
> Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> 2019. They all contributed many patches, code reviews and participated
> in many KIP discussions. We appreciate their contributions and are
> looking forward to many more to come.
>
> Congrats Colin, Vahid and Manikumar!
>
> Gwen, on behalf of Apache Kafka PMC
>

[jira] [Created] (KAFKA-9431) Expose API in KafkaStreams to fetch all local offset lags

[Vinoth Chandar (Jira)]

Vinoth Chandar created KAFKA-9431:
-

 Summary: Expose API in KafkaStreams to fetch all local offset lags
 Key: KAFKA-9431
 URL: https://issues.apache.org/jira/browse/KAFKA-9431
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Vinoth Chandar
Assignee: Vinoth Chandar
 Fix For: 2.5.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

[Colin McCabe]

Hi all,

I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access in 
Kafka Administrative Tools.

KIP:  https://cwiki.apache.org/confluence/x/Wg6dC

Discussion thread: 
https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E

cheers,
Colin

[jira] [Created] (KAFKA-9430) Tighten up lag estimates when source topic optimization is on

[Vinoth Chandar (Jira)]

Vinoth Chandar created KAFKA-9430:
-

 Summary: Tighten up lag estimates when source topic optimization 
is on 
 Key: KAFKA-9430
 URL: https://issues.apache.org/jira/browse/KAFKA-9430
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vinoth Chandar
Assignee: Vinoth Chandar


Right now, we use _endOffsets_ of the source topic for the computation. Since 
the source topics can also have user event produces, this is an over estimate

 

>From John:

For "optimized" changelogs, this will be wrong, strictly speaking, but it's an 
over-estimate (which seems better than an under-estimate), and it's also still 
an apples-to-apples comparison, since all replicas would use the same upper 
bound to compute their lags, so the "pick the freshest" replica is still going 
to pick the right one. We can add a new 2.5 blocker ticket to really fix it, 
and not worry about it until after this KSQL stuff is done.

 

For active: we need to use  consumed offsets and not end of source topic



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9429) Allow ability to control whether stale reads out of state stores are desirable

[Vinoth Chandar (Jira)]

Vinoth Chandar created KAFKA-9429:
-

 Summary: Allow ability to control whether stale reads out of state 
stores are desirable
 Key: KAFKA-9429
 URL: https://issues.apache.org/jira/browse/KAFKA-9429
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vinoth Chandar
Assignee: Vinoth Chandar
 Fix For: 2.5.0


>From John :

 

I also meant to talk with you about the change to allow querying recovering 
stores. I think you might have already talked with Matthias a little about this 
in the scope of KIP-216, but it's probably not ok to just change the default 
from only allowing query while running, since there are actually people 
depending on full-consistency queries for correctness right now.

 

What we can do is add an overload {{KafkaStreams.store(name, 
QueriableStoreType, QueriableStoreOptions)}}, with one option: 
{{queryStaleState(true/false)}} (your preference on the name, I just made that 
up right now). The default would be false, and KSQL would set it to true. While 
false, it would not allow querying recovering stores OR standbys. This 
basically allows a single switch to preserve existing behavior.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

[Colin McCabe]

Hi Dhruvil,

Thanks for the KIP.  I think there should be some way to turn this off, in case 
that becomes necessary.  I'm also curious how long we intend to wait between 
detecting the duplication and  deleting the extra logs.  The KIP says 
"scheduled for deletion" but doesn't give a time frame -- is it assumed to be 
immediate?

best,
Colin


On Tue, Jan 14, 2020, at 05:56, Dhruvil Shah wrote:
> If there are no more questions or concerns, I will start a vote thread
> tomorrow.
> 
> Thanks,
> Dhruvil
> 
> On Mon, Jan 13, 2020 at 6:59 PM Dhruvil Shah  wrote:
> 
> > Hi Nikhil,
> >
> > Thanks for looking at the KIP. The kind of race condition you mention is
> > not possible as stray partition detection is done synchronously while
> > handling the LeaderAndIsrRequest. In other words, we atomically evaluate
> > the partitions the broker must host and the extra partitions it is hosting
> > and schedule deletions based on that.
> >
> > One possible shortcoming of the KIP is that we do not have the ability to
> > detect a stray partition if the topic has been recreated since. We will
> > have the ability to disambiguate between different generations of a
> > partition with KIP-516.
> >
> > Thanks,
> > Dhruvil
> >
> > On Sat, Jan 11, 2020 at 11:40 AM Nikhil Bhatia 
> > wrote:
> >
> >> Thanks Dhruvil, the proposal looks reasonable to me.
> >>
> >> is there a potential of a race between a new topic being assigned to the
> >> same node that is still performing a cleanup of the stray partition ?
> >> Topic
> >> ID will definitely solve this issue.
> >>
> >> Thanks
> >> Nikhil
> >>
> >> On 2020/01/06 04:30:20, Dhruvil Shah  wrote:
> >> > Here is the link to the KIP:>
> >> >
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
> >> >
> >>
> >> >
> >> > On Mon, Jan 6, 2020 at 9:59 AM Dhruvil Shah 
> >> wrote:>
> >> >
> >> > > Hi all, I would like to kick off discussion for KIP-550 which proposes
> >> a>
> >> > > mechanism to detect and delete stray partitions on a broker.
> >> Suggestions>
> >> > > and feedback are welcome.>
> >> > >>
> >> > > - Dhruvil>
> >> > >>
> >> >
> >>
> >
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Bill Bejeck]

Congrats Colin, Vahid and Manikumar! Well deserved.

-Bill

On Tue, Jan 14, 2020 at 12:30 PM Gwen Shapira  wrote:

> Hi everyone,
>
> I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
> Reddy are now members of Apache Kafka PMC.
>
> Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> 2019. They all contributed many patches, code reviews and participated
> in many KIP discussions. We appreciate their contributions and are
> looking forward to many more to come.
>
> Congrats Colin, Vahid and Manikumar!
>
> Gwen, on behalf of Apache Kafka PMC
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Stanislav Kozlovski]

Congratulations to all!

Best,
Stanislav

On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira  wrote:

> Hi everyone,
>
> I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
> Reddy are now members of Apache Kafka PMC.
>
> Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> 2019. They all contributed many patches, code reviews and participated
> in many KIP discussions. We appreciate their contributions and are
> looking forward to many more to come.
>
> Congrats Colin, Vahid and Manikumar!
>
> Gwen, on behalf of Apache Kafka PMC
>


-- 
Best,
Stanislav

[jira] [Created] (KAFKA-9428) Expose standby information in KafkaStreams via queryMetadataForKey API

[Vinoth Chandar (Jira)]

Vinoth Chandar created KAFKA-9428:
-

 Summary: Expose standby information in KafkaStreams via 
queryMetadataForKey API
 Key: KAFKA-9428
 URL: https://issues.apache.org/jira/browse/KAFKA-9428
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vinoth Chandar
Assignee: Vinoth Chandar
 Fix For: 2.5.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Gwen Shapira]

Hi everyone,

I'm happy to announce that Colin McCabe, Vahid Hashemian and Manikumar
Reddy are now members of Apache Kafka PMC.

Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
2019. They all contributed many patches, code reviews and participated
in many KIP discussions. We appreciate their contributions and are
looking forward to many more to come.

Congrats Colin, Vahid and Manikumar!

Gwen, on behalf of Apache Kafka PMC

Re: Streams, Kafka windows

[Sachin Mittal]

You can try to convert the final resultant stream to table.
Check this page for more info:
https://docs.confluent.io/current/streams/faq.html#how-can-i-convert-a-kstream-to-a-ktable-without-an-aggregation-step

This way table would always contain the latest (single) record for a given
key.

Sachin




On Tue, Jan 14, 2020 at 10:11 PM Viktor Markvardt <
viktor.markva...@gmail.com> wrote:

> Hi,
>
> My name is Viktor. I'm currently working with Kafka streams and have
> several questions about Kafka and I can not find answers in the official
> docs.
>
> 1. Why suppress functionality does not work with Hopping windows? How to
> make it work?
>
> Example of the code:
>
> KStream finalStream = source
> .groupByKey()
>
>
> .windowedBy(TimeWindows.of(Duration.ofSeconds(30)).advanceBy(Duration.ofSeconds(10)))
> .reduce((aggValue, newValue) -> newValue,
> Materialized.with(Serdes.String(), Serdes.String()))
>
>
> .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
> .toStream();
>
> finalStream.print(Printed.toSysOut());
> finalStream.to(outputTopic);
>
> After I run the code above - output stream is empty. There were no
> errors/exceptions.
> NOTE: With Tumbling Window the code working as expected.
> Maybe I simply use it incorrectly?
>
> 2. Why with Hopping windows (without suppress) there are duplicates in the
> output stream?
> E.g., I send one record in the input kstream with Hopping window
> (duration=30s, advanceBy=2s) but get two same records (duplicate) in the
> output kstream.
> Is that an expected behavior? If so, how can I filter/switch off these
> duplicates?
>
> 3. Mainly I'm trying to solve this problem:
> I have kstream with events inside and events can be repeated (duplicates).
> In the output kstream I would like to receive only unique events for the
> last 24 hours (window duration) with 1 hour window overlay (window
> advanceBy).
> Could you recommend me any examples of code or docs please?
> I have already read official docs and examples but it was not enough to get
> full understanding of how I can achieve this.
>
> Best regards,
> Viktor Markvardt
>

Re: [DISCUSS] KIP-555: Deprecate Direct Zookeeper access in Kafka Administrative Tools

[Colin McCabe]

Hi Ron,

Yes, the KIP-455 command line changes will be in 2.5.

regards,
Colin


On Sat, Jan 11, 2020, at 03:21, Ron Dagostino wrote:
> << Wow, I totally missed that!  Great.
> 
> Would the deprecation for kafka-reassign-partitions depend on KIP 455 
> being merged?
> 
> Ron
> 
> > On Jan 10, 2020, at 8:25 PM, Mitchell  wrote:
> > 
> > I"m very pro this.  The sooner we can make people aware that zookeeper
> > is going away the better.
> > -mitch
> > 
> >> On Fri, Jan 10, 2020 at 4:37 PM Jose Garcia Sancio  
> >> wrote:
> >> 
> >> Comments below.
> >> 
> >>> On Thu, Jan 9, 2020 at 5:03 PM Colin McCabe  wrote:
> >>> 
> >>> That's a good question.  The current plan is for the 3.x releases to still
> >>> require ZooKeeper.  What we will drop in 3.x is direct ZK access in
> >>> command-line administrative tools (unless those tools are specifically
> >>> about administering ZK itself, like launching it, stopping it, editing its
> >>> internal settings, etc.)
> >> Makes sense to me. Thanks for the clarification.
> >> 
> >> --
> >> -Jose
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Colin McCabe]

Hi Ron,

Thanks for the explanation.  I guess thinking about it a little bit more, we 
should just add --zk-tls-config-file to all of these commands.

We will be removing this option (plus ZK in general) from these commands in the 
next major release, but ZK is still supported in 2.5, so we should just do the 
logical thing.  (The exception is ZkSecurityMigrator, which will stay).

best,
Colin


On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> Hi Colin.
> 
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
> 
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
> 
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
> 
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
> 
> Ron
> 
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through.  I am trying to get emails through now to move
> this discussion forward.
> 
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe  wrote:
> 
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone.  I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this one
> > would not make sense, since it would defeat the whole point of the tool :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here.  KIP-455
> > implementation has dragged a bit, but this should get done soon.  Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well.  I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > >   a) Kafka brokers
> > >   b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > >   zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > >   zookeeper.clientCnxnSocket
> > >   zookeeper.ssl.keyStore.location
> > >   zookeeper.ssl.keyStore.password
> > >   zookeeper.ssl.trustStore.location
> > >   zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one the
> > > brokers use.  We therefore also add the following optional configs, which
> > > override the corresponding ones from above when present:
> > >   authorizer.zookeeper.client.secure
> > >   authorizer.zookeeper.clientCnxnSocket
> > >   authorizer.zookeeper.ssl.keyStore.location
> > >   authorizer.zookeeper.ssl.keyStore.password
> > >   authorizer.zookeeper.ssl.trustStore.location
> 

[jira] [Created] (KAFKA-9427) StateRestoreListener.onRestoreEnd should return actual message count

[Chris Stromberger (Jira)]

Chris Stromberger created KAFKA-9427:


 Summary: StateRestoreListener.onRestoreEnd should return actual 
message count
 Key: KAFKA-9427
 URL: https://issues.apache.org/jira/browse/KAFKA-9427
 Project: Kafka
  Issue Type: Improvement
Reporter: Chris Stromberger


{{StateRestoreListener.onRestoreEnd appears to report the difference between 
offsets as "totalRestored", which may differ from the actual number of messages 
restored to a state store}}{{. Am assuming this is due to missing offsets in 
compacted topics. It would be more useful if 
}}{{StateRestoreListener.onRestoreEnd}}{{ reported the actual count of messages 
restored (sum of values reported by }}{{StateRestoreListener.onBatchRestored). 
}}

Was asked to create this ticket in Slack thread 
[https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1578956151094200]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[smiklos]

Hi,

Is there any update on this? I've done performance test with Avro data 
and Snappy compression.


Setting the buffer from 32kb to 128kb brings a rough 10% decrease in 
storage which is a big deal.


I could offer working on this as well.

Best regards,

Miklos

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Ron Dagostino]

Hi Rajini.  Thanks for pointing out kafka-acls as another bootstrap use
case. I will update the KIP to mention it.  (BTW, I sent an email on 1/8
stating that I had updated the KIP to reflect the current understanding and
discussion, but that email did not go through -- the KIP has undergone a
major rewrite, so interested parties should definitely take a look).

It is easy to support the bootstrap use case as I described previously, so
I think we should support it in both kafka-acls and kafka-configs.

Regarding SASL + SSL simultaneously, yes, the two can be used together in
ZooKeeper.  I will be sure to state this in any documentation, and I will
update the KIP to fully describe how this works, including the principal
that is used and whether there is any way to transform the DN in the
certificate.  Here is a quick summary:

1) ZooKeeper 3.5.6 requires a client certificate.  There is a sever-side
config "ssl.clientAuth" that the code recognizes (case-insensitively:
want/need/none are the valid options), but this config has no effect in
3.5.6 (https://issues.apache.org/jira/browse/ZOOKEEPER-3674).  I built from
source recently and confirmed that it worked in the 3.6 SNAPSHOT, but that
version is not yet released.
2) ZooKeeper will associate multiple identities with any session that
authenticates multiple ways (e.g. SSL and SASL).  A client is authorized if
it has at least 1 of the identities present in an ACL.
3) The SSL identity is the DN, and that can only be changed by implementing
and using a custom ZK authentication provider that overrides the
method protected
String getClientId(X509Certificate clientCert).

Given the above, if clients want to use client certificate authentication
with SASL, then there really isn't any difficulty -- brokers and ZK
security migrator will be identified by both their respective DNs and the
SASL identity, and at least the SASL identity will be common to all clients
as it is today, so it continues to work fine.

If clients want to use client certificate authentication without SASL
(which is I believe what Pere intended in the original KIP discussion),
then either all brokers and ZK security migrator will have to present the
same DN (with different SANs for the different hostnames) or ZooKeeper will
have to be running with a custom authProvider that maps the different DNs
to a single, common identity (this second approach is more difficult and
seems to be less secure than the first one that uses SANs since it creates
the possibility of the DN mapping being leveraged in a malicious way).

Since I am catching up on emails I sent that did not go through, I'll also
include the following two comments here:

1) The ZooKeeper project's command line interface has no facility for
accepting client TLS configuration except via Java system properties.  I've
opened ZooKeeper issue https://issues.apache.org/jira/browse/ZOOKEEPER-3689 for
this, but in the meantime we can address it in Kafka's
zookeeper-shell.{bat,sh} scripts by invoking a new, Kafka-specific subclass
of org.apache.zookeeper.ZooKeeperMain as opposed
to org.apache.zookeeper.ZooKeeperMain itself.  We would simply look for a
"-zk-tls-config-file " argument pair, create the TLS client config if
specified, and pass the arguments without the config information to the
superclass; then we would override the connectToZK(String) method to use
any created TLS client config.  While the subclass unfortunately needs to
be in the same org.apache.zookeeper package due to the package-private
run() method in ZooKeeperMain that it has to call, it is working fine
(confirmed via a system test).

2) Regarding Thomas Zhou's question on 1/10 about changing the way client
certificates are loaded (
https://lists.apache.org/thread.html/r3bf6c58b052bf47bc99c360dff86a8bd38c35a89a7387ba2a5f16dee%40%3Cdev.kafka.apache.org%3E),
when I updated the KIP I included all of the ZooKeeper-supported
client configurations related to TLS (not just keystore/trsustore locations
and passwords).  There is no intention to support anything beyond what
ZooKeeper supports, but the "zookeeper.ssl.context.supplier.class"
configuration is included and is useful for this particular use case.

Ron


On Tue, Jan 14, 2020 at 10:48 AM Rajini Sivaram 
wrote:

> Resending this note which wasn't delivered when it was sent on Jan 6th:
>
> Hi Ron,
>
> Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
> for ZooKeeper like we do in Kafka? If not, we need to describe that in the
> migration section. We should also document the principals used for
> authorization when SSL is used and any additional configs that may be used
> for transforming the DN from the certificate.
>
> For the CLI tools, the two that use ZooKeeper for bootstrapping are
> kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
> connectivity to ZooKeeper in order to store configs/ACLs prior to starting
> brokers. Are we going to support SSL for both?
>
> Regards,
>
> Rajini
>
> 

Streams, Kafka windows

[Viktor Markvardt]

Hi,

My name is Viktor. I'm currently working with Kafka streams and have
several questions about Kafka and I can not find answers in the official
docs.

1. Why suppress functionality does not work with Hopping windows? How to
make it work?

Example of the code:

KStream finalStream = source
.groupByKey()

.windowedBy(TimeWindows.of(Duration.ofSeconds(30)).advanceBy(Duration.ofSeconds(10)))
.reduce((aggValue, newValue) -> newValue,
Materialized.with(Serdes.String(), Serdes.String()))

.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
.toStream();

finalStream.print(Printed.toSysOut());
finalStream.to(outputTopic);

After I run the code above - output stream is empty. There were no
errors/exceptions.
NOTE: With Tumbling Window the code working as expected.
Maybe I simply use it incorrectly?

2. Why with Hopping windows (without suppress) there are duplicates in the
output stream?
E.g., I send one record in the input kstream with Hopping window
(duration=30s, advanceBy=2s) but get two same records (duplicate) in the
output kstream.
Is that an expected behavior? If so, how can I filter/switch off these
duplicates?

3. Mainly I'm trying to solve this problem:
I have kstream with events inside and events can be repeated (duplicates).
In the output kstream I would like to receive only unique events for the
last 24 hours (window duration) with 1 hour window overlay (window
advanceBy).
Could you recommend me any examples of code or docs please?
I have already read official docs and examples but it was not enough to get
full understanding of how I can achieve this.

Best regards,
Viktor Markvardt

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Sönke Liebau]

+1 (non-binding)

Thanks for creating this!

On Tue, 14 Jan 2020 at 17:36, Mitchell  wrote:

> +1 (non-binding)!
> Very useful kip.
> -mitch
>
> On Tue, Jan 14, 2020 at 10:26 AM Manikumar 
> wrote:
> >
> > +1 (binding).
> >
> > Thanks for the KIP.
> >
> >
> >
> > On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet 
> wrote:
> >
> > > +1 (non binding)
> > >
> > > On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
> > >
> > > > Hey Colin,
> > > >
> > > > +1 - Really useful for folks managing a cluster by themselves.
> > > >
> > > > M. MAnna
> > > >
> > > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio <
> jsan...@confluent.io>
> > > > wrote:
> > > >
> > > > > +1, LGTM.
> > > > >
> > > > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> > > wrote:
> > > > >
> > > > > > +1, thanks for driving this
> > > > > >
> > > > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe  >
> > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start the vote on KIP-551: Expose disk read and
> write
> > > > > > metrics.
> > > > > > >
> > > > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > > > >
> > > > > > > Discussion thread:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > > > >
> > > > > > > cheers,
> > > > > > > Colin
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
>


-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Mitchell]

+1 (non-binding)!
Very useful kip.
-mitch

On Tue, Jan 14, 2020 at 10:26 AM Manikumar  wrote:
>
> +1 (binding).
>
> Thanks for the KIP.
>
>
>
> On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet  wrote:
>
> > +1 (non binding)
> >
> > On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
> >
> > > Hey Colin,
> > >
> > > +1 - Really useful for folks managing a cluster by themselves.
> > >
> > > M. MAnna
> > >
> > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio 
> > > wrote:
> > >
> > > > +1, LGTM.
> > > >
> > > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> > wrote:
> > > >
> > > > > +1, thanks for driving this
> > > > >
> > > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe 
> > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to start the vote on KIP-551: Expose disk read and write
> > > > > metrics.
> > > > > >
> > > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > > >
> > > > > > Discussion thread:
> > > > > >
> > > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > >
> > > >
> > > >
> > > > --
> > > > -Jose
> > > >
> > >
> >

Build failed in Jenkins: kafka-trunk-jdk8 #4157

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Fix connect:mirror checkstyle (#7951)

[wangguoz] KAFKA-9159: The caller of sendListOffsetRequest need to handle 
retriable

[wangguoz] KAFKA-9346: Consumer back-off logic when fetching pending offsets


--
[...truncated 5.68 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > 

[jira] [Resolved] (KAFKA-6212) Kafka Streams - Incorrect partition rebalancing

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6212.
--
Resolution: Cannot Reproduce

> Kafka Streams - Incorrect partition rebalancing
> ---
>
> Key: KAFKA-6212
> URL: https://issues.apache.org/jira/browse/KAFKA-6212
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.1
> Environment: Linux
>Reporter: Ivan Atanasov
>Priority: Major
>
> Trying to use streaming with version 0.10.0.1 of kafka but it is not working 
> how I'd expect. I realize that this is a fairly old version now but it is 
> what we are running and are not in a position to upgrade right now.
> The particular problem I am having is when an extra instance of the streaming 
> app is run using the same application ID. What seems to happen is the newly 
> introduced instance takes half of the partitions available, which is expected 
> but the original instance drops all the partitions it was reading from. 
> therefore from then on, data is only read from half the partitions.
> Strangely offsets are still being committed for the other partitions but the 
> data from them is not consumed as expected.
> My topology is very simple for now, all it does is a print of the message. 
> Also I have tried making both instances use different client IDs and state 
> directories.
> Is this a known bug in 0.10.0.1?
> *Logs Below:*
> Instance 1:
> {quote}[2017-11-15 10:41:41,597] INFO [StreamThread-1] Setting newly assigned 
> partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, 
> rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group 
> kafka-stream-test 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:41:41,616] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_0 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_1 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_2 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_3 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_4 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_5 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_6 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_7 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_8 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:41:41,648] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_9 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Revoking previously assigned 
> partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9, 
> rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group 
> kafka-stream-test 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,682] INFO [StreamThread-1] Removing a task 0_0 
> (org.apache.kafka.streams.processor.internals.StreamThread){quote}
> Instance 2:
> {quote}[2017-11-15 10:42:08,827] INFO [StreamThread-1] Successfully joined 
> group kafka-stream-test with generation 2 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2017-11-15 10:42:08,829] INFO [StreamThread-1] Setting newly assigned 
> partitions [rawEvents-5, rawEvents-3, rawEvents-1, rawEvents-2, rawEvents-0] 
> for group kafka-stream-test 
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2017-11-15 10:42:08,840] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_0 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,869] INFO [StreamThread-1] Creating restoration consumer 
> client for stream task #0_1 
> (org.apache.kafka.streams.processor.internals.StreamTask)
> [2017-11-15 10:42:08,870] INFO [StreamThread-1] 

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Manikumar]

+1 (binding).

Thanks for the KIP.



On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet  wrote:

> +1 (non binding)
>
> On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
>
> > Hey Colin,
> >
> > +1 - Really useful for folks managing a cluster by themselves.
> >
> > M. MAnna
> >
> > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio 
> > wrote:
> >
> > > +1, LGTM.
> > >
> > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> wrote:
> > >
> > > > +1, thanks for driving this
> > > >
> > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe 
> > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to start the vote on KIP-551: Expose disk read and write
> > > > metrics.
> > > > >
> > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > >
> > > > > Discussion thread:
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > >
> > > > > cheers,
> > > > > Colin
> > > >
> > >
> > >
> > > --
> > > -Jose
> > >
> >
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Rajini Sivaram]

Resending this note which wasn't delivered when it was sent on Jan 6th:

Hi Ron,

Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
for ZooKeeper like we do in Kafka? If not, we need to describe that in the
migration section. We should also document the principals used for
authorization when SSL is used and any additional configs that may be used
for transforming the DN from the certificate.

For the CLI tools, the two that use ZooKeeper for bootstrapping are
kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
connectivity to ZooKeeper in order to store configs/ACLs prior to starting
brokers. Are we going to support SSL for both?

Regards,

Rajini

On Tue, Jan 14, 2020 at 3:39 PM Ron Dagostino  wrote:

> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through.  I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe  wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone.  I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for
> TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > would not make sense, since it would defeat the whole point of the tool
> :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > implementation has dragged a bit, but this should get done soon.
> Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well.  I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > >   a) Kafka brokers
> > >   b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > >   zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > >   zookeeper.clientCnxnSocket
> > >   zookeeper.ssl.keyStore.location
> > >   zookeeper.ssl.keyStore.password
> > >   zookeeper.ssl.trustStore.location
> > >   zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one
> the
> > > brokers use.  We therefore also add the following optional configs,
> which
> 

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Ron Dagostino]

Hi Colin.

<<< It seems like this [--zk-tls-config-file information] could just appear
in a configuration file, which all of these tools already accept (I think)

ZkSecurityMigrator has no such property file facility; adding a
"--zk-tls-config-file" parameter is exactly for this purpose.  If we add
that to ZkSecurityMigrator then it is trivial to add it to other commands
(the same code is simply reused; it ends up being just a few extra lines).
I do not see any parameter in the other two commands to adjust the ZK
connection; ConfigCommand accepts a "--command-config" flag, but according
to the code "This is used only with --bootstrap-server option for
describing and altering broker configs."

I do agree there would be no need to add "--zk-tls-config-file" to
ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
for the next release.

ConfigCommand supports the "--bootstrap-server" option and will have its
direct ZooKeeper access formally deprecated as per KIP-555, but the special
use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
prior to starting Kafka will still exist, so it feels like while
"--zk-tls-config-file" would never be used except for this use case it
could probably still be added for this particular situation.

Ron

P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
sent) did not go through.  I am trying to get emails through now to move
this discussion forward.

On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe  wrote:

> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > Hi everyone.  I would like to make the following changes to the KIP.
> >
> > MOTIVATION:
> > Include a statement that it will be difficult in the short term to
> > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
> > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > with encrypted passwords in Zookeeper is an explicitly-supported use
> case;
> > therefore it is in scope to be able to securely configure the CLI tools
> > that still leverage non-deprecated direct Zookeeper communication for TLS
> > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > zookeeper-security-migration.sh).
>
> Hi Ron,
>
> Thanks for the KIP.
>
> About deprecations:
>
> * zookeeper-security-migration: clearly, deprecating ZK access in this one
> would not make sense, since it would defeat the whole point of the tool :)
>
> * kafka-reassign-partitions: ZK access should be deprecated here.  KIP-455
> implementation has dragged a bit, but this should get done soon.  Certainly
> before the next release.
>
> * kafka-configs: I think ZK access should be deprecated here as well.  I
> agree there is a super-special bootstrapping case here, but that should
> have its own tool, not use kafka-configs.
>
> I will post a separate KIP for this, though.
>
> >
> > GOALS:
> > Support the secure configuration of TLS-encrypted communication between
> > Zookeeper and:
> >   a) Kafka brokers
> >   b) The three CLI tools mentioned above that still support direct,
> > non-deprecated communication to Zookeeper
> > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > communication in CLI tools as part of this KIP; such work will occur in
> > future KIPs instead.
> >
> > PUBLIC INTERFACES:
> > 1) The following new broker configurations will be recognized.
> >   zookeeper.client.secure (default value = false, for backwards
> > compatibility)
> >   zookeeper.clientCnxnSocket
> >   zookeeper.ssl.keyStore.location
> >   zookeeper.ssl.keyStore.password
> >   zookeeper.ssl.trustStore.location
> >   zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
> >
> > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> supports
> > the ability to connect to a different Zookeeper instance than the one the
> > brokers use.  We therefore also add the following optional configs, which
> > override the corresponding ones from above when present:
> >   authorizer.zookeeper.client.secure
> >   authorizer.zookeeper.clientCnxnSocket
> >   authorizer.zookeeper.ssl.keyStore.location
> >   authorizer.zookeeper.ssl.keyStore.password
> >   authorizer.zookeeper.ssl.trustStore.location
> >   authorizer.zookeeper.ssl.trustStore.password
> >
> > 3) The three CLI tools mentioned above will support a new
> --zk-tls-config-file
> > " option.  The following
> > properties will be recognized in that file, and unrecognized properties
> > will be ignored to allow the possibility of pointing zk-tls-config-file
> at
> > the broker's config file.
> >   zookeeper.client.secure (default value = false)
> >   zookeeper.clientCnxnSocket
> >   zookeeper.ssl.keyStore.location
> >   zookeeper.ssl.keyStore.password
> >   zookeeper.ssl.trustStore.location
> >   zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to 

[jira] [Created] (KAFKA-9426) OffsetsForLeaderEpochClient Use Switch Statement

[David Mollitor (Jira)]

David Mollitor created KAFKA-9426:
-

 Summary: OffsetsForLeaderEpochClient Use Switch Statement
 Key: KAFKA-9426
 URL: https://issues.apache.org/jira/browse/KAFKA-9426
 Project: Kafka
  Issue Type: Improvement
Reporter: David Mollitor


Use switch statement for Error Code Enum handling.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9425) InFlightRequests Class Uses Thread-Safe Counter Non-Thread-Safe Collection

[David Mollitor (Jira)]

David Mollitor created KAFKA-9425:
-

 Summary: InFlightRequests Class Uses Thread-Safe Counter 
Non-Thread-Safe Collection
 Key: KAFKA-9425
 URL: https://issues.apache.org/jira/browse/KAFKA-9425
 Project: Kafka
  Issue Type: Improvement
Reporter: David Mollitor


[https://github.com/apache/kafka/blob/d6ace7b2d7c4ad721dd8247fb2b3eff9f67fbfee/clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java#L34-L36]

 

Not sure why this needs the overheard of {{AtomicInteger}} when the collection 
being modified isn't itself thread-safe.  The comment of the counter says that 
it may lag as things currently stand.

 

Also, add a few more niceties since I'm opening it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

[Sean Glover]

Hi Habib,

Thank you for the reminder.  I'll update the KIP this week and address the
feedback from you and Gokul.

Regards,
Sean

On Tue, Jan 14, 2020 at 9:06 AM Habib Nahas  wrote:

> Any chance of an update on the KIP? We are interested in seeing this move
> forward.
>
> Thanks,
> Habib
> Sr SDE, AWS
>
> On Wed, Dec 18, 2019, at 3:27 PM, Habib Nahas wrote:
> > Thanks Sean. Look forward to the updated KIP.
> >
> > Regards,
> > Habib
> >
> > On Fri, Dec 13, 2019, at 6:22 AM, Sean Glover wrote:
> > > Hi,
> > >
> > > After my last reply I had a nagging feeling something wasn't right,
> and I
> > > remembered that epoch time is UTC. This makes the discussion about
> > > timezone irrelevant, since we're always using UTC. This makes the need
> for
> > > the LatencyTime interface that I proposed in the design irrelevant as
> well,
> > > since I can no longer think about how that might be useful. I'll update
> > > the KIP. I'll also review KIP-32 to understand message timestamps
> better
> > > so I can explain the different types of latency results that could be
> > > reported with this metric.
> > >
> > > Regards,
> > > Sean
> > >
> > > On Thu, Dec 12, 2019 at 6:25 PM Sean Glover  >
> > > wrote:
> > >
> > > > Hi Habib,
> > > >
> > > > Thanks for question! If the consumer is in a different timezone than
> the
> > > > timezone used to produce messages to a partition then you can use an
> > > > implementation of LatencyTime to return the current time of that
> timezone.
> > > > The current design assumes that messages produced to a partition
> must all
> > > > be produced from the same timezone. If timezone metadata were
> encoded into
> > > > each message then it would be possible to automatically determine the
> > > > source timezone and calculate latency, however the current design
> will not
> > > > pass individual messages into LatencyTime to retrieve message
> metadata.
> > > > Instead, the LatencyTime.getWallClockTime method is only called once
> per
> > > > fetch request response per partition and then the metric is recorded
> once
> > > > the latency calculation is complete. This follows the same design as
> the
> > > > current consumer lag metric which calculates offset lag based on the
> last
> > > > message of the fetch request response for a partition. Since the
> metric is
> > > > just an aggregate (max/mean) over some time window we only need to
> > > > occasionally calculate latency, which will have negligible impact on
> the
> > > > performance of consumer polling.
> > > >
> > > > A simple implementation of LatencyTime that returns wall clock time
> for
> > > > the Asia/Singapore timezone for all partitions:
> > > >
> > > > import java.time.*;
> > > >
> > > > class SingaporeTime implements LatencyTime {
> > > > ZoneId zoneSingapore = ZoneId.of("Asia/Singapore");
> > > > Clock clockSingapore = Clock.system(zoneSingapore);
> > > >
> > > > @Override
> > > > public long getWallClockTime(TopicPartition tp) {
> > > > return clockSingapore.instant.getEpochSecond();
> > > > }
> > > >
> > > > ...
> > > > }
> > > >
> > > > Regards,
> > > > Sean
> > > >
> > > >
> > > > On Thu, Dec 12, 2019 at 6:18 AM Habib Nahas  wrote:
> > > >
> > > >> Hi Sean,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> As I understand it users are free to set their own timestamp on
> > > >> ProducerRecord. What is the recommendation for the proposed metric
> in a
> > > >> scenario where the user sets this timestamp in timezone A and
> consumes the
> > > >> record in timezone B. Its not clear to me if a custom
> implementation of
> > > >> LatencyTime will help here.
> > > >>
> > > >> Thanks,
> > > >> Habib
> > > >>
> > > >> On Wed, Dec 11, 2019, at 4:52 PM, Sean Glover wrote:
> > > >> > Hello again,
> > > >> >
> > > >> > There has been some interest in this KIP recently. I'm bumping the
> > > >> thread
> > > >> > to encourage feedback on the design.
> > > >> >
> > > >> > Regards,
> > > >> > Sean
> > > >> >
> > > >> > On Mon, Jul 15, 2019 at 9:01 AM Sean Glover <
> sean.glo...@lightbend.com>
> > > >> > wrote:
> > > >> >
> > > >> > > To hopefully spark some discussion I've copied the motivation
> section
> > > >> from
> > > >> > > the KIP:
> > > >> > >
> > > >> > > Consumer lag is a useful metric to monitor how many records are
> > > >> queued to
> > > >> > > be processed. We can look at individual lag per partition or we
> may
> > > >> > > aggregate metrics. For example, we may want to monitor what the
> > > >> maximum lag
> > > >> > > of any particular partition in our consumer subscription so we
> can
> > > >> identify
> > > >> > > hot partitions, caused by an insufficient producing partitioning
> > > >> strategy.
> > > >> > > We may want to monitor a sum of lag across all partitions so we
> have a
> > > >> > > sense as to our total backlog of messages to consume. Lag in
> offsets
> > > >> is
> > > >> > > useful when you have a good understanding of your messages and
> > > >> processing
> > > >> > > characteristics, but it 

[jira] [Resolved] (KAFKA-7538) Improve locking model used to update ISRs and HW

[Rajini Sivaram (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-7538.
---
Fix Version/s: 2.5.0
 Reviewer: Jason Gustafson
   Resolution: Fixed

> Improve locking model used to update ISRs and HW
> 
>
> Key: KAFKA-7538
> URL: https://issues.apache.org/jira/browse/KAFKA-7538
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.5.0
>
>
> We currently use a ReadWriteLock in Partition to update ISRs and high water 
> mark for the partition. This can result in severe lock contention if there 
> are multiple producers writing a large amount of data into a single partition.
> The current locking model is:
>  # read lock while appending to log on every Produce request on the request 
> handler thread
>  # write lock on leader change, updating ISRs etc. on request handler or 
> scheduler thread
>  # write lock on every replica fetch request to check if ISRs need to be 
> updated and to update HW and ISR on the request handler thread
> 2) is infrequent, but 1) and 3) may be frequent and can result in lock 
> contention. If there are lots of produce requests to a partition from 
> multiple processes, on the leader broker we may see:
>  # one slow log append locks up one request thread for that produce while 
> holding onto the read lock
>  # (replicationFactor-1) request threads can be blocked waiting for write 
> lock to process replica fetch request
>  # potentially several other request threads processing Produce may be queued 
> up to acquire read lock because of the waiting writers.
> In a thread dump with this issue, we noticed several request threads blocked 
> waiting for write, possibly to due to replication fetch retries.
>  
> Possible fixes:
>  # Process `Partition#maybeExpandIsr` on a single scheduler thread similar to 
> `Partition#maybeShrinkIsr` so that only a single thread is blocked on the 
> write lock. But this will delay updating ISRs and HW.
>  # Change locking in `Partition#maybeExpandIsr` so that only read lock is 
> acquired to check if ISR needs updating and write lock is acquired only to 
> update ISRs. Also use a different lock for updating HW (perhaps just the 
> Partition object lock) so that typical replica fetch requests complete 
> without acquiring Partition write lock on the request handler thread.
> I will submit a PR for 2) , but other suggestions to fix this are welcome.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1079

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Fix connect:mirror checkstyle (#7951)

[wangguoz] KAFKA-9159: The caller of sendListOffsetRequest need to handle 
retriable

[wangguoz] KAFKA-9346: Consumer back-off logic when fetching pending offsets


--
[...truncated 2.83 MB...]

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task 

Re: [VOTE] KIP-216: IQ should throw different exceptions for different errors

[Navinder Brar]

+1 (non-binding) With a small comment which was mentioned by Vinoth as well. 
Did we fix on the flag for StreamsRebalancingException, I don't see it in the 
KIP.
-Navinder


On Tuesday, 14 January, 2020, 08:00:11 pm IST, Vito Jeng 
 wrote:  
 
 Hi, all,

I would like to start the vote for KIP-216.

Currently, IQ throws InvalidStateStoreException for any types of error.
With this KIP, user can distinguish different types of error.

KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors

Thanks

---
Vito
-- 


---
Vito
  

[VOTE] KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, all,

I would like to start the vote for KIP-216.

Currently, IQ throws InvalidStateStoreException for any types of error.
With this KIP, user can distinguish different types of error.

KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors

Thanks

---
Vito
-- 


---
Vito

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

[Habib Nahas]

Any chance of an update on the KIP? We are interested in seeing this move 
forward.

Thanks,
Habib
Sr SDE, AWS

On Wed, Dec 18, 2019, at 3:27 PM, Habib Nahas wrote:
> Thanks Sean. Look forward to the updated KIP.
> 
> Regards,
> Habib
> 
> On Fri, Dec 13, 2019, at 6:22 AM, Sean Glover wrote:
> > Hi,
> > 
> > After my last reply I had a nagging feeling something wasn't right, and I
> > remembered that epoch time is UTC. This makes the discussion about
> > timezone irrelevant, since we're always using UTC. This makes the need for
> > the LatencyTime interface that I proposed in the design irrelevant as well,
> > since I can no longer think about how that might be useful. I'll update
> > the KIP. I'll also review KIP-32 to understand message timestamps better
> > so I can explain the different types of latency results that could be
> > reported with this metric.
> > 
> > Regards,
> > Sean
> > 
> > On Thu, Dec 12, 2019 at 6:25 PM Sean Glover 
> > wrote:
> > 
> > > Hi Habib,
> > >
> > > Thanks for question! If the consumer is in a different timezone than the
> > > timezone used to produce messages to a partition then you can use an
> > > implementation of LatencyTime to return the current time of that timezone.
> > > The current design assumes that messages produced to a partition must all
> > > be produced from the same timezone. If timezone metadata were encoded into
> > > each message then it would be possible to automatically determine the
> > > source timezone and calculate latency, however the current design will not
> > > pass individual messages into LatencyTime to retrieve message metadata.
> > > Instead, the LatencyTime.getWallClockTime method is only called once per
> > > fetch request response per partition and then the metric is recorded once
> > > the latency calculation is complete. This follows the same design as the
> > > current consumer lag metric which calculates offset lag based on the last
> > > message of the fetch request response for a partition. Since the metric is
> > > just an aggregate (max/mean) over some time window we only need to
> > > occasionally calculate latency, which will have negligible impact on the
> > > performance of consumer polling.
> > >
> > > A simple implementation of LatencyTime that returns wall clock time for
> > > the Asia/Singapore timezone for all partitions:
> > >
> > > import java.time.*;
> > >
> > > class SingaporeTime implements LatencyTime {
> > > ZoneId zoneSingapore = ZoneId.of("Asia/Singapore");
> > > Clock clockSingapore = Clock.system(zoneSingapore);
> > >
> > > @Override
> > > public long getWallClockTime(TopicPartition tp) {
> > > return clockSingapore.instant.getEpochSecond();
> > > }
> > >
> > > ...
> > > }
> > >
> > > Regards,
> > > Sean
> > >
> > >
> > > On Thu, Dec 12, 2019 at 6:18 AM Habib Nahas  wrote:
> > >
> > >> Hi Sean,
> > >>
> > >> Thanks for the KIP.
> > >>
> > >> As I understand it users are free to set their own timestamp on
> > >> ProducerRecord. What is the recommendation for the proposed metric in a
> > >> scenario where the user sets this timestamp in timezone A and consumes 
> > >> the
> > >> record in timezone B. Its not clear to me if a custom implementation of
> > >> LatencyTime will help here.
> > >>
> > >> Thanks,
> > >> Habib
> > >>
> > >> On Wed, Dec 11, 2019, at 4:52 PM, Sean Glover wrote:
> > >> > Hello again,
> > >> >
> > >> > There has been some interest in this KIP recently. I'm bumping the
> > >> thread
> > >> > to encourage feedback on the design.
> > >> >
> > >> > Regards,
> > >> > Sean
> > >> >
> > >> > On Mon, Jul 15, 2019 at 9:01 AM Sean Glover 
> > >> > wrote:
> > >> >
> > >> > > To hopefully spark some discussion I've copied the motivation section
> > >> from
> > >> > > the KIP:
> > >> > >
> > >> > > Consumer lag is a useful metric to monitor how many records are
> > >> queued to
> > >> > > be processed. We can look at individual lag per partition or we may
> > >> > > aggregate metrics. For example, we may want to monitor what the
> > >> maximum lag
> > >> > > of any particular partition in our consumer subscription so we can
> > >> identify
> > >> > > hot partitions, caused by an insufficient producing partitioning
> > >> strategy.
> > >> > > We may want to monitor a sum of lag across all partitions so we have 
> > >> > > a
> > >> > > sense as to our total backlog of messages to consume. Lag in offsets
> > >> is
> > >> > > useful when you have a good understanding of your messages and
> > >> processing
> > >> > > characteristics, but it doesn’t tell us how far behind *in time* we
> > >> are.
> > >> > > This is known as wait time in queueing theory, or more informally 
> > >> > > it’s
> > >> > > referred to as latency.
> > >> > >
> > >> > > The latency of a message can be defined as the difference between 
> > >> > > when
> > >> > > that message was first produced to when the message is received by a
> > >> > > consumer. The latency of records in a partition correlates with lag,
> > >> but a
> > >> > > 

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

[Navinder Brar]

+1 (non-binding)
Navinder
On Tuesday, 14 January, 2020, 07:24:02 pm IST, Ryanne Dolan 
 wrote:  
 
 Bump. We've got 4 non-binding and one binding vote.

Ryanne

On Fri, Dec 13, 2019, 1:44 AM Tom Bentley  wrote:

> +1 (non-binding)
>
> On Thu, Dec 12, 2019 at 6:33 PM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > +1 (non-binding)
> >
> > On 12/12/2019, 14:20, "Mickael Maison" 
> wrote:
> >
> >    +1 (binding)
> >    Thanks for the KIP!
> >
> >    On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan 
> > wrote:
> >    >
> >    > Bump. We've got 2 non-binding votes so far.
> >    >
> >    > On Wed, Nov 13, 2019 at 3:32 PM Ning Zhang  >
> > wrote:
> >    >
> >    > > My current plan is to implement this in "MirrorCheckpointTask"
> >    > >
> >    > > On 2019/11/02 03:30:11, Xu Jianhai  wrote:
> >    > > > I think this kip will implement a task in sinkTask ? right?
> >    > > >
> >    > > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan <
> > ryannedo...@gmail.com>
> >    > > wrote:
> >    > > >
> >    > > > > Hey y'all, Ning Zhang and I would like to start the vote for
> > the
> >    > > following
> >    > > > > small KIP:
> >    > > > >
> >    > > > >
> >    > > > >
> >    > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> >    > > > >
> >    > > > > This is an elegant way to automatically write consumer group
> > offsets to
> >    > > > > downstream clusters without breaking existing use cases.
> > Currently, we
> >    > > rely
> >    > > > > on external tooling based on RemoteClusterUtils and
> >    > > kafka-consumer-groups
> >    > > > > command to write offsets. This KIP bakes this functionality
> > into MM2
> >    > > > > itself, reducing the effort required to failover/failback
> > workloads
> >    > > between
> >    > > > > clusters.
> >    > > > >
> >    > > > > Thanks for the votes!
> >    > > > >
> >    > > > > Ryanne
> >    > > > >
> >    > > >
> >    > >
> >
> >
> >
>  

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

[Dhruvil Shah]

If there are no more questions or concerns, I will start a vote thread
tomorrow.

Thanks,
Dhruvil

On Mon, Jan 13, 2020 at 6:59 PM Dhruvil Shah  wrote:

> Hi Nikhil,
>
> Thanks for looking at the KIP. The kind of race condition you mention is
> not possible as stray partition detection is done synchronously while
> handling the LeaderAndIsrRequest. In other words, we atomically evaluate
> the partitions the broker must host and the extra partitions it is hosting
> and schedule deletions based on that.
>
> One possible shortcoming of the KIP is that we do not have the ability to
> detect a stray partition if the topic has been recreated since. We will
> have the ability to disambiguate between different generations of a
> partition with KIP-516.
>
> Thanks,
> Dhruvil
>
> On Sat, Jan 11, 2020 at 11:40 AM Nikhil Bhatia 
> wrote:
>
>> Thanks Dhruvil, the proposal looks reasonable to me.
>>
>> is there a potential of a race between a new topic being assigned to the
>> same node that is still performing a cleanup of the stray partition ?
>> Topic
>> ID will definitely solve this issue.
>>
>> Thanks
>> Nikhil
>>
>> On 2020/01/06 04:30:20, Dhruvil Shah  wrote:
>> > Here is the link to the KIP:>
>> >
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
>> >
>>
>> >
>> > On Mon, Jan 6, 2020 at 9:59 AM Dhruvil Shah 
>> wrote:>
>> >
>> > > Hi all, I would like to kick off discussion for KIP-550 which proposes
>> a>
>> > > mechanism to detect and delete stray partitions on a broker.
>> Suggestions>
>> > > and feedback are welcome.>
>> > >>
>> > > - Dhruvil>
>> > >>
>> >
>>
>

[jira] [Created] (KAFKA-9424) Using AclCommand,avoid call the global method loadcache in SimpleAclAuthorizer

[Steven Lu (Jira)]

Steven Lu created KAFKA-9424:


 Summary: Using AclCommand,avoid call the global method loadcache 
in SimpleAclAuthorizer
 Key: KAFKA-9424
 URL: https://issues.apache.org/jira/browse/KAFKA-9424
 Project: Kafka
  Issue Type: Improvement
  Components: admin, tools
Affects Versions: 2.3.1, 2.4.0, 0.10.2.0
 Environment: Linux,JDK7+
Reporter: Steven Lu


In the class Named AclCommand,configure SimpleAclAuthorizer,but no need call 
loadCache.
now we have 20,000 topics in kafka cluster,everytime I run AclCommand,all these 
topics's Alcs need to be authed, it will be very slow.
The purpose of this optimization is：we can choose to not load the acl of all 
topics into memory, mainly for adding and deleting permissions.

PR Available here: [https://github.com/apache/kafka/pull/7706]

mainly for adding and deleting permissions,we can choose to not load the acl of 
all topics into memory,then we can add two args "--load-acl-cache" "false" in 
AclCommand.main;else you don't add these args, it will load the acl cache 
defaultly.

we can choose improve the running time from minutes to less than one second.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

[Ryanne Dolan]

Bump. We've got 4 non-binding and one binding vote.

Ryanne

On Fri, Dec 13, 2019, 1:44 AM Tom Bentley  wrote:

> +1 (non-binding)
>
> On Thu, Dec 12, 2019 at 6:33 PM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > +1 (non-binding)
> >
> > On 12/12/2019, 14:20, "Mickael Maison" 
> wrote:
> >
> > +1 (binding)
> > Thanks for the KIP!
> >
> > On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan 
> > wrote:
> > >
> > > Bump. We've got 2 non-binding votes so far.
> > >
> > > On Wed, Nov 13, 2019 at 3:32 PM Ning Zhang  >
> > wrote:
> > >
> > > > My current plan is to implement this in "MirrorCheckpointTask"
> > > >
> > > > On 2019/11/02 03:30:11, Xu Jianhai  wrote:
> > > > > I think this kip will implement a task in sinkTask ? right?
> > > > >
> > > > > On Sat, Nov 2, 2019 at 1:06 AM Ryanne Dolan <
> > ryannedo...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hey y'all, Ning Zhang and I would like to start the vote for
> > the
> > > > following
> > > > > > small KIP:
> > > > > >
> > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
> > > > > >
> > > > > > This is an elegant way to automatically write consumer group
> > offsets to
> > > > > > downstream clusters without breaking existing use cases.
> > Currently, we
> > > > rely
> > > > > > on external tooling based on RemoteClusterUtils and
> > > > kafka-consumer-groups
> > > > > > command to write offsets. This KIP bakes this functionality
> > into MM2
> > > > > > itself, reducing the effort required to failover/failback
> > workloads
> > > > between
> > > > > > clusters.
> > > > > >
> > > > > > Thanks for the votes!
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > >
> > > >
> >
> >
> >
>

Re: Issues with triggering the build

[Levani Kokhreidze]

Hi Bruno,

Thanks for trying. I’ve also tried with “retest this please” but didn’t do any 
good.
Seems like it affects other PRs as well.

- Levani

> On Jan 14, 2020, at 12:13 PM, Bruno Cadonna  wrote:
> 
> Hi,
> 
> I tried with "Retest this, please" but it didn't work.
> 
> Best,
> Bruno
> 
> On Tue, Jan 14, 2020 at 9:01 AM Levani Kokhreidze
>  wrote:
>> 
>> Hello,
>> 
>> Seems like there’re issues with triggering Jenkins builds. Latest commits 
>> for my PR doesn’t trigger any of the builds.
>> Any ideas how to fix the issue? Here’s the example PR: 
>> https://github.com/apache/kafka/pull/7170 
>> 
>> 
>> - Levani

[jira] [Created] (KAFKA-9423) Refine layout of configuration options on website and make individual settings directly linkable

[Jira]

Sönke Liebau created KAFKA-9423:
---

 Summary: Refine layout of configuration options on website and 
make individual settings directly linkable
 Key: KAFKA-9423
 URL: https://issues.apache.org/jira/browse/KAFKA-9423
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Sönke Liebau
 Attachments: image-2020-01-14-11-17-55-277.png, 
image-2020-01-14-11-18-12-190.png, image-2020-01-14-11-18-24-939.png, 
image-2020-01-14-11-18-36-825.png

KAKFA-8474 changed the layout of configuration options on the website from a 
table which over time ran out of horizontal space to a list.
This vastly improved readability but is not yet ideal. Further discussion was 
had in the [PR|https://github.com/apache/kafka/pull/6870] for KAFKA-8474.
This ticket is to move that discussion to a separate thread and make it more 
visible to other people and to give subsequent PRs a home.

Currently proposed options are listed below.

Option 1: 
 !image-2020-01-14-11-17-55-277.png|thumbnail! 

Option 2:
 !image-2020-01-14-11-18-12-190.png|thumbnail! 

Option 3:
 !image-2020-01-14-11-18-24-939.png|thumbnail! 

Option 4:
 !image-2020-01-14-11-18-36-825.png|thumbnail! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Issues with triggering the build

[Bruno Cadonna]

Hi,

I tried with "Retest this, please" but it didn't work.

Best,
Bruno

On Tue, Jan 14, 2020 at 9:01 AM Levani Kokhreidze
 wrote:
>
> Hello,
>
> Seems like there’re issues with triggering Jenkins builds. Latest commits for 
> my PR doesn’t trigger any of the builds.
> Any ideas how to fix the issue? Here’s the example PR: 
> https://github.com/apache/kafka/pull/7170 
> 
>
> - Levani

Issues with triggering the build

[Levani Kokhreidze]

Hello,

Seems like there’re issues with triggering Jenkins builds. Latest commits for 
my PR doesn’t trigger any of the builds.
Any ideas how to fix the issue? Here’s the example PR: 
https://github.com/apache/kafka/pull/7170 


- Levani