[jira] [Created] (KAFKA-10170) ReplicaManager should be responsible for checking delayed operations after appending to the log.

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10170:
--

 Summary: ReplicaManager should be responsible for checking delayed 
operations after appending to the log.
 Key: KAFKA-10170
 URL: https://issues.apache.org/jira/browse/KAFKA-10170
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This issue aims to refactor code to simplify the code of checking delayed 
operations. This issue is inspired by [~hachikuji] 
(https://github.com/apache/kafka/pull/8657#discussion_r426943627)

{quote}
Currently we have a somewhat convoluted model where ReplicaManager creates 
delayed operations, but we depend on lower level components like Partition to 
be aware of them and complete them. This breaks encapsulation.

Not something we should try to complete in this PR, but as an eventual goal, I 
think we can consider trying to factor delayed operations out of Partition so 
that they can be managed by ReplicaManager exclusively. If you assume that is 
the end state, then we could drop completeDelayedRequests and let 
ReplicaManager always be responsible for checking delayed operations after 
appending to the log.

Other than ReplicaManager, the only caller of this method is 
GroupMetadataManager which uses it during offset expiration. I think the only 
reason we do this is because we didn't want to waste purgatory space. I don't 
think that's a good enough reason to go outside the normal flow. It would be 
simpler to follow the same path. Potentially we could make the callback an 
Option so that we still have a way to avoid polluting the purgatory.
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Permission to create KIP

[Matthias J. Sax]

Done.

On 6/15/20 1:27 PM, Sam Pal wrote:
> Hi,
> 
> I'd like permission to create a KIP on the Apache Kafka Wiki. My WikiID is
> spal.
> 
> Best,
> Sam
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #1570

[Apache Jenkins Server]

See 


Changes:

[github] HOTFIX: checkstyle error in ProcessorStateManager (#8874)


--
[...truncated 1.89 MB...]

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest > testDecodeSeq STARTED

kafka.utils.json.JsonValueTest > testDecodeSeq PASSED

kafka.utils.json.JsonValueTest > testJsonObjectGet STARTED

kafka.utils.json.JsonValueTest > testJsonObjectGet PASSED

kafka.utils.json.JsonValueTest > testJsonValueEquals STARTED

kafka.utils.json.JsonValueTest > testJsonValueEquals PASSED

kafka.utils.json.JsonValueTest > testJsonArrayIterator STARTED

kafka.utils.json.JsonValueTest > testJsonArrayIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectApply STARTED

kafka.utils.json.JsonValueTest > testJsonObjectApply PASSED

kafka.utils.json.JsonValueTest > testDecodeBoolean STARTED

kafka.utils.json.JsonValueTest > testDecodeBoolean PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsOverwriteExisting 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsOverwriteExisting PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters STARTED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter PASSED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting STARTED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
PASSED

kafka.tools.CustomDeserializerTest > checkDeserializerTopicIsNotNull STARTED

kafka.tools.CustomDeserializerTest > checkDeserializerTopicIsNotNull PASSED

kafka.tools.CustomDeserializerTest > checkFormatterCallDeserializerWithHeaders 
STARTED

kafka.tools.CustomDeserializerTest > checkFormatterCallDeserializerWithHeaders 
PASSED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody STARTED

kafka.tools.ConsumerPerformanceTest > testDetailedHeaderMatchBody PASSED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption STARTED

kafka.tools.ConsumerPerformanceTest > testConfigWithUnrecognizedOption PASSED

kafka.tools.ConsumerPerformanceTest > testBrokerListOverride STARTED

kafka.tools.ConsumerPerformanceTest > testBrokerListOverride PASSED

kafka.tools.ConsumerPerformanceTest > testConfigBootStrapServer STARTED

kafka.tools.ConsumerPerformanceTest > testConfigBootStrapServer PASSED

kafka.tools.ConsumerPerformanceTest > testConfigBrokerList STARTED

kafka.tools.ConsumerPerformanceTest > testConfigBrokerList PASSED

kafka.tools.ConsumerPerformanceTest > 

[VOTE] KIP-626: Rename StreamsConfig config variable name

[Matthias J. Sax]

Hi,

I found a small inconsistency in our public API and propose a small KIP
to fix it. As the change is trivial, I skip the discussion and call
directly for a VOTE.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-626%3A+Rename+StreamsConfig+config+variable+name


-Matthias



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-10169) KafkaException: Failing batch since transaction was aborted

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-10169:
---

 Summary: KafkaException: Failing batch since transaction was 
aborted
 Key: KAFKA-10169
 URL: https://issues.apache.org/jira/browse/KAFKA-10169
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman
 Fix For: 2.6.0


We've seen the following exception in our eos-beta test application recently:
{code:java}
[2020-06-13T00:09:14-07:00] 
(streams-soak-2-6-all-fixes-eos-beta_soak_i-0ae30dd12c4fb7018_streamslog) 
org.apache.kafka.streams.errors.StreamsException: Error encountered sending 
record to topic 
stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-25-changelog for task 
1_2 due to: [2020-06-13T00:09:14-07:00] 
(streams-soak-2-6-all-fixes-eos-beta_soak_i-0ae30dd12c4fb7018_streamslog) 
org.apache.kafka.common.KafkaException: Failing batch since transaction was 
aborted [2020-06-13T00:09:14-07:00] 
(streams-soak-2-6-all-fixes-eos-beta_soak_i-0ae30dd12c4fb7018_streamslog) 
Exception handler choose to FAIL the processing, no more records would be sent. 
at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:213)
 at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$0(RecordCollectorImpl.java:185)
 at 
org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:1347)
 at 
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:231)
 at 
org.apache.kafka.clients.producer.internals.ProducerBatch.abort(ProducerBatch.java:159)
 at 
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortUndrainedBatches(RecordAccumulator.java:781)
 at 
org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:425)
 at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:313) 
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240) at 
java.lang.Thread.run(Thread.java:748) [2020-06-13T00:09:14-07:00] 
(streams-soak-2-6-all-fixes-eos-beta_soak_i-0ae30dd12c4fb7018_streamslog) 
Caused by: org.apache.kafka.common.KafkaException: Failing batch since 
transaction was aborted at 
org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:423)
 ... 3 more
{code}
Somewhat unclear if this is an issue with eos-beta specifically, or just eos in 
general. But several threads have died over the course of a few days in the 
eos-beta application, while none so far have died on the eos-alpha application.

It's also unclear (at least to me) whether this is definitely an issue in 
Streams or possibly a bug in the producer (or even the broker, although that 
seems unlikely)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10168) Rename public StreamsConfig variable

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10168:
---

 Summary: Rename public StreamsConfig variable
 Key: KAFKA-10168
 URL: https://issues.apache.org/jira/browse/KAFKA-10168
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


All Kafka Streams configuration parameter are exposed via public variables that 
all end with `_CONFIG` suffix. However, we added the variable of 
`topology.optimization` as `TOPOLOGY_OPTIMIZATION` instead of 
`TOPLOGY_OPTIMIZATION_CONFIG`. We should align the variable name.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10167) Streams EOS-Beta should not try to get end-offsets as read-committed

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-10167:
-

 Summary: Streams EOS-Beta should not try to get end-offsets as 
read-committed
 Key: KAFKA-10167
 URL: https://issues.apache.org/jira/browse/KAFKA-10167
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang


This is a bug discovered with the new EOS protocol (KIP-447), here's the 
context:

In Streams when we are assigned with the new active tasks, we would first try 
to restore the state from the changelog topic all the way to the log end 
offset, and then we can transit from the `restoring` to the `running` state to 
start processing the task.

Before KIP-447, the end-offset call is only triggered after we've passed the 
synchronization barrier at the txn-coordinator which would guarantee that the 
txn-marker has been sent and received (otherwise we would error with 
CONCURRENT_TRANSACTIONS and let the producer retry), and when the txn-marker is 
received, it also means that the marker has been fully replicated, which in 
turn guarantees that the data written before that marker has been fully 
replicated. As a result, when we send the list-offset with `read-committed` 
flag we are guaranteed that the returned offset == LSO == high-watermark.

After KIP-447 however, we do not fence on the txn-coordinator but on 
group-coordinator upon offset-fetch, and the group-coordinator would return the 
fetching offset right after it has received the replicated the txn-marker sent 
to it. However, since the txn-marker are sent to different brokers in parallel, 
and even within the same broker markers of different partitions are appended / 
replicated independently as well, so when the fetch-offset request returns it 
is NOT guaranteed that the LSO on other data partitions would have been 
advanced as well. And hence in that case the `endOffset` call may returned a 
smaller offset, causing data loss.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Jorge Esteban Quilcate Otoya]

Hi everyone, sorry for the late reply.

Thanks Matthias for your feedback. I think it makes sense to reconsider the
current design based on your input.

After digging deeper into the current implementation, I'd like to bring my
current understanding to be double-checked as it might be redefining the
KIP's scope:

1. There are 2 ranges been exposed by different stores:

a. Key Range
b. Timestamp Range

So far, we have discussed covering both.

2. Key Range functions do not provide ordering guarantees by design:

```ReadOnlyKeyValueStore.java
/**
 * Get an iterator over a given range of keys. This iterator must be
closed after use.
 * The returned iterator must be safe from {@link
java.util.ConcurrentModificationException}s
 * and must not return null values. No ordering guarantees are provided.
 * ...
 */
 KeyValueIterator range(K from, K to);
```

Therefore, I'd propose removing Key range operations from the scope.

3. Timestamp Range operations happen at the SegmentsStore level (internal)
API

AFAICT, Segments wrappers handle all Timestamp ranges queries.

I'd propose extending `Segments#segments(timeFrom, timeTo, backwards)` with
a flag for backwards operations.

As segments returned will be processed backwards, I'm not extending
KeyValueStores to query each segment backwards as previous point 2.

4. Extend WindowStores implementations with a new
WindowBackwardStore/ReadOnlyBackwardStore:

```java
public interface ReadOnlyWindowBackwardStore {
WindowStoreIterator backwardFetch(K key, Instant from, Instant to)
throws IllegalArgumentException;

KeyValueIterator, V> backwardFetch(K from, K to, Instant
fromTime, Instant toTime)
throws IllegalArgumentException;

KeyValueIterator, V> backwardFetchAll(Instant from, Instant
to) throws IllegalArgumentException;
```

5. SessionStore is a bit different as it has fetch/find sessions spread
between SessionStore and ReadOnlySessionStore.

I'd propose a new interface `SessionBackwardStore` to expose backward find
operations:

```java
public interface SessionBackwardStore {
KeyValueIterator, AGG> backwardFindSessions(final K key,
final long earliestSessionEndTime, final long latestSessionStartTime);

KeyValueIterator, AGG> backwardFindSessions(final K
keyFrom, final K keyTo, final long earliestSessionEndTime, final long
latestSessionStartTime);
}
```

If this understanding is correct I'll proceed to update the KIP based on
this.

Looking forward to your feedback.

Thanks,
Jorge.

On Fri, May 29, 2020 at 3:32 AM Matthias J. Sax  wrote:

> Hey,
>
> Sorry that I am late to the game. I am not 100% convinced about the
> current proposal. Using a new config as feature flag seems to be rather
> "nasty" to me, and flipping from/to is a little bit too fancy for my
> personal taste.
>
> I agree, that the original proposal using a "ReadDirection" enum is not
> ideal either.
>
> Thus, I would like to put out a new idea: We could add a new interface
> that offers new methods that return revers iterators.
>
> The KIP already proposes to add `reverseAll()` and it seems backward
> incompatible to just add this method to `ReadOnlyKeyValueStore` and
> `ReadOnlyWindowStore`. I don't think we could provide a useful default
> implementation for custom stores and thus either break compatibility or
> need add a default that just throws an exception. Neither seems to be a
> good option.
>
> Using a new interface avoid this issue and allows users implementing
> custom stores to opt-in by adding the interface to their stores.
> Furthermore, we don't need any config. In the end, we encapsulte the
> change into the store, and our runtime is agnostic to it (as it should be).
>
> The hierarchy becomes a little complex (but uses would not really see
> the complexity):
>
> // exsiting
> ReadOnlyKeyValueStore
> KeyValueStore extend StateStore, ReadOnlyKeyValueStore
>
>
> // helper interface; users don't care
> // need similar ones for other stores
> ReverseReadOnlyKeyValueStore {
> KeyValueIterator reverseRange(K from, K to);
> KeyValueIterator reverseAll();
> }
>
>
> // two new user facing interfaces for kv-store
> // need similar ones for other stores
> ReadOnlyKeyValueStoreWithReverseIterators extends ReadOnlyKeyValueStore,
> ReverseReadOnlyKeyValueStore
>
> KeyValueStoreWithReverseIterators extends KeyValueStore,
> ReverseReadOnlyKeyValueStore
>
>
> // updated (also internal)
> // also need to update other built-in stores
> RocksDB implements KeyValueStoreWithReverseIterators, BulkLoadingStore
>
>
> In the end, user would only care about the two (for kv-case) new
> interface that offer revers iterator (read only and regular) and can
> cast stores accordingly in their Processors/Transformers or via IQ.
>
>
> Btw: if we add revers iterator for KeyValue and Window store, should we
> do the same for Session store?
>
>
>
> This might be more code to write, but I believe it provides the better
> user experience. Thoughts?
>
>
>
> -Matthias

Re: First time patch submitter advice

[Michael Carter]

Great, thanks Luke.
I’ve undone the patch and added that comment.

Cheers,
Michael

> On 15 Jun 2020, at 6:07 pm, Luke Chen  wrote:
> 
> Hi Michael,
> The failed unit test has already handled here:
> https://issues.apache.org/jira/browse/KAFKA-10155
> https://issues.apache.org/jira/browse/KAFKA-10147
> 
> So, maybe you can ignore the test errors and mention the issue number in PR.
> Thanks.
> 
> Luke
> 
> On Mon, Jun 15, 2020 at 3:23 PM Michael Carter <
> michael.car...@instaclustr.com> wrote:
> 
>> Thanks for the response Gwen, that clarifies things for me.
>> 
>> Regarding the unit test (ReassignPartitionsUnitTest.
>> testModifyBrokerThrottles),  it appears to fail quite reliably on trunk as
>> well (at least on my machine).
>> It looks to me like a new override to
>> MockAdminClient.describeConfigs(Collection resources)
>> (MockAdminClient.java line 369) introduced in commit
>> 48b56e533b3ff22ae0e2cf7fcc649e7df19f2b06 changed the behaviour of this
>> method that the unit test relied on.
>> I’ve just now put a patch into my branch to make that test pass by calling
>> a slightly different version of describeConfigs (that avoids the overridden
>> behaviour). It’s probably arguable whether that constitutes a fix or not
>> though.
>> 
>> Cheers,
>> Michael
>> 
>>> On 15 Jun 2020, at 3:41 pm, Gwen Shapira  wrote:
>>> 
>>> Hi,
>>> 
>>> 1. Unfortunately, you need to get a committer to approve running the
>> tests.
>>> I just gave the green-light on your PR.
>>> 2. You can hope that committers will see your PR, but sometimes things
>> get
>>> lost. If you know someone who is familiar with that area of the code, it
>> is
>>> a good idea to ping them.
>>> 3. We do have some flaky tests. You can see that Jenkins will run 3
>>> parallel builds, if some of them pass and the committer confirms that
>>> failures are not related to your code, we are ok to merge. Obviously, if
>>> you end up tracking them down and fixing, everyone will be very grateful.
>>> 
>>> Hope this helps,
>>> 
>>> Gwen
>>> 
>>> On Sun, Jun 14, 2020 at 5:52 PM Michael Carter <
>>> michael.car...@instaclustr.com> wrote:
>>> 
 Hi all,
 
 I’ve submitted a patch for the first time(
 https://github.com/apache/kafka/pull/8844 <
 https://github.com/apache/kafka/pull/8844>), and I have a couple of
 questions that I’m hoping someone can help me answer.
 
 I’m a little unclear what happens after that patch has been submitted.
>> The
 coding guidelines say Jenkins will run tests automatically, but I don’t
>> see
 any results anywhere. Have I misunderstood what should happen, or do I
>> just
 not know where to look?
 Should I be attempting to find reviewers for the change myself, or is
>> that
 done independently of the patch submitter?
 
 Also, in resolving a couple of conflicts that have arisen after the
>> patch
 was first submitted, I noticed that there are now failing unit tests
>> that
 have nothing to do with my change. Is there a convention on how to deal
 with these? Should it be something that I try to fix on my branch?
 
 Any thoughts are appreciated.
 
 Thanks,
 Michael
>>> 
>>> 
>>> 
>>> --
>>> Gwen Shapira
>>> Engineering Manager | Confluent
>>> 650.450.2760 | @gwenshap
>>> Follow us: Twitter | blog
>> 
>> 

Build failed in Jenkins: kafka-trunk-jdk8 #4644

[Apache Jenkins Server]

See 


Changes:

[github] HOTFIX: checkstyle error in ProcessorStateManager (#8874)


--
[...truncated 3.15 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 

Build failed in Jenkins: kafka-2.6-jdk8 #45

[Apache Jenkins Server]

See 


Changes:

[wangguoz] HOTFIX: checkstyle error in ProcessorStateManager (#8874)


--
[...truncated 3.14 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task :streams:upgrade-system-tests-0102:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:testClasses
> Task 

Build failed in Jenkins: kafka-trunk-jdk14 #220

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix log message for transition from standby to active (#8872)

[github] HOTFIX: checkstyle error in ProcessorStateManager (#8874)


--
[...truncated 3.17 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

Re: Broker side round robin on topic partitions when receiving messages

[Vinicius Scheidegger]

I understand your point of view... My requirement is *exact* balancing -
parts of our current flow have a consumption processing of around 5
minutes... (and this is an important/expensive part - because it's CPU and
memory intensive and we'd like to avoid queueing) so we need EQUAL load
balancing - and we need to know when we need to scale/descale.
If you pay attention I'm always saying equal load balancing with multiple
producers.
By that I mean: if I have 10 partitions in a topic and send 10 messages
from different producers I expect the load to be exactly divided, 1 message
in each partition.

I thought about some possible solutions using Kafka as-is, although there
is always a drawback.

What Kafka offers out of the box:
A) RoundRobinPartitioner - cyclic round robin internal to producer - tends
to producer N messages in each single partition before total the number of
partition is met (where N is equal to the number of producers). Drawback:
unequal balance over short periods of time (depending on the number of
producers, where the messages are coming from - which producer, etc).
B) DefaultPartitioner - hash of the key modulus total number of partitions
- if a random key is used, mathematically (think of big number of messages)
should be equally distributed - Drawback: unequal balance over short
periods of time.
Is that correct / do you agree?

Possible options we could think of:

1) A custom partitioner using shared memory between producers to decide the
next partition; Drawback - all producers would need to be within the shared
memory boundary.
2) Creating a single dummy consumer/producer with RoundRobinpartitioner
between two topics "in", where the real producers would send the message to
and "out" with multiple partitions where the real consumers would listen
to. Drawbacks: Single point of failure (ok, we could have a single
partition "in" an extra consumer within the same consumer group that could
take if the consumer/producer fails) - I believe we could go far here -
making design, maintainability, monitoring, etc worse from an architecture
point of view.

I didn't understand the atomic counter - I guess maybe would look like
number 1?
And maybe fanout, like number 2?

I believe we should be able to do perfect load balancing, 10 messages
received in a topic being distributed to 10 partitions - 20 messages, 20
partitions, no matter who generated them.
The thing is that currently the broker receives messages on the partition
level only. No way to send them on a topic level and redistribute

We are currently paying extra idle machines - my ideas are either:
i) make sure we are not missing something (maybe some of our assumptions
are wrong and we have easy ootb options)
ii) if we are not missing something going with option 1 (and limiting our
producers to be within the shared mem boundaries)
iii) Checking the feasibility(how hard would it be?)/acceptance of the
community of doing this in Kafka by submitting a KIP

Thanks once again!



On Mon, Jun 15, 2020 at 9:09 PM Colin McCabe  wrote:

> This is a bit frustrating since you keep saying that the load is not
> balanced, but the load actually is balanced, it's just balanced in an
> approximate fashion.  If you need exact balancing (for example, because
> you're creating a job scheduler or something), then you need to use a
> different strategy.  One example would be using an external atomic counter
> to determine what partition the producers should send the messages to.
> Another would be using a single consumer with fanout.  I think this is
> outside the scope of Kafka, at least if I understand the problem here (?)
>
> best,
> Colin
>
> On Mon, Jun 15, 2020, at 11:32, Vinicius Scheidegger wrote:
> > Hi Collin,
> >
> > One producer shouldn't need to know about the other to distribute the
> load
> > equally, but what Kafka has now is roughly equal...
> > If you have a single producer RounRobinPartitioner works fine, if you
> have
> > 10 producers you can have 7/8 messages in one partition while another
> > partition has none (producers are in sync - which happened a couple times
> > in our tests).
> >
> > Producer0 getNext() = partition0
> > Producer1 getNext() = partition0
> > Producer2 getNext() = partition0
> >
> > A link to some of our test data prints:
> > https://imgur.com/a/ha9OQMj
> >
> > This, depending on how intensive (slow) your consumption rate is, may be
> a
> > problem as it will generate enqueuing.
> > We use Kafka as a messaging protocol in a big (and in some points heavy
> > load) machine learning flow - for high throughput (lightweight
> processing)
> > enqueuing is not an issue - añthough we saw it happening. but for heavy
> > processes we are unable to do equal load balance.
> >
> > We currently use the DefaultPartitioner and Kafka algorithm (murmur2 hash
> > of the key) to decide the partition.
> > We noticed enqueuing and timeouts while several consumers were idle -
> which
> > made us take a better look on how the load is 

Permission to create KIP

[Sam Pal]

Hi,

I'd like permission to create a KIP on the Apache Kafka Wiki. My WikiID is
spal.

Best,
Sam

Build failed in Jenkins: kafka-2.6-jdk8 #44

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Avoid WARN log message when re-init from checkpoint skipped

[wangguoz] MINOR: Fix log message for transition from standby to active (#8872)


--
[...truncated 2.21 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Boyang Chen]

Thanks for more feedback Colin! I have addressed them in the KIP.

Boyang

On Mon, Jun 15, 2020 at 11:29 AM Colin McCabe  wrote:

> On Fri, Jun 12, 2020, at 15:30, Boyang Chen wrote:
> > Thanks Colin for the suggestions!
> >
> > On Fri, Jun 12, 2020 at 2:40 PM Colin McCabe  wrote:
> >
> > > Hi Boyang,
> > >
> > > Thanks for the KIP!  I think it's getting close.
> > >
> > >  > For older requests that need redirection, forwarding
> > >  > broker will just use its own authorizer to verify the principals.
> When
> > > the
> > >  > request looks good, it will just forward the request with its own
> > >  > credentials, no second validation needed
> > >
> > > Just to be clear, the controller will still validate the request,
> right?
> > > But at that point the principal will be the broker principal.  It
> would be
> > > good to note that here.
> > >
> > > Sounds good, cleared in the KIP.
> >
> > > Internal CreateTopicsRequest Routing
> > >
> > > The forwarding broker is sending the request as the latest version,
> > > right?  It would be good to add a note of this.  This also prevents
> routing
> > > loops since the latest version is not forwardable (another good thing
> to
> > > add, I think...)
> > >
> > We are not bumping the CreateTopic RPC here, so it should be the latest
> > by default.
> >
>
> Sorry, CreateTopics was a bad example here, since it already must be sent
> to the controller.  Oops.
>
> >
> > And just to be clear, we are not "forwarding" but actually
> > sending a CreateTopicRequest from the receiving broker to the controller
> > broker.
> >
>
> Right.  I think we agree on this point.  But we do need a term to describe
> the broker which initially receives the user request and resends it to the
> controller.  Resending broker?
>
> And I do think it's important to note that the request we send to the
> controller can't be itself resent.
>
> >
> >  > As we discussed in the request routing section, to work with an older
> > >  > client, the first contacted broker need to act as a proxy to
> redirect
> > > the
> > >  > write request to the controller. To support the proxy of requests,
> we
> > > need
> > >  > to build a channel for brokers to talk directly to the controller.
> This
> > >  > part of the design is internal change only and won’t block the KIP
> > >  > progress.
> > >
> > > I think it's good to note that we eventually want a separate controller
> > > endpoint in KIP-500.  However, we don't need it to implement KIP-590,
> > > right?  The other brokers could forward to the existing internal
> endpoint
> > > for the controller.  So maybe it's best to discuss the separate
> endpoint in
> > > "future work" rather than here.
> > >
> > > I moved the new endpoint part towards the future work and addressed the
> > > usage of controller internal endpoint for routing requests.
> >
>
> Thanks.
>
> >
> > > > === Start Old Proposal  ===
> > >
> > > I'm glad the old proposal shows up here, but I think this is too much
> > > detail.  It would be better to just have a one or two paragraph
> summary of
> > > the main points.  As it is, the old proposal takes up 40% of the doc
> which
> > > is pretty confusing for someone reading through.  Let's also not forget
> > > that someone can just read the old version by using the "page history"
> > > function on the wiki.  So there's no need to keep that all here.
> > >
> > > Make sense, removed.
> >
>
> Thanks again.
>
> >
> >{ "name": "PrincipalName", "type": "string", "tag": 0,
> "taggedVersions": "2+", "ignorable": true,
> >  "about": "Optional value of the principal name when the request is
> redirected by a broker." },
> >
>
> Maybe "InitialPrincipalName" would be better here?  PrincipalName is a bit
> confusing since the message already has a principal name, after all...
>
> cheers,
> Colin
>

Build failed in Jenkins: kafka-trunk-jdk11 #1569

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Avoid WARN log message when re-init from checkpoint skipped

[github] MINOR: Fix log message for transition from standby to active (#8872)


--
[...truncated 2.24 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = 

Build failed in Jenkins: kafka-trunk-jdk14 #219

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Avoid WARN log message when re-init from checkpoint skipped


--
[...truncated 3.17 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #4643

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Avoid WARN log message when re-init from checkpoint skipped

[github] MINOR: Fix log message for transition from standby to active (#8872)


--
[...truncated 2.23 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0110:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0110:test
> Task :streams:upgrade-system-tests-10:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-10:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-10:classes UP-TO-DATE
> Task 

Re: Broker side round robin on topic partitions when receiving messages

[Colin McCabe]

This is a bit frustrating since you keep saying that the load is not balanced, 
but the load actually is balanced, it's just balanced in an approximate 
fashion.  If you need exact balancing (for example, because you're creating a 
job scheduler or something), then you need to use a different strategy.  One 
example would be using an external atomic counter to determine what partition 
the producers should send the messages to.  Another would be using a single 
consumer with fanout.  I think this is outside the scope of Kafka, at least if 
I understand the problem here (?)

best,
Colin

On Mon, Jun 15, 2020, at 11:32, Vinicius Scheidegger wrote:
> Hi Collin,
> 
> One producer shouldn't need to know about the other to distribute the load
> equally, but what Kafka has now is roughly equal...
> If you have a single producer RounRobinPartitioner works fine, if you have
> 10 producers you can have 7/8 messages in one partition while another
> partition has none (producers are in sync - which happened a couple times
> in our tests).
> 
> Producer0 getNext() = partition0
> Producer1 getNext() = partition0
> Producer2 getNext() = partition0
> 
> A link to some of our test data prints:
> https://imgur.com/a/ha9OQMj
> 
> This, depending on how intensive (slow) your consumption rate is, may be a
> problem as it will generate enqueuing.
> We use Kafka as a messaging protocol in a big (and in some points heavy
> load) machine learning flow - for high throughput (lightweight processing)
> enqueuing is not an issue - añthough we saw it happening. but for heavy
> processes we are unable to do equal load balance.
> 
> We currently use the DefaultPartitioner and Kafka algorithm (murmur2 hash
> of the key) to decide the partition.
> We noticed enqueuing and timeouts while several consumers were idle - which
> made us take a better look on how the load is balanced.
> 
> I believe the only way to perform equal load balance without having to know
> other producers would be to do it on the Broker side. Do you agree?
> 
> Thanks,
> 
> 
> 
> On Mon, Jun 15, 2020 at 7:32 PM Colin McCabe  wrote:
> 
> > Hi Vinicius,
> >
> > It's actually not necessary for one producer to know about the others to
> > get an even distribution across partitions, right?  All that's really
> > required is that all producers produce a roughly equal amount of data to
> > each partition, which is what RoundRobinPartitioner is designed to do.  In
> > mathematical terms, the sum of several uniform random variables is itself
> > uniformly random.
> >
> > (There is a bug in RRP right now, KAFKA-9965, but it's not related to what
> > we're talking about now and we have a fix ready.)
> >
> > cheers,
> > Colin
> >
> >
> > On Sun, Jun 14, 2020, at 14:26, Vinicius Scheidegger wrote:
> > > Hi Collin,
> > >
> > > Thanks for the reply. Actually the RoundRobinPartitioner won't do an
> > equal
> > > distribution when working with multiple producers. One producer does not
> > > know the others. If you consider that producers are randomly producing
> > > messages, in the worst case scenario all producers can be synced and one
> > > could have as many messages in a single partition as the number of
> > > producers.
> > > It's easy to generate evidences of it.
> > >
> > > I have asked this question on the users mail list too (and on Slack and
> > on
> > > Stackoverflow).
> > >
> > > Kafka currently does not have means to do a round robin across multiple
> > > producers or on the broker side.
> > >
> > > This means there is currently NO GUARANTEE of equal distribution across
> > > partitions as the partition election is decided by the producer.
> > >
> > > There result is an unbalanced consumption when working with consumer
> > groups
> > > and the options are: creating a custom shared partitioner, relying on
> > Kafka
> > > random partition or introducing a middle man between topics (all of them
> > > having big cons).
> > >
> > > I thought of asking here to see whether this is a topic that could
> > concern
> > > other developers (and maybe understand whether this could be a KIP
> > > discussion)
> > >
> > > Maybe I'm missing something... I would like to know.
> > >
> > > According to my interpretation of the code (just read through some
> > > classes), but there is currently no way to do partition balancing on the
> > > broker - the producer sends messages directly to partition leaders so
> > > partition currently needs to be defined on the producer.
> > >
> > > I understand that in order to perform round robin across partitions of a
> > > topic when working with multiple producers, some development needs to be
> > > done. Am I right?
> > >
> > >
> > > Thanks
> > >
> > >
> > > On Fri, Jun 12, 2020, 10:57 PM Colin McCabe  wrote:
> > >
> > > > HI Vinicius,
> > > >
> > > > This question seems like a better fit for the user mailing list rather
> > > > than the developer mailing list.
> > > >
> > > > Anyway, if I understand correctly, you are asking if the producer can
> > > > 

[jira] [Created] (KAFKA-10166) Excessive TaskCorruptedException seen in soak

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-10166:
---

 Summary: Excessive TaskCorruptedException seen in soak
 Key: KAFKA-10166
 URL: https://issues.apache.org/jira/browse/KAFKA-10166
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman
 Fix For: 2.6.0


As the title indicates. Seen occasionally in the ALOS (~20 times in two days on 
one soak), and very frequently in the EOS (many times per day)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Broker side round robin on topic partitions when receiving messages

[Vinicius Scheidegger]

Hi Collin,

One producer shouldn't need to know about the other to distribute the load
equally, but what Kafka has now is roughly equal...
If you have a single producer RounRobinPartitioner works fine, if you have
10 producers you can have 7/8 messages in one partition while another
partition has none (producers are in sync - which happened a couple times
in our tests).

Producer0 getNext() = partition0
Producer1 getNext() = partition0
Producer2 getNext() = partition0

A link to some of our test data prints:
https://imgur.com/a/ha9OQMj

This, depending on how intensive (slow) your consumption rate is, may be a
problem as it will generate enqueuing.
We use Kafka as a messaging protocol in a big (and in some points heavy
load) machine learning flow - for high throughput (lightweight processing)
enqueuing is not an issue - añthough we saw it happening. but for heavy
processes we are unable to do equal load balance.

We currently use the DefaultPartitioner and Kafka algorithm (murmur2 hash
of the key) to decide the partition.
We noticed enqueuing and timeouts while several consumers were idle - which
made us take a better look on how the load is balanced.

I believe the only way to perform equal load balance without having to know
other producers would be to do it on the Broker side. Do you agree?

Thanks,



On Mon, Jun 15, 2020 at 7:32 PM Colin McCabe  wrote:

> Hi Vinicius,
>
> It's actually not necessary for one producer to know about the others to
> get an even distribution across partitions, right?  All that's really
> required is that all producers produce a roughly equal amount of data to
> each partition, which is what RoundRobinPartitioner is designed to do.  In
> mathematical terms, the sum of several uniform random variables is itself
> uniformly random.
>
> (There is a bug in RRP right now, KAFKA-9965, but it's not related to what
> we're talking about now and we have a fix ready.)
>
> cheers,
> Colin
>
>
> On Sun, Jun 14, 2020, at 14:26, Vinicius Scheidegger wrote:
> > Hi Collin,
> >
> > Thanks for the reply. Actually the RoundRobinPartitioner won't do an
> equal
> > distribution when working with multiple producers. One producer does not
> > know the others. If you consider that producers are randomly producing
> > messages, in the worst case scenario all producers can be synced and one
> > could have as many messages in a single partition as the number of
> > producers.
> > It's easy to generate evidences of it.
> >
> > I have asked this question on the users mail list too (and on Slack and
> on
> > Stackoverflow).
> >
> > Kafka currently does not have means to do a round robin across multiple
> > producers or on the broker side.
> >
> > This means there is currently NO GUARANTEE of equal distribution across
> > partitions as the partition election is decided by the producer.
> >
> > There result is an unbalanced consumption when working with consumer
> groups
> > and the options are: creating a custom shared partitioner, relying on
> Kafka
> > random partition or introducing a middle man between topics (all of them
> > having big cons).
> >
> > I thought of asking here to see whether this is a topic that could
> concern
> > other developers (and maybe understand whether this could be a KIP
> > discussion)
> >
> > Maybe I'm missing something... I would like to know.
> >
> > According to my interpretation of the code (just read through some
> > classes), but there is currently no way to do partition balancing on the
> > broker - the producer sends messages directly to partition leaders so
> > partition currently needs to be defined on the producer.
> >
> > I understand that in order to perform round robin across partitions of a
> > topic when working with multiple producers, some development needs to be
> > done. Am I right?
> >
> >
> > Thanks
> >
> >
> > On Fri, Jun 12, 2020, 10:57 PM Colin McCabe  wrote:
> >
> > > HI Vinicius,
> > >
> > > This question seems like a better fit for the user mailing list rather
> > > than the developer mailing list.
> > >
> > > Anyway, if I understand correctly, you are asking if the producer can
> > > choose to assign partitions in a round-robin fashion rather than based
> on
> > > the key.  The answer is, you can, by using RoundRobinPartitioner.
> (again,
> > > if I'm understanding the question correctly).
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Jun 9, 2020, at 00:48, Vinicius Scheidegger wrote:
> > > > Anyone?
> > > >
> > > > On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger <
> > > > vinicius.scheideg...@gmail.com> wrote:
> > > >
> > > > > Does anyone know how could I perform a load balance to distribute
> > > equally
> > > > > the messages to all consumers within the same consumer group having
> > > > > multiple producers?
> > > > >
> > > > > Is this a conceptual flaw on Kafka, wasn't it thought for equal
> > > > > distribution with multiple producers or am I missing something?
> > > > > I've asked on Stack Overflow, on Kafka users mailing 

Re: [VOTE] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

[Colin McCabe]

On Fri, Jun 12, 2020, at 15:30, Boyang Chen wrote:
> Thanks Colin for the suggestions!
> 
> On Fri, Jun 12, 2020 at 2:40 PM Colin McCabe  wrote:
> 
> > Hi Boyang,
> >
> > Thanks for the KIP!  I think it's getting close.
> >
> >  > For older requests that need redirection, forwarding
> >  > broker will just use its own authorizer to verify the principals. When
> > the
> >  > request looks good, it will just forward the request with its own
> >  > credentials, no second validation needed
> >
> > Just to be clear, the controller will still validate the request, right?
> > But at that point the principal will be the broker principal.  It would be
> > good to note that here.
> >
> > Sounds good, cleared in the KIP.
> 
> > Internal CreateTopicsRequest Routing
> >
> > The forwarding broker is sending the request as the latest version,
> > right?  It would be good to add a note of this.  This also prevents routing
> > loops since the latest version is not forwardable (another good thing to
> > add, I think...)
> >
> We are not bumping the CreateTopic RPC here, so it should be the latest
> by default.
>

Sorry, CreateTopics was a bad example here, since it already must be sent to 
the controller.  Oops.

>
> And just to be clear, we are not "forwarding" but actually
> sending a CreateTopicRequest from the receiving broker to the controller
> broker.
> 

Right.  I think we agree on this point.  But we do need a term to describe the 
broker which initially receives the user request and resends it to the 
controller.  Resending broker?

And I do think it's important to note that the request we send to the 
controller can't be itself resent.

>
>  > As we discussed in the request routing section, to work with an older
> >  > client, the first contacted broker need to act as a proxy to redirect
> > the
> >  > write request to the controller. To support the proxy of requests, we
> > need
> >  > to build a channel for brokers to talk directly to the controller. This
> >  > part of the design is internal change only and won’t block the KIP
> >  > progress.
> >
> > I think it's good to note that we eventually want a separate controller
> > endpoint in KIP-500.  However, we don't need it to implement KIP-590,
> > right?  The other brokers could forward to the existing internal endpoint
> > for the controller.  So maybe it's best to discuss the separate endpoint in
> > "future work" rather than here.
> >
> > I moved the new endpoint part towards the future work and addressed the
> > usage of controller internal endpoint for routing requests.
> 

Thanks.

> 
> > > === Start Old Proposal  ===
> >
> > I'm glad the old proposal shows up here, but I think this is too much
> > detail.  It would be better to just have a one or two paragraph summary of
> > the main points.  As it is, the old proposal takes up 40% of the doc which
> > is pretty confusing for someone reading through.  Let's also not forget
> > that someone can just read the old version by using the "page history"
> > function on the wiki.  So there's no need to keep that all here.
> >
> > Make sense, removed.
> 

Thanks again.

>
>{ "name": "PrincipalName", "type": "string", "tag": 0, "taggedVersions": 
> "2+", "ignorable": true,
>  "about": "Optional value of the principal name when the request is 
> redirected by a broker." },
>

Maybe "InitialPrincipalName" would be better here?  PrincipalName is a bit 
confusing since the message already has a principal name, after all...

cheers,
Colin

Re: [DISCUSSION] KIP-619: Add internal topic creation support

[Colin McCabe]

Hi Cheng,

The link from the main KIP page is an "edit link" meaning that it drops you 
into the editor for the wiki page.  I think the link you meant to use is a 
"view link" that will just take you to view the page.

In general I'm not sure what I'm supposed to take away from the large UML 
diagram in the KIP.  This is just a description of the existing code, right?  
Seems like we should remove this.

I'm not sure why the controller classes are featured here since as far as I can 
tell, the controller doesn't need to care if a topic is internal.

> Kafka and its upstream applications treat internal topics differently from
> non-internal topics. For example:
> * Kafka handles topic creation response errors differently for internal topics
> * Internal topic partitions cannot be added to a transaction
> * Internal topic records cannot be deleted
> * Appending to internal topics might get rejected

I think we need to understand which of these limitations we will carry forward 
and which we will not.  We also have the option of putting limitations just on 
consumer offsets, but not on other internal topics.

Taking it one by one:

> * Kafka handles topic creation response errors differently for internal 
> topics.

Hmm.  Kafka doesn't currently allow you to create internal topics, so the 
difference here is that you always fail, right?  Or is there something else 
more subtle here?  Like do we specifically prevent you from creating topics 
named __consumer_offsets or something?  We need to spell this all out in the 
KIP.

> * Internal topic partitions cannot be added to a transaction

I don't think we should carry this limitation forward, or if we do, we should 
only do it for consumer-offsets.  Does anyone know why this limitation exists?

> * Internal topic records cannot be deleted

This seems like something that should be handled by ACLs rather than by 
treating internal topics specially.

> * Appending to internal topics might get rejected

We clearly need to use ACLs here rather than rejecting appends.  Otherwise, how 
will external systems like KSQL, streams, etc. use this feature?  This is the 
kind of information we need to have in the KIP.

> Public Interfaces
> 2. KafkaZkClient will have a new method getInternalTopics() which 
> returns a set of internal topic name strings.

KafkaZkClient isn't a public interface, so it doesn't need to be described here.

> There are no compatibility concerns in this KIP.

I think there are a fair number of compatibility concerns.  What's the result 
if someone tries to create a topic with the configuration internal = true right 
now?  Does it fail?  If not, that seems like a potential problem.

Are people going to be able to create or delete topics named __consumer_offsets 
or __transaction_state using this mechanism?  If so, how does the security 
model work for that?

best,
Colin

On Fri, May 29, 2020, at 01:09, Cheng Tan wrote:
> Hello developers,
> 
> 
> I’m proposing KIP-619 to add internal topic creation support. 
> 
> Kafka and its upstream applications treat internal topics differently 
> from non-internal topics. For example:
> 
>   • Kafka handles topic creation response errors differently for internal 
> topics
>   • Internal topic partitions cannot be added to a transaction
>   • Internal topic records cannot be deleted
>   • Appending to internal topics might get rejected
>   • ……
> 
> Clients and upstream applications may define their own internal topics. 
> For example, Kafka Connect defines `connect-configs`, 
> `connect-offsets`, and `connect-statuses`. Clients are fetching the 
> internal topics by sending the MetadataRequest (ApiKeys.METADATA).
> 
> However, clients and upstream application cannot register their own 
> internal topics in servers. As a result, servers have no knowledge 
> about client-defined internal topics. They can only test if a given 
> topic is internal or not simply by checking against a static set of 
> internal topic string, which consists of two internal topic names 
> `__consumer_offsets` and `__transaction_state`. As a result, 
> MetadataRequest cannot provide any information about client created 
> internal topics. 
> 
> To solve this pain point, I'm proposing support for clients to register 
> and query their own internal topics. 
> 
> Please feel free to join the discussion. Thanks in advance.
> 
> 
> Best, - Cheng Tan

[jira] [Created] (KAFKA-10165) Percentiles metric leaking memory

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-10165:
---

 Summary: Percentiles metric leaking memory
 Key: KAFKA-10165
 URL: https://issues.apache.org/jira/browse/KAFKA-10165
 Project: Kafka
  Issue Type: Bug
  Components: metrics, streams
Reporter: Sophie Blee-Goldman
Assignee: Sophie Blee-Goldman
 Fix For: 2.6.0


We've hit several OOM in our soak cluster lately. We were finally able to get a 
heap dump right after the OOM, and found over 3.5 GB of memory being retained 
by the percentiles (or specifically by the 1MB float[] used by the 
percentiles). 

The leak does seem specific to the Percentiles class, as we see ~3000 instances 
of the Percentiles object vs only ~500 instances of the Max object, which is 
also used in the same sensor as the Percentiles

We did recently lower the size from 1MB to 100kB, but it's clear there is a 
leak of some kind and a "smaller leak" is not an acceptable solution. If the 
cause fo the leak is not immediately obvious we should just revert the 
percentiles in 2.6 and work on stabilizing them for 2.7



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Broker side round robin on topic partitions when receiving messages

[Colin McCabe]

Hi Vinicius,

It's actually not necessary for one producer to know about the others to get an 
even distribution across partitions, right?  All that's really required is that 
all producers produce a roughly equal amount of data to each partition, which 
is what RoundRobinPartitioner is designed to do.  In mathematical terms, the 
sum of several uniform random variables is itself uniformly random.

(There is a bug in RRP right now, KAFKA-9965, but it's not related to what 
we're talking about now and we have a fix ready.)

cheers,
Colin


On Sun, Jun 14, 2020, at 14:26, Vinicius Scheidegger wrote:
> Hi Collin,
> 
> Thanks for the reply. Actually the RoundRobinPartitioner won't do an equal
> distribution when working with multiple producers. One producer does not
> know the others. If you consider that producers are randomly producing
> messages, in the worst case scenario all producers can be synced and one
> could have as many messages in a single partition as the number of
> producers.
> It's easy to generate evidences of it.
> 
> I have asked this question on the users mail list too (and on Slack and on
> Stackoverflow).
> 
> Kafka currently does not have means to do a round robin across multiple
> producers or on the broker side.
> 
> This means there is currently NO GUARANTEE of equal distribution across
> partitions as the partition election is decided by the producer.
> 
> There result is an unbalanced consumption when working with consumer groups
> and the options are: creating a custom shared partitioner, relying on Kafka
> random partition or introducing a middle man between topics (all of them
> having big cons).
> 
> I thought of asking here to see whether this is a topic that could concern
> other developers (and maybe understand whether this could be a KIP
> discussion)
> 
> Maybe I'm missing something... I would like to know.
> 
> According to my interpretation of the code (just read through some
> classes), but there is currently no way to do partition balancing on the
> broker - the producer sends messages directly to partition leaders so
> partition currently needs to be defined on the producer.
> 
> I understand that in order to perform round robin across partitions of a
> topic when working with multiple producers, some development needs to be
> done. Am I right?
> 
> 
> Thanks
> 
> 
> On Fri, Jun 12, 2020, 10:57 PM Colin McCabe  wrote:
> 
> > HI Vinicius,
> >
> > This question seems like a better fit for the user mailing list rather
> > than the developer mailing list.
> >
> > Anyway, if I understand correctly, you are asking if the producer can
> > choose to assign partitions in a round-robin fashion rather than based on
> > the key.  The answer is, you can, by using RoundRobinPartitioner. (again,
> > if I'm understanding the question correctly).
> >
> > best,
> > Colin
> >
> > On Tue, Jun 9, 2020, at 00:48, Vinicius Scheidegger wrote:
> > > Anyone?
> > >
> > > On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger <
> > > vinicius.scheideg...@gmail.com> wrote:
> > >
> > > > Does anyone know how could I perform a load balance to distribute
> > equally
> > > > the messages to all consumers within the same consumer group having
> > > > multiple producers?
> > > >
> > > > Is this a conceptual flaw on Kafka, wasn't it thought for equal
> > > > distribution with multiple producers or am I missing something?
> > > > I've asked on Stack Overflow, on Kafka users mailing group, here (on
> > Kafka
> > > > Devs) and on Slack - and still have no definitive answer (actually
> > most of
> > > > the time I got no answer at all)
> > > >
> > > > Would something like this even be possible in the way Kafka is
> > currently
> > > > designed?
> > > > How does proposing for a KIP work?
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > On Thu, May 28, 2020, 3:44 PM Vinicius Scheidegger <
> > > > vinicius.scheideg...@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I'm trying to understand a little bit more about how Kafka works.
> > > >> I have a design with multiple producers writing to a single topic and
> > > >> multiple consumers in a single Consumer Group consuming message from
> > this
> > > >> topic.
> > > >>
> > > >> My idea is to distribute the messages from all producers equally. From
> > > >> reading the documentation I understood that the partition is always
> > > >> selected by the producer. Is that correct?
> > > >>
> > > >> I'd also like to know if there is an out of the box option to assign
> > the
> > > >> partition via a round robin *on the broker side *to guarantee equal
> > > >> distribution of the load - if possible to each consumer, but if not
> > > >> possible, at least to each partition.
> > > >>
> > > >> If my understanding is correct, it looks like in a multiple producer
> > > >> scenario there is lack of support from Kafka regarding load balancing
> > and
> > > >> customers have to either stick to the hash of the key (random
> > distribution,
> > > >> although it would 

Re: [DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

[Colin McCabe]

Hi Tom,

It's an interesting idea.  Obviously protocol buffers does this for all numeric 
fields.

I have to admit I have some mixed feelings, since this is another thing that 
makes encoding more complex.  And it's not a clear win in all cases, although 
it is in some.

I assume that the performance numbers here are for the old Struct-based 
encoding / decoding system.  Do we know what the numbers are when using the 
generated read and write functions?

I don't think it makes sense for type to be version-dependent.  Type ultimately 
translates into what Java type we should use to represent the field when it's 
in POJO form.  We can't have two types there.

Making encoding version-dependent is reasonable.  I do sort of wonder if we 
should just have something like "packedVersions" : "9+" rather than the 
"encoding" thing, though.  The latter is more conceptually elegant but it seems 
like it would be a pain to use.  For example, for a new field you would have to 
type "encoding": { "0+" : "packed" } which is kind of ugly.

Another thing here is that if we are going to go all this way for optimization, 
we should certainly give people the choice of whether to use zigzag encoding or 
not.  For fields that can never be negative, zigzag encoding is a waste.  So 
you would then have the option of unpacked, signed packed, and unsigned packed.

Finally, the Kafka protocol has a lot of fields which can never be negative, 
except for some special cases where they are -1.  But no other negative numbers 
are allowed.  So we should consider making the "unsigned packed" option 
actually encode num + 1 just to support this usage.  That's what we did for 
string and bytes prefix length encoding and it worked well.

Before we do all this, though, one simpler improvement would be making all the 
"error" fields into tagged fields.  Most of them remain at 0 most of the time, 
so this could very well provide a big savings without any big encoding changes.

best,
Colin


On Mon, Jun 15, 2020, at 05:34, Tom Bentley wrote:
> Hi,
> 
> I'd like to start discussion on KIP-625: Richer encodings for
> integral-typed protocol fields.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-625%3A+Richer+encodings+for+integral-typed+protocol+fields
> 
> It's about allowing regular/required fields of protocol messages to use
> variable length encoding. If you have a moment please take a look.
> 
> Kind regards,
> 
> Tom
>

[DISCUSS]: KIP-625: Richer encodings for integral-typed protocol fields.

[Tom Bentley]

Hi,

I'd like to start discussion on KIP-625: Richer encodings for
integral-typed protocol fields.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-625%3A+Richer+encodings+for+integral-typed+protocol+fields

It's about allowing regular/required fields of protocol messages to use
variable length encoding. If you have a moment please take a look.

Kind regards,

Tom

Re: First time patch submitter advice

[Luke Chen]

Hi Michael,
The failed unit test has already handled here:
https://issues.apache.org/jira/browse/KAFKA-10155
https://issues.apache.org/jira/browse/KAFKA-10147

So, maybe you can ignore the test errors and mention the issue number in PR.
Thanks.

Luke

On Mon, Jun 15, 2020 at 3:23 PM Michael Carter <
michael.car...@instaclustr.com> wrote:

> Thanks for the response Gwen, that clarifies things for me.
>
> Regarding the unit test (ReassignPartitionsUnitTest.
> testModifyBrokerThrottles),  it appears to fail quite reliably on trunk as
> well (at least on my machine).
> It looks to me like a new override to
> MockAdminClient.describeConfigs(Collection resources)
> (MockAdminClient.java line 369) introduced in commit
> 48b56e533b3ff22ae0e2cf7fcc649e7df19f2b06 changed the behaviour of this
> method that the unit test relied on.
> I’ve just now put a patch into my branch to make that test pass by calling
> a slightly different version of describeConfigs (that avoids the overridden
> behaviour). It’s probably arguable whether that constitutes a fix or not
> though.
>
> Cheers,
> Michael
>
> > On 15 Jun 2020, at 3:41 pm, Gwen Shapira  wrote:
> >
> > Hi,
> >
> > 1. Unfortunately, you need to get a committer to approve running the
> tests.
> > I just gave the green-light on your PR.
> > 2. You can hope that committers will see your PR, but sometimes things
> get
> > lost. If you know someone who is familiar with that area of the code, it
> is
> > a good idea to ping them.
> > 3. We do have some flaky tests. You can see that Jenkins will run 3
> > parallel builds, if some of them pass and the committer confirms that
> > failures are not related to your code, we are ok to merge. Obviously, if
> > you end up tracking them down and fixing, everyone will be very grateful.
> >
> > Hope this helps,
> >
> > Gwen
> >
> > On Sun, Jun 14, 2020 at 5:52 PM Michael Carter <
> > michael.car...@instaclustr.com> wrote:
> >
> >> Hi all,
> >>
> >> I’ve submitted a patch for the first time(
> >> https://github.com/apache/kafka/pull/8844 <
> >> https://github.com/apache/kafka/pull/8844>), and I have a couple of
> >> questions that I’m hoping someone can help me answer.
> >>
> >> I’m a little unclear what happens after that patch has been submitted.
> The
> >> coding guidelines say Jenkins will run tests automatically, but I don’t
> see
> >> any results anywhere. Have I misunderstood what should happen, or do I
> just
> >> not know where to look?
> >> Should I be attempting to find reviewers for the change myself, or is
> that
> >> done independently of the patch submitter?
> >>
> >> Also, in resolving a couple of conflicts that have arisen after the
> patch
> >> was first submitted, I noticed that there are now failing unit tests
> that
> >> have nothing to do with my change. Is there a convention on how to deal
> >> with these? Should it be something that I try to fix on my branch?
> >>
> >> Any thoughts are appreciated.
> >>
> >> Thanks,
> >> Michael
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
>
>

[jira] [Created] (KAFKA-10164) Implement Admin side changes

[David Jacot (Jira)]

David Jacot created KAFKA-10164:
---

 Summary: Implement Admin side changes
 Key: KAFKA-10164
 URL: https://issues.apache.org/jira/browse/KAFKA-10164
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10162) Update Rate implementation to cope with spiky workload

[David Jacot (Jira)]

David Jacot created KAFKA-10162:
---

 Summary: Update Rate implementation to cope with spiky workload
 Key: KAFKA-10162
 URL: https://issues.apache.org/jira/browse/KAFKA-10162
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10163) Implement Broker side changes

[David Jacot (Jira)]

David Jacot created KAFKA-10163:
---

 Summary: Implement Broker side changes
 Key: KAFKA-10163
 URL: https://issues.apache.org/jira/browse/KAFKA-10163
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10161) Update Documentation

[David Jacot (Jira)]

David Jacot created KAFKA-10161:
---

 Summary: Update Documentation
 Key: KAFKA-10161
 URL: https://issues.apache.org/jira/browse/KAFKA-10161
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

[David Jacot]

Hi all,

The vote has passed with 5 binding votes (Gwen, Rajini, Mickael, Jun and
Colin)
and 2 non-binding votes (Tom, Anna).

Thank you all for the fruitful discussion! I'd like to particularly thank
Anna who has
heavily contributed to the design of this KIP.

Regards,
David

On Fri, Jun 12, 2020 at 10:08 PM Colin McCabe  wrote:

> +1.  Thanks, David!
>
> best,
> Colin
>
> On Thu, Jun 11, 2020, at 23:51, David Jacot wrote:
> > Colin, Jun,
> >
> > Do the proposed error code and the updated KIP look good to you guys? I’d
> > like to wrap up and close the vote.
> >
> > Thanks,
> > David
> >
> > Le mer. 10 juin 2020 à 14:50, David Jacot  a écrit
> :
> >
> > > Hi Colin and Jun,
> > >
> > > I have no problem if we have to rewrite part of it when the new
> controller
> > > comes
> > > out. I will be more than happy to help out.
> > >
> > > Regarding KIP-590, I think that we can cope with a principal as a
> string
> > > when the
> > > time comes. The user entity name is defined with a string already.
> > >
> > > Regarding the name of the error, you have made a good point. I do agree
> > > that it
> > > is important to differentiate the two cases. I propose the following
> two
> > > errors:
> > > - THROTTLING_QUOTA_EXCEEDED - Throttling is slightly better than rate
> as
> > > we have quotas which are not rate (e.g. request quota). This one is
> > > retryable
> > > once the throttle time is passed.
> > > - LIMIT_QUOTA_EXCEEDED - This one would indicate that the limit has
> been
> > > reached and is a final error.
> > > We only need the former in this KIP. What do you think?
> > >
> > > Jun, I have added a few examples in the KIP. The new name works exactly
> > > like
> > > the existing one once it is added to the accepted dynamic configs for
> the
> > > user
> > > and the client entities. I have added a "Kafka Config Command" chapter
> in
> > > the
> > > KIP. I will also open a Jira to not forget updating the AK
> documentation
> > > once
> > > the KIP gets merged.
> > >
> > > Thanks,
> > > David
> > >
> > > On Wed, Jun 10, 2020 at 3:03 AM Jun Rao  wrote:
> > >
> > >> Hi, Colin,
> > >>
> > >> Good point. Maybe sth like THROTTLING_QUOTA_VIOLATED will make this
> clear.
> > >>
> > >> Hi, David,
> > >>
> > >> We added a new quota name in the KIP. You chose not to bump up the
> version
> > >> of DESCRIBE_CLIENT_QUOTAS and ALTER_CLIENT_QUOTAS, which seems ok
> since
> > >> the
> > >> quota name is represented as a string. However, the new quota name
> can be
> > >> used in client tools for setting and listing the quota (
> > >> https://kafka.apache.org/documentation/#quotas). Could you document
> how
> > >> the
> > >> new name will be used in those tools?
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Jun 9, 2020 at 3:37 PM Colin McCabe 
> wrote:
> > >>
> > >> > On Tue, Jun 9, 2020, at 05:06, David Jacot wrote:
> > >> > > Hi Colin,
> > >> > >
> > >> > > Thank you for your feedback.
> > >> > >
> > >> > > Jun has summarized the situation pretty well. Thanks Jun! I would
> > >> like to
> > >> > > complement it with the following points:
> > >> > >
> > >> > > 1. Indeed, when the quota is exceeded, the broker will reject the
> > >> topic
> > >> > > creations, partition creations and topics deletions that are
> exceeding
> > >> > > with the new QUOTA_VIOLATED error. The ThrottleTimeMs field will
> > >> > > be populated accordingly to let the client know how long it must
> wait.
> > >> > >
> > >> > > 2. I do agree that we actually want a mechanism to apply back
> pressure
> > >> > > to the clients. The KIP basically proposes a mechanism to control
> and
> > >> to
> > >> > > limit the rate of operations before entering the controller. I
> think
> > >> that
> > >> > > it is similar to your thinking but is enforced based on a defined
> > >> > > instead of relying on the number of pending items in the
> controller.
> > >> > >
> > >> > > 3. You mentioned an alternative idea in your comments that, if I
> > >> > understood
> > >> > > correctly, would bound the queue to limit the overload and reject
> > >> work if
> > >> > > the queue is full. I have been thinking about this as well but I
> don't
> > >> > think
> > >> > > that it  works well in our case.
> > >> > > - The first reason is the one mentioned by Jun. We actually want
> to be
> > >> > able
> > >> > > to limit specific clients (the misbehaving ones) in a multi-tenant
> > >> > > environment.
> > >> > > - The second reason is that, at least in our current
> implementation,
> > >> the
> > >> > > length of the queue is not really a good characteristic to
> estimate
> > >> the
> > >> > load.
> > >> > > Coming back to your example of the CreateTopicsRequest. They
> create
> > >> path
> > >> > >  in ZK for each newly created topics which trigger a ChangeTopic
> event
> > >> > in
> > >> > > the controller. That single event could be for a single topic in
> some
> > >> > cases or
> > >> > > for a thousand topics in others.
> > >> > > These two reasons aside, bounding 

Re: First time patch submitter advice

[Michael Carter]

Thanks for the response Gwen, that clarifies things for me.

Regarding the unit test (ReassignPartitionsUnitTest. 
testModifyBrokerThrottles),  it appears to fail quite reliably on trunk as well 
(at least on my machine).
It looks to me like a new override to 
MockAdminClient.describeConfigs(Collection resources) 
(MockAdminClient.java line 369) introduced in commit 
48b56e533b3ff22ae0e2cf7fcc649e7df19f2b06 changed the behaviour of this method 
that the unit test relied on.
I’ve just now put a patch into my branch to make that test pass by calling a 
slightly different version of describeConfigs (that avoids the overridden 
behaviour). It’s probably arguable whether that constitutes a fix or not though.

Cheers,
Michael

> On 15 Jun 2020, at 3:41 pm, Gwen Shapira  wrote:
> 
> Hi,
> 
> 1. Unfortunately, you need to get a committer to approve running the tests.
> I just gave the green-light on your PR.
> 2. You can hope that committers will see your PR, but sometimes things get
> lost. If you know someone who is familiar with that area of the code, it is
> a good idea to ping them.
> 3. We do have some flaky tests. You can see that Jenkins will run 3
> parallel builds, if some of them pass and the committer confirms that
> failures are not related to your code, we are ok to merge. Obviously, if
> you end up tracking them down and fixing, everyone will be very grateful.
> 
> Hope this helps,
> 
> Gwen
> 
> On Sun, Jun 14, 2020 at 5:52 PM Michael Carter <
> michael.car...@instaclustr.com> wrote:
> 
>> Hi all,
>> 
>> I’ve submitted a patch for the first time(
>> https://github.com/apache/kafka/pull/8844 <
>> https://github.com/apache/kafka/pull/8844>), and I have a couple of
>> questions that I’m hoping someone can help me answer.
>> 
>> I’m a little unclear what happens after that patch has been submitted. The
>> coding guidelines say Jenkins will run tests automatically, but I don’t see
>> any results anywhere. Have I misunderstood what should happen, or do I just
>> not know where to look?
>> Should I be attempting to find reviewers for the change myself, or is that
>> done independently of the patch submitter?
>> 
>> Also, in resolving a couple of conflicts that have arisen after the patch
>> was first submitted, I noticed that there are now failing unit tests that
>> have nothing to do with my change. Is there a convention on how to deal
>> with these? Should it be something that I try to fix on my branch?
>> 
>> Any thoughts are appreciated.
>> 
>> Thanks,
>> Michael
> 
> 
> 
> -- 
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog

[jira] [Created] (KAFKA-10160) Kafka MM2 consumer configuration

[Pavol Ipoth (Jira)]

Pavol Ipoth created KAFKA-10160:
---

 Summary: Kafka MM2 consumer configuration
 Key: KAFKA-10160
 URL: https://issues.apache.org/jira/browse/KAFKA-10160
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.4.1, 2.5.0
Reporter: Pavol Ipoth


[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMakerConfig.java#L51,]
 according this producer/consumer level properties should be configured as e.g. 
somesource->sometarget.consumer.client.id, i try to set 
somesource->sometarget.consumer.auto.offset.reset=latest, but without success, 
consumer always tries to fetch earliest, not sure if bug or my 
misconfiguration, but then at least some update to docu would be useful



--
This message was sent by Atlassian Jira
(v8.3.4#803005)