Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

[Manikumar]

Thanks for the KIP. LGTM.

On Wed, Aug 5, 2020 at 3:43 AM Ron Dagostino  wrote:

> Yes, Rajinit, it looks good -- an excellently-written KIP.  Thanks!
>
> Ron
>
> On Mon, Aug 3, 2020 at 9:17 AM Andrew Otto  wrote:
>
> > Yes please!  I had to build custom tooling to automate generation and
> > distribution of java truststore and keystore files at Wikimedia for Kafka
> > brokers and clients.  PEM is much more standard outside of the Java world
> > and it would be simpler to not have to maintain custom tooling for
> > keystores.
> >
> > - Andrew Otto
> >
> > On Mon, Aug 3, 2020 at 7:29 AM Rajini Sivaram 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have submitted KIP-651 to support PEM format for SSL key and trust
> > > stores:
> > >
> > >-
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key
> > >
> > > This enables better integration with other Kafka features like dynamic
> > > config update and password protection.
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>

[GitHub] [kafka-site] showuon commented on pull request #284: MINOR: Change the arrow direction based on the view state is expanded or not

[GitBox]

showuon commented on pull request #284:
URL: https://github.com/apache/kafka-site/pull/284#issuecomment-669677637


   @scott-confluent @guozhangwang , could you help review this PR? Thanks.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] showuon opened a new pull request #284: MINOR: Change the arrow direction based on the view state is expanded or not

[GitBox]

showuon opened a new pull request #284:
URL: https://github.com/apache/kafka-site/pull/284


   Change the arrow direction based on the view state is expanded or not. We 
can identify it's in expanded mode or not by checking the return value of 
toggle method. Reference: 
https://developer.mozilla.org/en-US/docs/Web/API/DOMTokenList/toggle
   
   
![image](https://user-images.githubusercontent.com/43372967/89490258-00d72500-d7df-11ea-8582-8bae5005a21e.png)
   
   
   
   
![image](https://user-images.githubusercontent.com/43372967/89490628-dcc81380-d7df-11ea-8335-8c4551253790.png)
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Unmesh Joshi]

Hi,
I have built a small prototype of how the controller backed by consensus
implementation will look like.
https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/Kip631Controller.scala
This is a miniature Kafka like implementation I use to teach distributed
systems at ThoughtWorks, and also experiment with my work on patterns of
distributed systems
.
It has a working end to end test to demonstrate how the overall system will
possibly look like
https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/ProducerConsumerKIP500Test.scala
The Kip631Controller itself depends on a Consensus module, to demonstrate
how possible interactions with the consensus module will look like
 (The Consensus can be pluggable really, with an API to allow reading
replicated log upto HighWaterMark)
As of now I do not have complete state transition implementation of broker
states. But it has implementation of LeaseTracker
https://github.com/unmeshjoshi/distrib-broker/blob/master/src/main/scala/com/dist/simplekafka/kip500/LeaderLeaseTracker.scala
to demonstrate LeaseTracker's interaction with the consensus module. (It
needs to propose and commit a FenceBroker record for expired leases)
I am hoping to add more specific tests for broker state transition here
https://github.com/unmeshjoshi/distrib-broker/blob/master/src/test/scala/com/dist/simplekafka/kip500/Kip631ControllerTest.scala

I hope it will be helpful in driving some discussions and doing quick
experimentation outside of the Kafka codebase.

Thanks,
Unmesh






On Tue, Aug 4, 2020 at 7:52 AM Unmesh Joshi  wrote:

> Hi,
>
> The LeaseStartTimeMs is expected to be the broker's
> 'System.currentTimeMillis()' at the point of the request. The active
> controller will add its lease period to this in order to compute
> the LeaseEndTimeMs.
> I think this is a bit confusing.  Monotonic clock on the active controller
> should be used to track leases.
> For example,
> https://issues.apache.org/jira/browse/ZOOKEEPER-1616
>
> https://github.com/etcd-io/etcd/pull/6888/commits/e7f4010ccaf28b6ce64fe514d25a4b2fa459d114
>
> Then we will not need LeaseStartTimeMs?
> Instead of LeaseStartTimeMs, can we call it LeaseTTL? The active
> controller can then calculate LeaseEndTime = System.nanoTime() + LeaseTTL.
> In this case we might just drop LeaseEndTimeMs from the response, as the
> broker already knows about the TTL and can send heartbeats at some fraction
> of TTL, say every TTL/4 milliseconds.(elapsed time on the broker measured
> by System.nanoTime)
>
> Thanks,
> Unmesh
>
>
>
>
>
> On Tue, Aug 4, 2020 at 4:48 AM Colin McCabe  wrote:
>
>> On Mon, Aug 3, 2020, at 15:51, Jose Garcia Sancio wrote:
>> > Thanks for the KIP Colin,
>> >
>> > Here is a partial review:
>> >
>> > > 1. Even when a broker and a controller are co-located in the same
>> JVM, they must
>> > > have different node IDs
>> >
>> > Why? What problem are you trying to solve?
>> >
>>
>> Hi Jose,
>>
>> Thanks for the comments.
>>
>> We did talk about this a bit earlier in the thread.  The controller is
>> always on its own port, even when it is running in the same JVM as a
>> broker.  So it would not make sense to share the same ID here-- your
>> messages would not get through to the controller if you sent them to the
>> broker port instead.  And clearly if the controller is on a separate host
>> from any broker, it can't share a broker id.
>>
>> >
>> > > 2. Node IDs must be set in the configuration file for brokers and
>> controllers.
>> >
>> > I understand that controller IDs must be static and in the
>> > configuration file to be able to generate consensus in KIP-595. Why
>> > are the broker nodes which are observers in KIP-595 cannot discover
>> > their ID on first boot and persist their ID for consistency in future
>> > restarts?
>> >
>>
>> This is discussed in the rejected alternatives section.  Basically, node
>> ID auto-assignment is complicated and antiquated in the age of Kubernetes,
>> Chef, Puppet, Ansible, etc.
>>
>> >
>> > > 3. Controller processes will listen on a separate endpoint from
>> brokers
>> >
>> > Why is this? Kafka supports multi endpoints. For example, one broker
>> > can have one endpoint for listening to connections by other brokers
>> > and another endpoint for connections from admin, producer and consumer
>> > clients.
>> >
>>
>> The reason for having separate ports is discussed in KIP-590.  Basically,
>> it is so that control plane traffic can be isolated from data plane
>> traffic, as much as possible.  This is the existing situation with
>> ZooKeeper.  Since ZK is on a separate port, the client cannot disrupt the
>> cluster by flooding it with traffic (unless the admin has unwisely exposed
>> all internal ports, but this would cause bigger security issues).  We want
>> to preserve this property.
>>
>> > > 4. In the case of controller RPCs like 

Jenkins build is back to normal : kafka-2.6-jdk8 #106

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #4765

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Code cleanup in StreamsResetter (#9126)

[github] MINOR: Streams integration tests should not call exit (#9067)


--
[...truncated 6.36 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Jason Gustafson]

Hi Ryan,

Thanks for the proposal. Just a few quick questions:

1. I wonder if we need to bother with `enable.dynamic.config`, especially
if the default is going to be true anyway. I think users who don't want to
use this capability can just not set dynamic configs. The only case I can
see an explicit opt-out being useful is when users are trying to avoid
getting affected by dynamic defaults. And on that note, is there a strong
case for supporting default overrides? Many client configs are tied closely
to application behavior, so it feels a bit dangerous to give users the
ability to override the configuration for all applications.

2. Tying dynamic configurations to clientId has some downsides. It is
common for users to use a different clientId for every application in a
consumer group so that it is easier to tie group members back to where
the client is running. This makes setting configurations at an application
level cumbersome. The alternative is to use the default, but that means
hitting /all/ applications which I think is probably not a good idea. A
convenient alternative for consumers would be to use group.id, but we don't
have anything similar for the producer. I am wondering if we need to give
the clients a separate config label of some kind so that there is a
convenient way to group configurations. For example `config.group`. Note
that this would be another way to opt into dynamic config support.

3. I'm trying to understand the contract between brokers and clients to
support dynamic configurations. I imagine that once this is available,
users will have a hard time telling which applications support the
capability and which do not. Also, we would likely add new dynamic config
support over time which would make this even harder since we cannot
retroactively change clients to add support for new dynamic configs. I'm
wondering if there is anything we can do to make it easier for users to
tell which dynamic configs are available for each application.

4. In the case of `session.timeout.ms`, even if the config is updated, the
group will need to be rebalanced for it to take effect. This is because the
session timeout is sent to the group coordinator in the JoinGroup request.
I'm wondering if we need to change the JoinGroup behavior so that it can be
used to update the session timeout without triggering a rebalance.

Thanks,
Jason




On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn  wrote:

> Hi David,
>
> Here are some additional thoughts...
>
> > 1. Once dynamic configs have been loaded and resolved, how can a client
> > know what values are selected?
>
> A copy of the original user-provided configs is kept by the client.
> Currently these are used to revert to the user-provided config if a dynamic
> config is deleted. However, they can also be used to distinguish between
> dynamic and user-provided configs.
>
> > 3. Are there other configs we'd like to allow the broker to push up to
> the
> > clients? Did we consider making this mechanism generic so the broker
> could
> > push any consumer/producer config up to the clients via dynamic configs?
>
> Rephrasing my answer to this question:
>
> The mechanism for sending and altering configs is rather generic. However,
> the client-side handling of these configs is not. The reason for this is
> that configs affect the behavior of the clients in specific ways, so the
> client must reconfigure itself in a specific way for each different config.
>
> An example of this is that when session.timeout.ms is dynamically
> configured, the consumer must rejoin the group by sending a
> JoinGroupRequest. This is because the session timeout is sent in the
> initial JoinGroupRequest to the coordinator and stored with the rest of the
> group member's metadata. To reconfigure the client, the value in the
> coordinator must also be changed. This does not need to be done for
> heartbeat.interval.ms.
>
>
> On 2020/08/03 17:47:19, David Arthur  wrote:
> > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > questions
> >
> > 1. Once dynamic configs have been loaded and resolved, how can a client
> > know what values are selected? Will we log the actual resolved values
> once
> > they are loaded?
> >
> > 2. Do we want to support the case that we load dynamic configs but also
> > allow client overrides?
> >
> > 3. Are there other configs we'd like to allow the broker to push up to
> the
> > clients? Did we consider making this mechanism generic so the broker
> could
> > push any consumer/producer config up to the clients via dynamic configs?
> >
> > Also, minor note, in ConfigResource#Type, I think the byte values are
> used
> > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> >
> > -David
> >
> > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn 
> > wrote:
> >
> > > Thanks Jose,
> > > Here are the changes to the KIP:
> > >
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615=10=9
> > >
> > > 1. Done.

[jira] [Created] (KAFKA-10366) TimeWindowedDeserializer doesn't allow users to set a custom window size

[Leah Thomas (Jira)]

Leah Thomas created KAFKA-10366:
---

 Summary: TimeWindowedDeserializer doesn't allow users to set a 
custom window size
 Key: KAFKA-10366
 URL: https://issues.apache.org/jira/browse/KAFKA-10366
 Project: Kafka
  Issue Type: Bug
Reporter: Leah Thomas
Assignee: Leah Thomas


Related to [KAFKA-4468|https://issues.apache.org/jira/browse/KAFKA-4468], in 
timeWindowedDeserializer Long.MAX_VALUE is used as _windowSize_ for any 
deserializer that uses the default constructor. While streams apps can pass in 
a window size in serdes or while creating a timeWindowedDeserializer, the 
deserializer that is actually used in processing the messages is created by the 
Kafka consumer, without passing in the set windowSize. The deserializer the 
consumer creates uses the configs, but as there is no config for windowSize, 
the window size is always default.

See _KStreamAggregationIntegrationTest #ShouldReduceWindowed()_ as an example 
of this issue. Despite passing in the windowSize to both the serdes and the 
timeWindowedDeserializer, the window size is set to Long.MAX_VALUE. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk14 #342

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] rhauch commented on a change in pull request #277: 2.6.0 release

[GitBox]

rhauch commented on a change in pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#discussion_r466057952



##
File path: 26/upgrade.html
##
@@ -19,6 +19,48 @@
 
 

[GitHub] [kafka-site] rhauch commented on a change in pull request #277: 2.6.0 release

[GitBox]

rhauch commented on a change in pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#discussion_r466057579



##
File path: 26/upgrade.html
##
@@ -19,6 +19,50 @@
 
 

Build failed in Jenkins: kafka-trunk-jdk11 #1691

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Code cleanup in StreamsResetter (#9126)

[github] MINOR: Streams integration tests should not call exit (#9067)

[github] KAFKA-9274: Remove `retries` for global task (#9047)


--
[...truncated 3.21 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

[GitHub] [kafka-site] guozhangwang merged pull request #282: bug fixes

[GitBox]

guozhangwang merged pull request #282:
URL: https://github.com/apache/kafka-site/pull/282


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-10365) Cannot delete topic in Windows

[Nilesh Balu Jorwar (Jira)]

Nilesh Balu Jorwar created KAFKA-10365:
--

 Summary: Cannot delete topic in Windows
 Key: KAFKA-10365
 URL: https://issues.apache.org/jira/browse/KAFKA-10365
 Project: Kafka
  Issue Type: Bug
Reporter: Nilesh Balu Jorwar


Even after checked out latest kafka source code and built. When ran zookeeper 
and server, deleted the topic created, server fails and gives below error:

*Suppressed: java.nio.file.AccessDeniedException: 
E:\nil\logs\kf-logs\javatechie2-0 -> 
 E:\nil\logs\kf-logs\javatechie2-0.793787296b094bb69b4fa3135584a6f4-delete
at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:911)
... 14 more
[2020-08-05 16:12:30,383] WARN [ReplicaManager broker=0] Stopping serving 
replicas in dir E:\nil\logs\kf-logs (kafka.server.ReplicaManager)
[2020-08-05 16:12:30,515] INFO [ReplicaFetcherManager on broker 0] Removed 
fetcher for partitions Set(javatechie3-0, javatechie-0) 
(kafka.server.ReplicaFetcherManager)
[2020-08-05 16:12:30,517] INFO [ReplicaAlterLogDirsManager on broker 0] Removed 
fetcher for partitions Set(javatechie3-0, javatechie-0) 
(kafka.server.ReplicaAlterLogDirsManager)
[2020-08-05 16:12:30,525] WARN [ReplicaManager broker=0] Broker 0 stopped 
fetcher for partitions javatechie3-0,javatechie-0 and stopped moving logs for 
partitions  because they are in the failed log directory E:\nil\logs\kf-logs. 
(kafka.server.ReplicaManager)
[2020-08-05 16:12:30,527] WARN Stopping serving logs in dir E:\nil\logs\kf-logs 
(kafka.log.LogManager)
[2020-08-05 16:12:30,535] ERROR Shutdown broker because all log dirs in 
E:\nil\logs\kf-logs have failed (kafka.log.LogManager)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Preliminary blog post about the Apache 2.6.0 release

[Maulin Vasavada]

Thanks Randall. That clarifies things. Makes sense!

On Wed, Aug 5, 2020 at 11:38 AM Randall Hauch  wrote:

> Hi, Maulin.
>
> We have traditionally not mentioned *every* KIP in the project's blog posts
> -- unless a release just happens to involve a small number of KIPs. This
> means that we do have to choose a subset of the KIPs -- AK 2.6.0 had 30
> KIPs that were implemented, and a blog summarizing each would be too long
> and much less widely received. I tried to choose KIPs that clearly impacted
> the broadest set of users, and finding a subset wasn't that easy.
>
> I chose not to include KIP-519 in the blog simply because it requires
> installing and integration into the broker components that are not included
> in the official distribution. I hope that helps explain my thought process.
>
> Best regards,
>
> Randall
>
> On Wed, Aug 5, 2020 at 12:52 PM Maulin Vasavada  >
> wrote:
>
> > Hi Randall
> >
> > One question: Do we mention all KIPs/NewFeatures in the blog that are
> > listed in the release notes document -
> > https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?
> >
> > I see that [KAFKA-8890  >]
> > - KIP-519: Make SSL context/engine configuration extensible is missing
> from
> > the google doc that you shared.
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:
> >
> > > Thanks, Jason. We haven't done that for a few releases, but I think
> it's
> > a
> > > great idea. I've updated the blog post draft and the Google doc to
> > mention
> > > the 127 contributors by name that will also be mentioned in the email
> > > release announcement.
> > >
> > > I also linked to the project's downloads page in the opening sentence,
> > and
> > > tweaked the wording slightly in the first paragraph.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Randall,
> > > >
> > > > Thanks for putting this together. I think it would be great if the
> blog
> > > > included the names of the release contributors. It's an easy way to
> > give
> > > > some recognition. I know we have done that in the past.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch 
> > wrote:
> > > >
> > > > > I've prepared a preliminary blog post about the upcoming Apache
> Kafka
> > > > 2.6.0
> > > > > release.
> > > > > Please take a look and let me know via this thread if you want to
> > > > > add/modify details.
> > > > > Thanks to all who contributed to this blog post.
> > > > >
> > > > > Unfortunately, the preview is not currently publicly visible at
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > > > (I've
> > > > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in
> the
> > > > > interest of time I created a Google doc with read privilege for
> > > everyone
> > > > by
> > > > > pasting the preview content directly into:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > > > >
> > > > > I will continue to update this doc with any changes to the draft
> blog
> > > > post.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Randall
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-595: A Raft Protocol for the Metadata Quorum

[Jun Rao]

Hi, Jason,

Thanks for the KIP. +1

Just to confirm. For those newly added request types, will we expose the
existing latency metrics (total, local, remote, etc) with a new tag
request=[request-type]?

Jun

On Tue, Aug 4, 2020 at 3:00 PM Boyang Chen 
wrote:

> Thanks for the KIP Jason, +1 (binding) from me as well for sure :)
>
>
> On Tue, Aug 4, 2020 at 2:46 PM Colin McCabe  wrote:
>
> > On Mon, Aug 3, 2020, at 20:55, Jason Gustafson wrote:
> > > Hi Colin,
> > >
> > > Thanks for the responses.
> > >
> > > > I have a few lingering questions.  I still don't like the fact that
> the
> > > > leader epoch / fetch epoch is 31 bits.  What happens when this rolls
> > over?
> > > > Can we just make this 63 bits now so that we never have to worry
> about
> > it
> > > > again?  ZK has some awful bugs surrounding 32 bit rollover, due to a
> > > > similar decision to use a 32 bit counter in their log structure.
> > Doesn't
> > > > seem like a good tradeoff.
> > >
> > > This is a bit difficult to do at the moment since the leader epoch is 4
> > > bytes in the message format. One option that I have considered is
> > toggling
> > > a batch attribute that lets us turn the producerId into an 8-byte
> leader
> > > epoch instead since we do not have a use for it in the metadata quorum.
> > We
> > > would need another solution if we ever wanted to use Raft for partition
> > > replication, but perhaps by then we can make the case for a new message
> > > format.
> > >
> >
> > Hi Jason,
> >
> > Thanks for the explanation.  I suspected that there was a technical
> > limitation like this lurking somewhere.  I think a hack like the one you
> > suggested would be OK for now.  I just really want to avoid thinking
> about
> > rollover :)
> >
> > Regarding the epoch overflow, some offline discussions among Jason,
> Guozhang, Jose and I reached some conclusions:
>
> 1. The current default election timeout is 10 seconds, which means it takes
> hundreds of years to be exhausted if just bumping through election timeout.
> Even if the user sets it to 1 second, it still needs years to exhaust.
>
> 2. The most common case for fast epoch bumps is due to network partition.
> If a certain voter couldn't connect to the quorum, it will repeatedly start
> elections and do the epoch bump. To mitigate this concern, we already
> planned a follow-up KIP to add the `pre-vote` feature to Kafka Raft
> implementation as described in the literature to avoid rapid epoch
> increments in the algorithm level.
>
> 3. As you suggested, the leader epoch overflow was a common problem not
> just for Raft. We could kick off a separate KIP to address changing epoch
> from 4 bytes to 8 bytes through message format upgrade, to solve the issue
> for Kafka in a holistic manner.
>
>
>
> > >
> > > > Just like in bootstrap.servers, I don't think we want to manually
> > assign
> > > > IDs per hostname.  The hosts know their own IDs, after all.  Having
> to
> > > > manually specify the IDs also opens up the possibility of
> > > > misconfigurations: what I say the foobar server is node 2, but it's
> > > > actually node 3? This would make the logs extremely confusing.  I
> > realize
> > > > this may require a little finesse to do, but there's got to be a way
> > we can
> > > > avoid hard-coding IDs
> > >
> > > Fine. We can move this to KIP-631, but I think it would be a mistake to
> > > take IDs out of this configuration. For safety, the one thing that the
> > > configuration needs to tell us is what the IDs of the voters are.
> Without
> > > that, it's really easy for a cluster to get into a state where none of
> > > the quorum members agree on what the proper set of voters is. I think
> > > perhaps you are confused on the usage of these IDs. It is what enables
> > > validation of voter requests. Without it, a voter would have to accept
> a
> > > vote request from any ID. There is a reason that other consensus
> systems
> > > like zookeeper and etcd require ids when configured statically.
> > >
> >
> > I hadn't considered the fact that we need to validate incoming voter
> > requests.  The fact that nodes can have multiple DNS addresses does make
> > this difficult to do with just a list of hostnames.
> >
> > I guess you're right that we should keep the IDs.  But let's be careful
> to
> > validate that the node's ID really is what we think it is, and consider
> > that peer failed if it's not.
> >
> > >
> > > > Also, here's another case where we are saying "broker" when we mean
> > > > "controller."  It's really hard to break old habits.  :)
> > >
> > > I think we still have this basic disagreement on the KIP-500 vision :).
> > I'm
> > > not sure I understand why you are so eager to force users to think
> about
> > > the controller as a separate system. It's almost like Zookeeper is not
> > > going anywhere!
> > >
> >
> > Well, KIP-500 clearly does identify the controller as a separate system,
> > not as part of the broker, even if it runs in the same JVM.  :) A system
> > where 

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Jun Rao]

Hi, Jose,

Thanks for the KIP. A few comments blow.

10. I agree with Jason that it's useful to document the motivation a bit
clearer. Regarding semantic/performance, one benefit of snapshotting is
that it allows changes to be encoded incrementally instead of using the
full post image. For example, in KIP-631, each partition has multiple
fields like assigned replicas, leader, epoch, isr, etc. If only isr is
changed, the snapshotting approach allows the change to be represented with
just the new value in isr. Compaction will require all existing fields to
be included in order to represent just an isr change. This is just because
we can customize the combining logic with snapshotting. As for the
performance benefit, I guess in theory snapshotting allows the snapshot to
be updated in-place incrementally without having to read the full state in
the snapshot. BTW, during compaction, we only read the cleaned data once
instead of 3 times.

11. The KIP mentions topic id. Currently there is no topic id. Does this
KIP depend on KIP-516?

12. Is there a need to keep more than 1 snapshot? It seems we always expose
the latest snapshot to clients.

13. "During leader election, followers with incomplete or missing snapshot
will send a vote request and response as if they had an empty log." Hmm, a
follower may not have a snapshot created, but that doesn't imply its log is
empty.

14. "LBO is max.replication.lag.ms old." Not sure that I follow. How do we
compare an offset to a time?

15. "Followers and observers will increase their log begin offset to the
value sent on the fetch response as long as the local state machine has
generated a snapshot that includes to the log begin offset minus one." Does
the observer store the log? I thought it only needed to maintain a
snapshot. If so, the observer doesn't need to maintain LBO.

16. "There are two cases when the Kafka Controller needs to load the
snapshot: When it is booting. When the follower and observer replicas
finishes fetching a new snapshot from the leader." For faster failover, it
seems it's useful for a non-controller voter to maintain the in-memory
metadata state. In order to do that, it seems that every voter needs to
load the snapshot on booting?

17. "There is an invariant that every log must have at least one snapshot
where the offset of the snapshot is between LogBeginOffset - 1 and
High-Watermark. If this is not the case then the replica should assume that
the log is empty." Does that invariant hold when there is no snapshot
initially?

18. "The __cluster_metadata topic will have an cleanup.policy value of
snapshot" Is there a need to make this configurable if it's read-only?

19. OFFSET_OUT_OF_RANGE in FetchSnapshotResponse: It seems that
POSITION_OUT_OF_RANGE is more appropriate?

Thanks,

Jun

On Wed, Aug 5, 2020 at 12:13 PM Jose Garcia Sancio 
wrote:

> Once again, thanks for the feedback Jason,
>
> My changes to the KIP are here:
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=18=17
>
> And see my comments below...
>
> On Mon, Aug 3, 2020 at 1:57 PM Jason Gustafson  wrote:
> >
> > Hi Jose,
> >
> > Thanks for the proposal. I think there are three main motivations for
> > snapshotting over the existing compaction semantics.
> >
> > First we are arguing that compaction is a poor semantic fit for how we
> want
> > to model the metadata in the cluster. We are trying to view the changes
> in
> > the cluster as a stream of events, not necessarily as a stream of
> key/value
> > updates. The reason this is useful is that a single event may correspond
> to
> > a set of key/value updates. We don't need to delete each partition
> > individually for example if we are deleting the full topic. Outside of
> > deletion, however, the benefits of this approach are less obvious. I am
> > wondering if there are other cases where the event-based approach has
> some
> > benefit?
> >
>
> Yes. Another example of this is what KIP-631 calls FenceBroker. In the
> current implementation of the Kafka Controller and the implementation
> proposed in
> KIP-631, whenever a broker is fenced the controller removes the broker
> from the ISR and performs leader election if necessary. The impact of
> this operation on replication is documented in section "Amount of Data
> Replicated". I have also updated the KIP to reflect this.
>
> > The second motivation is from the perspective of consistency. Basically
> we
> > don't like the existing solution for the tombstone deletion problem,
> which
> > is just to add a delay before removal. The case we are concerned about
> > requires a replica to fetch up to a specific offset and then stall for a
> > time which is longer than the deletion retention timeout. If this
> happens,
> > then the replica might not see the tombstone, which would lead to an
> > inconsistent state. I think we are already talking about a rare case,
> but I
> > wonder if there are simple ways to tighten it further. For the sake of
> > 

[GitHub] [kafka-site] mjsax opened a new pull request #283: MINOR: fix HTML

[GitBox]

mjsax opened a new pull request #283:
URL: https://github.com/apache/kafka-site/pull/283


   Follow up to https://github.com/apache/kafka/pull/9127
   
   \cc @guozhangwang 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch commented on a change in pull request #277: 2.6.0 release

[GitBox]

rhauch commented on a change in pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#discussion_r465958151



##
File path: 26/upgrade.html
##
@@ -19,6 +19,50 @@
 
 

[GitHub] [kafka-site] rhauch commented on a change in pull request #277: 2.6.0 release

[GitBox]

rhauch commented on a change in pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#discussion_r465956697



##
File path: 26/upgrade.html
##
@@ -19,6 +19,50 @@
 
 

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Jose Garcia Sancio]

Once again, thanks for the feedback Jason,

My changes to the KIP are here:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=18=17

And see my comments below...

On Mon, Aug 3, 2020 at 1:57 PM Jason Gustafson  wrote:
>
> Hi Jose,
>
> Thanks for the proposal. I think there are three main motivations for
> snapshotting over the existing compaction semantics.
>
> First we are arguing that compaction is a poor semantic fit for how we want
> to model the metadata in the cluster. We are trying to view the changes in
> the cluster as a stream of events, not necessarily as a stream of key/value
> updates. The reason this is useful is that a single event may correspond to
> a set of key/value updates. We don't need to delete each partition
> individually for example if we are deleting the full topic. Outside of
> deletion, however, the benefits of this approach are less obvious. I am
> wondering if there are other cases where the event-based approach has some
> benefit?
>

Yes. Another example of this is what KIP-631 calls FenceBroker. In the
current implementation of the Kafka Controller and the implementation
proposed in
KIP-631, whenever a broker is fenced the controller removes the broker
from the ISR and performs leader election if necessary. The impact of
this operation on replication is documented in section "Amount of Data
Replicated". I have also updated the KIP to reflect this.

> The second motivation is from the perspective of consistency. Basically we
> don't like the existing solution for the tombstone deletion problem, which
> is just to add a delay before removal. The case we are concerned about
> requires a replica to fetch up to a specific offset and then stall for a
> time which is longer than the deletion retention timeout. If this happens,
> then the replica might not see the tombstone, which would lead to an
> inconsistent state. I think we are already talking about a rare case, but I
> wonder if there are simple ways to tighten it further. For the sake of
> argument, what if we had the replica start over from the beginning whenever
> there is a replication delay which is longer than tombstone retention time?
> Just want to be sure we're not missing any simple/pragmatic solutions
> here...
>

We explore the changes needed to log compaction and the fetch protocol
such that it results in a consistent replicated log in the rejected
sections. I changed the KIP to also mention it in the motivation
section by adding a section called "Consistent Log and Tombstones"

> Finally, I think we are arguing that compaction gives a poor performance
> tradeoff when the state is already in memory. It requires us to read and
> replay all of the changes even though we already know the end result. One
> way to think about it is that compaction works O(the rate of changes) while
> snapshotting is O(the size of data). Contrarily, the nice thing about
> compaction is that it works irrespective of the size of the data, which
> makes it a better fit for user partitions. I feel like this might be an
> argument we can make empirically or at least with back-of-the-napkin
> calculations. If we assume a fixed size of data and a certain rate of
> change, then what are the respective costs of snapshotting vs compaction? I
> think compaction fares worse as the rate of change increases. In the case
> of __consumer_offsets, which sometimes has to support a very high rate of
> offset commits, I think snapshotting would be a great tradeoff to reduce
> load time on coordinator failover. The rate of change for metadata on the
> other hand might not be as high, though it can be very bursty.
>

This is a very good observation. If you assume that the number of keys
doesn't change but that we have frequent updates to its values then I
think that after log compaction the size of the compacted section of
the log is O(size of the data) + O(size of the tombstones). And as you
point out the size of the snapshot is also O(size of the data). I
think this is a reasonable assumption for topics like
__cluster_metadata and __consumer_offsets.

The difference is the number of reads required. With in-memory
snapshot we only need to read the log once. With log compaction we
need to read the log 3 times: 1. to update the in-memory state, 2.
generate the map of key to offset and 3. compact the log using the map
of keys to offset. I have updated the KIP and go into a lot more
details in section "Loading State and Frequency of Compaction

-- 
-Jose

[GitHub] [kafka-site] hachikuji merged pull request #279: Add missing "to" to contributing page

[GitBox]

hachikuji merged pull request #279:
URL: https://github.com/apache/kafka-site/pull/279


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

The last blog post for 2.5.0 mentioned KIP-500, and I know we've made a lot
of progress on that and will have even more in upcoming releases. I'd like
to add the following paragraph to the opening section, just after the
paragraph mentioning JDK 14 and Scala 2.13:

"Finally, these accomplishments are only one part of a larger active
roadmap in the run up to Apache Kafka 3.0, which may be one of the most
significant releases in the project’s history. The work to replace
Zookeeper (link to KIP-500) with built-in RAFT-based consensus is well
underway with eight KIPs in active development. Kafka’s new RAFT protocol
for the metadata quorum is already available for review (link to
https://github.com/apache/kafka/pull/9130). Tiered Storage unlocks infinite
scaling and faster rebalance times via KIP-405 (link to KIP-405), and is up
and running in internal clusters at Uber."

I'ved incorporated this into the preview doc, but feedback is welcome.

Best regards,

Randall

On Wed, Aug 5, 2020 at 10:27 AM Randall Hauch  wrote:

> I've prepared a preliminary blog post about the upcoming Apache Kafka
>  2.6.0 release.
> Please take a look and let me know via this thread if you want to
> add/modify details.
> Thanks to all who contributed to this blog post.
>
> Unfortunately, the preview is not currently publicly visible at
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3 
> (I've
> logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> interest of time I created a Google doc with read privilege for everyone by
> pasting the preview content directly into:
>
>
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
>
> I will continue to update this doc with any changes to the draft blog post.
>
>
> Thanks,
> Randall
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

Hi, Maulin.

We have traditionally not mentioned *every* KIP in the project's blog posts
-- unless a release just happens to involve a small number of KIPs. This
means that we do have to choose a subset of the KIPs -- AK 2.6.0 had 30
KIPs that were implemented, and a blog summarizing each would be too long
and much less widely received. I tried to choose KIPs that clearly impacted
the broadest set of users, and finding a subset wasn't that easy.

I chose not to include KIP-519 in the blog simply because it requires
installing and integration into the broker components that are not included
in the official distribution. I hope that helps explain my thought process.

Best regards,

Randall

On Wed, Aug 5, 2020 at 12:52 PM Maulin Vasavada 
wrote:

> Hi Randall
>
> One question: Do we mention all KIPs/NewFeatures in the blog that are
> listed in the release notes document -
> https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?
>
> I see that [KAFKA-8890 ]
> - KIP-519: Make SSL context/engine configuration extensible is missing from
> the google doc that you shared.
>
> Thanks
> Maulin
>
>
> On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:
>
> > Thanks, Jason. We haven't done that for a few releases, but I think it's
> a
> > great idea. I've updated the blog post draft and the Google doc to
> mention
> > the 127 contributors by name that will also be mentioned in the email
> > release announcement.
> >
> > I also linked to the project's downloads page in the opening sentence,
> and
> > tweaked the wording slightly in the first paragraph.
> >
> > Best regards,
> >
> > Randall
> >
> > On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> > wrote:
> >
> > > Hey Randall,
> > >
> > > Thanks for putting this together. I think it would be great if the blog
> > > included the names of the release contributors. It's an easy way to
> give
> > > some recognition. I know we have done that in the past.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch 
> wrote:
> > >
> > > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > > 2.6.0
> > > > release.
> > > > Please take a look and let me know via this thread if you want to
> > > > add/modify details.
> > > > Thanks to all who contributed to this blog post.
> > > >
> > > > Unfortunately, the preview is not currently publicly visible at
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > > (I've
> > > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > > > interest of time I created a Google doc with read privilege for
> > everyone
> > > by
> > > > pasting the preview content directly into:
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > > >
> > > > I will continue to update this doc with any changes to the draft blog
> > > post.
> > > >
> > > >
> > > > Thanks,
> > > > Randall
> > > >
> > >
> >
>

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #282: bug fixes

[GitBox]

scott-confluent commented on a change in pull request #282:
URL: https://github.com/apache/kafka-site/pull/282#discussion_r465924056



##
File path: 25/documentation.html
##
@@ -31,7 +31,7 @@
 
   

-

Review comment:
   I believe the older versions SHOULD have this banner.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #282: bug fixes

[GitBox]

scott-confluent commented on a change in pull request #282:
URL: https://github.com/apache/kafka-site/pull/282#discussion_r465921715



##
File path: 25/streams/developer-guide/write-streams.html
##
@@ -90,22 +90,22 @@
   See the section Data Types and Serialization for more information about 
Serializers/Deserializers.
   
   Example pom.xml snippet when using Maven:
-  
-org.apache.kafka
-kafka-streams
-{{fullDotVersion}}
-
-
-org.apache.kafka
-kafka-clients
-{{fullDotVersion}}
-
+  dependency>
+groupId>org.apache.kafka/groupId>

Review comment:
   theoretically it should be generated with html entities for both gt and 
lt, but the opening is the only one that matters for highlighting the syntax. 
it's not an easy thing to find/replace so we'd have to do an audit of all the 
docs.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] guozhangwang commented on a change in pull request #282: bug fixes

[GitBox]

guozhangwang commented on a change in pull request #282:
URL: https://github.com/apache/kafka-site/pull/282#discussion_r465912196



##
File path: 25/documentation.html
##
@@ -31,7 +31,7 @@
 
   

-

Review comment:
   Ditto for other 25/xyx.html pages: does the same issue exist in older 
versions (24, 23...)?

##
File path: 25/streams/developer-guide/write-streams.html
##
@@ -90,22 +90,22 @@
   See the section Data Types and Serialization for more information about 
Serializers/Deserializers.
   
   Example pom.xml snippet when using Maven:
-  
-org.apache.kafka
-kafka-streams
-{{fullDotVersion}}
-
-
-org.apache.kafka
-kafka-clients
-{{fullDotVersion}}
-
+  dependency>
+groupId>org.apache.kafka/groupId>

Review comment:
   Should the `>` be replaced too?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] guozhangwang commented on pull request #277: 2.6.0 release

[GitBox]

guozhangwang commented on pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#issuecomment-669351468


   LGTM.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-10364) Use Token Bucket for all quotas

[David Jacot (Jira)]

David Jacot created KAFKA-10364:
---

 Summary: Use Token Bucket for all quotas
 Key: KAFKA-10364
 URL: https://issues.apache.org/jira/browse/KAFKA-10364
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Preliminary blog post about the Apache 2.6.0 release

[Maulin Vasavada]

Hi Randall

One question: Do we mention all KIPs/NewFeatures in the blog that are
listed in the release notes document -
https://home.apache.org/~rhauch/kafka-2.6.0-rc2/RELEASE_NOTES.html ?

I see that [KAFKA-8890 ]
- KIP-519: Make SSL context/engine configuration extensible is missing from
the google doc that you shared.

Thanks
Maulin


On Wed, Aug 5, 2020 at 9:49 AM Randall Hauch  wrote:

> Thanks, Jason. We haven't done that for a few releases, but I think it's a
> great idea. I've updated the blog post draft and the Google doc to mention
> the 127 contributors by name that will also be mentioned in the email
> release announcement.
>
> I also linked to the project's downloads page in the opening sentence, and
> tweaked the wording slightly in the first paragraph.
>
> Best regards,
>
> Randall
>
> On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson 
> wrote:
>
> > Hey Randall,
> >
> > Thanks for putting this together. I think it would be great if the blog
> > included the names of the release contributors. It's an easy way to give
> > some recognition. I know we have done that in the past.
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:
> >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.6.0
> > > release.
> > > Please take a look and let me know via this thread if you want to
> > > add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > > Unfortunately, the preview is not currently publicly visible at
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > > (I've
> > > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > > interest of time I created a Google doc with read privilege for
> everyone
> > by
> > > pasting the preview content directly into:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> > >
> > > I will continue to update this doc with any changes to the draft blog
> > post.
> > >
> > >
> > > Thanks,
> > > Randall
> > >
> >
>

[GitHub] [kafka-site] rhauch commented on pull request #277: 2.6.0 release

[GitBox]

rhauch commented on pull request #277:
URL: https://github.com/apache/kafka-site/pull/277#issuecomment-669332049


   @guozhangwang, @ijuma: I've updated the content to try to address your 
comments. Would you mind taking a look?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #278: Remove console.log()

[GitBox]

scott-confluent commented on pull request #278:
URL: https://github.com/apache/kafka-site/pull/278#issuecomment-669326459


   Nice. Thanks!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] guozhangwang commented on pull request #278: Remove console.log()

[GitBox]

guozhangwang commented on pull request #278:
URL: https://github.com/apache/kafka-site/pull/278#issuecomment-669312037


   LGTM



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

[Harsha Ch]

Thanks for the updates Gokul.  +1 (binding).

Thanks,

Harsha

On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian < 
gokul24...@gmail.com > wrote:

> 
> 
> 
> Hi.
> 
> 
> 
> I request the community to kindly resume the voting process for KIP-578.
> If you have further comments, we can address those as well.
> 
> 
> 
> Thanks and cheers.
> 
> 
> 
> On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul2411s@ gmail.
> com ( gokul24...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi.
>> 
>> 
>> 
>> Can we resume the voting process for KIP-578? I have addressed additional
>> comments by Boyang and Ismael.
>> 
>> 
>> 
>> Thanks.
>> 
>> 
>> 
>> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul2411s@ gmail.
>> com ( gokul24...@gmail.com ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi. Can we resume the voting process for KIP-578? Thanks.
>>> 
>>> 
>>> 
>>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul2411s@ 
>>> gmail.
>>> com ( gokul24...@gmail.com ) > wrote:
>>> 
>>> 
 
 
 Thanks Colin. Have updated the KIP per your recommendations. Let me know
 what you think.
 
 
 
 Thanks Harsha for the vote.
 
 
 
 On Wed, May 27, 2020 at 8:17 PM Colin McCabe < cmccabe@ apache. org (
 cmcc...@apache.org ) > wrote:
 
 
> 
> 
> Hi Gokul Ramanan Subramanian,
> 
> 
> 
> Thanks for the KIP.
> 
> 
> 
> Can you please modify the KIP to remove the reference to the deprecated
> --zookeeper flag? This is not how kafka-configs. sh (
> http://kafka-configs.sh/ ) is supposed to be used in new versions of 
> Kafka.
> You get a warning message if you do use this deprecated flag. As described
> in KIP-604, we are removing the --zookeeper flag in the Kafka 3.0 release.
> It also causes problems when people use the deprecated access mode-- for
> example, as you note in this KIP, it bypasses resource limits such as the
> ones described here.
> 
> 
> 
> Instead of WILL_EXCEED_PARTITION_LIMITS, how about RESOURCE_LIMIT_REACHED?
> Then the error string can contain the detailed message about which
> resource limit was hit (per broker limit, per cluster limit, whatever.) It
> would also be good to spell out that CreateTopicsPolicy plugins can also
> throw this exception, for consistency.
> 
> 
> 
> I realize that 2 billion partitions seems like a very big number. However,
> filesystems have had to transition to 64 bit inode numbers as time has
> gone on. There doesn't seem to be any performance reason why this should
> be a 31 bit number, so let's just make these configurations longs, not
> ints.
> 
> 
> 
> best,
> Colin
> 
> 
> 
> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
> 
> 
>> 
>> 
>> Thanks for the KIP Gokul. This will be really useful for our use
>> 
>> 
> 
> 
> 
> cases as
> 
> 
>> 
>> 
>> well.
>> +1 (binding).
>> 
>> 
>> 
>> -Harsha
>> 
>> 
>> 
>> On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < 
>> gokul2411s@ gmail.
>> com ( gokul24...@gmail.com ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi.
>>> 
>>> 
>>> 
>>> Any votes for this?
>>> 
>>> 
>>> 
>>> Thanks.
>>> 
>>> 
>>> 
>>> On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
>>> 
>>> 
>> 
>> 
> 
> 
> 
> gokul2411s@
> 
> 
>> 
>>> 
>>> 
>>> gmail. com ( http://gmail.com/ ) > wrote:
>>> 
>>> 
>>> 
>>> Hello,
>>> 
>>> 
>>> 
>>> I'd like to initialize voting on KIP-578:
>>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ (
>>> https://cwiki.apache.org/confluence/display/KAFKA/ ) 
>>> KIP-578%3A+Add+configuration+to+limit+number+of+partitions
>>> 
>>> .
>>> 
>>> 
>>> 
>>> Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
>>> 
>>> 
>> 
>> 
> 
> 
> 
> and Tom
> 
> 
>> 
>>> 
>>> 
>>> Bentley on the discussion thread. I have addressed their comments.
>>> 
>>> 
>> 
>> 
> 
> 
> 
> I want
> 
> 
>> 
>>> 
>>> 
>>> to thank them for their time.
>>> 
>>> 
>>> 
>>> If there are any more concerns about the KIP, I am happy to discuss
>>> 
>>> 
>> 
>> 
> 
> 
> 
> them
> 
> 
>> 
>>> 
>>> 
>>> further.
>>> 
>>> 
>>> 
>>> Thanks.
>>> 
>>> 
>> 
>> 
> 
> 
 
 
>>> 
>>> 
>> 
>> 
> 
> 
>

[GitHub] [kafka-site] guozhangwang commented on pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

guozhangwang commented on pull request #281:
URL: https://github.com/apache/kafka-site/pull/281#issuecomment-669310726


   Anyways, this change LGTM. Please feel free to merge afterwards



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] guozhangwang commented on pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

guozhangwang commented on pull request #281:
URL: https://github.com/apache/kafka-site/pull/281#issuecomment-669310491


   Yeah I think encouraging people to directly submit PRs is better.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent opened a new pull request #282: bug fixes

[GitBox]

scott-confluent opened a new pull request #282:
URL: https://github.com/apache/kafka-site/pull/282


   ### home page
   -  spelling
   - re-introduce see full list tooltip
   
   ### docs
   -  remove version banner



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

Thanks, Jason. We haven't done that for a few releases, but I think it's a
great idea. I've updated the blog post draft and the Google doc to mention
the 127 contributors by name that will also be mentioned in the email
release announcement.

I also linked to the project's downloads page in the opening sentence, and
tweaked the wording slightly in the first paragraph.

Best regards,

Randall

On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson  wrote:

> Hey Randall,
>
> Thanks for putting this together. I think it would be great if the blog
> included the names of the release contributors. It's an easy way to give
> some recognition. I know we have done that in the past.
>
> Thanks,
> Jason
>
> On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:
>
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.6.0
> > release.
> > Please take a look and let me know via this thread if you want to
> > add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> > Unfortunately, the preview is not currently publicly visible at
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > (I've
> > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > interest of time I created a Google doc with read privilege for everyone
> by
> > pasting the preview content directly into:
> >
> >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> >
> > I will continue to update this doc with any changes to the draft blog
> post.
> >
> >
> > Thanks,
> > Randall
> >
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

Thanks, Jason. We haven't done that for a few releases, but I think it's a
great idea. I've updated the blog post draft and the Google doc to mention
the 127 contributors by name that will also be mentioned in the email
release announcement.

I also linked to the project's downloads page in the opening sentence, and
tweaked the wording slightly in the first paragraph.

Best regards,

Randall

On Wed, Aug 5, 2020 at 11:21 AM Jason Gustafson  wrote:

> Hey Randall,
>
> Thanks for putting this together. I think it would be great if the blog
> included the names of the release contributors. It's an easy way to give
> some recognition. I know we have done that in the past.
>
> Thanks,
> Jason
>
> On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:
>
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.6.0
> > release.
> > Please take a look and let me know via this thread if you want to
> > add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> > Unfortunately, the preview is not currently publicly visible at
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> > (I've
> > logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> > interest of time I created a Google doc with read privilege for everyone
> by
> > pasting the preview content directly into:
> >
> >
> >
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
> >
> > I will continue to update this doc with any changes to the draft blog
> post.
> >
> >
> > Thanks,
> > Randall
> >
>

Re: Preliminary blog post about the Apache 2.6.0 release

[Jason Gustafson]

Hey Randall,

Thanks for putting this together. I think it would be great if the blog
included the names of the release contributors. It's an easy way to give
some recognition. I know we have done that in the past.

Thanks,
Jason

On Wed, Aug 5, 2020 at 8:25 AM Randall Hauch  wrote:

> I've prepared a preliminary blog post about the upcoming Apache Kafka 2.6.0
> release.
> Please take a look and let me know via this thread if you want to
> add/modify details.
> Thanks to all who contributed to this blog post.
>
> Unfortunately, the preview is not currently publicly visible at
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
> (I've
> logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
> interest of time I created a Google doc with read privilege for everyone by
> pasting the preview content directly into:
>
>
> https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing
>
> I will continue to update this doc with any changes to the draft blog post.
>
>
> Thanks,
> Randall
>

Preliminary blog post about the Apache 2.6.0 release

[Randall Hauch]

I've prepared a preliminary blog post about the upcoming Apache Kafka 2.6.0
release.
Please take a look and let me know via this thread if you want to
add/modify details.
Thanks to all who contributed to this blog post.

Unfortunately, the preview is not currently publicly visible at
https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache3
(I've
logged https://issues.apache.org/jira/browse/INFRA-20646), so in the
interest of time I created a Google doc with read privilege for everyone by
pasting the preview content directly into:

https://docs.google.com/document/d/1MQQoJk4ewedYgFfgSA2axx3VhRwgj_0rO1tf3yezV0c/edit?usp=sharing

I will continue to update this doc with any changes to the draft blog post.


Thanks,
Randall

Re: [VOTE] KIP-578: Add configuration to limit number of partitions

[Gokul Ramanan Subramanian]

Hi.

I request the community to kindly resume the voting process for KIP-578. If
you have further comments, we can address those as well.

Thanks and cheers.

On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> Hi.
>
> Can we resume the voting process for KIP-578? I have addressed additional
> comments by Boyang and Ismael.
>
> Thanks.
>
> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian <
> gokul24...@gmail.com> wrote:
>
>> Hi. Can we resume the voting process for KIP-578? Thanks.
>>
>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian <
>> gokul24...@gmail.com> wrote:
>>
>>> Thanks Colin. Have updated the KIP per your recommendations. Let me know
>>> what you think.
>>>
>>> Thanks Harsha for the vote.
>>>
>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe  wrote:
>>>
 Hi Gokul Ramanan Subramanian,

 Thanks for the KIP.

 Can you please modify the KIP to remove the reference to the deprecated
 --zookeeper flag?  This is not how kafka-configs.sh is supposed to be used
 in new versions of Kafka.  You get a warning message if you do use this
 deprecated flag.  As described in KIP-604, we are removing the --zookeeper
 flag in the Kafka 3.0 release.  It also causes problems when people use the
 deprecated access mode-- for example, as you note in this KIP, it bypasses
 resource limits such as the ones described here.

 Instead of WILL_EXCEED_PARTITION_LIMITS, how about
 RESOURCE_LIMIT_REACHED?  Then the error string can contain the detailed
 message about which resource limit was hit (per broker limit, per cluster
 limit, whatever.)  It would also be good to spell out that
 CreateTopicsPolicy plugins can also throw this exception, for consistency.

 I realize that 2 billion partitions seems like a very big number.
 However, filesystems have had to transition to 64 bit inode numbers as time
 has gone on.  There doesn't seem to be any performance reason why this
 should be a 31 bit number, so let's just make these configurations longs,
 not ints.

 best,
 Colin


 On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote:
 > Thanks for the KIP Gokul. This will be really useful for our use
 cases as
 > well.
 > +1 (binding).
 >
 > -Harsha
 >
 >
 > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian <
 > gokul24...@gmail.com> wrote:
 >
 > > Hi.
 > >
 > > Any votes for this?
 > >
 > > Thanks.
 > >
 > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian <
 gokul2411s@
 > > gmail.com> wrote:
 > >
 > > Hello,
 > >
 > > I'd like to initialize voting on KIP-578:
 > > https://cwiki.apache.org/confluence/display/KAFKA/
 > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions
 > > .
 > >
 > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez
 and Tom
 > > Bentley on the discussion thread. I have addressed their comments.
 I want
 > > to thank them for their time.
 > >
 > > If there are any more concerns about the KIP, I am happy to discuss
 them
 > > further.
 > >
 > > Thanks.
 > >
 > >
 >

>>>

[GitHub] [kafka-site] ottomata commented on pull request #280: Update Wikimedia powered-by text

[GitBox]

ottomata commented on pull request #280:
URL: https://github.com/apache/kafka-site/pull/280#issuecomment-669199664


   Thank you!
   
   On Wed, Aug 5, 2020 at 9:35 AM Ismael Juma  wrote:
   
   > Merged #280  into asf-site.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   >
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] KIP-653: Upgrade log4j to log4j2

[Dongjin Lee]

Hi, Kafka dev,

I hope to initiate the discussion of KIP-653, upgrading log4j to log4j2.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2

All kinds of feedbacks are greatly appreciated!

Best,
Dongjin

-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*




*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[GitHub] [kafka-site] ijuma commented on pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

ijuma commented on pull request #281:
URL: https://github.com/apache/kafka-site/pull/281#issuecomment-669197495


   Maybe we should ask people to submit a PR to this repo directly instead of 
the mailing list. What do you think @guozhangwang?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] ijuma merged pull request #280: Update Wikimedia powered-by text

[GitBox]

ijuma merged pull request #280:
URL: https://github.com/apache/kafka-site/pull/280


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] ijuma opened a new pull request #281: Remove mention of Twitter as a way to get added to "Powered By"

[GitBox]

ijuma opened a new pull request #281:
URL: https://github.com/apache/kafka-site/pull/281


   The mailing list seems more suitable.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] ottomata commented on pull request #280: Update Wikimedia powered-by text

[GitBox]

ottomata commented on pull request #280:
URL: https://github.com/apache/kafka-site/pull/280#issuecomment-669187906


   Let  me know if there is another process I don't know about other than PR to 
get this updated.  Thank you!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] ottomata opened a new pull request #280: Update Wikimedia powered-by text

[GitBox]

ottomata opened a new pull request #280:
URL: https://github.com/apache/kafka-site/pull/280


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[Dongjin Lee]

Hi All,

I updated the KIP

and the implementation, following the discussion here.

You must be working hard preparing the release of 2.6.0, so please have a
look after your work is done.

Thanks,
Dongjin

On Sun, Mar 8, 2020 at 12:20 PM John Roesler  wrote:

> Thanks Matthias,
>
> Good idea. I've changed the ticket name and added a note
> clarifying that this ticket is not the same as
> https://issues.apache.org/jira/browse/KAFKA-7224
>
> Incidentally, I learned that I never documented my reasons
> for abandoning my work on KAFKA-7224 ! I've now updated
> that ticket, too, so your question had an unexpected side-benefit.
>
> Thanks,
> -John
>
> On Sat, Mar 7, 2020, at 18:01, Matthias J. Sax wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA512
> >
> > Thanks for clarification.
> >
> > Can you maybe update the Jira ticket? Do we have a ticket for
> > spill-to-disk? Maybe link to it and explain that it's two different
> > things? Maybe even rename the ticket to something more clear, ie,
> > "make suppress result queryable" or simliar?
> >
> >
> > - -Matthias
> >
> > On 3/7/20 1:58 PM, John Roesler wrote:
> > > Hey Matthias,
> > >
> > > I’m sorry if the ticket was poorly stated. The ticket is to add a
> > DSL overload to pass a Materialized argument to suppress. As a result,
> > the result of the suppression would be queriable.
> > >
> > > This is unrelated to “persistent buffer” aka “spill-to-disk”.
> > >
> > > There was some confusion before about whether this ticket could be
> > implemented as “query the buffer”. Maybe it can, but not trivially.
> > The obvious way is just to add a new state store which we write the
> > results into just before we forward. I.e., it’s exactly like the
> > materialized variant of any stateless KTable operation.
> > >
> > > Thanks, John
> > >
> > > On Sat, Mar 7, 2020, at 15:32, Matthias J. Sax wrote: Thanks for
> > > the KIP Dongjin,
> > >
> > > I am still not sure if I can follow, what might also be caused by
> > > the backing JIRA ticket (maybe John can clarify the intent of the
> > > ticket as he created it):
> > >
> > > Currently, suppress() only uses an in-memory buffer and my
> > > understanding of the Jira is, to add the ability to use a
> > > persistent buffer (ie, spill to disk backed by RocksDB).
> > >
> > > Adding a persistent buffer is completely unrelated to allow
> > > querying the buffer. In fact, one could query an in-memory buffer,
> > > too. However, querying the buffer does not really seem to be useful
> > > as pointed out by John, as you can always query the upstream KTable
> > > store.
> > >
> > > Also note that for the emit-on-window-close case the result is
> > > deleted from the buffer when it is emitted, and thus cannot be
> > > queried any longe r.
> > >
> > >
> > > Can you please clarify if you intend to allow spilling to disk or
> > > if you intent to enable IQ (even if I don't see why querying make
> > > sense, as the data is either upstream or deleted). Also, if you
> > > want to enable IQ, why do we need all those new interfaces? The
> > > result of a suppress() is a KTable that is the same as any other
> > > key-value/windowed/sessions store?
> > >
> > > We should also have corresponding Jira tickets for different cases
> > > to avoid the confusion I am in atm :)
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 2/27/20 8:21 AM, John Roesler wrote:
> >  Hi Dongjin,
> > 
> >  No problem; glad we got it sorted out.
> > 
> >  Thanks again for picking this up! -John
> > 
> >  On Wed, Feb 26, 2020, at 09:24, Dongjin Lee wrote:
> > >> I was under the impression that you wanted to expand the
> > >> scope of the KIP
> > > to additionally allow querying the internal buffer, not
> > > just the result. Can you clarify whether you are proposing
> > > to allow querying the state of the internal buffer, the
> > > result, or both?
> > >
> > > Sorry for the confusion. As we already talked with, we only
> > > need to query the suppressed output, not the internal
> > > buffer. The current implementation is wrong. After refining
> > > the KIP and implementation accordingly I will notify you -
> > > I must be confused, also.
> > >
> > > Thanks, Dongjin
> > >
> > > On Tue, Feb 25, 2020 at 12:17 AM John Roesler
> > >  wrote:
> > >
> > >> Hi Dongjin,
> > >>
> > >> Ah, I think I may have been confused. I 100% agree that
> > >> we need a materialized variant for suppress(). Then, you
> > >> could do: ...suppress(...,
> > >> Materialized.as(“final-count”))
> > >>
> > >> If that’s your proposal, then we are on the same page.
> > >>
> > >> I was under the impression that you wanted to expand the
> > >> scope of the KIP to additionally allow querying the
> > >> internal buffer, not just the result. Can you clarify
> > 

Re: New Website Layout

[Luke Chen]

When entering streams doc, it'll always show:
*You're viewing documentation for an older version of Kafka - check out our
current documentation here.*



On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:

> Thanks for the PR and feedback Michael. Appreciated.
>
> On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> wrote:
>
> > Thank you, it looks great!
> >
> > I found a couple of small issues:
> > - It's not rendering correctly with http.
> > - It's printing "called" to the console. I opened a PR to remove the
> > console.log() call: https://github.com/apache/kafka-site/pull/278
> >
> > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
> > >
> > > The new website layout has gone live as you may have seen. There are a
> > > couple of rendering issues in the streams developer guide that we're
> > > getting addressed. If anyone spots anything else could they please
> reply
> > to
> > > this thread.
> > >
> > > Thanks
> > >
> > > Ben
> > >
> > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
> > >
> > > > Hey folks
> > > >
> > > > We've made some updates to the website's look and feel. There is a
> > staged
> > > > version in the link below.
> > > >
> > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > username: kafka
> > > > password: streaming
> > > >
> > > > Comments welcomed.
> > > >
> > > > Ben
> > > >
> > > >
> >
>
>
> --
>
> Ben Stopford
>
> Lead Technologist, Office of the CTO
>
> 
>

[GitHub] [kafka-site] showuon commented on pull request #278: Remove console.log()

[GitBox]

showuon commented on pull request #278:
URL: https://github.com/apache/kafka-site/pull/278#issuecomment-669134108


   Nice catch! LGTM!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: New Website Layout

[Ben Stopford]

Thanks for the PR and feedback Michael. Appreciated.

On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
wrote:

> Thank you, it looks great!
>
> I found a couple of small issues:
> - It's not rendering correctly with http.
> - It's printing "called" to the console. I opened a PR to remove the
> console.log() call: https://github.com/apache/kafka-site/pull/278
>
> On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
> >
> > The new website layout has gone live as you may have seen. There are a
> > couple of rendering issues in the streams developer guide that we're
> > getting addressed. If anyone spots anything else could they please reply
> to
> > this thread.
> >
> > Thanks
> >
> > Ben
> >
> > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
> >
> > > Hey folks
> > >
> > > We've made some updates to the website's look and feel. There is a
> staged
> > > version in the link below.
> > >
> > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > username: kafka
> > > password: streaming
> > >
> > > Comments welcomed.
> > >
> > > Ben
> > >
> > >
>


-- 

Ben Stopford

Lead Technologist, Office of the CTO

[jira] [Created] (KAFKA-10363) Broker try to connect to a new cluster when there are changes in zookeeper.connect properties

[Alexey Kornev (Jira)]

Alexey Kornev created KAFKA-10363:
-

 Summary: Broker try to connect to a new cluster when there are 
changes in zookeeper.connect properties
 Key: KAFKA-10363
 URL: https://issues.apache.org/jira/browse/KAFKA-10363
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.3.1, 2.4.0
 Environment: 3 Kafka brokers (v2.3.1, v2.4.0) with Zookeeper cluster 
(3.4.10)
Ubuntu 18.04 LTS
Reporter: Alexey Kornev


We've just successfully set up a Kafka cluster consists of 3 brokers and faced 
with the following issue: when we change order of zookeeper servers in 
zookeeper.connect property in server.properties files and restart Kafka broker 
then this Kafka broker tries to connect to a new Kafka cluster. As a result, 
Kafka broker throws an error and shutdown. 

For example, config server.properties on first broker:
{code:java}
broker.id=-1
...
zookeeper.connect=node_1:2181/kafka,node_2:2181/kafka,node_3:2181/kafka
{code}
 

 

 We changed it to 

 
{code:java}
broker.id=-1
...
zookeeper.connect=node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka
{code}
 

and restart Kafka broker. 

Logs:

{code:java}
[2020-08-05 09:07:55,658] INFO [ExpirationReaper-0-Heartbeat]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 
09:07:55,658] INFO [ExpirationReaper-0-Heartbeat]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 
09:07:55,658] INFO [ExpirationReaper-0-topic]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)[2020-08-05 
09:07:57,070] INFO Registered kafka:type=kafka.Log4jController MBean 
(kafka.utils.Log4jControllerRegistration$)[2020-08-05 09:07:57,656] INFO 
Registered signal handlers for TERM, INT, HUP 
(org.apache.kafka.common.utils.LoggingSignalHandler)[2020-08-05 09:07:57,657] 
INFO starting (kafka.server.KafkaServer)[2020-08-05 09:07:57,658] INFO 
Connecting to zookeeper on 
node_2:2181/kafka,node_3:2181/kafka,node_1:2181/kafka 
(kafka.server.KafkaServer)[2020-08-05 09:07:57,685] INFO [ZooKeeperClient Kafka 
server] Initializing a new session to node_2:2181. 
(kafka.zookeeper.ZooKeeperClient)[2020-08-05 09:07:57,690] INFO Client 
environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, 
built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)[2020-08-05 
09:07:57,693] INFO Client environment:host.name=localhost 
(org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,693] INFO Client 
environment:java.version=11.0.8 (org.apache.zookeeper.ZooKeeper)[2020-08-05 
09:07:57,696] INFO Client environment:java.vendor=Ubuntu 
(org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,696] INFO Client 
environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 
(org.apache.zookeeper.ZooKeeper)[2020-08-05 09:07:57,696] INFO Client 

[GitHub] [kafka-site] tom1299 opened a new pull request #279: Add missing "to" to contributing page

[GitBox]

tom1299 opened a new pull request #279:
URL: https://github.com/apache/kafka-site/pull/279


   Just added the missing word "to" the contribution guide



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: New Website Layout

[Mickael Maison]

Thank you, it looks great!

I found a couple of small issues:
- It's not rendering correctly with http.
- It's printing "called" to the console. I opened a PR to remove the
console.log() call: https://github.com/apache/kafka-site/pull/278

On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
>
> The new website layout has gone live as you may have seen. There are a
> couple of rendering issues in the streams developer guide that we're
> getting addressed. If anyone spots anything else could they please reply to
> this thread.
>
> Thanks
>
> Ben
>
> On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
>
> > Hey folks
> >
> > We've made some updates to the website's look and feel. There is a staged
> > version in the link below.
> >
> > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > username: kafka
> > password: streaming
> >
> > Comments welcomed.
> >
> > Ben
> >
> >

[GitHub] [kafka-site] mimaison opened a new pull request #278: Remove console.log()

[GitBox]

mimaison opened a new pull request #278:
URL: https://github.com/apache/kafka-site/pull/278


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: kafka-trunk-jdk11 #1690

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add additional shutdown log info (#9124)


--
[...truncated 6.41 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

Re: New Website Layout

[Ben Stopford]

The new website layout has gone live as you may have seen. There are a
couple of rendering issues in the streams developer guide that we're
getting addressed. If anyone spots anything else could they please reply to
this thread.

Thanks

Ben

On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:

> Hey folks
>
> We've made some updates to the website's look and feel. There is a staged
> version in the link below.
>
> https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> username: kafka
> password: streaming
>
> Comments welcomed.
>
> Ben
>
>

Build failed in Jenkins: kafka-trunk-jdk14 #341

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add additional shutdown log info (#9124)


--
[...truncated 3.21 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #4764

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add additional shutdown log info (#9124)


--
[...truncated 3.18 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

Re: [DISCUSS] KIP-616: Rename implicit Serdes instances in kafka-streams-scala

[Yuriy Badalyantc]

Hi John!

I understand your point. But, I'm still not sure that having a Unit serde
is a good idea because it could be ambiguous for the user. In a KStream
[Long, Unit] valid value could mean one the following:
1. Empty byte array
2. null
3. Any value (value ignored)

As a user of a library, I think all these options are more or less valid.
That's because the Unit type in my opinion has no clear representation in
terms of values and deserialization.

About Null serde - I'm ok with this. But, it should be a low priority
implicit to not intersect will other implicits. I will add it.

-Yuriy

On Sat, Jul 11, 2020 at 9:18 PM John Roesler  wrote:

> Thanks, Yuriy,
>
> IIRC, I added VoidSerde because there are times when the key or value is
> always null, but since ‘null’ has no type in Java, we had used the
> ByteArraySerde or some other proxy. This is confusing, because then the
> type is ‘byte[]’ although we intended it to always be null. It also delays
> finding bugs because serdes themselves force runtime type checks, but the
> ByteArraySerde accepts all data.
>
> Anyway, Void isn’t the type of null, but it’s close enough. Although
> Scala’s equivalent to “void” method is Unit methods, and like Void, Unit is
> uninstantiable, Unit has a value, also called Unit or “()”, whereas Void
> has no value in Java. But because Void is still a reference type, it can be
> null (I.e., it can only be null), so I’m some sense, it’s closer to the
> Scala type Null.
>
> I guess Scala users would benefit from both a NullSerde and a UnitSerde,
> but not a VoidSerde.
>
> Thanks for bringing it up, Matthias.
>
> -John
>
> On Fri, Jul 10, 2020, at 21:49, Yuriy Badalyantc wrote:
> > Ok, I mentioned adding missing serdes in the Proposed Change paragraph.
> >
> > About VoidSerde. I didn't add it intentionally. The semantic of the Unit
> > (scala's void) type is not clear in terms of the data. If kafka topic
> > contains messages of type Unit, what does it actually means? That there
> is
> > always null? Well, for that we have a Null type. That it's an empty byte
> > array? For that, we have an Array[Byte]. Empty string? No, it's a String.
> > So, I decided to not include Unit serde in the built-in Serdes. And if a
> > user will want to use the Unit type he can implement its own serde.
> >
> > -Yuriy
> >
> > On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax 
> wrote:
> >
> > > Thanks Yuriy!
> > >
> > > What about `VoidSerde` ? It's not listed.
> > >
> > > It might also be nice to add a short sentence and state that in
> addition
> > > to fixing the name collisions, the KIP will also close the gap of
> > > out-of-the-box serdes and add missing Serdes that are offered in Java
> to
> > > Scala.
> > >
> > >
> > > -Matthias
> > >
> > > On 7/10/20 7:51 AM, Yuriy Badalyantc wrote:
> > > > Oh, ok. I have done that. Just didn't know that it was necessary.
> > > >
> > > > -Yuriy
> > > >
> > > > On Fri, Jul 10, 2020 at 9:30 PM John Roesler 
> > > wrote:
> > > >
> > > >> Ah, thanks Yuriy,
> > > >>
> > > >> Sorry if this wasn't clear, but _all_ public API changes have to
> > > >> be explicitly included in the KIP. Can you just enumerate all
> > > >> the contents of the new API?
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote:
> > > >>> Hi, Matthias,
> > > >>>
> > > >>> It's not directly mentioned in the KIP, but I added all missing
> Java
> > > >>> serdes. I mentioned it in the pull request description:
> > > >>> https://github.com/apache/kafka/pull/8955
> > > >>>
> > > >>> And also, this KIP originally was based on a pull request where I
> added
> > > >>> missing java serdes :) https://github.com/apache/kafka/pull/8049
> > > >>>
> > > >>> -Yuriy
> > > >>>
> > > >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax 
> > > >> wrote:
> > > >>>
> > >  Yuriy,
> > > 
> > >  thanks for the KIP update. I have one follow up thought: I checked
> > > what
> > >  default Serdes we offer in the Java class
> > > 
> > >   `org.apache.kafka.common.serialization.Serdes`
> > > 
> > >  and I think it would be good if we could close the gap between the
> > > Java
> > >  and Scala code and add the missing Java Serdes in Scala, too.
> > > 
> > >  It seems we are missing `Short` (Java and Scala), `Void`, `UUID`,
> and
> > >  `ByterBuffer`.
> > > 
> > >  Can we add those in addition?
> > > 
> > > 
> > >  -Matthias
> > > 
> > >  On 7/8/20 6:45 AM, John Roesler wrote:
> > > > Hi Yuriy,
> > > >
> > > > Once it seems like there’s general agreement in the discussion,
> you
> > > >> can
> > >  start a voting thread. You can find examples on the mailing list
> of
> > > >> what to
> > >  say in the first message. It’s basically just a message with the
> > > >> subject
> > >  line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating
> that
> > >  you’d like to start the vote. It’s nice to link to the kip