Build failed in Jenkins: kafka-trunk-jdk10 #354

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6761: Construct Physical Plan using Graph, Reduce streams

[harsha] MINOR: Fixed default streams state dir location. (#5441)

--
[...truncated 1.54 MB...]
kafka.zookeeper.ZooKeeperClientTest > testSessionExpiryDuringClose PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout PASSED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler STARTED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler PASSED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString STARTED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData STARTED

kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone STARTED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Hey Guozhang,


thanks for this detailed proposal! Quickly I want to clarify, where this 
registry id is generated? My understanding is that the registry id is a unique 
id provided by user correct? This way even if multiple consumers are using the 
same member id, we are fencing against polluting the commit history. Very 
thoughtful approach! Furthermore, I feel we should export alerting metrics on 
catching duplicate consumer instances since multiple of them could still be 
reading the same topic partition and executing duplicate business logic.


Also a minor comment is that our current codebase has two different classes: 
the GroupCoordinator and ConsumerCoordinator. When talking about term 
"coordinator", it would be great to distinguish between the two, or explicitly 
nominate broker coordinator as default "coordinator".


Best,

Boyang


From: Guozhang Wang 
Sent: Thursday, August 2, 2018 1:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hello Boyang,

Thanks for the great summary.

Personally I think 1) is still doable as we are not necessarily have to
rely on source of truth to always fence "the right zombie"; in fact, my
point is that as long as we can fence "someone" when there are zombies
sharing the same member id so that users are notified of such issues and
then react to it, it is good enough. Letting first or last comer wins falls
in that consideration.

I've thought about Jason's "registry id" approach, and I think it may
actually be suitable for providing fencing while approach the goal of avoid
unnecessary rebalances. Here are some more concrete details (@Jason, if it
is not the same as what you had originally in mind, let me know):


a. In JoinGroupRequest, include a "registry id" in addition to the "member
id" to the coordinator. In all other requests like CommitOffset, etc, still
only include the "member id".

b. On the coordinator side, when receiving a JoinGroupRequest with a
registry id with empty member id, and this registry id is indeed in the
cache, it still generates a new member id and return it in the response;
additionally, and update the [registry-id, member-id] mapping in its member
metadata cache (i.e. blind-write to the cache regardless whether it already
contains this registry id); if both registry-id and member-id are
specified, check if it matches in the cache, if not or if the cache does
not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
the client reset its member id and reset again.

** NOTE ** this logic is actually "letting the last comer win", i.e.
whenever a new client comes with a registry id and empty member id, always
treat it as a valid consumer and replace with whatever existed with this
registry id, and the previous member id will be effectively removed from
the group.

c. On the coordinator side, when receiving any other group requests like
CommitOffset, make sure the member id is indeed in this cache (i.e. it is
contained in the [registry-id. member-id] mapping), otherwise reject it as
we do today.


--

So to give an example, if clientA joins group with registry id 1, it will
be assigned a member id X and coordinator remembers current mapping [1, X]
and "X" is the valid member belonging to this group; if another clientB
joins group but with the same registry id 1 with empty member id,
coordinator assigns it with a new member id Y and update the mapping [1,
Y]. Hence the coordinator always blindly trust the new comer as the valid
member and kicks out the previous one. At this time both clientA and
clientB may be fetching from the same assigned partitions, BUT when clientA
tries to commit the offset, by sending with member id X, it will be
rejected.

--

And then back to the original goal of 5/6, when a consumer shuts down and
restarts, it will send a JoinGroupRequest with the registry id with empty
member id, as long as coordinator still remembers this registry id, it will
sends back the original assigned partitions to the consumer, while updating
its cache with the newly generated member id. No rebalance will be
triggered, and if there are other zombies with the same registry ids and
already get the assignment before, although they can still fetch data, they
will not be able to commit.

--



Guozhang



On Wed, Aug 1, 2018 at 8:07 AM, Boyang Chen  wrote:

> Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm
> missing) input here! Looks like we have got more interesting thoughts on
> this KIP.
>
>
> When I draft KIP-345, the original proposal is targeted at solving "static
> membership", which is trying to address 1/2/5/6  proposed by Guozhang. In
> my mindset, the most important detail I'm trying to follow is the owner of
> member id generation or the membership metadata (proposed by John).
>
>
> Guozhang's static membership approach overall LGTM. It would make the
> 

Build failed in Jenkins: kafka-0.10.2-jdk7 #233

[Apache Jenkins Server]

See 


Changes:

[matthias] Bump version to 0.10.2.2

[jason] MINOR: Use kill_java_processes when killing ConsoleConsumer in system

[jason] Revert "MINOR: Use kill_java_processes when killing ConsoleConsumer in

[matthias] Bump version to 0.10.2.3-SNAPSHOT

[rajinisivaram] MINOR: Close timing window in SimpleAclAuthorizer startup 
(#5318)

[jason] MINOR: Fix race condition in TestVerifiableProducer sanity test

[jason] MINOR: Fix kafkatest snapshot version for 0.10.2.3

[github] MINOR: update for 0.10.2.2 release (#5327)

[github] MINOR: add release script (#5328)

[matthias] MINOR: Caching layer should forward record timestamp (#5423) (#5426)

--
[...truncated 1.32 MB...]

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldThrowNullPointerExceptionWhenTryingToAddNullElement STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldThrowNullPointerExceptionWhenTryingToAddNullElement PASSED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLowestAvailableTimestampFromAllInputs STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLowestAvailableTimestampFromAllInputs PASSED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnTimestampOfOnlyRecord STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnTimestampOfOnlyRecord PASSED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLowestAvailableTimestampAfterPreviousLowestRemoved STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLowestAvailableTimestampAfterPreviousLowestRemoved PASSED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldIgnoreNullRecordOnRemove STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldIgnoreNullRecordOnRemove PASSED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLastKnownTimestampWhenAllElementsHaveBeenRemoved STARTED

org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
shouldReturnLastKnownTimestampWhenAllElementsHaveBeenRemoved PASSED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldComputeGroupingForSingleGroupWithMultipleTopics STARTED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldComputeGroupingForSingleGroupWithMultipleTopics PASSED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldNotCreateAnyTasksBecauseOneTopicHasUnknownPartitions STARTED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldNotCreateAnyTasksBecauseOneTopicHasUnknownPartitions PASSED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldComputeGroupingForTwoGroups STARTED

org.apache.kafka.streams.processor.DefaultPartitionGrouperTest > 
shouldComputeGroupingForTwoGroups PASSED

org.apache.kafka.streams.processor.LogAndSkipOnInvalidTimestampTest > 
logAndSkipOnInvalidTimestamp STARTED

org.apache.kafka.streams.processor.LogAndSkipOnInvalidTimestampTest > 
logAndSkipOnInvalidTimestamp PASSED

org.apache.kafka.streams.processor.LogAndSkipOnInvalidTimestampTest > 
extractMetadataTimestamp STARTED

org.apache.kafka.streams.processor.LogAndSkipOnInvalidTimestampTest > 
extractMetadataTimestamp PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testSourceTopics 
STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > testSourceTopics PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSink STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullNameWhenAddingSink PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testNamedTopicMatchesAlreadyProvidedPattern STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
testNamedTopicMatchesAlreadyProvidedPattern PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAddInternalTopicConfigWithCompactAndDeleteSetForWindowStores STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAddInternalTopicConfigWithCompactAndDeleteSetForWindowStores PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldAddInternalTopicConfigWithCompactForNonWindowStores STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 

[jira] [Created] (KAFKA-7233) InMemoryKeyValueStore is not thread-safe for Interactive Queries

[Hashan Gayasri Udugahapattuwa (JIRA)]

Hashan Gayasri Udugahapattuwa created KAFKA-7233:


 Summary: InMemoryKeyValueStore is not thread-safe for Interactive 
Queries
 Key: KAFKA-7233
 URL: https://issues.apache.org/jira/browse/KAFKA-7233
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.0, 1.1.0
 Environment: Fedora 27
Reporter: Hashan Gayasri Udugahapattuwa


When querying externally from via the 
[ReadOnlyKeyValueStore|https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.java]
 API (Interactive Queries), the InMemoryKeyValueStore does not protect its 
internal map from concurrent reads and writes which may produce incorrect 
results to the  ReadOnlyKeyValueStore API. Note that reads(query thread) and 
writes(stream thread) can happen concurrently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2853

[Apache Jenkins Server]

See 


Changes:

[harsha] MINOR: Fixed default streams state dir location. (#5441)

--
[...truncated 425.38 KB...]
kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerGroupServiceWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAutoCommitDisabledWithAuthenticationFailure 
STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testManualAssignmentConsumerWithAutoCommitDisabledWithAuthenticationFailure 
PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

[DISCUSS] KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

[Ratish Ravindran]

Hi,

I would like to open a discussion thread on KIP-350:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-350%3A+Allow+kafka-topics.sh+to+take+brokerid+as+parameter+to+show+partitions+associated+with+it

This is KIP is to add the optional brokerid parameter in kafka-topics.sh,
to only get the partition associated with that specific broker. Current,ly
we use grep to do so.

Let me know your thoughts and suggestions.

Thanks,
Ratish

[jira] [Created] (KAFKA-7232) Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

[Ratish Ravindran (JIRA)]

Ratish Ravindran created KAFKA-7232:
---

 Summary: Allow kafka-topics.sh to take brokerid as parameter to 
show partitions associated with it
 Key: KAFKA-7232
 URL: https://issues.apache.org/jira/browse/KAFKA-7232
 Project: Kafka
  Issue Type: Improvement
Reporter: Ratish Ravindran


Currently with kafka-topics.sh if we want to get the list of partitions 
associated with a specific broker irrespective of whether it is leader or 
replica, we pipe the output and then do grep on it.

I am proposing the change to add option in TopicCommand.scala to pass the 
broker id.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Nishanth Pradeep]

The only issue I see with this is that Sink#topic would also need to be
Optional as was pointed out already. Since Sink#topic is a preexisting
method, changing its return type would break backwards compatibility.

On the other hand, it might be worth it to rip that bandaid now.

Best,
Nishanth Pradeep

On Wed, Aug 1, 2018 at 11:56 AM Guozhang Wang  wrote:

> For source node, only one of `Set topicsSet` and `TopicPattern
> topicPattern()` will be specified by the user. Similarly for sink node,
> only one of `String` and `TopicNameExtractor` will be specified by the
> user. Although I've not seen Nishanth's updated PR, I think when it is not
> specified today we will return null in that case.
>
> If we want to improve on this situation with Optional, we'd need to do it
> on all of these functions. Also note that for `Source#toString()` and
> `Sink#toString()` we should only include the specified field in the
> resulted representation.
>
>
> Guozhang
>
> On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy  wrote:
>
> > Ewen - no as I don't believe they are never null. Whereas the
> > topicNameExtractor method returns null if it is the default extractor or
> > the extractor. So i think this would be better to be optional as it is
> > optionally returning a TopicNameExtractor
> >
> > On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
> > wrote:
> >
> > > Generally +1 (binding)
> > >
> > > It would be helpful to just provide the full, updated interfaces in the
> > > KIP and mark things as new with comments if needed. I had to go back
> and
> > > read the discussion thread to make sure I was understanding the intent
> > > correctly.
> > >
> > > Damian -- if we make that Optional, shouldn't the methods on Source
> also
> > > be Optional types?
> > >
> > > -Ewen
> > >
> > > On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
> > wrote:
> > >
> > >> Hi Nishanth,
> > >>
> > >> I have one nit on the KIP. I think the topicNameExtractor method
> should
> > >> return Optional rather than null.
> > >> Sorry I'm late here.
> > >>
> > >> Thanks,
> > >> Damian
> > >>
> > >> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep 
> > >> wrote:
> > >>
> > >> > We need one more binding vote.
> > >> >
> > >> > Binding Votes:
> > >> >
> > >> >- Matthias J. Sax
> > >> >- Guozhang Wong
> > >> >
> > >> > Community Votes:
> > >> >
> > >> >- Bill Bejeck
> > >> >- Ted Yu
> > >> >
> > >> > Best,
> > >> > Nishanth Pradeep
> > >> >
> > >> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
> > wrote:
> > >> >
> > >> > > Thanks for the KIP!
> > >> > >
> > >> > > +1
> > >> > >
> > >> > > -Bill
> > >> > >
> > >> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang  >
> > >> > wrote:
> > >> > >
> > >> > > > +1
> > >> > > >
> > >> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
> > >> > matth...@confluent.io
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > +1 (binding)
> > >> > > > >
> > >> > > > > -Matthias
> > >> > > > >
> > >> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
> > >> > > > > > +1
> > >> > > > > >
> > >> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
> > >> > > > nishanth...@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > >> Hello,
> > >> > > > > >>
> > >> > > > > >> I'm calling a vote for KIP-321:
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 321%3A+Update+
> > >> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
> > >> > > > > >>
> > >> > > > > >> Best,
> > >> > > > > >> Nishanth Pradeep
> > >> > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > -- Guozhang
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
>
>
> --
> -- Guozhang
>

Build failed in Jenkins: kafka-trunk-jdk10 #353

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 6970, done.
remote: Compressing objects:   1% (1/67)   remote: Compressing objects: 
  2% (2/67)   remote: Compressing objects:   4% (3/67)   
remote: Compressing objects:   5% (4/67)   remote: Compressing objects: 
  7% (5/67)   remote: Compressing objects:   8% (6/67)   
remote: Compressing objects:  10% (7/67)   remote: Compressing objects: 
 11% (8/67)   remote: Compressing objects:  13% (9/67)   
remote: Compressing objects:  14% (10/67)   remote: Compressing 
objects:  16% (11/67)   remote: Compressing objects:  17% (12/67)   
remote: Compressing objects:  19% (13/67)   remote: Compressing 
objects:  20% (14/67)   remote: Compressing objects:  22% (15/67)   
remote: Compressing objects:  23% (16/67)   remote: Compressing 
objects:  25% (17/67)   remote: Compressing objects:  26% (18/67)   
remote: Compressing objects:  28% (19/67)   remote: Compressing 
objects:  29% (20/67)   remote: Compressing objects:  31% (21/67)   
remote: Compressing objects:  32% (22/67)   remote: Compressing 
objects:  34% (23/67)   remote: Compressing objects:  35% (24/67)   
remote: Compressing objects:  37% (25/67)   remote: Compressing 
objects:  38% (26/67)   remote: Compressing objects:  40% (27/67)   
remote: Compressing objects:  41% (28/67)   remote: Compressing 
objects:  43% (29/67)   remote: Compressing objects:  44% (30/67)   
remote: Compressing objects:  46% (31/67)   remote: Compressing 
objects:  47% (32/67)   remote: Compressing objects:  49% (33/67)   
remote: Compressing objects:  50% (34/67)   remote: Compressing 
objects:  52% (35/67)   remote: Compressing objects:  53% (36/67)   
remote: Compressing objects:  55% (37/67)   remote: Compressing 
objects:  56% (38/67)   remote: Compressing objects:  58% (39/67)   
remote: Compressing objects:  59% (40/67)   remote: Compressing 
objects:  61% (41/67)   remote: Compressing objects:  62% (42/67)   
remote: Compressing objects:  64% (43/67)   remote: Compressing 
objects:  65% (44/67)   remote: Compressing objects:  67% (45/67)   
remote: Compressing objects:  68% (46/67)   remote: Compressing 
objects:  70% (47/67)   remote: Compressing objects:  71% (48/67)   
remote: Compressing objects:  73% (49/67)   remote: Compressing 
objects:  74% (50/67)   remote: Compressing objects:  76% (51/67)   
remote: Compressing objects:  77% (52/67)   remote: Compressing 
objects:  79% (53/67)   remote: Compressing objects:  80% (54/67)   
remote: Compressing objects:  82% (55/67)   

Build failed in Jenkins: kafka-0.10.2-jdk7 #232

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H27 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: missing object referenced by 'refs/tags/1.1.1-rc0'
error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 7727, done.
remote: Compressing objects:   1% (1/69)   remote: Compressing objects: 
  2% (2/69)   remote: Compressing objects:   4% (3/69)   
remote: Compressing objects:   5% (4/69)   remote: Compressing objects: 
  7% (5/69)   remote: Compressing objects:   8% (6/69)   
remote: Compressing objects:  10% (7/69)   remote: Compressing objects: 
 11% (8/69)   remote: Compressing objects:  13% (9/69)   
remote: Compressing objects:  14% (10/69)   remote: Compressing 
objects:  15% (11/69)   remote: Compressing objects:  17% (12/69)   
remote: Compressing objects:  18% (13/69)   remote: Compressing 
objects:  20% (14/69)   remote: Compressing objects:  21% (15/69)   
remote: Compressing objects:  23% (16/69)   remote: Compressing 
objects:  24% (17/69)   remote: Compressing objects:  26% (18/69)   
remote: Compressing objects:  27% (19/69)   remote: Compressing 
objects:  28% (20/69)   remote: Compressing objects:  30% (21/69)   
remote: Compressing objects:  31% (22/69)   remote: Compressing 
objects:  33% (23/69)   remote: Compressing objects:  34% (24/69)   
remote: Compressing objects:  36% (25/69)   remote: Compressing 
objects:  37% (26/69)   remote: Compressing objects:  39% (27/69)   
remote: Compressing objects:  40% (28/69)   remote: Compressing 
objects:  42% (29/69)   remote: Compressing objects:  43% (30/69)   
remote: Compressing objects:  44% (31/69)   remote: Compressing 
objects:  46% (32/69)   remote: Compressing objects:  47% (33/69)   
remote: Compressing objects:  49% (34/69)   remote: Compressing 
objects:  50% (35/69)   remote: Compressing objects:  52% (36/69)   
remote: Compressing objects:  53% (37/69)   remote: Compressing 
objects:  55% (38/69)   remote: Compressing objects:  56% (39/69)   
remote: Compressing objects:  57% (40/69)   remote: Compressing 
objects:  59% (41/69)   remote: Compressing objects:  60% (42/69)   
remote: Compressing objects:  62% (43/69)   remote: Compressing 
objects:  63% (44/69)   remote: Compressing objects:  65% (45/69)   
remote: Compressing objects:  66% (46/69)   remote: Compressing 
objects:  68% (47/69)   remote: Compressing objects:  69% (48/69)   
remote: Compressing objects:  71% (49/69)   remote: Compressing 
objects:  72% (50/69)   remote: Compressing objects:  73% (51/69)   
remote: Compressing objects:  75% (52/69)   remote: Compressing 
objects:  76% (53/69)   remote: Compressing objects:  78% (54/69)   

Build failed in Jenkins: kafka-trunk-jdk10 #352

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H31 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
remote: Counting objects: 4100, done.
remote: Compressing objects:   1% (1/64)   remote: Compressing objects: 
  3% (2/64)   remote: Compressing objects:   4% (3/64)   
remote: Compressing objects:   6% (4/64)   remote: Compressing objects: 
  7% (5/64)   remote: Compressing objects:   9% (6/64)   
remote: Compressing objects:  10% (7/64)   remote: Compressing objects: 
 12% (8/64)   remote: Compressing objects:  14% (9/64)   
remote: Compressing objects:  15% (10/64)   remote: Compressing 
objects:  17% (11/64)   remote: Compressing objects:  18% (12/64)   
remote: Compressing objects:  20% (13/64)   remote: Compressing 
objects:  21% (14/64)   remote: Compressing objects:  23% (15/64)   
remote: Compressing objects:  25% (16/64)   remote: Compressing 
objects:  26% (17/64)   remote: Compressing objects:  28% (18/64)   
remote: Compressing objects:  29% (19/64)   remote: Compressing 
objects:  31% (20/64)   remote: Compressing objects:  32% (21/64)   
remote: Compressing objects:  34% (22/64)   remote: Compressing 
objects:  35% (23/64)   remote: Compressing objects:  37% (24/64)   
remote: Compressing objects:  39% (25/64)   remote: Compressing 
objects:  40% (26/64)   remote: Compressing objects:  42% (27/64)   
remote: Compressing objects:  43% (28/64)   remote: Compressing 
objects:  45% (29/64)   remote: Compressing objects:  46% (30/64)   
remote: Compressing objects:  48% (31/64)   remote: Compressing 
objects:  50% (32/64)   remote: Compressing objects:  51% (33/64)   
remote: Compressing objects:  53% (34/64)   remote: Compressing 
objects:  54% (35/64)   remote: Compressing objects:  56% (36/64)   
remote: Compressing objects:  57% (37/64)   remote: Compressing 
objects:  59% (38/64)   remote: Compressing objects:  60% (39/64)   
remote: Compressing objects:  62% (40/64)   remote: Compressing 
objects:  64% (41/64)   remote: Compressing objects:  65% (42/64)   
remote: Compressing objects:  67% (43/64)   remote: Compressing 
objects:  68% (44/64)   remote: Compressing objects:  70% (45/64)   
remote: Compressing objects:  71% (46/64)   remote: Compressing 
objects:  73% (47/64)   remote: Compressing objects:  75% (48/64)   
remote: Compressing objects:  76% (49/64)   remote: Compressing 
objects:  78% (50/64)   remote: Compressing objects:  79% (51/64)   
remote: Compressing objects:  81% (52/64)   remote: Compressing 
objects:  82% (53/64)   remote: Compressing objects:  84% (54/64)   
remote: Compressing objects:  85% (55/64)   

Re: Permission to create KIP

[Ratish Ravindran]

Thanks a lot.

-Ratish

On Thu, Aug 2, 2018 at 12:12 AM Matthias J. Sax 
wrote:

> Done. :)
>
> -Matthias
>
> On 8/1/18 11:19 AM, Ratish Ravindran wrote:
> > Userid is *rravindran*
> >
> > -Ratish
> >
> > On Wed, Aug 1, 2018 at 10:22 PM Matthias J. Sax 
> > wrote:
> >
> >> Can you provide you login ID so we can grant permission?
> >>
> >> (The screenshot was striped btw)
> >>
> >> -Matthias
> >>
> >>
> >> On 7/31/18 9:56 PM, Ratish Ravindran wrote:
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>
> >>> When tried to click on Create KIP
> >>> <
> >>
> https://cwiki.apache.org/confluence?templateId=54329345=KAFKA=KAFKA=KIP-NEXT%3A+Insert+Title+Here
> >>> ,
> >>> got
> >>>
> >>> Screen Shot 2018-08-01 at 10.25.35 AM.png
> >>>
> >>> Thanks,
> >>> Ratish
> >>>
> >>> On Wed, Aug 1, 2018 at 6:25 AM Matthias J. Sax  >>> > wrote:
> >>>
> >>> What is your Wiki ID?
> >>>
> >>> -Matthias
> >>>
> >>> On 7/30/18 8:55 PM, Ratish Ravindran wrote:
> >>> > Hi,
> >>> >
> >>> > I am trying to create a KIP, but I don't access to do so. Can
> >> anyone
> >>> > provide me with the access ?
> >>> >
> >>> > Thanks,
> >>> > Ratish
> >>> >
> >>>
> >>
> >>
> >
>
>

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

[Brett Rann]

Thanks Jason.

>From what we've seen so far adding a max time should be the easy part
because there's already a block of checks for if compaction should be run.
The active segment issue can be dealt with by specifying a time based
segment.

Triggering compaction is the harder part given it's not really triggered
but evaluated by the log cleaner.  We might just go with the config setting
but will look further into triggering.


On Thu, Aug 2, 2018 at 10:56 AM Jason Gustafson  wrote:

> Hey Brett,
>
> I'd suggest going ahead with a KIP if you already have some ideas about it.
> One of the complications you may run into is that the active segment is not
> cleaned at present.
>
> I went ahead and gave you wiki permission. I assumed your username is
> brettrann, so let me know if it's different.
>
> Thanks,
> Jason
>
> On Wed, Aug 1, 2018 at 5:38 PM, Brett Rann 
> wrote:
>
> > Ping on this. We're prepared to do the work i'm just not sure of the
> right
> > way to start. If nobody wants to discuss it now should we just write up a
> > KIP?
> >
> > On Thu, Jul 19, 2018 at 10:57 AM Brett Rann  wrote:
> >
> > > re: https://issues.apache.org/jira/browse/KAFKA-7137
> 
> > >
> > > My team is investigating what would be involved in code changes to give
> > > some finer control over when compaction runs. Detail in the ticket but
> > > essentially the current way is that dirty.ratio is hijacked to be set
> to
> > 0
> > > to give guarantees that a tombstone record will result in previous
> record
> > > being deleted within a set time.
> > >
> > > This results in a lot of unnecessary compaction, and we would like to
> add
> > > the ability to provide a max delay for compaction to be used in
> > combination
> > > with a "healthy" dirty.ratio setting to meet this requirement instead.
> > >
> > > And to consider an API to trigger compaction (but that is looking
> > > difficult because compaction isn't really triggered, it's evaluated by
> > the
> > > log cleaner thread).
> > >
> > > We'd like to get some discussion going around this. Should we draft up
> a
> > > KIP first then kick off a discussion? (if yes, can I get KIP access
> for:
> > > brettr...@gmail.com )
> > >
> > > Thanks!
> > >
> > > --
> > >
> > > Brett Rann
> > >
> > > Senior DevOps Engineer
> > >
> > >
> > > Zendesk International Ltd
> > >
> > > 395 Collins Street, Melbourne VIC 3000 Australia
> > >
> > > Mobile: +61 (0) 418 826 017
> > >
> >
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Build failed in Jenkins: kafka-trunk-jdk10 #351

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 6964, done.
remote: Compressing objects:   1% (1/65)   remote: Compressing objects: 
  3% (2/65)   remote: Compressing objects:   4% (3/65)   
remote: Compressing objects:   6% (4/65)   remote: Compressing objects: 
  7% (5/65)   remote: Compressing objects:   9% (6/65)   
remote: Compressing objects:  10% (7/65)   remote: Compressing objects: 
 12% (8/65)   remote: Compressing objects:  13% (9/65)   
remote: Compressing objects:  15% (10/65)   remote: Compressing 
objects:  16% (11/65)   remote: Compressing objects:  18% (12/65)   
remote: Compressing objects:  20% (13/65)   remote: Compressing 
objects:  21% (14/65)   remote: Compressing objects:  23% (15/65)   
remote: Compressing objects:  24% (16/65)   remote: Compressing 
objects:  26% (17/65)   remote: Compressing objects:  27% (18/65)   
remote: Compressing objects:  29% (19/65)   remote: Compressing 
objects:  30% (20/65)   remote: Compressing objects:  32% (21/65)   
remote: Compressing objects:  33% (22/65)   remote: Compressing 
objects:  35% (23/65)   remote: Compressing objects:  36% (24/65)   
remote: Compressing objects:  38% (25/65)   remote: Compressing 
objects:  40% (26/65)   remote: Compressing objects:  41% (27/65)   
remote: Compressing objects:  43% (28/65)   remote: Compressing 
objects:  44% (29/65)   remote: Compressing objects:  46% (30/65)   
remote: Compressing objects:  47% (31/65)   remote: Compressing 
objects:  49% (32/65)   remote: Compressing objects:  50% (33/65)   
remote: Compressing objects:  52% (34/65)   remote: Compressing 
objects:  53% (35/65)   remote: Compressing objects:  55% (36/65)   
remote: Compressing objects:  56% (37/65)   remote: Compressing 
objects:  58% (38/65)   remote: Compressing objects:  60% (39/65)   
remote: Compressing objects:  61% (40/65)   remote: Compressing 
objects:  63% (41/65)   remote: Compressing objects:  64% (42/65)   
remote: Compressing objects:  66% (43/65)   remote: Compressing 
objects:  67% (44/65)   remote: Compressing objects:  69% (45/65)   
remote: Compressing objects:  70% (46/65)   remote: Compressing 
objects:  72% (47/65)   remote: Compressing objects:  73% (48/65)   
remote: Compressing objects:  75% (49/65)   remote: Compressing 
objects:  76% (50/65)   remote: Compressing objects:  78% (51/65)   
remote: Compressing objects:  80% (52/65)   remote: Compressing 
objects:  81% (53/65)   remote: Compressing objects:  83% (54/65)   
remote: Compressing objects:  84% (55/65)   

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

[Jason Gustafson]

Hey Brett,

I'd suggest going ahead with a KIP if you already have some ideas about it.
One of the complications you may run into is that the active segment is not
cleaned at present.

I went ahead and gave you wiki permission. I assumed your username is
brettrann, so let me know if it's different.

Thanks,
Jason

On Wed, Aug 1, 2018 at 5:38 PM, Brett Rann 
wrote:

> Ping on this. We're prepared to do the work i'm just not sure of the right
> way to start.  If nobody wants to discuss it now should we just write up a
> KIP?
>
> On Thu, Jul 19, 2018 at 10:57 AM Brett Rann  wrote:
>
> > re: https://issues.apache.org/jira/browse/KAFKA-7137
> >
> > My team is investigating what would be involved in code changes to give
> > some finer control over when compaction runs.  Detail in the ticket but
> > essentially the current way is that dirty.ratio is hijacked to be set to
> 0
> > to give guarantees that a tombstone record will result in previous record
> > being deleted within a set time.
> >
> > This results in a lot of unnecessary compaction, and we would like to add
> > the ability to provide a max delay for compaction to be used in
> combination
> > with a "healthy" dirty.ratio setting to meet this requirement instead.
> >
> > And to consider an API to trigger compaction (but that is looking
> > difficult because compaction isn't really triggered, it's evaluated by
> the
> > log cleaner thread).
> >
> > We'd like to get some discussion going around this.  Should we draft up a
> > KIP first then kick off a discussion?  (if yes, can I get KIP access for:
> > brettr...@gmail.com )
> >
> > Thanks!
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>

Build failed in Jenkins: kafka-0.10.2-jdk7 #231

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
error: Could not read 3af2cecb9d84369021549946db95dbef9bfe68fa
remote: Counting objects: 3946, done.
remote: Compressing objects:   1% (1/62)   remote: Compressing objects: 
  3% (2/62)   remote: Compressing objects:   4% (3/62)   
remote: Compressing objects:   6% (4/62)   remote: Compressing objects: 
  8% (5/62)   remote: Compressing objects:   9% (6/62)   
remote: Compressing objects:  11% (7/62)   remote: Compressing objects: 
 12% (8/62)   remote: Compressing objects:  14% (9/62)   
remote: Compressing objects:  16% (10/62)   remote: Compressing 
objects:  17% (11/62)   remote: Compressing objects:  19% (12/62)   
remote: Compressing objects:  20% (13/62)   remote: Compressing 
objects:  22% (14/62)   remote: Compressing objects:  24% (15/62)   
remote: Compressing objects:  25% (16/62)   remote: Compressing 
objects:  27% (17/62)   remote: Compressing objects:  29% (18/62)   
remote: Compressing objects:  30% (19/62)   remote: Compressing 
objects:  32% (20/62)   remote: Compressing objects:  33% (21/62)   
remote: Compressing objects:  35% (22/62)   remote: Compressing 
objects:  37% (23/62)   remote: Compressing objects:  38% (24/62)   
remote: Compressing objects:  40% (25/62)   remote: Compressing 
objects:  41% (26/62)   remote: Compressing objects:  43% (27/62)   
remote: Compressing objects:  45% (28/62)   remote: Compressing 
objects:  46% (29/62)   remote: Compressing objects:  48% (30/62)   
remote: Compressing objects:  50% (31/62)   remote: Compressing 
objects:  51% (32/62)   remote: Compressing objects:  53% (33/62)   
remote: Compressing objects:  54% (34/62)   remote: Compressing 
objects:  56% (35/62)   remote: Compressing objects:  58% (36/62)   
remote: Compressing objects:  59% (37/62)   remote: Compressing 
objects:  61% (38/62)   remote: Compressing objects:  62% (39/62)   
remote: Compressing objects:  64% (40/62)   remote: Compressing 
objects:  66% (41/62)   remote: Compressing objects:  67% (42/62)   
remote: Compressing objects:  69% (43/62)   remote: Compressing 
objects:  70% (44/62)   remote: Compressing objects:  72% (45/62)   
remote: Compressing objects:  74% (46/62)   remote: Compressing 
objects:  75% (47/62)   remote: Compressing objects:  77% (48/62)   
remote: Compressing objects:  79% (49/62)   remote: Compressing 
objects:  80% (50/62)   remote: Compressing objects:  82% (51/62)   
remote: Compressing objects:  83% (52/62)   remote: Compressing 
objects:  85% (53/62)   remote: Compressing objects:  87% (54/62)   

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

I talked with Colin offline. I think specifically he felt the version bump
on the broker was overkill since the broker still has to support the empty
group id for older versions. I had thought that eventually we would be able
to remove those old versions, but it's true that this may not happen until
indefinitely far in the future. I think the main improvement here is
changing the default group.id to null instead of "". I could go either way
on whether bumping the protocol is useful. I do think it is helpful though
to signal clearly that it its use is deprecated and discouraged, especially
in light of the ACL problem. I guess we could just deprecate the use on the
client. What do you think?

Thanks,
Jason

On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian  wrote:

> Thanks Jason for responding to Colin's concerns.
>
> If there are no other comment / feedback / objection I'll start a vote
> soon.
>
> Thanks.
> --Vahid
>
>
>
> From:   Jason Gustafson 
> To: dev 
> Date:   07/27/2018 10:38 AM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Colin,
>
> The problem is both that the empty group id is the default value and that
> it is actually accepted by the broker for offset commits. Combine that
> with
> the fact that auto commit is enabled by default and you users get
> surprising behavior. If you look at a random Kafka cluster, you'll
> probably
> find a bunch of inadvertent offset commits for the empty group id. I was
> hoping we could distinguish between users who are using the empty group id
> as an accident of the default configuration and those who use it
> intentionally. By default, there will be no group id and the consumer will
> not commit offsets. If a user has actually intentionally used the empty
> group id, however, it will continue to work. I actually think there are
> probably very few people doing this (maybe even no one), but I thought we
> might err on the side of compatibility.
>
> The big incompatible change here is having brokers reject using
> assign(...)
> > with empty / null group.id.
>
>
> This is not correct. In the proposal, the broker will only reject the
> empty
> group id for the new version of OffsetCommit. Older clients, which cannot
> be changed, will continue to work because the old versions of the
> OffsetCommit API still accept the empty group id. The null group id is
> different from the empty group id: it is not allowed in any version of the
> API. It is basically a way to indicate that the consumer has no dependence
> on the coordinator at all, which we actually have a surprising number of
> use cases for. Furthermore, if a user has an actual need for the empty
> group id, it will still be allowed. We are just deprecating it.
>
> -Jason
>
> On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  wrote:
>
> > Sorry if this is a silly question, but what's the rationale for
> switching
> > to using null for the default group id, rather than the empty string?
> > Continuing to use the empty string seems like less churn.  And after
> all,
> > we're not using the empty string group name for anything else.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...) with empty / null group.id.  If I understand correctly, the
> > KIP proposes that this change be made on the brokers on the next
> > incompatible Kafka release.  But that has nothing to do with client
> > versions.  Why not just have a broker config which controls this?  Maybe
> "
> > allow.assign.empty.group.id", or something like that.  At first, the
> > default will be true, and then eventually we can flip it over to false.
> >
> > It seems like the main rationale for tying this behavior to the Kafka
> > client version is to force people to stop using the empty group id so
> that
> > they can upgrade their clients.  But it's also possible that people will
> > stop upgrading their Kafka clients instead.  That would be pretty
> negative
> > since  they'd miss out on any efficiency and feature improvements in the
> > new clients and eventually have to do more protocol downgrading, etc.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > Hi Jason,
> > >
> > > That makes sense.
> > > I have updated the KIP based on the recent feedback.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Jason Gustafson 
> > > To: dev 
> > > Date:   07/25/2018 02:23 PM
> > > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hi Vahid,
> > >
> > > I was thinking we'd only use the old API version if we had to. That
> is,
> > > only if the user has explicitly configured "" as the group.id.
> > Otherwise,
> > > we'd just use the new one. Another option is to just drop support in
> the
> > > client for the empty group id, but usually we allow a deprecation
> period
> > > for changes like this.
> > >
> > 

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

[Brett Rann]

Ping on this. We're prepared to do the work i'm just not sure of the right
way to start.  If nobody wants to discuss it now should we just write up a
KIP?

On Thu, Jul 19, 2018 at 10:57 AM Brett Rann  wrote:

> re: https://issues.apache.org/jira/browse/KAFKA-7137
>
> My team is investigating what would be involved in code changes to give
> some finer control over when compaction runs.  Detail in the ticket but
> essentially the current way is that dirty.ratio is hijacked to be set to 0
> to give guarantees that a tombstone record will result in previous record
> being deleted within a set time.
>
> This results in a lot of unnecessary compaction, and we would like to add
> the ability to provide a max delay for compaction to be used in combination
> with a "healthy" dirty.ratio setting to meet this requirement instead.
>
> And to consider an API to trigger compaction (but that is looking
> difficult because compaction isn't really triggered, it's evaluated by the
> log cleaner thread).
>
> We'd like to get some discussion going around this.  Should we draft up a
> KIP first then kick off a discussion?  (if yes, can I get KIP access for:
> brettr...@gmail.com )
>
> Thanks!
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Jason Gustafson]

Hey Stanislav,

Just to make sure I understood you right - you propose not exposing any new
> exception types but rather the interface itself only?


Yes, exactly. Our exception hierarchy is a bit of a mess to be honest.
Interfaces are more flexible and here it simplifies the error handling.

Regardless, I believe this is best left out for another KIP as I feel it
> would warrant a bigger discussion


Ok, that's fair. I thought I'd suggest it here just to see if there was any
interest in the community. At least with this KIP, users have a viable way
to skip past bad data if they wish.

-Jason

On Tue, Jul 31, 2018 at 2:42 AM, Stanislav Kozlovski  wrote:

> Hey Jason,
>
> Just to make sure I understood you right - you propose not exposing any new
> exception types but rather the interface itself only? So a sample code
> dealing with this would be something like:
>
> try {
> // ...
> } catch (KafkaException e) {
> if (e instanceof UnconsumableRecordException) {
>   // handle retry
> }
> }
>
> If that is the case, I like it better.
>
>
> In regards to automatic handling of unconsumable messages - I like that
> idea too. To me, a callback seems like the more straightforward approach. A
> config such as `seek.past.unconsumable.record` limits the behavior too
> much
> in my opinion, I believe giving them the option to implement a (or use the
> default) callback is better in that way.
> Regardless, I believe this is best left out for another KIP as I feel it
> would warrant a bigger discussion
>
> Best,
> Stanislav
>
> On Mon, Jul 30, 2018 at 9:34 PM Jason Gustafson 
> wrote:
>
> > Hey Stanislav,
> >
> > Thanks for the KIP. I think the goal is to allow users to seek past a
> > records which cannot be parsed for whatever reason. However, it's a
> little
> > annoying that you need to catch two separate types to handle this. I'm
> > wondering if it makes sense to expose an interface like
> > `UnconsumableRecordException` or something like that. The consumer could
> > then have separate internal exception types which extend from
> > InvalidRecordException and SerializationException respectively and
> > implement `UnconsumableRecordException`. That would simplify the handling
> > and users could check the cause if they cared which case it was.
> >
> > Another question for consideration. I'd imagine some users would find it
> > helpful to seek past failed messages automatically. If there is a corrupt
> > record, for example, there's almost nothing you can do except seek past
> it
> > anyway. I'm wondering if there should be a config for this or if users
> > should be able to install a callback of some sorts to handle failed
> > records. Not sure if this is that big of a problem for users, but
> > interested to hear others thoughts.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Jul 20, 2018 at 6:32 PM, Stanislav Kozlovski <
> > stanis...@confluent.io
> > > wrote:
> >
> > > Hi Ted,
> > >
> > > I do plan to start one. When is the appropriate time? My reasoning was
> > that
> > > people would like to view the changes first
> > >
> > > On Fri, Jul 20, 2018, 6:21 PM Ted Yu  wrote:
> > >
> > > > Hi, Stanislav:
> > > > Do you plan to start VOTE thread ?
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Jul 20, 2018 at 6:11 PM Stanislav Kozlovski <
> > > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hey group,
> > > > >
> > > > > I added a Pull Request for this KIP - here it is
> > > > > https://github.com/apache/kafka/pull/5410
> > > > > Please take a look.
> > > > >
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > > On Thu, Jul 5, 2018 at 11:06 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Yes, the Scala consumers have been removed in 2.0.0, which
> > simplifies
> > > > > some
> > > > > > of this. The following commit was an initial step in unifying the
> > > > > exception
> > > > > > handling:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > https://github.com/apache/kafka/commit/
> 96bcfdfc7c9aac075635b2034e65e4
> > > 12a725672e
> > > > > >
> > > > > > But more can be done as you mentioned.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On 5 Jul 2018 9:36 am, "Stanislav Kozlovski" <
> > stanis...@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > Hey Ismael,
> > > > > >
> > > > > > It is only slightly related - my PR would attach two new
> attributes
> > > and
> > > > > > also touch upon deserialization exceptions.
> > > > > >
> > > > > > But this PR did provide me with some insight:
> > > > > > Maybe the best approach would be to make
> `InvalidRecordException` a
> > > > > public
> > > > > > exception instead of introducing a new one - I did not realize it
> > was
> > > > not
> > > > > > publicly exposed.
> > > > > > Does the following:
> > > > > >
> > > > > >  InvalidMessageException extends CorruptRecordException for
> > temporary
> > > > > > compatibility with the old Scala clients.
> > > > > >  * We want to update the server side code to use and catch the
> new
> 

Build failed in Jenkins: kafka-trunk-jdk10 #350

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 6964, done.
remote: Compressing objects:   1% (1/65)   remote: Compressing objects: 
  3% (2/65)   remote: Compressing objects:   4% (3/65)   
remote: Compressing objects:   6% (4/65)   remote: Compressing objects: 
  7% (5/65)   remote: Compressing objects:   9% (6/65)   
remote: Compressing objects:  10% (7/65)   remote: Compressing objects: 
 12% (8/65)   remote: Compressing objects:  13% (9/65)   
remote: Compressing objects:  15% (10/65)   remote: Compressing 
objects:  16% (11/65)   remote: Compressing objects:  18% (12/65)   
remote: Compressing objects:  20% (13/65)   remote: Compressing 
objects:  21% (14/65)   remote: Compressing objects:  23% (15/65)   
remote: Compressing objects:  24% (16/65)   remote: Compressing 
objects:  26% (17/65)   remote: Compressing objects:  27% (18/65)   
remote: Compressing objects:  29% (19/65)   remote: Compressing 
objects:  30% (20/65)   remote: Compressing objects:  32% (21/65)   
remote: Compressing objects:  33% (22/65)   remote: Compressing 
objects:  35% (23/65)   remote: Compressing objects:  36% (24/65)   
remote: Compressing objects:  38% (25/65)   remote: Compressing 
objects:  40% (26/65)   remote: Compressing objects:  41% (27/65)   
remote: Compressing objects:  43% (28/65)   remote: Compressing 
objects:  44% (29/65)   remote: Compressing objects:  46% (30/65)   
remote: Compressing objects:  47% (31/65)   remote: Compressing 
objects:  49% (32/65)   remote: Compressing objects:  50% (33/65)   
remote: Compressing objects:  52% (34/65)   remote: Compressing 
objects:  53% (35/65)   remote: Compressing objects:  55% (36/65)   
remote: Compressing objects:  56% (37/65)   remote: Compressing 
objects:  58% (38/65)   remote: Compressing objects:  60% (39/65)   
remote: Compressing objects:  61% (40/65)   remote: Compressing 
objects:  63% (41/65)   remote: Compressing objects:  64% (42/65)   
remote: Compressing objects:  66% (43/65)   remote: Compressing 
objects:  67% (44/65)   remote: Compressing objects:  69% (45/65)   
remote: Compressing objects:  70% (46/65)   remote: Compressing 
objects:  72% (47/65)   remote: Compressing objects:  73% (48/65)   
remote: Compressing objects:  75% (49/65)   remote: Compressing 
objects:  76% (50/65)   remote: Compressing objects:  78% (51/65)   
remote: Compressing objects:  80% (52/65)   remote: Compressing 
objects:  81% (53/65)   remote: Compressing objects:  83% (54/65)   
remote: Compressing objects:  84% (55/65)   

Build failed in Jenkins: kafka-0.10.2-jdk7 #230

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H27 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: missing object referenced by 'refs/tags/1.1.1-rc0'
error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 7721, done.
remote: Compressing objects:   1% (1/67)   remote: Compressing objects: 
  2% (2/67)   remote: Compressing objects:   4% (3/67)   
remote: Compressing objects:   5% (4/67)   remote: Compressing objects: 
  7% (5/67)   remote: Compressing objects:   8% (6/67)   
remote: Compressing objects:  10% (7/67)   remote: Compressing objects: 
 11% (8/67)   remote: Compressing objects:  13% (9/67)   
remote: Compressing objects:  14% (10/67)   remote: Compressing 
objects:  16% (11/67)   remote: Compressing objects:  17% (12/67)   
remote: Compressing objects:  19% (13/67)   remote: Compressing 
objects:  20% (14/67)   remote: Compressing objects:  22% (15/67)   
remote: Compressing objects:  23% (16/67)   remote: Compressing 
objects:  25% (17/67)   remote: Compressing objects:  26% (18/67)   
remote: Compressing objects:  28% (19/67)   remote: Compressing 
objects:  29% (20/67)   remote: Compressing objects:  31% (21/67)   
remote: Compressing objects:  32% (22/67)   remote: Compressing 
objects:  34% (23/67)   remote: Compressing objects:  35% (24/67)   
remote: Compressing objects:  37% (25/67)   remote: Compressing 
objects:  38% (26/67)   remote: Compressing objects:  40% (27/67)   
remote: Compressing objects:  41% (28/67)   remote: Compressing 
objects:  43% (29/67)   remote: Compressing objects:  44% (30/67)   
remote: Compressing objects:  46% (31/67)   remote: Compressing 
objects:  47% (32/67)   remote: Compressing objects:  49% (33/67)   
remote: Compressing objects:  50% (34/67)   remote: Compressing 
objects:  52% (35/67)   remote: Compressing objects:  53% (36/67)   
remote: Compressing objects:  55% (37/67)   remote: Compressing 
objects:  56% (38/67)   remote: Compressing objects:  58% (39/67)   
remote: Compressing objects:  59% (40/67)   remote: Compressing 
objects:  61% (41/67)   remote: Compressing objects:  62% (42/67)   
remote: Compressing objects:  64% (43/67)   remote: Compressing 
objects:  65% (44/67)   remote: Compressing objects:  67% (45/67)   
remote: Compressing objects:  68% (46/67)   remote: Compressing 
objects:  70% (47/67)   remote: Compressing objects:  71% (48/67)   
remote: Compressing objects:  73% (49/67)   remote: Compressing 
objects:  74% (50/67)   remote: Compressing objects:  76% (51/67)   
remote: Compressing objects:  77% (52/67)   remote: Compressing 
objects:  79% (53/67)   remote: Compressing objects:  80% (54/67)   

Build failed in Jenkins: kafka-trunk-jdk10 #349

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 6964, done.
remote: Compressing objects:   1% (1/70)   remote: Compressing objects: 
  2% (2/70)   remote: Compressing objects:   4% (3/70)   
remote: Compressing objects:   5% (4/70)   remote: Compressing objects: 
  7% (5/70)   remote: Compressing objects:   8% (6/70)   
remote: Compressing objects:  10% (7/70)   remote: Compressing objects: 
 11% (8/70)   remote: Compressing objects:  12% (9/70)   
remote: Compressing objects:  14% (10/70)   remote: Compressing 
objects:  15% (11/70)   remote: Compressing objects:  17% (12/70)   
remote: Compressing objects:  18% (13/70)   remote: Compressing 
objects:  20% (14/70)   remote: Compressing objects:  21% (15/70)   
remote: Compressing objects:  22% (16/70)   remote: Compressing 
objects:  24% (17/70)   remote: Compressing objects:  25% (18/70)   
remote: Compressing objects:  27% (19/70)   remote: Compressing 
objects:  28% (20/70)   remote: Compressing objects:  30% (21/70)   
remote: Compressing objects:  31% (22/70)   remote: Compressing 
objects:  32% (23/70)   remote: Compressing objects:  34% (24/70)   
remote: Compressing objects:  35% (25/70)   remote: Compressing 
objects:  37% (26/70)   remote: Compressing objects:  38% (27/70)   
remote: Compressing objects:  40% (28/70)   remote: Compressing 
objects:  41% (29/70)   remote: Compressing objects:  42% (30/70)   
remote: Compressing objects:  44% (31/70)   remote: Compressing 
objects:  45% (32/70)   remote: Compressing objects:  47% (33/70)   
remote: Compressing objects:  48% (34/70)   remote: Compressing 
objects:  50% (35/70)   remote: Compressing objects:  51% (36/70)   
remote: Compressing objects:  52% (37/70)   remote: Compressing 
objects:  54% (38/70)   remote: Compressing objects:  55% (39/70)   
remote: Compressing objects:  57% (40/70)   remote: Compressing 
objects:  58% (41/70)   remote: Compressing objects:  60% (42/70)   
remote: Compressing objects:  61% (43/70)   remote: Compressing 
objects:  62% (44/70)   remote: Compressing objects:  64% (45/70)   
remote: Compressing objects:  65% (46/70)   remote: Compressing 
objects:  67% (47/70)   remote: Compressing objects:  68% (48/70)   
remote: Compressing objects:  70% (49/70)   remote: Compressing 
objects:  71% (50/70)   remote: Compressing objects:  72% (51/70)   
remote: Compressing objects:  74% (52/70)   remote: Compressing 
objects:  75% (53/70)   remote: Compressing objects:  77% (54/70)   
remote: Compressing objects:  78% (55/70)   

Jenkins build is back to normal : kafka-0.11.0-jdk7 #394

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-0.10.2-jdk7 #229

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
error: Could not read 3af2cecb9d84369021549946db95dbef9bfe68fa
remote: Counting objects: 3946, done.
remote: Compressing objects:   1% (1/66)   remote: Compressing objects: 
  3% (2/66)   remote: Compressing objects:   4% (3/66)   
remote: Compressing objects:   6% (4/66)   remote: Compressing objects: 
  7% (5/66)   remote: Compressing objects:   9% (6/66)   
remote: Compressing objects:  10% (7/66)   remote: Compressing objects: 
 12% (8/66)   remote: Compressing objects:  13% (9/66)   
remote: Compressing objects:  15% (10/66)   remote: Compressing 
objects:  16% (11/66)   remote: Compressing objects:  18% (12/66)   
remote: Compressing objects:  19% (13/66)   remote: Compressing 
objects:  21% (14/66)   remote: Compressing objects:  22% (15/66)   
remote: Compressing objects:  24% (16/66)   remote: Compressing 
objects:  25% (17/66)   remote: Compressing objects:  27% (18/66)   
remote: Compressing objects:  28% (19/66)   remote: Compressing 
objects:  30% (20/66)   remote: Compressing objects:  31% (21/66)   
remote: Compressing objects:  33% (22/66)   remote: Compressing 
objects:  34% (23/66)   remote: Compressing objects:  36% (24/66)   
remote: Compressing objects:  37% (25/66)   remote: Compressing 
objects:  39% (26/66)   remote: Compressing objects:  40% (27/66)   
remote: Compressing objects:  42% (28/66)   remote: Compressing 
objects:  43% (29/66)   remote: Compressing objects:  45% (30/66)   
remote: Compressing objects:  46% (31/66)   remote: Compressing 
objects:  48% (32/66)   remote: Compressing objects:  50% (33/66)   
remote: Compressing objects:  51% (34/66)   remote: Compressing 
objects:  53% (35/66)   remote: Compressing objects:  54% (36/66)   
remote: Compressing objects:  56% (37/66)   remote: Compressing 
objects:  57% (38/66)   remote: Compressing objects:  59% (39/66)   
remote: Compressing objects:  60% (40/66)   remote: Compressing 
objects:  62% (41/66)   remote: Compressing objects:  63% (42/66)   
remote: Compressing objects:  65% (43/66)   remote: Compressing 
objects:  66% (44/66)   remote: Compressing objects:  68% (45/66)   
remote: Compressing objects:  69% (46/66)   remote: Compressing 
objects:  71% (47/66)   remote: Compressing objects:  72% (48/66)   
remote: Compressing objects:  74% (49/66)   remote: Compressing 
objects:  75% (50/66)   remote: Compressing objects:  77% (51/66)   
remote: Compressing objects:  78% (52/66)   remote: Compressing 
objects:  80% (53/66)   remote: Compressing objects:  81% (54/66)   

Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol

[Vahid S Hashemian]

If there is no additional feedback on this KIP I'll start a vote soon.

Thanks.
--Vahid



From:   "Vahid S Hashemian" 
To: dev@kafka.apache.org
Date:   07/18/2018 04:47 PM
Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data 
Protocol



The KIP is updated to follow the suggestion of using consumer group 
generation.

--Vahid



From:   "Vahid S Hashemian" 
To: dev@kafka.apache.org
Date:   07/17/2018 02:32 PM
Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data 
Protocol



Hi Jason,

Thanks for the feedback. That sounds good to me.
I'll update the KIP accordingly and move the local generation option to 
"Rejected Alternatives".
I'll send a note when the revised KIP is ready.

--Vahid



From:   Jason Gustafson 
To: dev 
Date:   07/17/2018 02:22 PM
Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data 
Protocol



Hey Vahid,

I'm with Mike that it seems simpler to just use the consumer group
generation. Even if you can figure out how to reason about the local
generation, it still seems confusing to have something called "generation"
which is not the consumer group generation. There doesn't seem to be much
downside to using the actual generation as far as I can tell and it is 
much
easier to reason about. I was thinking we could expose it in
PartitionAssignor.Assignment as an additional field. I think this wouldn't
impact existing assignors, though it would allow for other stateful
assignors to use the field in a similar way.

It's a pity we didn't add a version to this schema, by the way. There's 
not
much we can do if we can only add fields at the end of the schema, but I
guess we're hoping this is the last change ;-)

Thanks,
Jason

On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Mike,
>
>
>
> Thanks a lot for reviewing the KIP and sharing your feedback.
>
> I agree that such an issue could surface with option 1, but the existing
>
> PR (that currently implements this option) checks for such duplicate
>
> assignments and ignores one in favor of the other. So at the end there
>
> will be valid (non-duplicate) assignments to consumers, but they might
>
> deviate a bit from the ideal assignment.
>
> If rare scenarios like this are deemed troublesome and we want to avoid
>
> them, option 2 would probably be the way to go. In that case and in my
>
> opinion, option 2 would a better solution compared to introducing 
another
>
> field (e.g. timestamp).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   Mike Freyberger 
>
> To: "dev@kafka.apache.org" 
>
> Date:   07/13/2018 08:42 PM
>
> Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User 
Data
>
> Protocol
>
>
>
>
>
>
>
> This is great!
>
>
>
> For the client side implementation, I think it’s still possible for 
there
>
> to be a duplication. I’ll try to walk through the example here.
>
>
>
> Let’s says there are 2 consumers, 1 topic with 2 partitions.
>
>
>
> After the initial rebalance, generation 0:
>
> Consumer A has partition 0
>
> Consumer B has partition 1
>
>
>
> Let’s say consumer B leaves the group (long debug, GC pause). This leads
>
> to another rebalance. This rebalance will be considered generation 1 and
>
> will result in:
>
>
>
> Generation 1, Consumer A owns partition 0,1
>
>
>
> Now let’s say Consumer B is still out of the group and then Consumer A
>
> leaves as well. While Consumer A is out of the group, Consumer B rejoins
>
> the group. During this rebalance, the only previous state would be the
>
> initial generation 0 assignment. So this assignment would be considered
>
> generation 1 as well and would result in:
>
>
>
> Generation 1, Consumer B owns partition 0,1
>
>
>
> When A rejoins the group, both consumers would claim ownership of both
>
> partitions and they would report the assignment was from generation 1.
>
> This gets us back into the same issue as before because the generation
>
> number cannot help at all. You could add a timestamp in addition to the
>
> generation marker, but that’d still be vulnerable to clock skew.
>
>
>
> Would hooking into the existing generation marker protect the assignor 
for
>
> this kind of situation? We need to make sure the selected implantation 
is
>
> protected against the kind of failure mentioned above.
>
>
>
> Also, I have been working on KIP-315, which is another Sticky Assignor,
>
> which also requires some kind of epoch/generation marker to be protected
>
> against zombies. So, I’d be in favor of a generic solution here that 
other
>
> assignors can leverage.
>
>
>
> Best,
>
>
>
> Mike Freyberger
>
>
>
> > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian
>
>  wrote:
>
> >
>
> > Hi all,
>
> >
>
> > I create a short KIP to address an issue in Sticky Assignor assignment
>
> > logic:
>
> >
>
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-



> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
>
>
>
> > Please take a look and share your 

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Vahid S Hashemian]

Thanks Jason for responding to Colin's concerns.

If there are no other comment / feedback / objection I'll start a vote 
soon.

Thanks.
--Vahid



From:   Jason Gustafson 
To: dev 
Date:   07/27/2018 10:38 AM
Subject:Re: [DISCUSS] KIP-289: Improve the default group id 
behavior in KafkaConsumer



Hey Colin,

The problem is both that the empty group id is the default value and that
it is actually accepted by the broker for offset commits. Combine that 
with
the fact that auto commit is enabled by default and you users get
surprising behavior. If you look at a random Kafka cluster, you'll 
probably
find a bunch of inadvertent offset commits for the empty group id. I was
hoping we could distinguish between users who are using the empty group id
as an accident of the default configuration and those who use it
intentionally. By default, there will be no group id and the consumer will
not commit offsets. If a user has actually intentionally used the empty
group id, however, it will continue to work. I actually think there are
probably very few people doing this (maybe even no one), but I thought we
might err on the side of compatibility.

The big incompatible change here is having brokers reject using 
assign(...)
> with empty / null group.id.


This is not correct. In the proposal, the broker will only reject the 
empty
group id for the new version of OffsetCommit. Older clients, which cannot
be changed, will continue to work because the old versions of the
OffsetCommit API still accept the empty group id. The null group id is
different from the empty group id: it is not allowed in any version of the
API. It is basically a way to indicate that the consumer has no dependence
on the coordinator at all, which we actually have a surprising number of
use cases for. Furthermore, if a user has an actual need for the empty
group id, it will still be allowed. We are just deprecating it.

-Jason

On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  wrote:

> Sorry if this is a silly question, but what's the rationale for 
switching
> to using null for the default group id, rather than the empty string?
> Continuing to use the empty string seems like less churn.  And after 
all,
> we're not using the empty string group name for anything else.
>
> The big incompatible change here is having brokers reject using
> assign(...) with empty / null group.id.  If I understand correctly, the
> KIP proposes that this change be made on the brokers on the next
> incompatible Kafka release.  But that has nothing to do with client
> versions.  Why not just have a broker config which controls this?  Maybe 
"
> allow.assign.empty.group.id", or something like that.  At first, the
> default will be true, and then eventually we can flip it over to false.
>
> It seems like the main rationale for tying this behavior to the Kafka
> client version is to force people to stop using the empty group id so 
that
> they can upgrade their clients.  But it's also possible that people will
> stop upgrading their Kafka clients instead.  That would be pretty 
negative
> since  they'd miss out on any efficiency and feature improvements in the
> new clients and eventually have to do more protocol downgrading, etc.
>
> best,
> Colin
>
>
> On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > Hi Jason,
> >
> > That makes sense.
> > I have updated the KIP based on the recent feedback.
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev 
> > Date:   07/25/2018 02:23 PM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > I was thinking we'd only use the old API version if we had to. That 
is,
> > only if the user has explicitly configured "" as the group.id.
> Otherwise,
> > we'd just use the new one. Another option is to just drop support in 
the
> > client for the empty group id, but usually we allow a deprecation 
period
> > for changes like this.
> >
> > -Jason
> >
> > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for additional clarification.
> > >
> > > So the next version of the OffsetCommit API will return an
> > > INVALID_GROUP_ID error for empty group ids; but on the client side 
we
> > call
> > > the older version of the client until the next major release.
> > > The table below should summarize this.
> > >
> > > +-+
> > >   | Client (group.id="") |
> > > +-+
> > >   | pre-2.1 |   2.1  |   3.0 |
> > >
> > +-+---+-++--
> +
> > > | | V5 (cur.) | works   | works  | works |
> > > + API
> > +---+-++--+
> > > | | V6| N/A | 

Build failed in Jenkins: kafka-trunk-jdk10 #348

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 0f3affc0f40751dc8fd064b36b6e859728f63e37
error: Could not read 95fbb2e03f4fe79737c71632e0ef2dfdcfb85a69
error: Could not read 08c465028d057ac23cdfe6d57641fe40240359dd
remote: Counting objects: 6941, done.
remote: Compressing objects:   1% (1/64)   remote: Compressing objects: 
  3% (2/64)   remote: Compressing objects:   4% (3/64)   
remote: Compressing objects:   6% (4/64)   remote: Compressing objects: 
  7% (5/64)   remote: Compressing objects:   9% (6/64)   
remote: Compressing objects:  10% (7/64)   remote: Compressing objects: 
 12% (8/64)   remote: Compressing objects:  14% (9/64)   
remote: Compressing objects:  15% (10/64)   remote: Compressing 
objects:  17% (11/64)   remote: Compressing objects:  18% (12/64)   
remote: Compressing objects:  20% (13/64)   remote: Compressing 
objects:  21% (14/64)   remote: Compressing objects:  23% (15/64)   
remote: Compressing objects:  25% (16/64)   remote: Compressing 
objects:  26% (17/64)   remote: Compressing objects:  28% (18/64)   
remote: Compressing objects:  29% (19/64)   remote: Compressing 
objects:  31% (20/64)   remote: Compressing objects:  32% (21/64)   
remote: Compressing objects:  34% (22/64)   remote: Compressing 
objects:  35% (23/64)   remote: Compressing objects:  37% (24/64)   
remote: Compressing objects:  39% (25/64)   remote: Compressing 
objects:  40% (26/64)   remote: Compressing objects:  42% (27/64)   
remote: Compressing objects:  43% (28/64)   remote: Compressing 
objects:  45% (29/64)   remote: Compressing objects:  46% (30/64)   
remote: Compressing objects:  48% (31/64)   remote: Compressing 
objects:  50% (32/64)   remote: Compressing objects:  51% (33/64)   
remote: Compressing objects:  53% (34/64)   remote: Compressing 
objects:  54% (35/64)   remote: Compressing objects:  56% (36/64)   
remote: Compressing objects:  57% (37/64)   remote: Compressing 
objects:  59% (38/64)   remote: Compressing objects:  60% (39/64)   
remote: Compressing objects:  62% (40/64)   remote: Compressing 
objects:  64% (41/64)   remote: Compressing objects:  65% (42/64)   
remote: Compressing objects:  67% (43/64)   remote: Compressing 
objects:  68% (44/64)   remote: Compressing objects:  70% (45/64)   
remote: Compressing objects:  71% (46/64)   remote: Compressing 
objects:  73% (47/64)   remote: Compressing objects:  75% (48/64)   
remote: Compressing objects:  76% (49/64)   remote: Compressing 
objects:  78% (50/64)   remote: Compressing objects:  79% (51/64)   
remote: Compressing objects:  81% (52/64)   remote: Compressing 
objects:  82% (53/64)   remote: Compressing objects:  84% (54/64)   
remote: Compressing objects:  85% (55/64)   

Jenkins build is back to normal : kafka-1.0-jdk7 #222

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-0.10.2-jdk7 #228

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
error: Could not read 3af2cecb9d84369021549946db95dbef9bfe68fa
remote: Counting objects: 3856, done.
remote: Compressing objects:   3% (1/31)   remote: Compressing objects: 
  6% (2/31)   remote: Compressing objects:   9% (3/31)   
remote: Compressing objects:  12% (4/31)   remote: Compressing objects: 
 16% (5/31)   remote: Compressing objects:  19% (6/31)   
remote: Compressing objects:  22% (7/31)   remote: Compressing objects: 
 25% (8/31)   remote: Compressing objects:  29% (9/31)   
remote: Compressing objects:  32% (10/31)   remote: Compressing 
objects:  35% (11/31)   remote: Compressing objects:  38% (12/31)   
remote: Compressing objects:  41% (13/31)   remote: Compressing 
objects:  45% (14/31)   remote: Compressing objects:  48% (15/31)   
remote: Compressing objects:  51% (16/31)   remote: Compressing 
objects:  54% (17/31)   remote: Compressing objects:  58% (18/31)   
remote: Compressing objects:  61% (19/31)   remote: Compressing 
objects:  64% (20/31)   remote: Compressing objects:  67% (21/31)   
remote: Compressing objects:  70% (22/31)   remote: Compressing 
objects:  74% (23/31)   remote: Compressing objects:  77% (24/31)   
remote: Compressing objects:  80% (25/31)   remote: Compressing 
objects:  83% (26/31)   remote: Compressing objects:  87% (27/31)   
remote: Compressing objects:  90% (28/31)   remote: Compressing 
objects:  93% (29/31)   remote: Compressing objects:  96% (30/31)   
remote: Compressing objects: 100% (31/31)   remote: Compressing 
objects: 100% (31/31), done.
Receiving objects:   0% (1/3856)   Receiving objects:   1% (39/3856)   
Receiving objects:   2% (78/3856)   Receiving objects:   3% (116/3856)   
Receiving objects:   4% (155/3856)   Receiving objects:   5% (193/3856)   
Receiving objects:   6% (232/3856)   Receiving objects:   7% (270/3856)   
Receiving objects:   8% (309/3856)   Receiving objects:   9% (348/3856)   
Receiving objects:  10% (386/3856)   Receiving objects:  11% (425/3856)   
Receiving objects:  12% (463/3856)   Receiving objects:  13% (502/3856)   
Receiving objects:  14% (540/3856)   Receiving objects:  15% (579/3856)   
Receiving objects:  16% (617/3856)   Receiving objects:  17% (656/3856)   
Receiving objects:  18% (695/3856)   Receiving objects:  19% (733/3856)   
Receiving objects:  20% (772/3856)   Receiving objects:  21% (810/3856)   
Receiving objects:  22% (849/3856)   Receiving objects:  23% (887/3856)   
Receiving objects:  24% (926/3856)   Receiving objects:  25% (964/3856)   
Receiving objects:  26% (1003/3856)   Receiving objects:  27% (1042/3856)   
Receiving objects:  28% (1080/3856)   Receiving objects:  29% (1119/3856)   
Receiving objects:  30% 

Build failed in Jenkins: kafka-trunk-jdk10 #347

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Remove 1 minute minimum segment interval (#5323)

[wangguoz] KAFKA-7161: check invariant: oldValue is in the state (#5366)

[github] MINOR: Producers should set delivery timeout instead of retries  
(#5425)

--
[...truncated 1.98 MB...]

org.apache.kafka.streams.TopologyTest > 
sessionWindowNamedMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideNonPrefixedCustomConfigsWithPrefixedConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideNonPrefixedCustomConfigsWithPrefixedConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfNotAtLestOnceOrExactlyOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfNotAtLestOnceOrExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #2851

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Remove 1 minute minimum segment interval (#5323)

[wangguoz] KAFKA-7161: check invariant: oldValue is in the state (#5366)

[github] MINOR: Producers should set delivery timeout instead of retries  
(#5425)

--
[...truncated 426.03 KB...]
kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAuthorizeWithPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDeleteAllAclOnWildcardResource PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAddAclsOnWildcardResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesExtendedAclChangeEventWhenInterBrokerProtocolAtLeastKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testWritesLiteralAclChangeEventWhenInterBrokerProtocolIsKafkaV2 PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnPrefixedResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnPrefixedResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testEmptyAclThrowsException 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testEmptyAclThrowsException PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testSuperUserWithCustomPrincipalHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testAllowAccessWithCustomPrincipal STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testAllowAccessWithCustomPrincipal PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnWildcardResource 
STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDeleteAclOnWildcardResource 
PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testChangeListenerTiming STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testChangeListenerTiming 

Jenkins build is back to normal : kafka-2.0-jdk8 #97

[Apache Jenkins Server]

See 

Re: Discussion: New components in JIRA?

[Ray Chiang]

I haven't seen any comments.  Let me know if/when you add the new 
components.  Thanks.


-Ray


On 7/27/18 9:54 PM, Guozhang Wang wrote:

Hello Ray,

Any PMC member of the project can add more components in the JIRA system.
If there is no objection in the next 72 hours I can just go ahead and add
them.


Guozhang


On Thu, Jul 26, 2018 at 1:50 PM, Ray Chiang  wrote:


Thanks Guozhang.  I'm good with the way the documentation is now.

Is there any other procedure to follow to get "logging" and "mirrormaker"
added as components or can we just request a JIRA admin to do that on this
list?

-Ray


On 7/23/18 4:56 PM, Guozhang Wang wrote:


I've just updated the web docs on http://kafka.apache.org/contributing
accordingly.

On Mon, Jul 23, 2018 at 3:30 PM, khaireddine Rezgui <
khaireddine...@gmail.com> wrote:

Good job Ray for the wiki, it's clear enough.

Le 23 juil. 2018 10:17 PM, "Ray Chiang"  a écrit :

Okay, I've created a wiki page Reporting Issues in Apache Kafka
<
https://cwiki.apache.org/confluence/display/KAFKA/
Reporting+Issues+in+Apache+Kafka>.

I'd appreciate any feedback.  If this is good enough, I can file a JIRA
to change the link under "Bugs" in the "Project information" page.


-Ray


On 7/23/18 11:28 AM, Ray Chiang wrote:


Good point.  I'll look into adding some JIRA guidelines to the
documentation/wiki.

-Ray

On 7/22/18 10:23 AM, Guozhang Wang wrote:


Hello Ray,

Thanks for brining this up. I'm generally +1 on the first two, while
for
the last category, personally I felt leaving them as part of `tools` is
fine, but I'm also open for other opinions.

A more general question though, is that today we do not have any
guidelines
to ask JIRA reporters to set the right component, i.e. it is purely
best-effort, and we cannot disallow reporters to add any new component
names. And so far the project does not really have a tradition to
manage
JIRA reports per-component, as the goal is to not "separate" the
project
into silos but recommending everyone to get hands on every aspect of
the
project.


Guozhang


On Fri, Jul 20, 2018 at 2:44 PM, Ray Chiang 
wrote:

I've been doing a little bit of component cleanup in JIRA.  What do

people
think of adding
one or more of the following components?

- logging: For any consumer/producer/broker logging (i.e. log4j). This
should help disambiguate from the "log" component (i.e. Kafka
messages).

- mirrormaker: There are enough requests specific to MirrorMaker
that it
could be put into its own component.

- scripts: I'm a little more ambivalent about this one, but any of the
bin/*.sh script fixes could belong in their own category.  I'm not
sure if
other people feel strongly for how the "tools" component should be
used
w.r.t. the run scripts.

Any thoughts?

-Ray

[jira] [Created] (KAFKA-7231) NetworkClient.newClientRequest() ignores custom request timeout in favor of the default

[Ron Dagostino (JIRA)]

Ron Dagostino created KAFKA-7231:


 Summary: NetworkClient.newClientRequest() ignores custom request 
timeout in favor of the default
 Key: KAFKA-7231
 URL: https://issues.apache.org/jira/browse/KAFKA-7231
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.0.0
Reporter: Ron Dagostino


The below code in {{org.apache.kafka.clients.KafkaClient}} is not passing in 
the provided {{requestTimeoutMs}} -- it is ignoring it in favor of the 
{{defaultRequestTimeoutMs}} value.
{code:java}
@Override
public ClientRequest newClientRequest(String nodeId,
  AbstractRequest.Builder requestBuilder,
  long createdTimeMs,
  boolean expectResponse,
  int requestTimeoutMs,
  RequestCompletionHandler callback) {
 return new ClientRequest(nodeId, requestBuilder, correlation++, clientId, 
createdTimeMs, expectResponse,
defaultRequestTimeoutMs, callback);
}
{code}

This is an easy fix, but the impact of fixing it is difficult to quantify.  
Clients that set a custom timeout are getting the default timeout of 1000 ms -- 
fixing this will suddenly cause the custom timeout to take effect.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Permission to create KIP

[Matthias J. Sax]

Done. :)

-Matthias

On 8/1/18 11:19 AM, Ratish Ravindran wrote:
> Userid is *rravindran*
> 
> -Ratish
> 
> On Wed, Aug 1, 2018 at 10:22 PM Matthias J. Sax 
> wrote:
> 
>> Can you provide you login ID so we can grant permission?
>>
>> (The screenshot was striped btw)
>>
>> -Matthias
>>
>>
>> On 7/31/18 9:56 PM, Ratish Ravindran wrote:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>
>>> When tried to click on Create KIP
>>> <
>> https://cwiki.apache.org/confluence?templateId=54329345=KAFKA=KAFKA=KIP-NEXT%3A+Insert+Title+Here
>>> ,
>>> got
>>>
>>> Screen Shot 2018-08-01 at 10.25.35 AM.png
>>>
>>> Thanks,
>>> Ratish
>>>
>>> On Wed, Aug 1, 2018 at 6:25 AM Matthias J. Sax >> > wrote:
>>>
>>> What is your Wiki ID?
>>>
>>> -Matthias
>>>
>>> On 7/30/18 8:55 PM, Ratish Ravindran wrote:
>>> > Hi,
>>> >
>>> > I am trying to create a KIP, but I don't access to do so. Can
>> anyone
>>> > provide me with the access ?
>>> >
>>> > Thanks,
>>> > Ratish
>>> >
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[James Cheng]

I’m a little confused about something. Is this KIP focused on log cleaner 
exceptions in general, or focused on log cleaner exceptions due to disk 
failures?

Will max.uncleanable.partitions apply to all exceptions (including log cleaner 
logic errors) or will it apply to only disk I/o exceptions?

I can understand taking the disk offline if there have been “N” I/O exceptions. 
Disk errors are user fixable (by replacing the affected disk). It turns an 
invisible (soft?) failure into a visible hard failure. And the I/O exceptions 
are possibly already causing problems, so it makes sense to limit their impact.

But I’m not sure if it makes sense to take a disk offline after”N” logic errors 
in the log cleaner. If a log cleaner logic error happens, it’s rarely user 
fixable. And it will likely several partitions at once, so you’re likely to 
bump up against the max.uncleanable.partitions limit more quickly. If a disk 
was taken due to logic errors, I’m not sure what the user would do.

-James

Sent from my iPhone

> On Aug 1, 2018, at 9:11 AM, Stanislav Kozlovski  
> wrote:
> 
> Yes, good catch. Thank you, James!
> 
> Best,
> Stanislav
> 
>> On Wed, Aug 1, 2018 at 5:05 PM James Cheng  wrote:
>> 
>> Can you update the KIP to say what the default is for
>> max.uncleanable.partitions?
>> 
>> -James
>> 
>> Sent from my iPhone
>> 
>>> On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski 
>> wrote:
>>> 
>>> Hey group,
>>> 
>>> I am planning on starting a voting thread tomorrow. Please do reply if
>> you
>>> feel there is anything left to discuss.
>>> 
>>> Best,
>>> Stanislav
>>> 
>>> On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <
>> stanis...@confluent.io>
>>> wrote:
>>> 
 Hey, Ray
 
 Thanks for pointing that out, it's fixed now
 
 Best,
 Stanislav
 
> On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
> 
> Thanks.  Can you fix the link in the "KIPs under discussion" table on
> the main KIP landing page
> <
> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#
>>> ?
> 
> I tried, but the Wiki won't let me.
> 
> -Ray
> 
>> On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
>> Hey guys,
>> 
>> @Colin - good point. I added some sentences mentioning recent
> improvements
>> in the introductory section.
>> 
>> *Disk Failure* - I tend to agree with what Colin said - once a disk
> fails,
>> you don't want to work with it again. As such, I've changed my mind
>> and
>> believe that we should mark the LogDir (assume its a disk) as offline
>> on
>> the first `IOException` encountered. This is the LogCleaner's current
>> behavior. We shouldn't change that.
>> 
>> *Respawning Threads* - I believe we should never re-spawn a thread.
>> The
>> correct approach in my mind is to either have it stay dead or never
>> let
> it
>> die in the first place.
>> 
>> *Uncleanable-partition-names metric* - Colin is right, this metric is
>> unneeded. Users can monitor the `uncleanable-partitions-count` metric
> and
>> inspect logs.
>> 
>> 
>> Hey Ray,
>> 
>>> 2) I'm 100% with James in agreement with setting up the LogCleaner to
>>> skip over problematic partitions instead of dying.
>> I think we can do this for every exception that isn't `IOException`.
> This
>> will future-proof us against bugs in the system and potential other
> errors.
>> Protecting yourself against unexpected failures is always a good thing
> in
>> my mind, but I also think that protecting yourself against bugs in the
>> software is sort of clunky. What does everybody think about this?
>> 
>>> 4) The only improvement I can think of is that if such an
>>> error occurs, then have the option (configuration setting?) to
>> create a
>>> .skip file (or something similar).
>> This is a good suggestion. Have others also seen corruption be
>> generally
>> tied to the same segment?
>> 
>> On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 
> wrote:
>> 
>>> For the cleaner thread specifically, I do not think respawning will
> help at
>>> all because we are more than likely to run into the same issue again
> which
>>> would end up crashing the cleaner. Retrying makes sense for transient
>>> errors or when you believe some part of the system could have healed
>>> itself, both of which I think are not true for the log cleaner.
>>> 
>>> On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 
> wrote:
>>> 
 << in
>>> an
 infinite loop which consumes resources and fires off continuous log
 messages.
 Hi Colin.  In case it could be relevant, one way to mitigate this
> effect
>>> is
 to implement a backoff mechanism (if a second respawn is to occur
>> then
>>> wait
 for 1 minute before doing it; then if a 

Re: Permission to create KIP

[Ratish Ravindran]

Userid is *rravindran*

-Ratish

On Wed, Aug 1, 2018 at 10:22 PM Matthias J. Sax 
wrote:

> Can you provide you login ID so we can grant permission?
>
> (The screenshot was striped btw)
>
> -Matthias
>
>
> On 7/31/18 9:56 PM, Ratish Ravindran wrote:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > When tried to click on Create KIP
> > <
> https://cwiki.apache.org/confluence?templateId=54329345=KAFKA=KAFKA=KIP-NEXT%3A+Insert+Title+Here
> >,
> > got
> >
> > Screen Shot 2018-08-01 at 10.25.35 AM.png
> >
> > Thanks,
> > Ratish
> >
> > On Wed, Aug 1, 2018 at 6:25 AM Matthias J. Sax  > > wrote:
> >
> > What is your Wiki ID?
> >
> > -Matthias
> >
> > On 7/30/18 8:55 PM, Ratish Ravindran wrote:
> > > Hi,
> > >
> > > I am trying to create a KIP, but I don't access to do so. Can
> anyone
> > > provide me with the access ?
> > >
> > > Thanks,
> > > Ratish
> > >
> >
>
>

Build failed in Jenkins: kafka-trunk-jdk8 #2850

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H20 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
error: Could not read 3af2cecb9d84369021549946db95dbef9bfe68fa
remote: Counting objects: 2479, done.
remote: Compressing objects:   1% (1/77)   remote: Compressing objects: 
  2% (2/77)   remote: Compressing objects:   3% (3/77)   
remote: Compressing objects:   5% (4/77)   remote: Compressing objects: 
  6% (5/77)   remote: Compressing objects:   7% (6/77)   
remote: Compressing objects:   9% (7/77)   remote: Compressing objects: 
 10% (8/77)   remote: Compressing objects:  11% (9/77)   
remote: Compressing objects:  12% (10/77)   remote: Compressing 
objects:  14% (11/77)   remote: Compressing objects:  15% (12/77)   
remote: Compressing objects:  16% (13/77)   remote: Compressing 
objects:  18% (14/77)   remote: Compressing objects:  19% (15/77)   
remote: Compressing objects:  20% (16/77)   remote: Compressing 
objects:  22% (17/77)   remote: Compressing objects:  23% (18/77)   
remote: Compressing objects:  24% (19/77)   remote: Compressing 
objects:  25% (20/77)   remote: Compressing objects:  27% (21/77)   
remote: Compressing objects:  28% (22/77)   remote: Compressing 
objects:  29% (23/77)   remote: Compressing objects:  31% (24/77)   
remote: Compressing objects:  32% (25/77)   remote: Compressing 
objects:  33% (26/77)   remote: Compressing objects:  35% (27/77)   
remote: Compressing objects:  36% (28/77)   remote: Compressing 
objects:  37% (29/77)   remote: Compressing objects:  38% (30/77)   
remote: Compressing objects:  40% (31/77)   remote: Compressing 
objects:  41% (32/77)   remote: Compressing objects:  42% (33/77)   
remote: Compressing objects:  44% (34/77)   remote: Compressing 
objects:  45% (35/77)   remote: Compressing objects:  46% (36/77)   
remote: Compressing objects:  48% (37/77)   remote: Compressing 
objects:  49% (38/77)   remote: Compressing objects:  50% (39/77)   
remote: Compressing objects:  51% (40/77)   remote: Compressing 
objects:  53% (41/77)   remote: Compressing objects:  54% (42/77)   
remote: Compressing objects:  55% (43/77)   remote: Compressing 
objects:  57% (44/77)   remote: Compressing objects:  58% (45/77)   
remote: Compressing objects:  59% (46/77)   remote: Compressing 
objects:  61% (47/77)   remote: Compressing objects:  62% (48/77)   
remote: Compressing objects:  63% (49/77)   remote: Compressing 
objects:  64% (50/77)   remote: Compressing objects:  66% (51/77)   
remote: Compressing objects:  67% (52/77)   remote: Compressing 
objects:  68% (53/77)   remote: Compressing objects:  70% (54/77)   

Build failed in Jenkins: kafka-trunk-jdk10 #346

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H30 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1794)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/kafka.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: error: Could not read 24d0d3351a43f52f6f688143672d7c57c5f19595
error: Could not read 1e98b2aa5027eb60ee1a112a7f8ec45b9c47
error: Could not read 689bac3c6fcae927ccee89c8d4fd54c2fae83be1
remote: Counting objects: 2505, done.
remote: Compressing objects:   1% (1/77)   remote: Compressing objects: 
  2% (2/77)   remote: Compressing objects:   3% (3/77)   
remote: Compressing objects:   5% (4/77)   remote: Compressing objects: 
  6% (5/77)   remote: Compressing objects:   7% (6/77)   
remote: Compressing objects:   9% (7/77)   remote: Compressing objects: 
 10% (8/77)   remote: Compressing objects:  11% (9/77)   
remote: Compressing objects:  12% (10/77)   remote: Compressing 
objects:  14% (11/77)   remote: Compressing objects:  15% (12/77)   
remote: Compressing objects:  16% (13/77)   remote: Compressing 
objects:  18% (14/77)   remote: Compressing objects:  19% (15/77)   
remote: Compressing objects:  20% (16/77)   remote: Compressing 
objects:  22% (17/77)   remote: Compressing objects:  23% (18/77)   
remote: Compressing objects:  24% (19/77)   remote: Compressing 
objects:  25% (20/77)   remote: Compressing objects:  27% (21/77)   
remote: Compressing objects:  28% (22/77)   remote: Compressing 
objects:  29% (23/77)   remote: Compressing objects:  31% (24/77)   
remote: Compressing objects:  32% (25/77)   remote: Compressing 
objects:  33% (26/77)   remote: Compressing objects:  35% (27/77)   
remote: Compressing objects:  36% (28/77)   remote: Compressing 
objects:  37% (29/77)   remote: Compressing objects:  38% (30/77)   
remote: Compressing objects:  40% (31/77)   remote: Compressing 
objects:  41% (32/77)   remote: Compressing objects:  42% (33/77)   
remote: Compressing objects:  44% (34/77)   remote: Compressing 
objects:  45% (35/77)   remote: Compressing objects:  46% (36/77)   
remote: Compressing objects:  48% (37/77)   remote: Compressing 
objects:  49% (38/77)   remote: Compressing objects:  50% (39/77)   
remote: Compressing objects:  51% (40/77)   remote: Compressing 
objects:  53% (41/77)   remote: Compressing objects:  54% (42/77)   
remote: Compressing objects:  55% (43/77)   remote: Compressing 
objects:  57% (44/77)   remote: Compressing objects:  58% (45/77)   
remote: Compressing objects:  59% (46/77)   remote: Compressing 
objects:  61% (47/77)   remote: Compressing objects:  62% (48/77)   
remote: Compressing objects:  63% (49/77)   remote: Compressing 
objects:  64% (50/77)   remote: Compressing objects:  66% (51/77)   
remote: Compressing objects:  67% (52/77)   remote: Compressing 
objects:  68% (53/77)   remote: Compressing objects:  70% (54/77)   
remote: Compressing objects:  71% (55/77)   

[jira] [Resolved] (KAFKA-7161) KTable Reduce should check for invalid conditions

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-7161.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> KTable Reduce should check for invalid conditions
> -
>
> Key: KAFKA-7161
> URL: https://issues.apache.org/jira/browse/KAFKA-7161
> Project: Kafka
>  Issue Type: Improvement
>Reporter: John Roesler
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.1.0
>
>
> KTableReduce has the opportunity to explicitly check if the state is 
> inconsistent with the oldValues arriving from the stream. If it did so, it 
> could help detect topology changes that needed an app reset and fail fast 
> before any data corruption occurs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Guozhang Wang]

Hello Boyang,

Thanks for the great summary.

Personally I think 1) is still doable as we are not necessarily have to
rely on source of truth to always fence "the right zombie"; in fact, my
point is that as long as we can fence "someone" when there are zombies
sharing the same member id so that users are notified of such issues and
then react to it, it is good enough. Letting first or last comer wins falls
in that consideration.

I've thought about Jason's "registry id" approach, and I think it may
actually be suitable for providing fencing while approach the goal of avoid
unnecessary rebalances. Here are some more concrete details (@Jason, if it
is not the same as what you had originally in mind, let me know):


a. In JoinGroupRequest, include a "registry id" in addition to the "member
id" to the coordinator. In all other requests like CommitOffset, etc, still
only include the "member id".

b. On the coordinator side, when receiving a JoinGroupRequest with a
registry id with empty member id, and this registry id is indeed in the
cache, it still generates a new member id and return it in the response;
additionally, and update the [registry-id, member-id] mapping in its member
metadata cache (i.e. blind-write to the cache regardless whether it already
contains this registry id); if both registry-id and member-id are
specified, check if it matches in the cache, if not or if the cache does
not include this mapping at all, return an ILLEGAL_MEMBER_ID error to let
the client reset its member id and reset again.

** NOTE ** this logic is actually "letting the last comer win", i.e.
whenever a new client comes with a registry id and empty member id, always
treat it as a valid consumer and replace with whatever existed with this
registry id, and the previous member id will be effectively removed from
the group.

c. On the coordinator side, when receiving any other group requests like
CommitOffset, make sure the member id is indeed in this cache (i.e. it is
contained in the [registry-id. member-id] mapping), otherwise reject it as
we do today.


--

So to give an example, if clientA joins group with registry id 1, it will
be assigned a member id X and coordinator remembers current mapping [1, X]
and "X" is the valid member belonging to this group; if another clientB
joins group but with the same registry id 1 with empty member id,
coordinator assigns it with a new member id Y and update the mapping [1,
Y]. Hence the coordinator always blindly trust the new comer as the valid
member and kicks out the previous one. At this time both clientA and
clientB may be fetching from the same assigned partitions, BUT when clientA
tries to commit the offset, by sending with member id X, it will be
rejected.

--

And then back to the original goal of 5/6, when a consumer shuts down and
restarts, it will send a JoinGroupRequest with the registry id with empty
member id, as long as coordinator still remembers this registry id, it will
sends back the original assigned partitions to the consumer, while updating
its cache with the newly generated member id. No rebalance will be
triggered, and if there are other zombies with the same registry ids and
already get the assignment before, although they can still fetch data, they
will not be able to commit.

--



Guozhang



On Wed, Aug 1, 2018 at 8:07 AM, Boyang Chen  wrote:

> Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm
> missing) input here! Looks like we have got more interesting thoughts on
> this KIP.
>
>
> When I draft KIP-345, the original proposal is targeted at solving "static
> membership", which is trying to address 1/2/5/6  proposed by Guozhang. In
> my mindset, the most important detail I'm trying to follow is the owner of
> member id generation or the membership metadata (proposed by John).
>
>
> Guozhang's static membership approach overall LGTM. It would make the
> whole protocol much clear with group member come and go explicitly in a
> static setting. The static approach appoints Consumer/Stream app as the
> owner of member id generation, which brought up multiple validation
> concerns. As the host:port approach is vetoed, and I have thought for a
> while for other validation strategies but failed, I think it's time to
> decide whether we want to focus our next step discussion on
>
>
>   1.  how to let server validate id uniqueness generated by client, or
>   2.  how to let client materialize the id provided by the server
>
>
> Let me know your favor of the two topics here. Personally I'm inclined to
> 2 as 1 is intrinsically complex (no source of truth model). Hope we are on
> the same page now.
>
>
> Best,
>
> Boyang
>
> 
> From: John Roesler 
> Sent: Wednesday, August 1, 2018 5:28 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Overall, this seems like a good 

[jira] [Created] (KAFKA-7230) Empty Record created when producer failed due to RecordTooLargeException

[Julien Fabre (JIRA)]

Julien Fabre created KAFKA-7230:
---

 Summary: Empty Record created when producer failed due to 
RecordTooLargeException
 Key: KAFKA-7230
 URL: https://issues.apache.org/jira/browse/KAFKA-7230
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Julien Fabre


When a producer try to produce a RecordBatch which is bigger than the 
message.max.bytes value, it fails with the error 
{code:java}org.apache.kafka.common.errors.RecordTooLargeException{code}
but an empty Record gets created.

While hitting the RecordTooLargeException is expected, I was not expecting 
seeing a new offset with an empty Record in the Topic.

Is that a problem with Kafka or should the consumer handle this case ?

Test setup :
- Kafka 2.11-1.1.0
- The producer is written in Go, using a 
[SyncProducer|https://godoc.org/github.com/Shopify/sarama#SyncProducer] from 
the Sarama library.
- The consumer is kafkacat version 1.3.1-13-ga6b599

Debugs logs from Kafka :
{code}
[2018-08-01 17:21:11,201] DEBUG Accepted connection from /172.17.0.1:33718 on 
/172.17.0.3:9092 and assigned it to processor 1, sendBufferSize 
[actual|requested]: [102400|102400] recvBufferSize [actual|requested]: 
[102400|102400] (kafka.network.Acceptor)
[2018-08-01 17:21:11,201] DEBUG Processor 1 listening to new connection from 
/172.17.0.1:33718 (kafka.network.Processor)
[2018-08-01 17:21:11,203] DEBUG [ReplicaManager broker=1001] Request key 
events-0 unblocked 0 fetch requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,203] DEBUG [Partition events-0 broker=1001] High watermark 
updated to 2 [0 : 136] (kafka.cluster.Partition)
[2018-08-01 17:21:11,203] DEBUG Sessionless fetch context returning 1 
partition(s) (kafka.server.SessionlessFetchContext)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key 
events-0 unblocked 1 fetch requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key 
events-0 unblocked 0 producer requests. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Request key 
events-0 unblocked 0 DeleteRecordsRequest. (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,204] DEBUG [ReplicaManager broker=1001] Produce to local 
log in 2 ms (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,205] DEBUG Created a new full FetchContext with 1 
partition(s). Will not try to create a new session. (kafka.server.FetchManager)
[2018-08-01 17:21:11,210] DEBUG [ReplicaManager broker=1001] Produce to local 
log in 0 ms (kafka.server.ReplicaManager)
[2018-08-01 17:21:11,210] DEBUG [KafkaApi-1001] Produce request with 
correlation id 1 from client sarama on partition events-0 failed due to 
org.apache.kafka.common.errors.RecordTooLargeException (kafka.server.KafkaApis)
{code}

Debug logs from kafkacat :
{code}
%7|1533144071.204|SEND|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Sent FetchRequest (v4, 70 bytes @ 0, CorrId 89)
%7|1533144071.309|RECV|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Received FetchResponse (v4, 50 bytes, CorrId 89, rtt 
104.62ms)
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Topic events [0] MessageSet size 0, error "Success", 
MaxOffset 2, Ver 2/2
%7|1533144071.309|BACKOFF|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: events [0]: Fetch backoff for 500ms: Broker: No more 
messages
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Topic events [0] in state active at offset 0 (1/10 
msgs, 0/100 kb queued, opv 2) is not fetchable: fetch backed off
%7|1533144071.309|FETCHADD|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Removed events [0] from fetch list (0 entries, opv 2)
%7|1533144071.309|FETCH|rdkafka#consumer-1| [thrd:localhost:9092/1001]: 
localhost:9092/1001: Fetch backoff for 499ms
% Reached end of topic events [0] at offset 2
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Guozhang Wang]

For source node, only one of `Set topicsSet` and `TopicPattern
topicPattern()` will be specified by the user. Similarly for sink node,
only one of `String` and `TopicNameExtractor` will be specified by the
user. Although I've not seen Nishanth's updated PR, I think when it is not
specified today we will return null in that case.

If we want to improve on this situation with Optional, we'd need to do it
on all of these functions. Also note that for `Source#toString()` and
`Sink#toString()` we should only include the specified field in the
resulted representation.


Guozhang

On Wed, Aug 1, 2018 at 5:08 AM, Damian Guy  wrote:

> Ewen - no as I don't believe they are never null. Whereas the
> topicNameExtractor method returns null if it is the default extractor or
> the extractor. So i think this would be better to be optional as it is
> optionally returning a TopicNameExtractor
>
> On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
> wrote:
>
> > Generally +1 (binding)
> >
> > It would be helpful to just provide the full, updated interfaces in the
> > KIP and mark things as new with comments if needed. I had to go back and
> > read the discussion thread to make sure I was understanding the intent
> > correctly.
> >
> > Damian -- if we make that Optional, shouldn't the methods on Source also
> > be Optional types?
> >
> > -Ewen
> >
> > On Mon, Jul 30, 2018 at 11:13 PM Damian Guy 
> wrote:
> >
> >> Hi Nishanth,
> >>
> >> I have one nit on the KIP. I think the topicNameExtractor method should
> >> return Optional rather than null.
> >> Sorry I'm late here.
> >>
> >> Thanks,
> >> Damian
> >>
> >> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep 
> >> wrote:
> >>
> >> > We need one more binding vote.
> >> >
> >> > Binding Votes:
> >> >
> >> >- Matthias J. Sax
> >> >- Guozhang Wong
> >> >
> >> > Community Votes:
> >> >
> >> >- Bill Bejeck
> >> >- Ted Yu
> >> >
> >> > Best,
> >> > Nishanth Pradeep
> >> >
> >> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck 
> wrote:
> >> >
> >> > > Thanks for the KIP!
> >> > >
> >> > > +1
> >> > >
> >> > > -Bill
> >> > >
> >> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang 
> >> > wrote:
> >> > >
> >> > > > +1
> >> > > >
> >> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
> >> > matth...@confluent.io
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > > +1 (binding)
> >> > > > >
> >> > > > > -Matthias
> >> > > > >
> >> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
> >> > > > > > +1
> >> > > > > >
> >> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
> >> > > > nishanth...@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > >> Hello,
> >> > > > > >>
> >> > > > > >> I'm calling a vote for KIP-321:
> >> > > > > >>
> >> > > > > >>
> >> > > > > >>
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 321%3A+Update+
> >> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
> >> > > > > >>
> >> > > > > >> Best,
> >> > > > > >> Nishanth Pradeep
> >> > > > > >>
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > -- Guozhang
> >> > > >
> >> > >
> >> >
> >>
> >
>



-- 
-- Guozhang

[jira] [Resolved] (KAFKA-7229) Failed to dynamically update kafka certificate in kafka 2.0.0

[Yu Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Yang resolved KAFKA-7229.

Resolution: Not A Problem

> Failed to dynamically update kafka certificate in kafka 2.0.0
> -
>
> Key: KAFKA-7229
> URL: https://issues.apache.org/jira/browse/KAFKA-7229
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.0.0
> Environment: Ubuntu 14.04.5 LTS
>Reporter: Yu Yang
>Priority: Critical
>
> In kafka 1.1, we use the following command in a cron job to dynamically 
> update the certificate that kafka uses :
> kafka-configs.sh --bootstrap-server localhost:9093 --command-config 
> /var/pinterest/kafka/client.properties --alter --add-config 
> listener.name.ssl.ssl.keystore.location=/var/certs/kafka/kafka.keystore.jks.1533141082.38
>  --entity-type brokers --entity-name 9 
> In kafka 2.0.0, the command fails with the following exception: 
> [2018-08-01 16:38:01,480] ERROR [AdminClient clientId=adminclient-1] 
> Connection to node -1 failed authentication due to: SSL handshake failed 
> (org.apache.kafka.clients.NetworkClient)
> Error while executing config command with args '--bootstrap-server 
> localhost:9093 --command-config /var/pinterest/kafka/client.properties 
> --alter --add-config 
> listener.name.ssl.ssl.keystore.location=/var/pinterest/kafka/kafka.keystore.jks.1533141082.38
>  --entity-type brokers --entity-name 9'
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake 
> failed
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
>   at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
>   at kafka.admin.ConfigCommand$.brokerConfig(ConfigCommand.scala:346)
>   at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:304)
>   at 
> kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:290)
>   at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:83)
>   at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL 
> handshake failed
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>   at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
>   at 
> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>   at 
> sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>   at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>   at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>   at 
> org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
>   at 
> org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
>   at 
> org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
>   at 
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
>   at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
>   at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
>   at 
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>   at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
>   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
>   at 
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
>   at 
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
>   at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
>   at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
>   at 
> org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
>   at 
> 

Re: Permission to create KIP

[Matthias J. Sax]

Can you provide you login ID so we can grant permission?

(The screenshot was striped btw)

-Matthias


On 7/31/18 9:56 PM, Ratish Ravindran wrote:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> 
> When tried to click on Create KIP
> ,
> got 
> 
> Screen Shot 2018-08-01 at 10.25.35 AM.png
> 
> Thanks,
> Ratish
> 
> On Wed, Aug 1, 2018 at 6:25 AM Matthias J. Sax  > wrote:
> 
> What is your Wiki ID?
> 
> -Matthias
> 
> On 7/30/18 8:55 PM, Ratish Ravindran wrote:
> > Hi,
> >
> > I am trying to create a KIP, but I don't access to do so. Can anyone
> > provide me with the access ?
> >
> > Thanks,
> > Ratish
> >
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-7229) Failed to dynamically update kafka certificate in kafka 2.0.0

[Yu Yang (JIRA)]

Yu Yang created KAFKA-7229:
--

 Summary: Failed to dynamically update kafka certificate in kafka 
2.0.0
 Key: KAFKA-7229
 URL: https://issues.apache.org/jira/browse/KAFKA-7229
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 2.0.0
 Environment: Ubuntu 14.04.5 LTS
Reporter: Yu Yang


In kafka 1.1, we use the following command to dynamically update the 
certificate that kafka uses :

kafka-configs.sh --bootstrap-server localhost:9093 --command-config 
/var/pinterest/kafka/client.properties --alter --add-config 
listener.name.ssl.ssl.keystore.location=/var/certs/kafka/kafka.keystore.jks.1533141082.38
 --entity-type brokers --entity-name 9 

In kafka 2.0.0, the command fails with the following exception: 



[2018-08-01 16:38:01,480] ERROR [AdminClient clientId=adminclient-1] Connection 
to node -1 failed authentication due to: SSL handshake failed 
(org.apache.kafka.clients.NetworkClient)
Error while executing config command with args '--bootstrap-server 
localhost:9093 --command-config /var/pinterest/kafka/client.properties --alter 
--add-config 
listener.name.ssl.ssl.keystore.location=/var/pinterest/kafka/kafka.keystore.jks.1533141082.38
 --entity-type brokers --entity-name 9'
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
at kafka.admin.ConfigCommand$.brokerConfig(ConfigCommand.scala:346)
at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:304)
at 
kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:290)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:83)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL 
handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at 
sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at 
sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at 
org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
at 
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
at 
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at 
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at 
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1116)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at 
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
at 
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at 
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at 
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
at 
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
... 7 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS 
name matching localhost found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:204)
at 

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Stanislav Kozlovski]

Yes, good catch. Thank you, James!

Best,
Stanislav

On Wed, Aug 1, 2018 at 5:05 PM James Cheng  wrote:

> Can you update the KIP to say what the default is for
> max.uncleanable.partitions?
>
> -James
>
> Sent from my iPhone
>
> > On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski 
> wrote:
> >
> > Hey group,
> >
> > I am planning on starting a voting thread tomorrow. Please do reply if
> you
> > feel there is anything left to discuss.
> >
> > Best,
> > Stanislav
> >
> > On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> > wrote:
> >
> >> Hey, Ray
> >>
> >> Thanks for pointing that out, it's fixed now
> >>
> >> Best,
> >> Stanislav
> >>
> >>> On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
> >>>
> >>> Thanks.  Can you fix the link in the "KIPs under discussion" table on
> >>> the main KIP landing page
> >>> <
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#
> >?
> >>>
> >>> I tried, but the Wiki won't let me.
> >>>
> >>> -Ray
> >>>
>  On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
>  Hey guys,
> 
>  @Colin - good point. I added some sentences mentioning recent
> >>> improvements
>  in the introductory section.
> 
>  *Disk Failure* - I tend to agree with what Colin said - once a disk
> >>> fails,
>  you don't want to work with it again. As such, I've changed my mind
> and
>  believe that we should mark the LogDir (assume its a disk) as offline
> on
>  the first `IOException` encountered. This is the LogCleaner's current
>  behavior. We shouldn't change that.
> 
>  *Respawning Threads* - I believe we should never re-spawn a thread.
> The
>  correct approach in my mind is to either have it stay dead or never
> let
> >>> it
>  die in the first place.
> 
>  *Uncleanable-partition-names metric* - Colin is right, this metric is
>  unneeded. Users can monitor the `uncleanable-partitions-count` metric
> >>> and
>  inspect logs.
> 
> 
>  Hey Ray,
> 
> > 2) I'm 100% with James in agreement with setting up the LogCleaner to
> > skip over problematic partitions instead of dying.
>  I think we can do this for every exception that isn't `IOException`.
> >>> This
>  will future-proof us against bugs in the system and potential other
> >>> errors.
>  Protecting yourself against unexpected failures is always a good thing
> >>> in
>  my mind, but I also think that protecting yourself against bugs in the
>  software is sort of clunky. What does everybody think about this?
> 
> > 4) The only improvement I can think of is that if such an
> > error occurs, then have the option (configuration setting?) to
> create a
> > .skip file (or something similar).
>  This is a good suggestion. Have others also seen corruption be
> generally
>  tied to the same segment?
> 
>  On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 
> >>> wrote:
> 
> > For the cleaner thread specifically, I do not think respawning will
> >>> help at
> > all because we are more than likely to run into the same issue again
> >>> which
> > would end up crashing the cleaner. Retrying makes sense for transient
> > errors or when you believe some part of the system could have healed
> > itself, both of which I think are not true for the log cleaner.
> >
> > On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 
> >>> wrote:
> >
> >> << >>> in
> > an
> >> infinite loop which consumes resources and fires off continuous log
> >> messages.
> >> Hi Colin.  In case it could be relevant, one way to mitigate this
> >>> effect
> > is
> >> to implement a backoff mechanism (if a second respawn is to occur
> then
> > wait
> >> for 1 minute before doing it; then if a third respawn is to occur
> wait
> > for
> >> 2 minutes before doing it; then 4 minutes, 8 minutes, etc. up to
> some
> >>> max
> >> wait time).
> >>
> >> I have no opinion on whether respawn is appropriate or not in this
> > context,
> >> but a mitigation like the increasing backoff described above may be
> >> relevant in weighing the pros and cons.
> >>
> >> Ron
> >>
> >> On Wed, Jul 25, 2018 at 1:26 PM Colin McCabe 
> >>> wrote:
> >>
>  On Mon, Jul 23, 2018, at 23:20, James Cheng wrote:
>  Hi Stanislav! Thanks for this KIP!
> 
>  I agree that it would be good if the LogCleaner were more tolerant
> >>> of
>  errors. Currently, as you said, once it dies, it stays dead.
> 
>  Things are better now than they used to be. We have the metric
>    kafka.log:type=LogCleanerManager,name=time-since-last-run-ms
>  which we can use to tell us if the threads are dead. And as of
> >>> 1.1.0,
> >> we
>  have KIP-226, which allows you to restart the log cleaner thread,
>  without requiring a broker restart.

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[James Cheng]

Can you update the KIP to say what the default is for 
max.uncleanable.partitions?

-James

Sent from my iPhone

> On Jul 31, 2018, at 9:56 AM, Stanislav Kozlovski  
> wrote:
> 
> Hey group,
> 
> I am planning on starting a voting thread tomorrow. Please do reply if you
> feel there is anything left to discuss.
> 
> Best,
> Stanislav
> 
> On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski 
> wrote:
> 
>> Hey, Ray
>> 
>> Thanks for pointing that out, it's fixed now
>> 
>> Best,
>> Stanislav
>> 
>>> On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
>>> 
>>> Thanks.  Can you fix the link in the "KIPs under discussion" table on
>>> the main KIP landing page
>>> <
>>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#>?
>>> 
>>> I tried, but the Wiki won't let me.
>>> 
>>> -Ray
>>> 
 On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
 Hey guys,
 
 @Colin - good point. I added some sentences mentioning recent
>>> improvements
 in the introductory section.
 
 *Disk Failure* - I tend to agree with what Colin said - once a disk
>>> fails,
 you don't want to work with it again. As such, I've changed my mind and
 believe that we should mark the LogDir (assume its a disk) as offline on
 the first `IOException` encountered. This is the LogCleaner's current
 behavior. We shouldn't change that.
 
 *Respawning Threads* - I believe we should never re-spawn a thread. The
 correct approach in my mind is to either have it stay dead or never let
>>> it
 die in the first place.
 
 *Uncleanable-partition-names metric* - Colin is right, this metric is
 unneeded. Users can monitor the `uncleanable-partitions-count` metric
>>> and
 inspect logs.
 
 
 Hey Ray,
 
> 2) I'm 100% with James in agreement with setting up the LogCleaner to
> skip over problematic partitions instead of dying.
 I think we can do this for every exception that isn't `IOException`.
>>> This
 will future-proof us against bugs in the system and potential other
>>> errors.
 Protecting yourself against unexpected failures is always a good thing
>>> in
 my mind, but I also think that protecting yourself against bugs in the
 software is sort of clunky. What does everybody think about this?
 
> 4) The only improvement I can think of is that if such an
> error occurs, then have the option (configuration setting?) to create a
> .skip file (or something similar).
 This is a good suggestion. Have others also seen corruption be generally
 tied to the same segment?
 
 On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 
>>> wrote:
 
> For the cleaner thread specifically, I do not think respawning will
>>> help at
> all because we are more than likely to run into the same issue again
>>> which
> would end up crashing the cleaner. Retrying makes sense for transient
> errors or when you believe some part of the system could have healed
> itself, both of which I think are not true for the log cleaner.
> 
> On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 
>>> wrote:
> 
>> <<>> in
> an
>> infinite loop which consumes resources and fires off continuous log
>> messages.
>> Hi Colin.  In case it could be relevant, one way to mitigate this
>>> effect
> is
>> to implement a backoff mechanism (if a second respawn is to occur then
> wait
>> for 1 minute before doing it; then if a third respawn is to occur wait
> for
>> 2 minutes before doing it; then 4 minutes, 8 minutes, etc. up to some
>>> max
>> wait time).
>> 
>> I have no opinion on whether respawn is appropriate or not in this
> context,
>> but a mitigation like the increasing backoff described above may be
>> relevant in weighing the pros and cons.
>> 
>> Ron
>> 
>> On Wed, Jul 25, 2018 at 1:26 PM Colin McCabe 
>>> wrote:
>> 
 On Mon, Jul 23, 2018, at 23:20, James Cheng wrote:
 Hi Stanislav! Thanks for this KIP!
 
 I agree that it would be good if the LogCleaner were more tolerant
>>> of
 errors. Currently, as you said, once it dies, it stays dead.
 
 Things are better now than they used to be. We have the metric
   kafka.log:type=LogCleanerManager,name=time-since-last-run-ms
 which we can use to tell us if the threads are dead. And as of
>>> 1.1.0,
>> we
 have KIP-226, which allows you to restart the log cleaner thread,
 without requiring a broker restart.
 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
 <
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration
>>> 
 I've only read about this, I haven't personally tried it.
>>> Thanks for pointing this out, James!  Stanislav, we should probably
> add a
>>> 

[jira] [Created] (KAFKA-7228) DeadLetterQueue throws a NullPointerException

[Arjun Satish (JIRA)]

Arjun Satish created KAFKA-7228:
---

 Summary: DeadLetterQueue throws a NullPointerException
 Key: KAFKA-7228
 URL: https://issues.apache.org/jira/browse/KAFKA-7228
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Arjun Satish


Using the dead letter queue results in a NPE: 

{code:java}
[2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task 
threw an uncaught and unrecoverable exception 
(org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at 
org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
at 
org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
at 
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is 
being killed and will not recover until manually restarted 
(org.apache.kafka.connect.runtime.WorkerTask)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

[Manikumar]

Hi all,

If there are no concerns, I will start the voting process soon.

Thanks

On Tue, Jul 31, 2018 at 9:08 AM Manikumar  wrote:

> Hi Colin,
>
> Yes,  "--authorizer-properties" option is not required with
> "--bootstrap-server" option. Updated the KIP.
>
>
> Thanks,
>
> On Tue, Jul 31, 2018 at 1:30 AM Ted Yu  wrote:
>
>> Look good to me.
>>
>> On Mon, Jul 23, 2018 at 7:30 AM Manikumar 
>> wrote:
>>
>> > Hi all,
>> >
>> > I have created a KIP to use AdminClient API in AclCommand
>> (kafka-acls.sh)
>> >
>> > *
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API*
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-332%3A+Update+AclCommand+to+use+AdminClient+API
>> > >
>> >
>> > Please take a look.
>> >
>> > Thanks,
>> >
>>
>

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Stanislav Kozlovski]

Hey group,

I just wanted to note that I have an implementation ready for review. Feel
free to take a quick look and raise any concerns you might have in due
time. I plan on starting the voting thread tomorrow.

Best,
Stanislav

On Wed, Aug 1, 2018 at 10:01 AM Stanislav Kozlovski 
wrote:

> Hey Ray,
>
> * Yes, we'd have the logDir as a tag in the metric
> * The intention is to have Int.MaxValue as the maximum uncleanable
> partitions count
> * My idea is to store the marked logs (actually partitions) in memory
> instead of the ".skip" files to keep the change simple. I have also decided
> to omit any retries from the implementation - once a partition is marked as
> "uncleanable" it stays so until a broker restart
>
> Please do let me know if you are okay with this description. I should have
> the code available for review soon
>
> Thanks,
> Stanislav
>
> On Tue, Jul 31, 2018 at 6:30 PM Ray Chiang  wrote:
>
>> I had one question that I was trying to do some investigation before I
>> asked, but I'm having some issues with my JMX browser right now.
>>
>>   * For the uncleanable-partitions-count metric, is that going to be
>> per-logDir entry?
>>   * For max.uncleanable.partitions, is the intention to have -1 be
>> "infinite" or are we going to use Int.MaxValue as a practical
>> equivalent?
>>   * In this sentence: "When evaluating which logs to compact, skip the
>> marked ones.", should we define what "marking" will be?  If we're
>> going with the ".skip" file or equivalent, can we also add how
>> successful retries will behave?
>>
>> -Ray
>>
>> On 7/31/18 9:56 AM, Stanislav Kozlovski wrote:
>> > Hey group,
>> >
>> > I am planning on starting a voting thread tomorrow. Please do reply if
>> you
>> > feel there is anything left to discuss.
>> >
>> > Best,
>> > Stanislav
>> >
>> > On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <
>> stanis...@confluent.io>
>> > wrote:
>> >
>> >> Hey, Ray
>> >>
>> >> Thanks for pointing that out, it's fixed now
>> >>
>> >> Best,
>> >> Stanislav
>> >>
>> >> On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
>> >>
>> >>> Thanks.  Can you fix the link in the "KIPs under discussion" table on
>> >>> the main KIP landing page
>> >>> <
>> >>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#
>> >?
>> >>>
>> >>> I tried, but the Wiki won't let me.
>> >>>
>> >>> -Ray
>> >>>
>> >>> On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
>>  Hey guys,
>> 
>>  @Colin - good point. I added some sentences mentioning recent
>> >>> improvements
>>  in the introductory section.
>> 
>>  *Disk Failure* - I tend to agree with what Colin said - once a disk
>> >>> fails,
>>  you don't want to work with it again. As such, I've changed my mind
>> and
>>  believe that we should mark the LogDir (assume its a disk) as
>> offline on
>>  the first `IOException` encountered. This is the LogCleaner's current
>>  behavior. We shouldn't change that.
>> 
>>  *Respawning Threads* - I believe we should never re-spawn a thread.
>> The
>>  correct approach in my mind is to either have it stay dead or never
>> let
>> >>> it
>>  die in the first place.
>> 
>>  *Uncleanable-partition-names metric* - Colin is right, this metric is
>>  unneeded. Users can monitor the `uncleanable-partitions-count` metric
>> >>> and
>>  inspect logs.
>> 
>> 
>>  Hey Ray,
>> 
>> > 2) I'm 100% with James in agreement with setting up the LogCleaner
>> to
>> > skip over problematic partitions instead of dying.
>>  I think we can do this for every exception that isn't `IOException`.
>> >>> This
>>  will future-proof us against bugs in the system and potential other
>> >>> errors.
>>  Protecting yourself against unexpected failures is always a good
>> thing
>> >>> in
>>  my mind, but I also think that protecting yourself against bugs in
>> the
>>  software is sort of clunky. What does everybody think about this?
>> 
>> > 4) The only improvement I can think of is that if such an
>> > error occurs, then have the option (configuration setting?) to
>> create a
>> > .skip file (or something similar).
>>  This is a good suggestion. Have others also seen corruption be
>> generally
>>  tied to the same segment?
>> 
>>  On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 
>> >>> wrote:
>> > For the cleaner thread specifically, I do not think respawning will
>> >>> help at
>> > all because we are more than likely to run into the same issue again
>> >>> which
>> > would end up crashing the cleaner. Retrying makes sense for
>> transient
>> > errors or when you believe some part of the system could have healed
>> > itself, both of which I think are not true for the log cleaner.
>> >
>> > On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 
>> >>> wrote:
>> >> <<> you
>> >>> in
>> > an
>> >> infinite loop which consumes resources and 

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Boyang Chen]

Thanks for everyone's (Guozhang, Jason, Mike, John and someone in case I'm 
missing) input here! Looks like we have got more interesting thoughts on this 
KIP.


When I draft KIP-345, the original proposal is targeted at solving "static 
membership", which is trying to address 1/2/5/6  proposed by Guozhang. In my 
mindset, the most important detail I'm trying to follow is the owner of member 
id generation or the membership metadata (proposed by John).


Guozhang's static membership approach overall LGTM. It would make the whole 
protocol much clear with group member come and go explicitly in a static 
setting. The static approach appoints Consumer/Stream app as the owner of 
member id generation, which brought up multiple validation concerns. As the 
host:port approach is vetoed, and I have thought for a while for other 
validation strategies but failed, I think it's time to decide whether we want 
to focus our next step discussion on


  1.  how to let server validate id uniqueness generated by client, or
  2.  how to let client materialize the id provided by the server


Let me know your favor of the two topics here. Personally I'm inclined to 2 as 
1 is intrinsically complex (no source of truth model). Hope we are on the same 
page now.


Best,

Boyang


From: John Roesler 
Sent: Wednesday, August 1, 2018 5:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by 
specifying member id

Hi Boyang,

Overall, this seems like a good addition to the consumer.

I agree with the others that we should attempt to validate the uniqueness
of member.id usage. FWIW, Jason's idea of using a configured logical id +
assigned unique id seems to be suitably flexible and free of assumptions,
as opposed to picking machine characteristics like hostname.

It seems like a survey of solutions to the same problem in other software
might help to guide this discussion. Kafka Consumer isn't the first
stateful distributed system to encounter the need for stable shard
assignment. I would think that any mature distributed database and any
stateful stream processing framework would have some algorithm to solve
this problem, some of which might offer unique advantages over what we've
discussed so far.

Hopefully it wouldn't pollute the discussion if I threw out an alternative
idea: It seems like the purpose behind this proposal is to allow stateful
consumers to re-acquire their previous partitions when they return to the
group. In this scenario, it seems reliable that the consumers would know
what partitions they previously had assigned (since they are stateful).
Instead of reporting a member.id, they could just include their prior list
of partitions in the join-group request. This would solve the problem just
as well as member.id, while allowing more advanced techniques like
splitting state between two instances. For example, an instance I1 with TP1
and TP2 state shuts down, and we copy the state for TP2 to I2 (and delete
it from I1); then we start I1 and I2 up, and I1 only reports that it wants
TP1 while I2 only reports that it wants TP2. Thus, we can scale out or in
without incurring costly state reconstruction.


On a separate note, what is the impact of changing the default for
LEAVE_GROUP_ON_CLOSE_CONFIG? It seems like could potentially harm
applications not using member.id. Can we circumvent the issue without
changing that default globally? Such as ignoring that config when member.id
is set?

Thanks for the proposal!
-John

On Mon, Jul 30, 2018 at 9:57 PM Guozhang Wang  wrote:

> @Jason,
>
> Good point about disconnects. And with that I think I agree that a registry
> id maybe a better idea to enable fencing than validating on host / port.
>
>
> Guozhang
>
>
> On Mon, Jul 30, 2018 at 5:40 PM, Jason Gustafson 
> wrote:
>
> > Hey Guozhang,
> >
> > Thanks for the detailed response. Really quick about the fencing issue, I
> > think host/port will not be sufficient because it cannot handle
> > disconnects. For example, if the coordinator moves to another broker,
> then
> > there is no way we'd be able to guarantee the same host/port. Currently
> we
> > try to avoid rebalancing when the coordinator moves. That said, I agree
> in
> > principle with the "first comer wins" approach you've suggested.
> Basically
> > a member is only removed from the group if its session expires or it
> leaves
> > the group explicitly.
> >
> > -Jason
> >
> > On Mon, Jul 30, 2018 at 4:24 PM, Mike Freyberger <
> mfreyber...@appnexus.com
> > >
> > wrote:
> >
> > > Guozhang,
> > >
> > > Thanks for giving us a great starting point.
> > >
> > > A few questions that come to mind right away:
> > >
> > > 1) What do you think a reasonable group-change-timeout would be? I am
> > > thinking on the order of minutes (5 minutes?)
> > >
> > > 2) Will the nodes that are still alive continue to make progress
> during a
> > > static membership rebalance? I believe during a rebalance today all
> > 

[jira] [Created] (KAFKA-7227) Kafka cluster should has a default rack id if not set

[Genmao Yu (JIRA)]

Genmao Yu created KAFKA-7227:


 Summary: Kafka cluster should has a default rack id if not set
 Key: KAFKA-7227
 URL: https://issues.apache.org/jira/browse/KAFKA-7227
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 2.0.0, 1.0.0
Reporter: Genmao Yu


In current behavior, if partial of broker machines have "server.rack",  we will 
failed to create topic with exception like "Not all brokers have rack 
information". There is no big problem in local cluster. But, this behavior may 
not be very suitable for cloud environment. In cloud environment, there may be 
different types of machine, some has rack info, while others do not has rack 
info. It could happen for many reasons, like understock of specific machine. We 
can give each broker a default rack id. 
 * if not set any rack info, it is the same with previous behavior
 * if partial of brokers are set with rack info, we can think of the remaining 
brokers as same rack.

As far as know, if not set rack id, DataNode/NodeManager has a default rack id 
in Hadoop. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7226) kafka-console-consumer.sh doesn't use security.protocol provided in config file

[Alexandre GRIFFAUT (JIRA)]

Alexandre GRIFFAUT created KAFKA-7226:
-

 Summary: kafka-console-consumer.sh doesn't use security.protocol 
provided in config file
 Key: KAFKA-7226
 URL: https://issues.apache.org/jira/browse/KAFKA-7226
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 1.0.0
Reporter: Alexandre GRIFFAUT


It is confusing that kafka-console-consumer.sh doesn't read security.protocol 
when provided in config file with --producer.config, wheras 
kafka-console-consumer.sh does...

 

With debug activated:

$ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list 
$(hostname):6668 --topic test --producer.config 
/etc/kafka/ssl/kafka.client.properties

[2018-08-01 14:17:18,505] INFO ProducerConfig values:
...

    security.protocol = PLAINTEXT
...

> abc

..

java.io.EOFException
    at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:147)
    at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
    at 
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
    at 
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
    at 
org.apache.kafka.common.network.Selector.attemptRead(Selector.java:538)
    at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:482)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
    at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
    at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
    at java.lang.Thread.run(Thread.java:745)

...

 

The only way produce with SSL, is to use --security-protocol SSL

kafka-console-consumer.sh correctly read security.protocol from config file



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

[Damian Guy]

Ewen - no as I don't believe they are never null. Whereas the
topicNameExtractor method returns null if it is the default extractor or
the extractor. So i think this would be better to be optional as it is
optionally returning a TopicNameExtractor

On Tue, 31 Jul 2018 at 23:01 Ewen Cheslack-Postava 
wrote:

> Generally +1 (binding)
>
> It would be helpful to just provide the full, updated interfaces in the
> KIP and mark things as new with comments if needed. I had to go back and
> read the discussion thread to make sure I was understanding the intent
> correctly.
>
> Damian -- if we make that Optional, shouldn't the methods on Source also
> be Optional types?
>
> -Ewen
>
> On Mon, Jul 30, 2018 at 11:13 PM Damian Guy  wrote:
>
>> Hi Nishanth,
>>
>> I have one nit on the KIP. I think the topicNameExtractor method should
>> return Optional rather than null.
>> Sorry I'm late here.
>>
>> Thanks,
>> Damian
>>
>> On Tue, 31 Jul 2018 at 01:14 Nishanth Pradeep 
>> wrote:
>>
>> > We need one more binding vote.
>> >
>> > Binding Votes:
>> >
>> >- Matthias J. Sax
>> >- Guozhang Wong
>> >
>> > Community Votes:
>> >
>> >- Bill Bejeck
>> >- Ted Yu
>> >
>> > Best,
>> > Nishanth Pradeep
>> >
>> > On Fri, Jul 27, 2018 at 10:02 AM Bill Bejeck  wrote:
>> >
>> > > Thanks for the KIP!
>> > >
>> > > +1
>> > >
>> > > -Bill
>> > >
>> > > On Thu, Jul 26, 2018 at 2:39 AM Guozhang Wang 
>> > wrote:
>> > >
>> > > > +1
>> > > >
>> > > > On Wed, Jul 25, 2018 at 11:13 PM, Matthias J. Sax <
>> > matth...@confluent.io
>> > > >
>> > > > wrote:
>> > > >
>> > > > > +1 (binding)
>> > > > >
>> > > > > -Matthias
>> > > > >
>> > > > > On 7/25/18 7:47 PM, Ted Yu wrote:
>> > > > > > +1
>> > > > > >
>> > > > > > On Wed, Jul 25, 2018 at 7:24 PM Nishanth Pradeep <
>> > > > nishanth...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hello,
>> > > > > >>
>> > > > > >> I'm calling a vote for KIP-321:
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-321%3A+Update+
>> > > > > TopologyDescription+to+better+represent+Source+and+Sink+Nodes
>> > > > > >>
>> > > > > >> Best,
>> > > > > >> Nishanth Pradeep
>> > > > > >>
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > -- Guozhang
>> > > >
>> > >
>> >
>>
>

[jira] [Created] (KAFKA-7225) Kafka Connect ConfigProvider not invoked before validation

[Nacho Munoz (JIRA)]

Nacho Munoz created KAFKA-7225:
--

 Summary: Kafka Connect ConfigProvider not invoked before validation
 Key: KAFKA-7225
 URL: https://issues.apache.org/jira/browse/KAFKA-7225
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Nacho Munoz


When trying to register a JDBC connector with externalised secrets (e.g. 
connection.password) the validation fails and the endpoint returns a 500. I 
think that the problem is that the config transformer is not being invoked 
before the validation so trying to exercise the credentials against the 
database fails. I have checked that publishing the connector configuration 
directly to the connect-config topic to skip the validation and restarting the 
server is enough to get the connector working so that confirms that we are just 
missing to call config transformer before validating the connector. Please let 
me know if you need further information.

I'm happy to open a PR to address this issue given that I think that this is 
easy enough to fix for a new contributor to the project. So please feel free to 
assign me the resolution of the bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

[Stanislav Kozlovski]

Hey Ray,

* Yes, we'd have the logDir as a tag in the metric
* The intention is to have Int.MaxValue as the maximum uncleanable
partitions count
* My idea is to store the marked logs (actually partitions) in memory
instead of the ".skip" files to keep the change simple. I have also decided
to omit any retries from the implementation - once a partition is marked as
"uncleanable" it stays so until a broker restart

Please do let me know if you are okay with this description. I should have
the code available for review soon

Thanks,
Stanislav

On Tue, Jul 31, 2018 at 6:30 PM Ray Chiang  wrote:

> I had one question that I was trying to do some investigation before I
> asked, but I'm having some issues with my JMX browser right now.
>
>   * For the uncleanable-partitions-count metric, is that going to be
> per-logDir entry?
>   * For max.uncleanable.partitions, is the intention to have -1 be
> "infinite" or are we going to use Int.MaxValue as a practical
> equivalent?
>   * In this sentence: "When evaluating which logs to compact, skip the
> marked ones.", should we define what "marking" will be?  If we're
> going with the ".skip" file or equivalent, can we also add how
> successful retries will behave?
>
> -Ray
>
> On 7/31/18 9:56 AM, Stanislav Kozlovski wrote:
> > Hey group,
> >
> > I am planning on starting a voting thread tomorrow. Please do reply if
> you
> > feel there is anything left to discuss.
> >
> > Best,
> > Stanislav
> >
> > On Fri, Jul 27, 2018 at 11:05 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> > wrote:
> >
> >> Hey, Ray
> >>
> >> Thanks for pointing that out, it's fixed now
> >>
> >> Best,
> >> Stanislav
> >>
> >> On Fri, Jul 27, 2018 at 9:43 PM Ray Chiang  wrote:
> >>
> >>> Thanks.  Can you fix the link in the "KIPs under discussion" table on
> >>> the main KIP landing page
> >>> <
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#
> >?
> >>>
> >>> I tried, but the Wiki won't let me.
> >>>
> >>> -Ray
> >>>
> >>> On 7/26/18 2:01 PM, Stanislav Kozlovski wrote:
>  Hey guys,
> 
>  @Colin - good point. I added some sentences mentioning recent
> >>> improvements
>  in the introductory section.
> 
>  *Disk Failure* - I tend to agree with what Colin said - once a disk
> >>> fails,
>  you don't want to work with it again. As such, I've changed my mind
> and
>  believe that we should mark the LogDir (assume its a disk) as offline
> on
>  the first `IOException` encountered. This is the LogCleaner's current
>  behavior. We shouldn't change that.
> 
>  *Respawning Threads* - I believe we should never re-spawn a thread.
> The
>  correct approach in my mind is to either have it stay dead or never
> let
> >>> it
>  die in the first place.
> 
>  *Uncleanable-partition-names metric* - Colin is right, this metric is
>  unneeded. Users can monitor the `uncleanable-partitions-count` metric
> >>> and
>  inspect logs.
> 
> 
>  Hey Ray,
> 
> > 2) I'm 100% with James in agreement with setting up the LogCleaner to
> > skip over problematic partitions instead of dying.
>  I think we can do this for every exception that isn't `IOException`.
> >>> This
>  will future-proof us against bugs in the system and potential other
> >>> errors.
>  Protecting yourself against unexpected failures is always a good thing
> >>> in
>  my mind, but I also think that protecting yourself against bugs in the
>  software is sort of clunky. What does everybody think about this?
> 
> > 4) The only improvement I can think of is that if such an
> > error occurs, then have the option (configuration setting?) to
> create a
> > .skip file (or something similar).
>  This is a good suggestion. Have others also seen corruption be
> generally
>  tied to the same segment?
> 
>  On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah 
> >>> wrote:
> > For the cleaner thread specifically, I do not think respawning will
> >>> help at
> > all because we are more than likely to run into the same issue again
> >>> which
> > would end up crashing the cleaner. Retrying makes sense for transient
> > errors or when you believe some part of the system could have healed
> > itself, both of which I think are not true for the log cleaner.
> >
> > On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino 
> >>> wrote:
> >> << >>> in
> > an
> >> infinite loop which consumes resources and fires off continuous log
> >> messages.
> >> Hi Colin.  In case it could be relevant, one way to mitigate this
> >>> effect
> > is
> >> to implement a backoff mechanism (if a second respawn is to occur
> then
> > wait
> >> for 1 minute before doing it; then if a third respawn is to occur
> wait
> > for
> >> 2 minutes before doing it; then 4 minutes, 8 minutes, etc. up to
> some
> >>> max
> >> wait time).
> >>
> 

Re: [EXTERNAL] [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect

[McCaig, Rhys]

Hi All,

I’ve updated the proposal to include the improvements suggested by Stephane.

I have also submitted a PR to implement this functionality into Kafka. 
https://github.com/apache/kafka/pull/5438

I don’t have a benchmark against MirrorMaker yet, as I only currently have a 
local docker stack available to me, though I have seen very good performance in 
that test stack (200k messages/sec@100bytes on limited compute resource 
containers). Further benchmarking might take a few days.

Review and comments would be appreciated.

Cheers,
Rhys


On Jun 18, 2018, at 9:00 AM, McCaig, Rhys 
mailto:rhys_mcc...@cable.comcast.com>> wrote:

Hi Stephane,

Thanks for your feedback and apologies for the delay in my response.

Are there any performance benchmarks against Mirror Maker available? I'm
interested to know if this is more performant / scalable.
Regarding the implementation, here's some feedback:


Currently I don’t have any performance benchmarks, but I think this is a great 
idea, ill see if I can set up something one the next week or so.

- I think it's worth mentioning that this solution does not rely on
consumer groups, and therefore tracking progress may be tricky. Can you
think of a way to expose that?

This is a reasonable concern. I’m not sure how to track this other than looking 
at the Kafka connect offsets. Once a messages is passed to the framework, I'm 
unaware of a way to get at the commit offsets on the producer side. Any 
thoughts?

- Some code can be in config Validator I believe:
https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47

- I think your kip mentions `source.admin.` and `source.consumer.` but I
don't see it reflected yet in the code

- Is there a way to be flexible and merge list and regex, or offer the two
simultaneously ? source_topics=my_static_topic,prefix.* ?

Agree on all of the above - I will incorporate into the code later this week as 
ill get some time back to work on this.

Cheers,
Rhys



On Jun 6, 2018, at 7:16 PM, Stephane Maarek 
mailto:steph...@simplemachines.com.au>> wrote:

Hi Rhys,

I think this will be a great addition.

Are there any performance benchmarks against Mirror Maker available? I'm
interested to know if this is more performant / scalable.
Regarding the implementation, here's some feedback:

- I think it's worth mentioning that this solution does not rely on
consumer groups, and therefore tracking progress may be tricky. Can you
think of a way to expose that?


- Some code can be in config Validator I believe:
https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47

- I think your kip mentions `source.admin.` and `source.consumer.` but I
don't see it reflected yet in the code

- Is there a way to be flexible and merge list and regex, or offer the two
simultaneously ? source_topics=my_static_topic,prefix.* ?

Hope that helps
Stephane

Kind regards,
Stephane

[image: Simple Machines]

Stephane Maarek | Developer

+61 416 575 980
steph...@simplemachines.com.au
simplemachines.com.au
Level 2, 145 William Street, Sydney NSW 2010

On 5 June 2018 at 09:04, McCaig, Rhys  wrote:

Hi All,

As I didn’t get any comment on this KIP and there has since been an
additional 2 KIP’s created numbered 308 since, I'm bumping this and
renaming the KIP to 310 to remove the duplication:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-
310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect

Let me know if you have any comments or feedback, would love to hear them.

Cheers,
Rhys

On May 28, 2018, at 10:23 PM, McCaig, Rhys 
wrote:

Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/
confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+
Connector+to+Kafka+Connect

On May 28, 2018, at 10:19 PM, McCaig, Rhys 
wrote:

Hi All,

I added a KIP to include a Kafka Source Connector with Kafka Connect.
Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect

Looking forward to your feedback and suggestions.

Cheers,
Rhys