[jira] [Created] (KAFKA-6643) Warm up new replicas from scratch when changelog topic has retention time

[Navinder Brar (JIRA)]

Navinder Brar created KAFKA-6643:


 Summary: Warm up new replicas from scratch when changelog topic 
has retention time
 Key: KAFKA-6643
 URL: https://issues.apache.org/jira/browse/KAFKA-6643
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Navinder Brar


In the current scenario, Kafka Streams has changelog Kafka topics(internal 
topics having all the data for the store) which are used to build the state of 
replicas. So, if we keep the number of standby replicas as 1, we still have 
more availability for persistent state stores as changelog Kafka topics are 
also replicated depending upon broker replication policy but that also means we 
are using at least 4 times the space(1 master store, 1 replica store, 1 
changelog, 1 changelog replica). 

Now if we have an year's data in persistent stores(rocksdb), we don't want the 
changelog topics to have an year's data as it will put an unnecessary burden on 
brokers(in terms of space). If we have to scale our kafka streams 
application(having 200-300 TB's of data) we have to scale the kafka brokers as 
well. We want to reduce this dependency and find out ways to just use changelog 
topic as a queue, having just 2 or 3 days of data and warm up the replicas from 
scratch in some other way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6624) log segment deletion could cause a disk to be marked offline incorrectly

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-6624.

   Resolution: Fixed
Fix Version/s: 1.1.0

> log segment deletion could cause a disk to be marked offline incorrectly
> 
>
> Key: KAFKA-6624
> URL: https://issues.apache.org/jira/browse/KAFKA-6624
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.1.0
>Reporter: Jun Rao
>Assignee: Dong Lin
>Priority: Major
> Fix For: 1.1.0
>
>
> Saw the following log.
> [2018-03-06 23:12:20,721] ERROR Error while flushing log for topic1-0 in dir 
> /data01/kafka-logs with offset 80993 (kafka.server.LogDirFailureChannel)
> java.nio.channels.ClosedChannelException
>         at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:379)
>         at 
> org.apache.kafka.common.record.FileRecords.flush(FileRecords.java:163)
>         at 
> kafka.log.LogSegment$$anonfun$flush$1.apply$mcV$sp(LogSegment.scala:375)
>         at kafka.log.LogSegment$$anonfun$flush$1.apply(LogSegment.scala:374)
>         at kafka.log.LogSegment$$anonfun$flush$1.apply(LogSegment.scala:374)
>         at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
>         at kafka.log.LogSegment.flush(LogSegment.scala:374)
>         at 
> kafka.log.Log$$anonfun$flush$1$$anonfun$apply$mcV$sp$4.apply(Log.scala:1374)
>         at 
> kafka.log.Log$$anonfun$flush$1$$anonfun$apply$mcV$sp$4.apply(Log.scala:1373)
>         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
>         at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>         at kafka.log.Log$$anonfun$flush$1.apply$mcV$sp(Log.scala:1373)
>         at kafka.log.Log$$anonfun$flush$1.apply(Log.scala:1368)
>         at kafka.log.Log$$anonfun$flush$1.apply(Log.scala:1368)
>         at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
>         at kafka.log.Log.flush(Log.scala:1368)
>         at 
> kafka.log.Log$$anonfun$roll$2$$anonfun$apply$1.apply$mcV$sp(Log.scala:1343)
>         at 
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
>         at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>         at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> [2018-03-06 23:12:20,722] INFO [ReplicaManager broker=0] Stopping serving 
> replicas in dir /data01/kafka-logs (kafka.server.ReplicaManager)
> It seems that topic1 was being deleted around the time when flushing was 
> called. Then flushing hit an IOException, which caused the disk to be marked 
> offline incorrectly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6642) Rack aware replica assignment in kafka streams

[Ashish Surana (JIRA)]

Ashish Surana created KAFKA-6642:


 Summary: Rack aware replica assignment in kafka streams
 Key: KAFKA-6642
 URL: https://issues.apache.org/jira/browse/KAFKA-6642
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Ashish Surana


We have rack aware replica assignment in kafka broker ([KIP-36 Rack aware 
replica 
assignment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment]).

This request is to have a similar feature for kafka streams applications. 
Standby tasks/standby replica assignment in kafka streams is currently not rack 
aware, and this request is to make it rack aware for better availability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk9 #467

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Fix incorrect JavaDoc (type mismatch) (#4632)

--
[...truncated 1.48 MB...]
kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerConfigUpdateTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
PASSED

kafka.log.ProducerStateManagerTest > testTakeSnapshot STARTED

kafka.log.ProducerStateManagerTest > testTakeSnapshot PASSED

kafka.log.ProducerStateManagerTest > 

Jenkins build is back to normal : kafka-trunk-jdk8 #2468

[Apache Jenkins Server]

See 

How to set offset of a (topic,partition,group) without closing other consumers?

[York Zhang]

I have some running consumers.  I want to set the offset of a specific
topic-partition-group with a new consumer instance in the same group. I
have tried to use subscribe(), but kafka randomly assign partition to the
new consumer, I can't operate the partition I want. Then I tried to use
assign(), but when I tried to commit offset, I got exceptions :
"Error UNKNOWN_MEMBER_ID occurred while committing offsets for group
xxx" "Commit cannot be completed due to group rebalance".

The codes are:
consumer.poll(0);
consumer.seekToEnd(partition);
long endOffset =  consumer.position( partition);
cosumer.commitSync(Collections.singletonMap(partition, new
OffsetAndMetadata(endOffset)));

I'm using kafka 0.9.0. Is there any way I can set offset correctly ?

Build failed in Jenkins: kafka-trunk-jdk7 #3246

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Fix incorrect JavaDoc (type mismatch) (#4632)

--
[...truncated 417.57 KB...]

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testKafkaAdminClientWithAuthenticationFailure PASSED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure STARTED

kafka.api.SaslClientsWithInvalidCredentialsTest > 
testConsumerWithAuthenticationFailure PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithDescribeAcl PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.DelegationTokenEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoProduceWithoutDescribeAcl PASSED

kafka.api.TransactionsTest > testBasicTransactions STARTED

kafka.api.TransactionsTest > testBasicTransactions PASSED

kafka.api.TransactionsTest > testFencingOnSendOffsets STARTED

kafka.api.TransactionsTest > testFencingOnSendOffsets PASSED

kafka.api.TransactionsTest > testFencingOnAddPartitions STARTED

kafka.api.TransactionsTest > testFencingOnAddPartitions PASSED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration STARTED

kafka.api.TransactionsTest > testFencingOnTransactionExpiration PASSED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction STARTED

kafka.api.TransactionsTest > testDelayedFetchIncludesAbortedTransaction PASSED

kafka.api.TransactionsTest > 

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jun Rao]

Hi, Jan,

Thanks for the reply. A few more comments below.

50. Ok, we can think a bit harder for supporting compacted topics.

51. This is a fundamental design question. In the more common case, the
reason why someone wants to increase the number of partitions is that the
consumer application is slow and one wants to run more consumer instances
to increase the degree of parallelism. So, fixing the number of running
consumer instances when expanding the partitions won't help this case. If
we do need to increase the number of consumer instances, we need to somehow
reshuffle the state of the consumer across instances. What we have been
discussing in this KIP is whether we can do this more effectively through
the KStream library (e.g. through a 2-phase partition expansion). This will
add some complexity, but it's probably better than everyone doing this in
the application space. The recopying approach that you mentioned doesn't
seem to address the consumer state management issue when the consumer
switches from an old to a new topic.

52. As for your example, it depends on whether the join key is the same
between (A,B) and (B,C). If the join key is the same, we can do a 2-phase
partition expansion of A, B, and C together. If the join keys are
different, one would need to repartition the data on a different key for
the second join, then the partition expansion can be done independently
between (A,B) and (B,C).

53. If you always fix the number of consumer instances, we you described
works. However, as I mentioned in #51, I am not sure how your proposal
deals with consumer states when the number of consumer instances grows.
Also, it just seems that it's better to avoid re-copying the existing data.

60. "just want to throw in my question from the longer email in the other
Thread here. How will the bloom filter help a new consumer to decide to
apply the key or not?" Not sure that I fully understood your question. The
consumer just reads whatever key is in the log. The bloom filter just helps
clean up the old keys.

61. "Why can we afford having a topic where its apparently not possible to
start a new application on? I think this is an overall flaw of the
discussed idea here. Not playing attention to the overall architecture."
Could you explain a bit more when one can't start a new application?

Jun



On Sat, Mar 10, 2018 at 1:40 AM, Jan Filipiak 
wrote:

> Hi Jun, thanks for your mail.
>
> Thank you for your questions!
> I think they are really good and tackle the core of the problem I see.
>
> I will answer inline, mostly but still want to set the tone here.
>
> The core strength of kafka is what Martin once called the
> kappa-Architecture. How does this work?
> You have everything as a log as in kafka. When you need to change
> something.
> You create the new version of your application and leave it running in
> parallel.
> Once the new version is good you switch your users to use the new
> application.
>
> The online reshuffling effectively breaks this architecture and I think
> the switch in thinking here is more harmful
> than any details about the partitioning function to allow such a change. I
> feel with my suggestion we are the closest to
> the original and battle proven architecture and I can only warn to move
> away from it.
>
> I might have forgotten something, sometimes its hard for me to getting all
> the thoughts captured in a mail, but I hope the comments inline will
> further make my concern clear, and put some emphasis on why I prefer my
> solution ;)
>
> One thing we should all be aware of when discussing this, and I think Dong
> should have mentioned it (maybe he did).
> We are not discussing all of this out of thin air but there is an effort
> in the Samza project.
>
> https://cwiki.apache.org/confluence/display/SAMZA/SEP-5%3A+
> Enable+partition+expansion+of+input+streams
> https://issues.apache.org/jira/browse/SAMZA-1293
>
> To be clear. I think SEP-5 (state of last week, dont know if it adapted to
> this discussion) is on a way better path than KIP-253, and I can't really
> explain why.
>
> Best Jan,
>
> nice weekend everyone
>
>
> On 09.03.2018 03:36, Jun Rao wrote:
>
>> Hi, Jan,
>>
>> Thanks for the feedback. Just some comments on the earlier points that you
>> mentioned.
>>
>> 50. You brought up the question of whether existing data needs to be
>> copied
>> during partition expansion. My understand of your view is that avoid
>> copying existing data will be more efficient, but it doesn't work well
>> with
>> compacted topics since some keys in the original partitions will never be
>> cleaned. It would be useful to understand your use case of compacted
>> topics
>> a bit more. In the common use case, the data volume in a compacted topic
>> may not be large. So, I am not sure if there is a strong need to expand
>> partitions in a compacted topic, at least initially.
>>
> I do agree. State is usually smaller. Update rates might be also
> competitively high.
> 

Jenkins build is back to normal : kafka-trunk-jdk9 #466

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6641) Consider auto repartitioning for Stream.transform() API

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6641:


 Summary: Consider auto repartitioning for Stream.transform() API
 Key: KAFKA-6641
 URL: https://issues.apache.org/jira/browse/KAFKA-6641
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


Today with map / mapValues of Streams DSL, we will set a flag / not set a flag 
for the underlying topology builder; but for transform / transformValues, we do 
not make such marking choices. Maybe the topology builder can still make such 
flagging for transform() to indicate that since the key maybe changed, we 
should issue a repartition for the downstream stateful operators when necessary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk7 #3245

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Fix wrong message in `bin/kafka-run-class.sh` (#4682)

[jason] KAFKA-3368; Add documentation for old message format (#3425)

--
[...truncated 417.61 KB...]

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrLive STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrLive PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionLastIsrShuttingDown STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionLastIsrShuttingDown PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElection STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElection PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaInIsrNotLive STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaInIsrNotLive PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithNoLiveIsr STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithNoLiveIsr PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElection STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElection PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElection STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElection PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElection STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElection PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithEmptyIsr STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithEmptyIsr PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic STARTED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired STARTED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents STARTED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize STARTED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

Jenkins build is back to normal : kafka-1.1-jdk7 #75

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6640) Improve efficiency of KafkaAdminClient.describeTopics()

[Dong Lin (JIRA)]

Dong Lin created KAFKA-6640:
---

 Summary: Improve efficiency of KafkaAdminClient.describeTopics()
 Key: KAFKA-6640
 URL: https://issues.apache.org/jira/browse/KAFKA-6640
 Project: Kafka
  Issue Type: Improvement
Reporter: Dong Lin
Assignee: Dong Lin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3368) Add the Message/Record set protocol to the protocol docs

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-3368.

   Resolution: Fixed
Fix Version/s: 1.2.0

> Add the Message/Record set protocol to the protocol docs
> 
>
> Key: KAFKA-3368
> URL: https://issues.apache.org/jira/browse/KAFKA-3368
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Andras Beni
>Priority: Major
> Fix For: 1.2.0
>
>
> The message/Record format is not a part of the standard Protocol.java class. 
> This should be added to the protocol or manually added to the doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6639) Follower may have sparse index if catching up

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-6639:
--

 Summary: Follower may have sparse index if catching up
 Key: KAFKA-6639
 URL: https://issues.apache.org/jira/browse/KAFKA-6639
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Dhruvil Shah


When a follower is catching up, it may fetch considerably more data than the 
size of the offset index interval. Since we only write to the index once for 
every append, this could lead to a sparsely populated index, which may have a 
performance impact if the follower ever becomes leader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk9 #465

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6560: Add docs for KIP-261 (#4685)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H24 (ubuntu xenial) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
 > git rev-list --no-walk 825bfe5adefe4b5f967068b332cb0feedd7dc4fc # timeout=10
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:592)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
at 

Build failed in Jenkins: kafka-trunk-jdk7 #3244

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6560: Add docs for KIP-261 (#4685)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H29 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
 > git rev-list --no-walk 825bfe5adefe4b5f967068b332cb0feedd7dc4fc # timeout=10
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:592)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:592)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 

Build failed in Jenkins: kafka-trunk-jdk8 #2467

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6560: Add docs for KIP-261 (#4685)

--
Started by an SCM change
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H24 (ubuntu xenial) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
 > git rev-list --no-walk 825bfe5adefe4b5f967068b332cb0feedd7dc4fc # timeout=10
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:592)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1724)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 8e84961661e4d033a68dfff8498436d14b396bb3 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e84961661e4d033a68dfff8498436d14b396bb3
Commit message: "KAFKA-6560: Add docs for KIP-261 (#4685)"
java.util.concurrent.TimeoutException: Timeout waiting for task.
at 
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
at 
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
at 
com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
at 
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
at 
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:135)
at 
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:43)
at 
hudson.model.listeners.SCMListener.onChangeLogParsed(SCMListener.java:120)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:590)
Caused: java.io.IOException: Failed to parse changelog
  

Re: Log Retention Period of Kafka Messages

[James Cheng]

> On Mar 12, 2018, at 10:13 AM, Kyle Tinker  
> wrote:
> 
> You have a couple options:
> 1) You can adjust log.segment.bytes to make the segments smaller so that 
> individual segments can be trimmed
> 2) You can set log.roll.hours, which will roll to a new log segment even if 
> the size hasn't been reached
> Note: #1 and #2 also have per-topic setting controls
> 

About #2, it's possible that segments will only roll when a message has been 
received, after log.roll.hours (or log.roll.ms, etc). That is, if the time has 
passed but no new message has been received, the log roll might not happen.

You should do some testing to double check.

-James


> https://kafka.apache.org/08/documentation/#brokerconfigs
> 
> Thanks,
> --
> 
> 
> KYLE TINKER
> Lead Architect/Team Lead | WorkForce Software 
> 
> T: +1 734 742 2616 | E: ktin...@workforcesoftware.com
> 
> 
> -Original Message-
> From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganand...@fluke.com] 
> Sent: Monday, March 12, 2018 1:19 AM
> To: dev@kafka.apache.org
> Subject: Log Retention Period of Kafka Messages
> 
> Hi Team,
> 
> Form the documentation I could see that log retention period is relative to 
> the partition segments. So unless the segment is closed (1Gb/1 Week) the 
> message will never be deleted.
> 
> Consider I have a very critical use case where I will need to delete the data 
> every day and my segment reaches 1GB after three days. So according the above 
> statement, then there is no way I could configure Kafka to delete the data in 
> one day.
> 
> Is my understanding correct ?
> 
> Regards,
> AK
> 
> 
> Please be advised that this email may contain confidential information. If 
> you are not the intended recipient, please notify us by email by replying to 
> the sender and delete this message. The sender disclaims that the content of 
> this email constitutes an offer to enter into, or the acceptance of, any 
> agreement; provided that the foregoing does not invalidate the binding effect 
> of any digital or other electronic reproduction of a manual signature that is 
> included in any attachment.
> 
> 
> 
> This message is intended exclusively for the individual or entity to which it 
> is addressed. This communication may contain information that is proprietary, 
> privileged, confidential or otherwise legally exempt from disclosure. If you 
> are not the named addressee, or have been inadvertently and erroneously 
> referenced in the address line, you are not authorized to read, print, 
> retain, copy or disseminate this message or any part of it. If you have 
> received this message in error, please notify the sender immediately by 
> e-mail and delete all copies of the message. (ID m031214)

Re: [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

[Damian Guy]

+1
On Fri, 9 Mar 2018 at 21:30, Matthias J. Sax  wrote:

> Guozhang, renaming the class is an internal change and I think it's not
> required to cover it in the KIP.
>
>
> +1 (binding)
>
>
> -Matthias
>
> On 3/9/18 1:26 PM, Guozhang Wang wrote:
> > Hi John,
> >
> > You mentioned you are going to do the renaming for the existing class
> > to InternalMockProcessorContext
> > but that seems not reflected on the KIP wiki, could you add that in the
> > proposed changes section?
> >
> > Other than that, I'm +1 on the KIP.
> >
> > Thanks!
> >
> > Guozhang
> >
> > On Fri, Mar 9, 2018 at 12:39 PM, Ted Yu  wrote:
> >
> >> +1
> >>  Original message From: Bill Bejeck 
> >> Date: 3/9/18  12:29 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Re:
> >> [VOTE] KIP-267: Add Processor Unit Test Support to Kafka Streams Test
> Utils
> >> Thanks for the KIP.
> >>
> >> +1
> >>
> >> -Bill
> >>
> >> On Fri, Mar 9, 2018 at 3:16 PM, John Roesler  wrote:
> >>
> >>> Hello all,
> >>>
> >>> I'd like to start voting on KIP-267, to introduce a
> MockProcessorContent
> >>> enabling Processor, Transformer, and ValueTransformer authors to
> >> unit-test
> >>> their components.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+
> >>> Test+Utils?src=jira
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>
> >
> >
> >
>
>

[jira] [Created] (KAFKA-6638) Controller should remove replica from ISR if the replica is removed from the replica set

[Dong Lin (JIRA)]

Dong Lin created KAFKA-6638:
---

 Summary: Controller should remove replica from ISR if the replica 
is removed from the replica set
 Key: KAFKA-6638
 URL: https://issues.apache.org/jira/browse/KAFKA-6638
 Project: Kafka
  Issue Type: Improvement
Reporter: Dong Lin
Assignee: Dong Lin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6637) if set topic config segment.ms=0 Kafka broker won't be able to start

[Chong Wang (JIRA)]

Chong Wang created KAFKA-6637:
-

 Summary: if set topic config segment.ms=0 Kafka broker won't be 
able to start
 Key: KAFKA-6637
 URL: https://issues.apache.org/jira/browse/KAFKA-6637
 Project: Kafka
  Issue Type: Bug
Reporter: Chong Wang


If set topic config segment.ms to 0, Kafka server won't be able to start 
because of a FATAL error:

[2018-03-12 19:05:40,196] FATAL [KafkaServer id=2] Fatal error during 
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) 
java.lang.ArithmeticException: / by zero at 
kafka.log.LogConfig.randomSegmentJitter(LogConfig.scala:100) at 
kafka.log.Log.loadSegments(Log.scala:419) at 
kafka.log.Log.(Log.scala:203) at kafka.log.Log$.apply(Log.scala:1734) at 
kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:221) at 
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$8$$anonfun$apply$16$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:292)
 at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61) at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at 
java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748)

[https://github.com/apache/kafka/blob/1.0/core/src/main/scala/kafka/log/LogConfig.scala#L100]

So the minimum value shouldn't be 0
https://kafka.apache.org/documentation/#topicconfigs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6636) ReplicaFetcherThread should not die if hw < 0

[Dong Lin (JIRA)]

Dong Lin created KAFKA-6636:
---

 Summary: ReplicaFetcherThread should not die if hw < 0
 Key: KAFKA-6636
 URL: https://issues.apache.org/jira/browse/KAFKA-6636
 Project: Kafka
  Issue Type: Improvement
Reporter: Dong Lin
Assignee: Dong Lin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[Guozhang Wang]

Hello Matthias, thanks for your replies.


1) About the config names: actually I was trying to not expose
implementation details :) My main concern was that in your proposal the
values need to cover the span of all the versions that are actually using
the same version, i.e. "0.10.1.x-1.1.x". So if I (as a user) am upgrading
from any versions within this range I need to remember to use the value
"0.10.1.x-1.1.x" than just specifying my old version. In my suggestion I
was trying to argue the benefit of just letting users to specify the actual
Kafka version she's trying to upgrade from, than specifying a range of
versions. I was not suggesting to use "v1, v2, v3" etc as the values, but
still using Kafka versions like broker's `internal.version` config. But if
you were suggesting the same thing, i.e. by "0.10.1.x-1.1.x" you meant to
say users can just specify "0.10.1" or "0.10.2" or "0.11.0" or "1.1" which
are all recognizable config values then I think we are actually on the same
page.

2) About the "multi-assignment" idea: yes it would increase the network
footprint, but not the message size, IF I'm not mis-understanding your idea
of registering multiple assignment. More details:

In the JoinGroupRequest, in the protocols field we can encode multiple
protocols each with their different metadata. The coordinator will pick the
common one that everyone supports (if there are no common one, it will send
an error back; if there are multiple ones, it will pick the one with most
votes, i.e. the one which was earlier in the encoded list). Since our
current Streams rebalance protocol is still based on the consumer
coordinator, it means our protocol_type would be "consumer", but instead
the protocol type we can have multiple protocols like "streams",
"streams_v2", "streams_v3" etc. The downside is that we need to implement a
different assignor class for each version and register all of them in
consumer's PARTITION_ASSIGNMENT_STRATEGY_CONFIG. In the future if we
re-factor our implementation to have our own client coordinator layer like
Connect did, we can simplify this part of the implementation. But even for
now with the above approach this is still doable.

On the broker side, the group coordinator will only persist a group with
the selected protocol and its subscription metadata, e.g. if coordinator
decides to pick "streams_v2" it will only sends that protocol's metadata
from everyone to the leader to assign, AND when completing the rebalance it
will also only write the group metadata with that protocol and the
assignment only. In a word, although the network traffic maybe increased a
bit, it would not be a bummer in our trade-off. One corner situation we
need to consider is how to stop registering very old assignors to avoid the
network traffic from increasing indefinitely, e.g. if you are rolling
bounce from v2 to v3, then you'd not need to register v1 assignor anymore,
but that would unfortunately still require some configs.

3) About the  "version probing" idea, I think that's a promising approach
as well, but if we are going to do the multi-assignment its value seems
subsumed? But I'm thinking maybe it can be added on top of multi-assignment
to save us from still requiring the config to avoid registering all the
metadata for all version. More details:

In the JoinGroupRequest, we still register all the assignor but for all old
assignors we do not encode any metadata, i.e. the encoded data would be:

"streams_vN" : "encoded metadata"
"streams_vN-1":empty
"streams_vN-2":empty
..
"streams_0":empty

So the coordinator can still safely choose the latest common version; and
then when leaders receive the subscription (note it should always recognize
that version), let's say it is streams_vN-2, if one of the subscriptions
are empty bytes, it will send the empty assignment with that version number
encoded in the metadata. So in the second auto-triggered all members would
send the metadata with that version:

"streams_vN" : empty
"streams_vN-1" : empty
"streams_vN-2" : "encoded metadata"
..
"streams_0":empty


By doing this we would not require any configs for users.


4) About the "in_place" upgrade on rocksDB, I'm not clear about the details
so probably we'd need to fill that out before making a call. For example,
you mentioned "If we detect this situation, the Streams application closes
corresponding active tasks as well as "hot standby" tasks, and re-creates
the new active tasks using the new store." How could we guarantee that the
gap between these two stores will keep decreasing than increasing so we'll
eventually achieve the flip point? And also the longer we are before the
flip point, the larger we are doubling the storage space, etc.



Guozhang



On Sun, Mar 11, 2018 at 4:06 PM, Matthias J. Sax 
wrote:

> @John, Guozhang,
>
> thanks a lot for your comments. Very long reply...
>
>
> About upgrading the rebalance metadata:
>
> Another possibility to do this, would be to register multiple 

[jira] [Resolved] (KAFKA-4984) Unable to produce or consume when enabling authentication SASL/Kerberos

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4984.
--
Resolution: Cannot Reproduce

{color:#00}Closing inactive issue. Please reopen if you think the issue 
still exists{color}
 

> Unable to produce or consume when enabling authentication SASL/Kerberos
> ---
>
> Key: KAFKA-4984
> URL: https://issues.apache.org/jira/browse/KAFKA-4984
> Project: Kafka
>  Issue Type: Bug
> Environment: Ubuntu 16.04LTS running in VirtualBox
>Reporter: Ait haj Slimane
>Priority: Critical
> Attachments: Screenshot from 2017-03-30 15-36-30.png, logKafka.txt, 
> logZookeeper.txt
>
>
> I have a problem while trying to produce or consume on kerberos enabled 
> cluster.
> I launched a single broker and a console producer,
> using the SASL authentication between producer and broker.
> When i run the producer ,I got the result attached below
> Any advice on what can cause the problem.
> Thanks!
> 
> configuration used:
> server.properties:
> listeners=SASL_PLAINTEXT://kafka.example.com:9092
> security.inter.broker.protocol=SASL_PLAINTEXT
> sasl.mechanism.inter.broker.protocol=GSSAPI
> sasl.enabled.mechanism=GSSAPI
> sasl.kerberos.service.name=kafka
> producer.properties
> bootstrap.servers=kafka.example.com:9092
> sasl.kerberos.service.name=kafka
> security.protocol=SASL_PLAINTEXT
> kafka_client_jaas.conf
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/kafka/keytabs/kafkaclient.keytab"
> principal="kafkaclient/kafka.example@example.com";
> };



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5504) Kafka controller is not getting elected

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5504.
--
Resolution: Not A Problem

Closing as per above comments.

> Kafka controller is not getting elected
> ---
>
> Key: KAFKA-5504
> URL: https://issues.apache.org/jira/browse/KAFKA-5504
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.9.0.1
>Reporter: Ashish Kumar
>Priority: Major
>
> I am having a kafka cluster of 20 nodes and I was facing the issue of 
> under-replicated topics issue for last few days so decided to restart the 
> broker which was working as a controller but after restart getting below logs 
> in all the brokers (It seems controller is not finalized and leader election 
> is happening continuously):
> [2017-06-23 02:59:50,388] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:50,396] INFO New leader is 12 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:50,410] INFO Rolled new log segment for 
> 'dpe_feedback_rating_history-4' in 0 ms. (kafka.log.Log)
> [2017-06-23 02:59:51,585] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:51,590] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:51,609] INFO New leader is 11 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:52,792] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:52,799] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:52,808] INFO New leader is 12 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:54,122] INFO New leader is 3 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:55,504] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:55,512] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:55,520] INFO New leader is 11 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:56,695] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:56,701] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:56,709] INFO New leader is 11 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:57,949] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:57,955] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:57,965] INFO New leader is 12 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> [2017-06-23 02:59:59,378] INFO Creating /controller (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:59,384] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2017-06-23 02:59:59,395] INFO New leader is 12 
> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
> .
> .
> .
> Tried deleting controller znode (/controller) but no luck. Please let me know 
> if any fix is possible here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6022) mirror maker stores offset in zookeeper

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6022.
--
Resolution: Not A Problem

Please reopen if you think the issue still exists

> mirror maker stores offset in zookeeper
> ---
>
> Key: KAFKA-6022
> URL: https://issues.apache.org/jira/browse/KAFKA-6022
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I happened to notice that the mirror maker stores its offset in zookeeper. 
> I do not see it using:
> bin/kafka-consumer-groups.sh --bootstrap-server pi1:9092 --new-consumer --list
> I do however see consumers that store their offset in kafka.
> I would guess that storing the offset in zookeeper is old style?
> Would it be an idea to update the mirror maker to the new consumer style?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

RE: Log Retention Period of Kafka Messages

[Kyle Tinker]

You have a couple options:
1) You can adjust log.segment.bytes to make the segments smaller so that 
individual segments can be trimmed
2) You can set log.roll.hours, which will roll to a new log segment even if the 
size hasn't been reached
Note: #1 and #2 also have per-topic setting controls

https://kafka.apache.org/08/documentation/#brokerconfigs

Thanks,
--


KYLE TINKER
Lead Architect/Team Lead | WorkForce Software 

T: +1 734 742 2616 | E: ktin...@workforcesoftware.com


-Original Message-
From: Muruganandham, Ashokkumar [mailto:ashokkumar.muruganand...@fluke.com] 
Sent: Monday, March 12, 2018 1:19 AM
To: dev@kafka.apache.org
Subject: Log Retention Period of Kafka Messages

Hi Team,

Form the documentation I could see that log retention period is relative to the 
partition segments. So unless the segment is closed (1Gb/1 Week) the message 
will never be deleted.

Consider I have a very critical use case where I will need to delete the data 
every day and my segment reaches 1GB after three days. So according the above 
statement, then there is no way I could configure Kafka to delete the data in 
one day.

Is my understanding correct ?

Regards,
AK


Please be advised that this email may contain confidential information. If you 
are not the intended recipient, please notify us by email by replying to the 
sender and delete this message. The sender disclaims that the content of this 
email constitutes an offer to enter into, or the acceptance of, any agreement; 
provided that the foregoing does not invalidate the binding effect of any 
digital or other electronic reproduction of a manual signature that is included 
in any attachment.



This message is intended exclusively for the individual or entity to which it 
is addressed. This communication may contain information that is proprietary, 
privileged, confidential or otherwise legally exempt from disclosure. If you 
are not the named addressee, or have been inadvertently and erroneously 
referenced in the address line, you are not authorized to read, print, retain, 
copy or disseminate this message or any part of it. If you have received this 
message in error, please notify the sender immediately by e-mail and delete all 
copies of the message. (ID m031214)

[jira] [Resolved] (KAFKA-5467) setting offset retention minutes to a lower value is not reflecting

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5467.
--
Resolution: Cannot Reproduce

Closing inactive issue. Please reopen if you think the issue exists.

> setting offset retention minutes to a lower value is not reflecting
> ---
>
> Key: KAFKA-5467
> URL: https://issues.apache.org/jira/browse/KAFKA-5467
> Project: Kafka
>  Issue Type: Bug
>  Components: offset manager
>Affects Versions: 0.10.1.1
>Reporter: Divya
>Priority: Major
>
> We have been observing offsets to be unknown and saw that our offset 
> retention time was lesser than the log retention period. Inorder to recreate 
> the same in test environment, we set the offset.retention.minutes to 1 minute 
> and the log retention time to 168 hours. There were no events written for 
> more than an hour but still the offsets were not turning to unknown. (The 
> offset clean interval was 10 minutes.) I would like to know the reason on why 
> the offset did not turn to unknown in an hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Log Retention Period of Kafka Messages

[Muruganandham, Ashokkumar]

Hi Team,

Form the documentation I could see that log retention period is relative to the 
partition segments. So unless the segment is closed (1Gb/1 Week) the message 
will never be deleted.

Consider I have a very critical use case where I will need to delete the data 
every day and my segment reaches 1GB after three days. So according the above 
statement, then there is no way I could configure Kafka to delete the data in 
one day.

Is my understanding correct ?

Regards,
AK


Please be advised that this email may contain confidential information. If you 
are not the intended recipient, please notify us by email by replying to the 
sender and delete this message. The sender disclaims that the content of this 
email constitutes an offer to enter into, or the acceptance of, any agreement; 
provided that the foregoing does not invalidate the binding effect of any 
digital or other electronic reproduction of a manual signature that is included 
in any attachment.

Dots in topic names in metric labels

[Reftel, Magnus]

Hi,

Dots in topic names in labels are replaced with underscores in consumer metrics 
(it appears it has been that way all the way back since KAFKA-3233), but not in 
broker or producer metrics. We'd like to make visualizations that are 
parameterized by topic name, and show overview information about the whole 
chain of systems that interact with that topic, but this gets a bit tricky 
since we have dots in our topic names. Is there any particular reason that dots 
in topic names are replaced with underscores in consumer metrics labels? If 
not, would it be possible to change this, or would that be considered an 
incompatible change of an API?

Best Regards
Magnus Reftel


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den 
institusjon eller person den er rettet til og kan vaere belagt med lovbestemt 
taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt 
Skatteetaten.
The contents of this email message and any attachments are intended solely for 
the addressee(s) and may contain confidential information and may be legally 
protected from disclosure. If you are not the intended recipient of this 
message, please immediately delete the message and alert the Norwegian Tax 
Administration.

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

[John Roesler]

This all sounds awesome to me.

One (very minor) thought about the config parameters. You might consider
throwing an exception if "upgrade.from" is set but "upgrade.mode" is not.
This would eliminate the risk that folks forget to unset "upgrade.from" and
just leave it in their configs indefinitely. If that parameter name is
reused in the future, it could cause problems down the line.

On Sun, Mar 11, 2018 at 4:06 PM, Matthias J. Sax 
wrote:

> @John, Guozhang,
>
> thanks a lot for your comments. Very long reply...
>
>
> About upgrading the rebalance metadata:
>
> Another possibility to do this, would be to register multiple assignment
> strategies for the 1.2 applications. For this case, new instances would
> be configured to support both and the broker would pick the version that
> all instances understand. The disadvantage would be, that we send much
> more data (ie, two subscriptions) in each rebalance as long as no second
> rebalance is done disabling the old protocol. Thus, using this approach
> would allow to avoid a second rebalance trading-off an increased
> rebalance network footprint (I also assume that this would increase the
> message size that is written into __consumer_offsets topic?). Overall, I
> am not sure if this would be a good tradeoff, but it could avoid a
> second rebalance (I have some more thoughts about stores below that are
> relevant for single rebalance upgrade).
>
> For future upgrades we might be able to fix this though. I was thinking
> about the following:
>
> In the current implementation, the leader fails if it gets a
> subscription it does not understand (ie, newer version). We could change
> this behavior and let the leader send an empty assignment plus error
> code (including supported version) back to the instance sending the
> "bad" subscription. This would allow the following logic for an
> application instance:
>
>  - on startup, always send the latest subscription format
>  - if leader understands it, we get an assignment back an start processing
>  - if leader does not understand it, we get an empty assignment and
> supported version back
>  - the application unsubscribe()/subscribe()/poll() again and sends a
> subscription using the leader's supported version
>
> This protocol would allow to do a single rolling bounce, and implements
> a "version probing" step, that might result in two executed rebalances.
> The advantage would be, that the user does not need to set any configs
> or do multiple rolling bounces, as Streams takes care of this
> automatically.
>
> One disadvantage would be, that two rebalances happen and that for an
> error case during rebalance, we loose the information about the
> supported leader version and the "probing step" would happen a second time.
>
> If the leader is eventually updated, it will include it's own supported
> version in all assignments, to allow a "down graded" application to
> upgrade its version later. Also, if a application fails, the first
> probing would always be successful and only a single rebalance happens.
> If we use this protocol, I think we don't need any configuration
> parameter for future upgrades.
>
>
> About "upgrade.from" vs "internal.protocol.version":
>
> Users would set "upgrade.from" to the release version the current/old
> application is using. I think this is simpler, as users know this
> version. If we use "internal.protocol.version" instead, we expose
> implementation details and users need to know the protocol version (ie,
> they need to map from the release version to the protocol version; ie,
> "I am run 0.11.0 that runs with metadata protocol version 2").
>
> Also the KIP states that for the second rolling bounce, the
> "upgrade.mode" config should be set back to `null` -- and thus,
> "upgrade.from" would not have any effect and is ignored (I will update
> the KIP to point out this dependency).
>
>
>
> About your second point: I'll update the KIP accordingly to describe
> future updates as well. Both will be different.
>
>
>
> One more point about upgrading the store format. I was thinking about
> avoiding the second rolling bounce all together in the future: (1) the
> goal is to achieve an upgrade with zero downtime (2) this required to
> prepare the stores as "hot standbys" before we do the switch and delete
> the old stores. (3) the current proposal does the switch "globally" --
> this is simpler and due to the required second rebalance no disadvantage.
> However, a global consistent switch over might actually not be required.
> For "in_place" upgrade, following the protocol from above, we could
> decouple the store switch and each instance could switch its store
> independently from all other instances. After the rolling bounce, it
> seems to be ok to switch from the old store to the new store "under the
> hood" whenever the new store is ready (this could even be done, before
> we switch to the new metadata version). Each time we update the "hot
> standby" we check 

[jira] [Resolved] (KAFKA-6337) Error for partition [__consumer_offsets,15] to broker

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6337.
--
Resolution: Cannot Reproduce

Please reopen if you think the issue still exists

> Error for partition [__consumer_offsets,15] to broker
> -
>
> Key: KAFKA-6337
> URL: https://issues.apache.org/jira/browse/KAFKA-6337
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
> Environment: Windows running Kafka(0.10.2.0)
> 3 ZK Instances running on 3 different Windows Servers, 7 Kafka Broker nodes 
> running on single windows machine with different disk for logs directory.
>Reporter: Abhi
>Priority: Blocker
>  Labels: windows
>
> Hello *
> I am running Kafka(0.10.2.0) on windows from the past one year ...
> But off late there has been unique Broker issues that I have observed 4-5 
> times in
> last 4 months.
> Kafka setup cofig...
> 3 ZK Instances running on 3 different Windows Servers, 7 Kafka Broker nodes 
> running on single windows machine with different disk for logs directory
> My Kafka has 2 Topics with partition size 50 each , and replication factor of 
> 3.
> My partition logic selection: Each message has a unique ID and logic of 
> selecting partition is ( unique ID % 50), and then calling Kafka producer API 
> to route a specific message to a particular topic partition .
> My Each Broker Properties look like this
> {{broker.id=0
> port:9093
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> offsets.retention.minutes=360
> advertised.host.name=1.1.1.2
> advertised.port:9093
> ctories under which to store log files
> log.dirs=C:\\kafka_2.10-0.10.2.0-SNAPSHOT\\data\\kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=1
> log.retention.minutes=360
> log.segment.bytes=52428800
> log.retention.check.interval.ms=30
> log.cleaner.enable=true
> log.cleanup.policy=delete
> log.cleaner.min.cleanable.ratio=0.5
> log.cleaner.backoff.ms=15000
> log.segment.delete.delay.ms=6000
> auto.create.topics.enable=false
> zookeeper.connect=1.1.1.2:2181,1.1.1.3:2182,1.1.1.4:2183
> zookeeper.connection.timeout.ms=6000
> }}
> But of-late there has been a unique case that's cropping out in Kafka broker 
> nodes,
> _[2017-12-02 02:47:40,024] ERROR [ReplicaFetcherThread-0-4], Error for 
> partition [__consumer_offsets,15] to broker 
> 4:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server 
> is not the leader for that topic-partition. 
> (kafka.server.ReplicaFetcherThread)_
> The entire server.log is filled with these logs, and its very huge too , 
> please help me in understanding under what circumstances these can occur, and 
> what measures I need to take.. 
> Please help me this is the third time in last three Saturdays i faced the 
> similar issue. 
> Courtesy
> Abhi
> !wq 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk7 #3243

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Revert to ZooKeeper 3.4.10 due to ZOOKEEPER-2960 (#4678)

--
[...truncated 417.97 KB...]

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningNewConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured STARTED

kafka.tools.ConsoleConsumerTest > 
shouldSetAutoResetToSmallestWhenFromBeginningConfigured PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetEarliest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidOldConsumerConfigWithAutoOffsetResetLargest PASSED

kafka.tools.ConsoleConsumerTest > 
shouldResetUnConsumedOffsetsBeforeExitForNewConsumer STARTED

kafka.tools.ConsoleConsumerTest > 
shouldResetUnConsumedOffsetsBeforeExitForNewConsumer PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
STARTED

kafka.tools.ConsoleConsumerTest > groupIdsProvidedInDifferentPlacesMustMatch 
PASSED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 STARTED

kafka.tools.ConsoleConsumerTest > 
shouldExitOnInvalidConfigWithAutoOffsetResetAndConflictingFromBeginningOldConsumer
 PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewConsumerConfigWithAutoOffsetResetLatest PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageTest > testChecksum STARTED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testInvalidTimestamp STARTED

kafka.message.MessageTest > testInvalidTimestamp PASSED

kafka.message.MessageTest > testIsHashable STARTED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination STARTED

kafka.message.MessageTest > testInvalidTimestampAndMagicValueCombination PASSED

kafka.message.MessageTest > testExceptionMapping STARTED

kafka.message.MessageTest > testExceptionMapping PASSED

kafka.message.MessageTest > testFieldValues STARTED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testInvalidMagicByte STARTED

kafka.message.MessageTest > testInvalidMagicByte PASSED

kafka.message.MessageTest > testEquality STARTED

kafka.message.MessageTest > testEquality PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially STARTED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

Build failed in Jenkins: kafka-1.1-jdk7 #74

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Tag AWS instances with Jenkins build url (#4657)

[ismael] MINOR: Revert to ZooKeeper 3.4.10 due to ZOOKEEPER-2960 (#4678)

--
[...truncated 416.60 KB...]

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates STARTED

kafka.zk.KafkaZkClientTest > testGetTopicPartitionStates PASSED

kafka.zk.KafkaZkClientTest > testCreateConfigChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateConfigChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testDelegationTokenMethods STARTED

kafka.zk.KafkaZkClientTest > testDelegationTokenMethods PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics STARTED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testWindowsStyleTagNames STARTED

kafka.metrics.MetricsTest > testWindowsStyleTagNames PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled STARTED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

Jenkins build is back to normal : kafka-trunk-jdk9 #464

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-1.0-jdk7 #166

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #2466

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Revert to ZooKeeper 3.4.10 due to ZOOKEEPER-2960 (#4678)

--
[...truncated 416.54 KB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards PASSED

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Jan Filipiak]

Hey,

just want to throw in my question from the longer email in the other 
Thread here.
How will the bloom filter help a new consumer to decide to apply the key 
or not?


Why can we afford having a topic where its apparently not possible to 
start a new application on?
I think this is an overall flaw of the discussed idea here. Not playing 
attention to the overall architecture.


Best Jan


On 12.03.2018 00:09, Dong Lin wrote:

Hey Jason,

This is a good solution on the server side for log compacted topic.
Thinking about this more, there maybe another probably simpler solution on
the client side for log compacted topics. This solution is now specified in
the section "Changes in how producer produced keyed messages to log
compacted topics" of the KIP. The client-side solution seems simpler with
less performance overhead than the server-side solution. What do you think?

Thanks,
Dong


On Sat, Mar 10, 2018 at 10:55 AM, Jason Gustafson 
wrote:


Hey Dong,

I was thinking a bit about log compaction after a partition split. I think
the best you could hope for in terms of efficiency is that the network
overhead would be proportional to the number of remapped keys that need
cleaning. One thought I had which gets close to this is to propagate a
bloom filter covering the keys in the log prior to the split to all
partitions that might contain some of the remapped keys. As a simple
example, suppose we have a single partition which is split into two at
offset N. Say that broker 0 owns partition 0 and broker 1 owns partition 1.
Some subset of the keys prior to N will move to partition 1 and the rest
will remain on partition 0. The idea is something like this:

1. Every time we clean partition 0 on broker 0, we compute a bloom filter
for the keys in the log prior to offset N.
2. The bloom filter is propagated to broker 1 and cached.
3. The next time broker 1 cleans the log, it uses the bloom filter to
collect a set of possible matches.
4. When the cleaning completes, the matching keys are propagated to broker
0, where they are cached until the next cleaning.
5. The next time broker 0 cleans the log, it can remove all keys that have
been cached from the region prior to the split.

This incremental approach allows us to tradeoff cleaning latency to reduce
network traffic and memory overhead. A few points:

- The accuracy of bloom filters decreases as you add more elements to them.
We would probably choose to propagate the bloom filter for a subset of the
keys once it had reached a certain capacity to avoid having too many false
positives.
- We can limit the number of bloom filter matches that we will collect and
return in a single round of cleaning. These keys have to be cached in the
broker for a little while (until the next cleaning), so this lets us keep
the memory usage bounded.

There is probably some room for cleverness as well to avoid repeating work.
For example, the broker matching the bloom filter can also send the offset
of the last key that was matched against the filter. The next time we send
a bloom filter for a certain range of keys, we can send the starting offset
for matching. It's kind of like our "dirty offset" notion.

Needs a bit of investigation to work out the details (e.g. handling
multiple splits), but seems like it could work. What do you think?

-Jason



On Fri, Mar 9, 2018 at 1:23 PM, Matthias J. Sax 
wrote:


Thanks for your comment Clemens. It make sense what you are saying.
However, your described pattern is to split partitions and use linear
hashing to avoid random key distribution. But this is what Jan thinks we
should not do...

Also, I just picked an example with 2 -> 3 partitions, but if you don't
use linear hashing I think the same issue occurs if you double the
number of partitions.

I am in favor of using linear hashing. Still think, it is also useful to
split single partitions, too, in case load is not balanced and some
partitions are hot spots while others are "idle".

-Matthias


On 3/9/18 5:41 AM, Clemens Valiente wrote:

I think it's fair to assume that topics will always be increased by an

integer factor - e.g. from 2 partitions to 4 partitions. Then the mapping
is much easier.

Why anyone would increase partitions by lass than x2 is a mystery to

me.

If your two partitions cannot handle the load, then with three partitions
each one will still get 67% of that load which is still way too

dangerous.


So in your case we go from

part1: A B C D

part2: E F G H


to


part1: A C

part2: B D

part3: E F

part4: G H



From: Matthias J. Sax 
Sent: 09 March 2018 07:53
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-253: Support in-order message delivery with

partition expansion

@Jan: You suggest to copy the data from one topic to a new topic, and
provide an "offset mapping" from the old to the new topic for the
consumers. I don't quite understand how this would work.

Let's