Re: Some questions about providing corrections to documentation.

[Dhwani Katagade]

Happy New Year Team :-),

Could you please look at https://github.com/apache/kafka-site/pull/38 
and let me know if this looks good. This is my first PR and I am just 
checking to see if I got the process right.


thanks
-dhwani

On 12/31/2016 01:59 PM, Dhwani Katagade wrote:

Thanks Gouzhang,

I have created a PR here https://github.com/apache/kafka-site/pull/38

Also I believe my other assumption about the site docs process is also 
correct. That when the next release happens a new directory will be 
added under kafka-site:asf-site.


thanks
-dhwani

On 12/30/2016 11:19 AM, Guozhang Wang wrote:

Hello Dhwani,

Yes since it is a hotfix on the website, you need to submit a patch to
kafka-site on all the error directories instead of on kafka repo's docs
folder.

Details about web docs contribution can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes 




Guozhang


On Thu, Dec 29, 2016 at 9:46 AM, Dhwani Katagade <
dhwani_katag...@persistent.co.in> wrote:


Hi,

_Observations:_

While going through the design documentation I noticed a minor error of
duplication in points 4 and 5 here https://kafka.apache.org/docum
entation/#design_compactionguarantees. When I looked up the files in 
git,

I see the duplication in all of the below files in kafka-site repo

https://github.com/apache/kafka-site/blob/asf-site/081/design.html
https://github.com/apache/kafka-site/blob/asf-site/082/design.html
https://github.com/apache/kafka-site/blob/asf-site/090/design.html
https://github.com/apache/kafka-site/blob/asf-site/0100/design.html
https://github.com/apache/kafka-site/blob/asf-site/0101/design.html

And also in the following branches under kafka repo
https://github.com/apache/kafka/blob/0.9.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.0/docs/design.html
https://github.com/apache/kafka/blob/0.10.1/docs/design.html

But the same is corrected under trunk here 
https://github.com/apache/kafk

a/blob/trunk/docs/design.html

_Questions:_

1. If I have to provide a patch/PR to cleanup this documentation error,
should I provide the fix in files corresponding to all the versions
under kafka-site?
2. When the next release happens, as I understand, a new directory will
be added under kafka-site:asf-site. Since this would come from
kafka:trunk it will have the correction. Is my understanding 
correct?

3. As I understand, older branches under kafka repo are release
branches, and hence we should not make any new changes under the
docs directory on these branches. Is my understanding correct?
4. As I understand, this fix does not require a JIRA issue to be
logged. Is my understanding correct?

Thanks in advance for the clarifications.

-dhwani


DISCLAIMER
==
This e-mail may contain privileged and confidential information 
which is
the property of Persistent Systems Ltd. It is intended only for the 
use of

the individual or entity to which it is addressed. If you are not the
intended recipient, you are not authorized to read, retain, copy, 
print,
distribute or use this message. If you have received this 
communication in

error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus 
infected

mails.







DISCLAIMER
==
This e-mail may contain privileged and confidential information which 
is the property of Persistent Systems Ltd. It is intended only for the 
use of the individual or entity to which it is addressed. If you are 
not the intended recipient, you are not authorized to read, retain, 
copy, print, distribute or use this message. If you have received this 
communication in error, please notify the sender and delete all copies 
of this message. Persistent Systems Ltd. does not accept any liability 
for virus infected mails.





DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

[jira] [Work started] (KAFKA-3209) Support single message transforms in Kafka Connect

[Shikhar Bhushan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3209 started by Shikhar Bhushan.
--
> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4576) Log segments close to max size break on fetch

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15794042#comment-15794042
 ] 

huxi commented on KAFKA-4576:
-

Seems FileChannel.read does not fill up the 12-byte buffer although there are 
enough bytes to read.

The javadoc does not exactly specify how many bytes the buffer will be filled 
in. In practice, we are likely to always get a full buffer when using 
FileChannel, but actually Java does not make a guarantee about that. Instead, 
it depends on the OS implementation.

Maybe we should use a while loop to make sure 12 bytes have been filled into 
the buffer without violating the corner case check. Does it make sense? 
[~guozhang] [~ijuma] [~becket_qin]

> Log segments close to max size break on fetch
> -
>
> Key: KAFKA-4576
> URL: https://issues.apache.org/jira/browse/KAFKA-4576
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.1.1
>Reporter: Ivan Babrou
>
> We are running Kafka 0.10.1.1~rc1 (it's the same as 0.10.1.1).
> Max segment size is set to 2147483647 globally, that's 1 byte less than max 
> signed int32.
> Every now and then we see failures like this:
> {noformat}
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: ERROR [Replica Manager on 
> Broker 1006]: Error processing fetch operation on partition [mytopic,11], 
> offset 483579108587 (kafka.server.ReplicaManager)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: 
> java.lang.IllegalStateException: Failed to read complete buffer for 
> targetOffset 483686627237 startPosition 2145701130 in 
> /disk/data0/kafka-logs/mytopic-11/483571890786.log
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.log.FileMessageSet.searchForOffsetWithSize(FileMessageSet.scala:145)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.log.LogSegment.translateOffset(LogSegment.scala:128)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.log.LogSegment.read(LogSegment.scala:180)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.log.Log.read(Log.scala:563)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.ReplicaManager.kafka$server$ReplicaManager$$read$1(ReplicaManager.scala:567)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:606)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:605)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> scala.collection.Iterator$class.foreach(Iterator.scala:893)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:605)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:469)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:534)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> Dec 25 18:20:47 myhost kafka-run-class.sh[2054]: at 
> java.lang.Thread.run(Thread.java:745)
> {noformat}
> {noformat}
> ...
> -rw-r--r-- 1 kafka kafka  0 Dec 25 15:15 
> 483557418204.timeindex
> -rw-r--r-- 1 kafka kafka   9496 Dec 25 15:26 483564654488.index
> -rw-r--r-- 1 kafka kafka 2145763964 Dec 25 15:26 483564654488.log
> -rw-r--r-- 1 kafka kafka  0 Dec 25 15:26 
> 483564654488.timeindex
> -rw-r--r-- 1 kafka kafka   9576 Dec 25 15:37 483571890786.index
> -rw-r--r-- 1 kafka kafka 2147483644 Dec 25 15:37 483571890786.log
> -rw-r--r-- 1 kafka kafka  0 Dec 25 15:37 
> 483571890786.timeindex
> -rw-r--r-- 1 kafka kafka   9568 Dec 25 15:48 483579135712.index
> -rw-r--r-- 1 kafka kafka 2146791360 Dec 25 15:48 483579135712.log
> -rw-r--r-- 1 kafka kafka  0 Dec 25 15:48 
> 483579135712.timeindex
> -rw-r--r-- 1 kafka kafka   9408 Dec 25 15:59 483586374164.index
> ...
> {noformat}
> Here 483571890786.log is just 3 bytes below the max size.

[jira] [Commented] (KAFKA-4564) When the destination brokers are down or misconfigured in config, Streams should fail fast

[Umesh Chaudhary (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793810#comment-15793810
 ] 

Umesh Chaudhary commented on KAFKA-4564:


Thank you [~ewencp].

> When the destination brokers are down or misconfigured in config, Streams 
> should fail fast
> --
>
> Key: KAFKA-4564
> URL: https://issues.apache.org/jira/browse/KAFKA-4564
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
>
> Today if Kafka is down or users misconfigure the bootstrap list, Streams may 
> just hangs for a while without any error messages even with the log4j 
> enabled, which is quite confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2322) Use Java 7 features to improve code quality

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2322.

Resolution: Fixed

We have done this incrementally via various PRs. Marking as Fixed.

> Use Java 7 features to improve code quality
> ---
>
> Key: KAFKA-2322
> URL: https://issues.apache.org/jira/browse/KAFKA-2322
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Attachments: KAFKA-2322.patch
>
>
> Once KAFKA-2316 is merged, we should take advantage of Java 7 features that 
> improve code quality (readability, safety, etc.).
> Examples:
> * Diamond operator
> * Try with resources
> * Multi-catch
> * String in switch (maybe)
> * Suppressed exceptions (maybe)
> This issue is for simple and mechanical improvements. More complex changes  
> should be considered in separate issues (using nio.2, new concurrency 
> classes, etc.).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3172:
---
Fix Version/s: (was: 0.10.2.0)

> Consumer threads stay in 'Watiting' status and are blocked at consumer poll 
> method
> --
>
> Key: KAFKA-3172
> URL: https://issues.apache.org/jira/browse/KAFKA-3172
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: linux
>Reporter: Dany Benjamin
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: jmx_info.png, jstack.png, lagSample.png
>
>
> When running multiple consumers on same group (400 - for a 400 partitioned 
> topic), the application for all threads blocks at consumer.poll() method. The 
> timeout parameter sent in is 1.
> Stack dump:
> "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 
> waiting on condition [0x7f63df8f7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000605812710> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 
> tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0006058283e8> (a sun.nio.ch.Util$2)
> - locked <0x0006058283d8> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:425)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793692#comment-15793692
 ] 

Ismael Juma commented on KAFKA-3172:


Thanks [~oleg_gorobets]. JDK 8u112 includes the fix for the referenced bug. Has 
anyone else been able to reproduce the issue with JDK 8u112? I am clearing the 
"Fix version" field as the issue doesn't seem to be in Kafka.

> Consumer threads stay in 'Watiting' status and are blocked at consumer poll 
> method
> --
>
> Key: KAFKA-3172
> URL: https://issues.apache.org/jira/browse/KAFKA-3172
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: linux
>Reporter: Dany Benjamin
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: jmx_info.png, jstack.png, lagSample.png
>
>
> When running multiple consumers on same group (400 - for a 400 partitioned 
> topic), the application for all threads blocks at consumer.poll() method. The 
> timeout parameter sent in is 1.
> Stack dump:
> "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 
> waiting on condition [0x7f63df8f7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000605812710> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 
> tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0006058283e8> (a sun.nio.ch.Util$2)
> - locked <0x0006058283d8> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:425)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4459) rat license check not running in Jenkins

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793651#comment-15793651
 ] 

Ismael Juma commented on KAFKA-4459:


[~ewencp], the command for the job I linked to is:

{code}
./gradlew --no-daemon -Dorg.gradle.project.maxParallelForks=1 
-Dorg.gradle.project.testLoggingEvents=started,passed,skipped,failed clean 
testAll
{code}

It hasn't changed recently. I only changed the PR jobs.


> rat license check not running in Jenkins
> 
>
> Key: KAFKA-4459
> URL: https://issues.apache.org/jira/browse/KAFKA-4459
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.10.2.0
>
>
> The following commit introduced files without a license, which causes rat to 
> fail when run locally:
> https://github.com/apache/kafka/commit/e035fc039598127e88f31739458f705290b1fdba
> However, the build passed in Jenkins:
> https://builds.apache.org/job/kafka-trunk-jdk7/1710/
> Not sure why, but a potential culprit is the check for `.git` in the build:
> {code}
> if (new File('.git').exists()) {
>   apply from: file('gradle/rat.gradle')
>   rat {
> // Exclude everything under the directory that git should be ignoring via 
> .gitignore or that isn't checked in. These
> // restrict us only to files that are checked in or are staged.
> def repo = Grgit.open(project.file('.'))
> excludes = new ArrayList(repo.clean(ignore: false, directories: 
> true, dryRun: true))
> // And some of the files that we have checked in should also be excluded 
> from this check
> excludes.addAll([
> '**/.git/**',
> '**/build/**',
> 'CONTRIBUTING.md',
> 'gradlew',
> 'gradlew.bat',
> '**/README.md'
> ])
>   }
> }
> {code}
> If the current directory in Jenkins is not the repo directory, it could fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793643#comment-15793643
 ] 

Ismael Juma commented on KAFKA-3173:


Since we still don't know the source of this, removed the "Fix version".

> Error while moving some partitions to OnlinePartition state 
> 
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Critical
> Attachments: KAFKA-3173-race-repro.patch
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this 
> time the error appeared in the partition state machine. In KAFKA-2300, we 
> haven't cleaned up the state in {{PartitionStateMachine}} and 
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: 
> Error while moving some partitions to OnlinePartition state 
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: 
> Controller to broker state change requests batch is not empty while creating 
> a new one. 
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
>  might be lostat 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at 
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at 
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)at 
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at 
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)at 
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3173:
---
Fix Version/s: (was: 0.10.2.0)

> Error while moving some partitions to OnlinePartition state 
> 
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Critical
> Attachments: KAFKA-3173-race-repro.patch
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this 
> time the error appeared in the partition state machine. In KAFKA-2300, we 
> haven't cleaned up the state in {{PartitionStateMachine}} and 
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: 
> Error while moving some partitions to OnlinePartition state 
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: 
> Controller to broker state change requests batch is not empty while creating 
> a new one. 
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
>  might be lostat 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at 
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at 
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)at 
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at 
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)at 
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4440) Make producer RecordMetadata non-final

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4440:
-
Fix Version/s: 0.10.2.0

> Make producer RecordMetadata non-final
> --
>
> Key: KAFKA-4440
> URL: https://issues.apache.org/jira/browse/KAFKA-4440
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.10.0.1
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> ProducerRecord and ConsumerRecord were made non-final in KAFKA-4250. It will 
> be good to make RecordMetadata also non-final for the same reason of 
> extensibility of Producer/Consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4459) rat license check not running in Jenkins

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793625#comment-15793625
 ] 

Ewen Cheslack-Postava commented on KAFKA-4459:
--

[~ijuma] Seems like the build logs have been deleted, so hard to tell what's 
going on here. What is the build command that's running on jenkins? And has 
that changed recently with the new builders for jenkins using different JVMs & 
targets? Various targets manage to skip the tests such that the rat tests 
attached to them won't run; seems like the key issue here is making sure that 
the full set of tests that the rat tests are attached to are always executed as 
part of the jenkins tests.

> rat license check not running in Jenkins
> 
>
> Key: KAFKA-4459
> URL: https://issues.apache.org/jira/browse/KAFKA-4459
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.10.2.0
>
>
> The following commit introduced files without a license, which causes rat to 
> fail when run locally:
> https://github.com/apache/kafka/commit/e035fc039598127e88f31739458f705290b1fdba
> However, the build passed in Jenkins:
> https://builds.apache.org/job/kafka-trunk-jdk7/1710/
> Not sure why, but a potential culprit is the check for `.git` in the build:
> {code}
> if (new File('.git').exists()) {
>   apply from: file('gradle/rat.gradle')
>   rat {
> // Exclude everything under the directory that git should be ignoring via 
> .gitignore or that isn't checked in. These
> // restrict us only to files that are checked in or are staged.
> def repo = Grgit.open(project.file('.'))
> excludes = new ArrayList(repo.clean(ignore: false, directories: 
> true, dryRun: true))
> // And some of the files that we have checked in should also be excluded 
> from this check
> excludes.addAll([
> '**/.git/**',
> '**/build/**',
> 'CONTRIBUTING.md',
> 'gradlew',
> 'gradlew.bat',
> '**/README.md'
> ])
>   }
> }
> {code}
> If the current directory in Jenkins is not the repo directory, it could fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 0.10.2.0 release planning

[Ismael Juma]

Thanks for the update and for managing the next release Ewen. :)

Ismael

On Mon, Jan 2, 2017 at 10:22 PM, Ewen Cheslack-Postava 
wrote:

> Hey folks,
>
> We're quickly approaching our next time-based release. If you missed any of
> the updates on the new time-based releases we'll be following, see
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> for an explanation.
>
> I've started to organize the 0.10.2.0 release, scheduled for mid-February,
> here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.0
>
> Here are the important dates (also documented in the wiki):
>
> * Feature Freeze: Jan 13, 2017 (major features merged & working on
> stabilization, minor features have PR, release branch cut; anything not in
> this state will be automatically moved to the next release in JIRA)
> * Code Freeze: Jan 27, 2017 (first RC created now)
> * Release: Feb 14, 2017
>
> Please note that we expect these Feb releases to be a bit lighter than
> average because so many folks are on vacation over the winter holidays.
> However, we still have lots of good stuff happening!
>
> * KIPs! We have 11 adopted with 5 already committed and 6 with patches in
> flight. Let's plan to get all of these merged before the feature freeze in
> ~2w. Also, I did a quick pass over the VOTE threads to move some KIPs into
> the adopted section. But there are some that I saw just didn't have enough
> committer votes yet. If you have a KIP outstanding that is just waiting for
> votes & already has a PR prepared, please revive the thread and push it
> through. We'll cut them off by the feature freeze, but KIPs that are just
> waiting on votes could still make it in.
> * Open JIRAs. Oh my, we have a lot.
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.0%20AND%
> 20status%20not%20in%20(resolved%2C%20closed)%20ORDER%
> 20BY%20priority%20DESC%20
> More than 200 at the moment. I will be culling these and cleaning them up,
> although I won't remove the majority of them until we hit feature freeze
> (except for cases where we know they absolutely cannot make it). Note that
> if you have something you're aiming to get into 0.10.2.0, *you should mark
> it with that version now*.
> * Closed JIRAs: So far ~130 closed tickets for 0.10.2.0
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.0%20AND%
> 20status%20%3D%20resolved%20ORDER%20BY%20priority%20DESC%20
> * Release features:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.0
> has
> a "Release Features" section that I will include with the release
> notes/email for the release. I've populated it with notes about important
> KIPs, but key bug fixes or features could also be relevant. Please add to
> this list anything you think is worth noting -- it's a struggle keeping up
> with everything happening on the -dev list, so please chime in!
>
> I'll plan to give another update in ~1w as we get closer to the freeze
> dates, and with increasing frequency after that.
>
> -Ewen
>

0.10.2.0 release planning

[Ewen Cheslack-Postava]

Hey folks,

We're quickly approaching our next time-based release. If you missed any of
the updates on the new time-based releases we'll be following, see
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
for an explanation.

I've started to organize the 0.10.2.0 release, scheduled for mid-February,
here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.0

Here are the important dates (also documented in the wiki):

* Feature Freeze: Jan 13, 2017 (major features merged & working on
stabilization, minor features have PR, release branch cut; anything not in
this state will be automatically moved to the next release in JIRA)
* Code Freeze: Jan 27, 2017 (first RC created now)
* Release: Feb 14, 2017

Please note that we expect these Feb releases to be a bit lighter than
average because so many folks are on vacation over the winter holidays.
However, we still have lots of good stuff happening!

* KIPs! We have 11 adopted with 5 already committed and 6 with patches in
flight. Let's plan to get all of these merged before the feature freeze in
~2w. Also, I did a quick pass over the VOTE threads to move some KIPs into
the adopted section. But there are some that I saw just didn't have enough
committer votes yet. If you have a KIP outstanding that is just waiting for
votes & already has a PR prepared, please revive the thread and push it
through. We'll cut them off by the feature freeze, but KIPs that are just
waiting on votes could still make it in.
* Open JIRAs. Oh my, we have a lot.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%20
More than 200 at the moment. I will be culling these and cleaning them up,
although I won't remove the majority of them until we hit feature freeze
(except for cases where we know they absolutely cannot make it). Note that
if you have something you're aiming to get into 0.10.2.0, *you should mark
it with that version now*.
* Closed JIRAs: So far ~130 closed tickets for 0.10.2.0
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.0%20AND%20status%20%3D%20resolved%20ORDER%20BY%20priority%20DESC%20
* Release features:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.0 has
a "Release Features" section that I will include with the release
notes/email for the release. I've populated it with notes about important
KIPs, but key bug fixes or features could also be relevant. Please add to
this list anything you think is worth noting -- it's a struggle keeping up
with everything happening on the -dev list, so please chime in!

I'll plan to give another update in ~1w as we get closer to the freeze
dates, and with increasing frequency after that.

-Ewen

[jira] [Updated] (KAFKA-3304) KIP-35 - Retrieving protocol version

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3304:
-
Labels: kip  (was: )

> KIP-35 - Retrieving protocol version
> 
>
> Key: KAFKA-3304
> URL: https://issues.apache.org/jira/browse/KAFKA-3304
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>Priority: Critical
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> Uber JIRA to track adding of functionality to retrieve protocol versions. 
> More discussion can be found on 
> [KIP-35|https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-90 Remove zkClient dependency from Streams

[Ewen Cheslack-Postava]

Looks like this passes with 4 binding and 8 total +1 votes.

Hojjat, I've moved it into the adopted KIPs table and marked the JIRA for
0.10.2.0 if we can get it into that release.

-Ewen

On Wed, Dec 21, 2016 at 12:55 AM, Matthias J. Sax 
wrote:

> +1
>
>
> On 12/21/16 12:45 AM, Edoardo Comar wrote:
> > +1 (non-binding)
> > thanks!
> > --
> > Edoardo Comar
> > IBM MessageHub
> > eco...@uk.ibm.com
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> > IBM United Kingdom Limited Registered in England and Wales with number
> > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > 3AU
> >
> >
> >
> > From:   Ismael Juma 
> > To: dev@kafka.apache.org
> > Date:   20/12/2016 22:59
> > Subject:Re: [VOTE] KIP-90 Remove zkClient dependency from Streams
> > Sent by:isma...@gmail.com
> >
> >
> >
> > Thanks for the KIP, +1 (binding).
> >
> > On Tue, Dec 20, 2016 at 1:01 PM, Hojjat Jafarpour 
> > wrote:
> >
> >> Hi all,
> >>
> >> Seems that there is no opposition to this KIP. This email is to start
> > the
> >> voting for this KIP.
> >> Once again the KIP is for removing zkClient dependency from Streams.
> > Please
> >> check out the KIP page:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 90+-+Remove+zkClient+
> >> dependency+from+Streams
> >>
> >> Thanks,
> >> --Hojjat
> >>
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>
>

Re: [VOTE] KIP-100 - Relax Type constraints in Kafka Streams API

[Ewen Cheslack-Postava]

This passes with 4 binding and 7 total +1 votes.

Xavier, I've marked the JIRA for 0.10.2.0, which I assume we'll be able to
make, and moved it as WIP into the adopted KIPs section.

-Ewen

On Mon, Dec 19, 2016 at 2:33 PM, Guozhang Wang  wrote:

> +1.
>
> On Sat, Dec 17, 2016 at 3:27 AM, Ismael Juma  wrote:
>
> > Thanks Xavier. +1 (binding)
> >
> > Ismael
> >
> > On Fri, Dec 16, 2016 at 8:15 PM, Xavier Léauté 
> > wrote:
> >
> > > Ismael made a good point so I updated KIP-100 and expanded its scope to
> > > include covariant result types for functions applied to streams.
> > > I will update the discussion thread accordingly.
> > >
> > > On Tue, Dec 13, 2016 at 12:13 AM Ismael Juma 
> wrote:
> > >
> > > > Hi Xavier,
> > > >
> > > > Thanks for the KIP. If Java had declaration site variance (proposed
> > for a
> > > > future Java version[1]), we'd mark function parameters as
> contravariant
> > > > (i.e. "super") and the result as covariant (i.e. "extends"). In the
> > > > meantime, we have to use the wildcards at use site as per your
> > proposal.
> > > > However, it seems that only the first case is covered by your
> proposal.
> > > > This is an improvement, but is there any reason not to do the latter
> as
> > > > well? It would be good to get it completely right this time.
> > > >
> > > > Ismael
> > > >
> > > > [1] http://openjdk.java.net/jeps/300
> > > >
> > > > On Fri, Dec 9, 2016 at 6:27 PM, Xavier Léauté 
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to start the vote for KIP-100 unless there are any
> more
> > > > > comments.
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+
> > > > > Relax+Type+constraints+in+Kafka+Streams+API
> > > > >
> > > > > corresponding PR here https://github.com/apache/kafka/pull/2205
> > > > >
> > > > > Thanks,
> > > > > Xavier
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

[jira] [Updated] (KAFKA-4001) Improve Kafka Streams Join Semantics (KIP-77)

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4001:
-
Labels: kip  (was: )

> Improve Kafka Streams Join Semantics (KIP-77)
> -
>
> Key: KAFKA-4001
> URL: https://issues.apache.org/jira/browse/KAFKA-4001
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> Kafka Streams supports three types of joins:
> * KStream-KStream
> * KStream-KTable
> * KTable-KTable
> Furthermore, Kafka Streams supports the join variant, namely
> * inner join
> * left join
> * outer join
> Not all combination of "type" and "variant" are supported.
> *The problem is, that the semantics of the different joins do use different 
> semantics (and are thus inconsistent).*
> With this ticket, we want to
> * introduce unique semantics over all joins
> * improve handling of "null"
> * add missing inner KStream-KTable join
> See KIP-76 for more details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76%3A+Improve+Kafka+Streams+Join+Semantics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4259) Enable JAAS configuration for Kafka clients without a config file

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4259:
-
Labels: kip  (was: )

> Enable JAAS configuration for Kafka clients without a config file
> -
>
> Key: KAFKA-4259
> URL: https://issues.apache.org/jira/browse/KAFKA-4259
> Project: Kafka
>  Issue Type: New Feature
>  Components: security
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> See KIP-85 for details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4458) Add per partition metrics for in-sync and assigned replica count

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4458:
-
Labels: kip  (was: )

> Add per partition metrics for in-sync and assigned replica count
> 
>
> Key: KAFKA-4458
> URL: https://issues.apache.org/jira/browse/KAFKA-4458
> Project: Kafka
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count
>  for details on proposed changes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4161) Decouple flush and offset commits

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4161:
-
Labels: kip needs-kip  (was: needs-kip)

> Decouple flush and offset commits
> -
>
> Key: KAFKA-4161
> URL: https://issues.apache.org/jira/browse/KAFKA-4161
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Assignee: Shikhar Bhushan
>  Labels: kip, needs-kip
> Fix For: 0.10.2.0
>
>
> It is desirable to have, in addition to the time-based flush interval, volume 
> or size-based commits. E.g. a sink connector which is buffering in terms of 
> number of records may want to request a flush when the buffer is full, or 
> when sufficient amount of data has been buffered in a file.
> Having a method like say {{requestFlush()}} on the {{SinkTaskContext}} would 
> allow for connectors to have flexible policies around flushes. This would be 
> in addition to the time interval based flushes that are controlled with 
> {{offset.flush.interval.ms}}, for which the clock should be reset when any 
> kind of flush happens.
> We should probably also support requesting flushes via the 
> {{SourceTaskContext}} for consistency though a use-case doesn't come to mind 
> off the bat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4393) Improve invalid/negative TS handling

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4393:
-
Labels: kip  (was: )

> Improve invalid/negative TS handling
> 
>
> Key: KAFKA-4393
> URL: https://issues.apache.org/jira/browse/KAFKA-4393
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> Currently, Kafka Streams does not handle invalid/negative timestamps returned 
> from the {{TimestampExtractor}} gracefully, but fails with an exception, 
> because negative timestamps cannot get handled in a meaningful way for any 
> time based (ie, window) operators like window aggregates and joins.
> We want to change Streams to a auto-drop behavior for negative timestamps for 
> those records (without any further user notification about dropped record) to 
> enable users to "step over" those records and keep going (instead of an 
> exception). To guard the user from silently dropping messages by default (and 
> kept current fail-fast behavior), we change the default extractor 
> {{ConsumerRecordTimestampExtractor}} to check the extracted meta-data record 
> timestamp and raise an exception if it is negative. Furthermore, we add a 
> "drop-and-log" extractor, as this seems to be a common behavior user might 
> want to have. For any other behavior, users can still provide a custom 
> TS-Extractor implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3452) Support session windows

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3452:
-
Labels: api kip  (was: api)

> Support session windows
> ---
>
> Key: KAFKA-3452
> URL: https://issues.apache.org/jira/browse/KAFKA-3452
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api, kip
> Fix For: 0.10.2.0
>
>
> The Streams DSL currently does not provide session window as in the DataFlow 
> model. We have seen some common use cases for this feature and it's better 
> adding this support asap.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4381:
-
Labels: kip  (was: )

> Add per partition lag metric to KafkaConsumer.
> --
>
> Key: KAFKA-4381
> URL: https://issues.apache.org/jira/browse/KAFKA-4381
> Project: Kafka
>  Issue Type: Task
>  Components: clients, consumer
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> Currently KafkaConsumer only has a metric of max lag across all the 
> partitions. It would be useful to know per partition lag as well.
> I remember there was a ticket created before but did not find it. So I am 
> creating this ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4462) Improved Kafka Client Compatibility Policy

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4462:
-
Labels: kip  (was: )

> Improved Kafka Client Compatibility Policy
> --
>
> Key: KAFKA-4462
> URL: https://issues.apache.org/jira/browse/KAFKA-4462
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.1.1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> A proposal to improve the compatibility policy of the Kafka client by 
> supporting the combination of new client, old broker.  See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4462) Improved Kafka Client Compatibility Policy

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4462:
-
Fix Version/s: 0.10.2.0

> Improved Kafka Client Compatibility Policy
> --
>
> Key: KAFKA-4462
> URL: https://issues.apache.org/jira/browse/KAFKA-4462
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.1.1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> A proposal to improve the compatibility policy of the Kafka client by 
> supporting the combination of new client, old broker.  See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3853:
-
Component/s: tools
 admin

> Report offsets for empty groups in ConsumerGroupCommand
> ---
>
> Key: KAFKA-3853
> URL: https://issues.apache.org/jira/browse/KAFKA-3853
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, tools
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> We ought to be able to display offsets for groups which either have no active 
> members or which are not using group management. The owner column can be left 
> empty or set to "N/A". If a group is active, I'm not sure it would make sense 
> to report all offsets, in particular when partitions are unassigned, but if 
> it seems problematic to do so, we could enable the behavior with a flag (e.g. 
> --include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3853:
-
Labels: kip  (was: )

> Report offsets for empty groups in ConsumerGroupCommand
> ---
>
> Key: KAFKA-3853
> URL: https://issues.apache.org/jira/browse/KAFKA-3853
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, tools
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> We ought to be able to display offsets for groups which either have no active 
> members or which are not using group management. The owner column can be left 
> empty or set to "N/A". If a group is active, I'm not sure it would make sense 
> to report all offsets, in particular when partitions are unassigned, but if 
> it seems problematic to do so, we could enable the behavior with a flag (e.g. 
> --include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4481) Relax Kafka Streams API type constraints

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4481:
-
Labels: kip usability  (was: usability)

> Relax Kafka Streams API type constraints
> 
>
> Key: KAFKA-4481
> URL: https://issues.apache.org/jira/browse/KAFKA-4481
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>  Labels: kip, usability
> Fix For: 0.10.2.0
>
>
> Streams API methods that apply transformations to streams are currently 
> invariant in the key and value types, when they should probably be 
> contravariant in those types.
> For instance, {{KStream.filter(Predicate predicate)}} should be 
> {{KStream.filter(Predicate predicate)}} to accept 
> predicates that can act on any supertype of K, or V.
> Same thing applies to method that take {{Aggregator}}, {{StreamPartitioner}}, 
> {{KeyValueMapper}}, {{ValueMapper}}, {{ProcessorSupplier}}, {{ValueJoiner}}, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4481) Relax Kafka Streams API type constraints

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4481:
-
Fix Version/s: 0.10.2.0

> Relax Kafka Streams API type constraints
> 
>
> Key: KAFKA-4481
> URL: https://issues.apache.org/jira/browse/KAFKA-4481
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>  Labels: kip, usability
> Fix For: 0.10.2.0
>
>
> Streams API methods that apply transformations to streams are currently 
> invariant in the key and value types, when they should probably be 
> contravariant in those types.
> For instance, {{KStream.filter(Predicate predicate)}} should be 
> {{KStream.filter(Predicate predicate)}} to accept 
> predicates that can act on any supertype of K, or V.
> Same thing applies to method that take {{Aggregator}}, {{StreamPartitioner}}, 
> {{KeyValueMapper}}, {{ValueMapper}}, {{ProcessorSupplier}}, {{ValueJoiner}}, 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4060) Remove ZkClient dependency in Kafka Streams

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4060:
-
Labels: kip  (was: )

> Remove ZkClient dependency in Kafka Streams
> ---
>
> Key: KAFKA-4060
> URL: https://issues.apache.org/jira/browse/KAFKA-4060
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Hojjat Jafarpour
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> In Kafka Streams we need to dynamically create or update those internal 
> topics (i.e. repartition topics) upon rebalance, inside 
> {{InternalTopicManager}} which is triggered by {{StreamPartitionAssignor}}. 
> Currently we are using {{ZkClient}} to talk to ZK directly for such actions.
> With create and delete topics request merged in by [~granthenke] as part of 
> KIP-4, we should now be able to remove the ZkClient dependency and directly 
> use these requests.
> Related: 
> 1. KIP-4. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> 2. Consumer Reblance Protocol. 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4060) Remove ZkClient dependency in Kafka Streams

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4060:
-
Fix Version/s: 0.10.2.0

> Remove ZkClient dependency in Kafka Streams
> ---
>
> Key: KAFKA-4060
> URL: https://issues.apache.org/jira/browse/KAFKA-4060
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Hojjat Jafarpour
>  Labels: kip
> Fix For: 0.10.2.0
>
>
> In Kafka Streams we need to dynamically create or update those internal 
> topics (i.e. repartition topics) upon rebalance, inside 
> {{InternalTopicManager}} which is triggered by {{StreamPartitionAssignor}}. 
> Currently we are using {{ZkClient}} to talk to ZK directly for such actions.
> With create and delete topics request merged in by [~granthenke] as part of 
> KIP-4, we should now be able to remove the ZkClient dependency and directly 
> use these requests.
> Related: 
> 1. KIP-4. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> 2. Consumer Reblance Protocol. 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4466) Add support to ducktape to run only a part of all tests

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793443#comment-15793443
 ] 

Ewen Cheslack-Postava commented on KAFKA-4466:
--

[~raghavgautam] Has this been resolved with the 0.6.0 release of ducktape that 
lets you split things into subsets of tests, in addition to the existing 
support for running tests via listing directories/files/test classes/test 
methods in the ducktape command?

> Add support to ducktape to run only a part of all tests
> ---
>
> Key: KAFKA-4466
> URL: https://issues.apache.org/jira/browse/KAFKA-4466
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4470) Exception (java.lang.RuntimeException) encountered during startup: org.codehaus.jackson.JsonParseException:

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4470.
--
Resolution: Invalid
  Assignee: Ewen Cheslack-Postava

Bug was filed on the wrong project.

> Exception (java.lang.RuntimeException) encountered during startup: 
> org.codehaus.jackson.JsonParseException: 
> 
>
> Key: KAFKA-4470
> URL: https://issues.apache.org/jira/browse/KAFKA-4470
> Project: Kafka
>  Issue Type: Bug
>Reporter: JianwenSun
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
>
> upgrade 2.1.13 to 3.9 error.
> we upgrade 1.2.4 thrift tables to 2.1.13 without any problems, but when 
> upgrade to higher 3.x it looks something wrong.
> any help?
> [root@bj-dev-infra-001 cassandra]# apache-cassandra-3.9/bin/cassandra -R -f
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset 
> (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset 
> (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize 
> (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
> CompilerOracle: inline 
> org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody 
> (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
> CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
> CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds 
> (JJ)V
> CompilerOracle: inline 
> org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary 
> (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan 
> (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
> CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes 
> (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
> CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes 
> (JJIJ[J)V
> CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
> (Ljava/nio/ByteBuffer;[B)I
> CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
> ([BLjava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned 
> (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/lang/Object;JILjava/lang/Object;JI)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt 
> (JI)[B
> INFO  06:05:28 Configuration location: 
> file:/usr/local/cassandra/apache-cassandra-3.9/conf/cassandra.yaml
> INFO  06:05:28 Node configuration:[allocate_tokens_for_keyspace=null; 
> authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
> auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; 
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
> broadcast_address=null; broadcast_rpc_address=null; 
> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; 
> cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
> cdc_raw_directory=/usr/local/cassandra/data/cdc_raw; 
> cdc_total_space_in_mb=null; client_encryption_options=; 
> cluster_name=TestCluster; column_index_cache_size_in_kb=2; 
> column_index_size_in_kb=64; 

[jira] [Commented] (KAFKA-3959) __consumer_offsets wrong number of replicas at startup

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793433#comment-15793433
 ] 

Ewen Cheslack-Postava commented on KAFKA-3959:
--

[~onurkaraman] Marked for 0.10.2.0 as this seems like an important fix, but 
given the new behavior could be considered a breaking change, we might need to 
defer until 0.11.0.0. Thoughts?

> __consumer_offsets wrong number of replicas at startup
> --
>
> Key: KAFKA-3959
> URL: https://issues.apache.org/jira/browse/KAFKA-3959
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, offset manager, replication
>Affects Versions: 0.9.0.1, 0.10.0.0
> Environment: Brokers of 3 kafka nodes running Red Hat Enterprise 
> Linux Server release 7.2 (Maipo)
>Reporter: Alban Hurtaud
>Assignee: Onur Karaman
>  Labels: reliability
> Fix For: 0.10.2.0
>
>
> When creating a stack of 3 kafka brokers, the consumer is starting faster 
> than kafka nodes and when trying to read a topic, only one kafka node is 
> available.
> So the __consumer_offsets is created with a replication factor set to 1 
> (instead of configured 3) :
> offsets.topic.replication.factor=3
> default.replication.factor=3
> min.insync.replicas=2
> Then, other kafka nodes go up and we have exceptions because the replicas # 
> for __consumer_offsets is 1 and min insync is 2. So exceptions are thrown.
> What I missed is : Why the __consumer_offsets is created with replication to 
> 1 (when 1 broker is running) whereas in server.properties it is set to 3 ?
> To reproduce : 
> - Prepare 3 kafka nodes with the 3 lines above added to servers.properties.
> - Run one kafka,
> - Run one consumer (the __consumer_offsets is created with replicas =1)
> - Run 2 more kafka nodes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3959) __consumer_offsets wrong number of replicas at startup

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3959:
-
Fix Version/s: 0.10.2.0

> __consumer_offsets wrong number of replicas at startup
> --
>
> Key: KAFKA-3959
> URL: https://issues.apache.org/jira/browse/KAFKA-3959
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, offset manager, replication
>Affects Versions: 0.9.0.1, 0.10.0.0
> Environment: Brokers of 3 kafka nodes running Red Hat Enterprise 
> Linux Server release 7.2 (Maipo)
>Reporter: Alban Hurtaud
>Assignee: Onur Karaman
>  Labels: reliability
> Fix For: 0.10.2.0
>
>
> When creating a stack of 3 kafka brokers, the consumer is starting faster 
> than kafka nodes and when trying to read a topic, only one kafka node is 
> available.
> So the __consumer_offsets is created with a replication factor set to 1 
> (instead of configured 3) :
> offsets.topic.replication.factor=3
> default.replication.factor=3
> min.insync.replicas=2
> Then, other kafka nodes go up and we have exceptions because the replicas # 
> for __consumer_offsets is 1 and min insync is 2. So exceptions are thrown.
> What I missed is : Why the __consumer_offsets is created with replication to 
> 1 (when 1 broker is running) whereas in server.properties it is set to 3 ?
> To reproduce : 
> - Prepare 3 kafka nodes with the 3 lines above added to servers.properties.
> - Run one kafka,
> - Run one consumer (the __consumer_offsets is created with replicas =1)
> - Run 2 more kafka nodes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4569:
-
Assignee: Umesh Chaudhary

> Transient failure in 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable
> -
>
> Key: KAFKA-4569
> URL: https://issues.apache.org/jira/browse/KAFKA-4569
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
>
> One example is:
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/370/testReport/junit/org.apache.kafka.clients.consumer/KafkaConsumerTest/testWakeupWithFetchDataAvailable/
> {code}
> Stacktrace
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.fail(Assert.java:95)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable(KafkaConsumerTest.java:679)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
>   at 
> 

[jira] [Commented] (KAFKA-4564) When the destination brokers are down or misconfigured in config, Streams should fail fast

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793379#comment-15793379
 ] 

Ewen Cheslack-Postava commented on KAFKA-4564:
--

[~umesh9...@gmail.com] I have assigned you, and also given you contributor 
permissions on the project so you can assign yourself to tickets in the future.

> When the destination brokers are down or misconfigured in config, Streams 
> should fail fast
> --
>
> Key: KAFKA-4564
> URL: https://issues.apache.org/jira/browse/KAFKA-4564
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
>
> Today if Kafka is down or users misconfigure the bootstrap list, Streams may 
> just hangs for a while without any error messages even with the log4j 
> enabled, which is quite confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4564) When the destination brokers are down or misconfigured in config, Streams should fail fast

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4564:
-
Assignee: Umesh Chaudhary

> When the destination brokers are down or misconfigured in config, Streams 
> should fail fast
> --
>
> Key: KAFKA-4564
> URL: https://issues.apache.org/jira/browse/KAFKA-4564
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Umesh Chaudhary
>  Labels: newbie
>
> Today if Kafka is down or users misconfigure the bootstrap list, Streams may 
> just hangs for a while without any error messages even with the log4j 
> enabled, which is quite confusing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Multiple Consumer Group In Single Topic

[Ewen Cheslack-Postava]

Rico,

Every consumer group will see all messages for the topics they are
subscribed to. If you want to filter these messages by consumer group,
you'd need to perform that filtering yourself after the messages are
returned by the consumer. To do so, you'd need to include enough
information in the messages to perform this filtering (e.g. you might
include metadata indicating which consumer group the message is intended
for, although if you are doing that it might be better to simply use
different topics for the two sets of messages).

-Ewen

On Fri, Dec 16, 2016 at 12:28 AM, Rico Lugod  wrote:

> Hi Guys,
>
> Good day!
>
> I have question regarding how to consume a specific message belongs to that
> consumer group?
>
> Here's the scenario:
>
> Publish message "A" to topic "X"
> Consume by Consumer Group A the message "A" from topic "X"
>
> Publish message "B" to topic "X"
> Consume by Consumer Group B the message "B" from topic "X"
>
> How can achieve this in scala? Because both consumers will consume the
> message even if it assigned a groupId.
>
> Here is my producer code:
>
> def main(args: Array[String]): Unit = {
>   val zookeeperUrl: String = "localhost:2182"
>   val kafkaServerUrl: String = "localhost:9092,localhost:9093"
>   val topic: String = "Topic_X"
>   val groupId: String = "Consumer_Group_A"
>   val eventType: String = "delete"
>   val deleteRetention: Int = 524
>   val producerConfig =
> ProducerService.createProducerConfig(zookeeperUrl, kafkaServerUrl,
> topic, groupId, deleteRetention.toString)
>   val producer = new KafkaProducer[String, String](producerConfig)
>   val producerPayload = ProducerPayload("{\"batch_id\":\"" +
> UUID.randomUUID().toString()
> + "\", \"document_id\":\"" + UUID.randomUUID().toString()
> + "\", \"type\":\"" + eventType
> + "\"}", "", topic, groupId, deleteRetention)
>   ProducerService.sendMessage(producer, zookeeperUrl, kafkaServerUrl,
> producerPayload)
>   logger.info("Done.")
> }
>
>
> Here is my consumer code:
>
> def consumeMessage = Action { implicit rs =>
>   val zookeeperUrl: String = "localhost:2182"
>   val kafkaServerUrl: String = "localhost:9092,localhost:9093"
>   val topic: String = "Topic_X"
>   val groupId: String = "Consumer_Group_A"
>   val config = ConsumerService.createConsumerConfig(zookeeperUrl,
> kafkaServerUrl, groupId)
>   val consumer = kafka.consumer.Consumer.create(config)
>   val consumerMap = consumer.createMessageStreams(Map(topic -> 1))
>   val streams = consumerMap.get(topic).get
>   val it = streams(0).iterator()
>   while (it.hasNext()) {
> val msg = new String(it.next().message())
> logger.info(s"Message successfully consumed from topic ${topic} => "
> + msg)
>   }
>   consumer.shutdown()
>   logger.info("Done.")
>   Ok
> }
>
>
>
>
> Your help is much appreciated. Thank you*Sincerely yours,*
>
>
> *Rico Nodalo Lugod*
> Senior Java / J2EE / SOA - Developer
> Cebu City, Philippines 6000
>
> Email: rnl2...@gmail.com
>

Re: New question / request on kafka consumer

[Ewen Cheslack-Postava]

This sounds related to https://issues.apache.org/jira/browse/KAFKA-1894
Currently the consumer won't timeout when it cannot contact the brokers.

-Ewen

On Wed, Dec 14, 2016 at 5:21 AM, Costache, Vlad <
vlad.costa...@metrosystems.net> wrote:

> Hello,
>
>
>
> We are trying to make a consumer for kafka, (client code alone and camel
> integrated) and we ended in a blocking point.
>
> Can you please give us an advice, or any other idea?
>
>
>
> Our problem:
>
> -  We create a kafka consumer that connects to a wrong server
> (wrong ip/port), and the consumer get stuck in “poll” method even the
> connection is not created.
>
> -  We tried also with camel, but the same problem (as the same
> kafka client is called)
>
> -  It seems that there is a bug in kafka java client that the
> connection ends up in a loop
>
> -  For producer everything is fine.
>
>
>
> Do you have any advice, or can you confirm that there is a bug? Do you
> plan to fix this? Our production code will be with camel, so there will be
> needed an exception to be thrown so we can make our data error handling.
>
>
>
> *How can we make the consumer throw exception if the connection to server
> is not created successfully or the connection is lost at some point?*
>
>
>
>
>
> Our environment:
>
> Standalone:
>
> 
>
>   
>
>   org.apache.kafka
>
>   kafka_2.11
>
>   0.10.0.0
>
>   
>
>
>
> With camel:
>
>   
>
>  org.apache.camel
>
>  camel-*kafka*
>
>  2.17.0.redhat-630187
>
>   
>
>
>
>
>
> The dummy code example is attached.
>
>
>
> Or camel:
>
>
>
>//server do not exist
>
> from("kafka:10.97.210.222:8093?topic=testTopic_
> mip133=testing=earliest=1"
> ).process(*new* Processor(
>
> ) {
>
>
>
> @Override
>
> *public* *void* process(Exchange exchange) *throws* Exception
> {
>
> *LOG*.info(*new* MsbHeaderImpl(), "Am primit header: " +
> exchange.getIn().getHeaders());
>
> *LOG*.info(*new* MsbHeaderImpl(), "Am primit body: " +
> exchange.getIn().getBody());
>
>
>
> }
>
> });
>
>
>
>
>
> Kafka server version: kafka_2.10-0.10.1.0
>
>
>
>
>
> Thanks,
>
> Vlad
>
> Geschaeftsanschrift/Business Address: METRO SYSTEMS GmbH, Metro-Strasse
> 12, 40235 Duesseldorf, Germany
> Aufsichtsrat/Supervisory Board: Heiko Hutmacher (Vorsitzender/ Chairman)
> Geschaeftsfuehrung/Management Board: Dr. Dirk Toepfer (Vorsitzender/CEO),
> Wim van Herwijnen
> Sitz Duesseldorf, Amtsgericht Duesseldorf, HRB 18232/Registered Office
> Duesseldorf, Commercial Register of the Duesseldorf Local Court, HRB 18232
>
> ---
> Betreffend Mails von *@metrosystems.net
>
> Die in dieser E-Mail enthaltenen Nachrichten und Anhaenge sind
> ausschliesslich fuer den bezeichneten Adressaten bestimmt.
> Sie koennen rechtlich geschuetzte, vertrauliche Informationen enthalten.
> Falls Sie nicht der bezeichnete Empfaenger oder zum Empfang dieser E-Mail
> nicht berechtigt sind, ist die Verwendung, Vervielfaeltigung oder
> Weitergabe der Nachrichten und Anhaenge untersagt. Falls Sie diese E-Mail
> irrtuemlich erhalten haben, informieren Sie bitte unverzueglich den
> Absender und vernichten Sie die E-Mail.
>
> Regarding mails from *@metrosystems.net
>
> This e-mail message and any attachment are intended exclusively for the
> named addressee.
> They may contain confidential information which may also be protected by
> professional secrecy. Unless you are the named addressee (or authorised to
> receive for the addressee) you may not copy or use this message or any
> attachment or disclose the contents to anyone else. If this e-mail was sent
> to you by mistake please notify the sender immediately and delete this
> e-mail.
>

[jira] [Commented] (KAFKA-4402) Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fash

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793141#comment-15793141
 ] 

ASF GitHub Bot commented on KAFKA-4402:
---

GitHub user yaojuncn reopened a pull request:

https://github.com/apache/kafka/pull/2128

KAFKA-4402: make the KafkaProducer true round robin per topic



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yaojuncn/kafka 
KAFKA-4402-client-producer-round-robin-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2128


commit d74b0bd4d712223da38b3f2e666e4611bf65b455
Author: yaojuncn 
Date:   2016-11-13T21:46:44Z

KAFKA-4402: make the KafkaProducer true round robin per topic

commit fdd946a0969ef9aed9f8e4bcb8ea095b81107411
Author: yaojuncn 
Date:   2016-11-13T22:18:09Z

KAFKA-4402: fix some style issues

commit f978b50b58ff14ee41a0d7b360073bf4671c2bc2
Author: yaojuncn 
Date:   2016-11-14T05:26:00Z

[KAFKA-4402] add performance test for round-robin default partitioner,
the perf test result on my own laptop is: "avg partition 
time(ms)=0.000109", so it's pretty fast with very low overhead that can be 
ignored,
anyway, from code perspective it only added one ConcurrentHashMap.get(topic)

commit f9ca41152b6241ead91b663289198a58ef0454d5
Author: yaojuncn 
Date:   2016-11-19T05:38:37Z

Merge branch 'trunk' into KAFKA-4402-client-producer-round-robin-fix

commit 82421e4b8281bc3236880a5296c3bd7c511e6c33
Author: yaojuncn 
Date:   2017-01-01T20:38:20Z

KAFKA-4402: fix some style issues in DefaultPartitionerTest.java




> Kafka Producer's DefaultPartitioner is actually not round robin as said in 
> the code comments "If no partition or key is present choose a partition in a 
> round-robin fashion"
> 
>
> Key: KAFKA-4402
> URL: https://issues.apache.org/jira/browse/KAFKA-4402
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Yao
>Priority: Minor
>
> From this code comments, it is said that Kafka client  Producer's 
> DefaultPartitioner will do round robin if "no partition or key is present", 
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a 
> counter and try to increase it every time and then will decide which 
> partition to go to; 
> However the issue here is the counter is a global counter that is shared by 
> all the topics, so  it is actually not round robin per topic and sometimes 
> caused unbalanced routing among different partitions. 
> Although we can pass a custom implementation of interface 
> "org.apache.kafka.clients.producer.Partitioner", it might be still good to 
> make the default implementation true round robin as comment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2128: KAFKA-4402: make the KafkaProducer true round robi...

[yaojuncn]

GitHub user yaojuncn reopened a pull request:

https://github.com/apache/kafka/pull/2128

KAFKA-4402: make the KafkaProducer true round robin per topic



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yaojuncn/kafka 
KAFKA-4402-client-producer-round-robin-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2128


commit d74b0bd4d712223da38b3f2e666e4611bf65b455
Author: yaojuncn 
Date:   2016-11-13T21:46:44Z

KAFKA-4402: make the KafkaProducer true round robin per topic

commit fdd946a0969ef9aed9f8e4bcb8ea095b81107411
Author: yaojuncn 
Date:   2016-11-13T22:18:09Z

KAFKA-4402: fix some style issues

commit f978b50b58ff14ee41a0d7b360073bf4671c2bc2
Author: yaojuncn 
Date:   2016-11-14T05:26:00Z

[KAFKA-4402] add performance test for round-robin default partitioner,
the perf test result on my own laptop is: "avg partition 
time(ms)=0.000109", so it's pretty fast with very low overhead that can be 
ignored,
anyway, from code perspective it only added one ConcurrentHashMap.get(topic)

commit f9ca41152b6241ead91b663289198a58ef0454d5
Author: yaojuncn 
Date:   2016-11-19T05:38:37Z

Merge branch 'trunk' into KAFKA-4402-client-producer-round-robin-fix

commit 82421e4b8281bc3236880a5296c3bd7c511e6c33
Author: yaojuncn 
Date:   2017-01-01T20:38:20Z

KAFKA-4402: fix some style issues in DefaultPartitionerTest.java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4402) Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fash

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793139#comment-15793139
 ] 

ASF GitHub Bot commented on KAFKA-4402:
---

Github user yaojuncn closed the pull request at:

https://github.com/apache/kafka/pull/2128


> Kafka Producer's DefaultPartitioner is actually not round robin as said in 
> the code comments "If no partition or key is present choose a partition in a 
> round-robin fashion"
> 
>
> Key: KAFKA-4402
> URL: https://issues.apache.org/jira/browse/KAFKA-4402
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Yao
>Priority: Minor
>
> From this code comments, it is said that Kafka client  Producer's 
> DefaultPartitioner will do round robin if "no partition or key is present", 
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a 
> counter and try to increase it every time and then will decide which 
> partition to go to; 
> However the issue here is the counter is a global counter that is shared by 
> all the topics, so  it is actually not round robin per topic and sometimes 
> caused unbalanced routing among different partitions. 
> Although we can pass a custom implementation of interface 
> "org.apache.kafka.clients.producer.Partitioner", it might be still good to 
> make the default implementation true round robin as comment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2128: KAFKA-4402: make the KafkaProducer true round robi...

[yaojuncn]

Github user yaojuncn closed the pull request at:

https://github.com/apache/kafka/pull/2128


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---