Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Gwen Shapira]

Looking at "existing solutions", it looks like Zookeeper allows plugging in
any SASL mechanism, but the server will only support one mechanism at a
time.
If this is good enough for our use-case (do we actually need to support
multiple mechanisms at once?), it will simplify life a lot for us (
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL)

On Mon, Feb 1, 2016 at 8:47 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> A few comments below.
>
> On Mon, Feb 1, 2016 at 4:30 PM, Gwen Shapira  wrote:
>
> > Thanks for clarifying, Ismael and Rajini. And I'm sorry for reopenning a
> > point that was clearly discussed already.
>
>
> Your input is definitely welcome. :) There was an initial discussion in the
> KIP meeting, but it was understood that a wider and more detailed
> discussion was needed before we could agree on the right solution.
>
> 1) Having two different ways to decide on the protocol (port + negotiation)
> > is needlessly confusing for administrators. Its just one more complexity
> to
> > figure out when authentication using SASL is already the most complex
> > administrative task one has in Kafka (judging by the number of questions
> we
> > get).
> >
>
> Yes, this is definitely a concern. It would be good to contrast this with
> how the multiple ports option could look like (given pluggable mechanisms,
> it won't be as simple as the existing config for multiple ports).
>
> 2) Troubleshooting. Especially in customer-support (and mailing-list
> > support) environment. Asking for two configuration files and two netstat
> > results is a completely different story than using tcp dump (which
> requires
> > root privileges), catching the correct negotiation packets and decoding
> > them to figure out what went wrong.
> >
>
> Right. Could we mitigate this somewhat with appropriate logging since we
> control the negotiation process?
>
> Ismael
>
> P.S. It is interesting that Cassandra went in the other direction and made
> it possible to use a single port for both encrypted and non-encrypted
> traffic a few months ago:
>
>
> https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=commit;h=535c3ac7
>

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[Rajiv Kurian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajiv Kurian updated KAFKA-3159:

Attachment: Screen Shot 2016-02-01 at 11.09.32 AM.png

CPU break down of the patched client. Some notes:
1. 40.58% of the process' CPU profile is on these poll calls which are done 
with a timeout of 5 seconds.
2. A lot of cpu is spent on hash map operations.
3. The rest of the cpu seems to be spent mostly in NetworkClient.poll().

> Kafka consumer 0.9.0.0  client poll is very CPU intensive under certain 
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
> Attachments: Screen Shot 2016-02-01 at 11.09.32 AM.png
>
>
> We are using the new kafka consumer with the following config (as logged by 
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id = 
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call 
> poll in a loop.  We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get 
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this 
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread 
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all 
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here timeout is about 10 seconds, so it is pretty big.
> records = consumer.poll(timeout);
> } catch (Exception e) {
>// This never hits for us
> logger.error("Exception polling Kafka ", e);
> records = null;
> }
> if (records != null) {
> for (ConsumerRecord record : records) {
>// The handler puts the byte array on a very fast ring buffer 
> so it barely takes any time.
> handler.handleMessage(ByteBuffer.wrap(record.value()));
> }
> }
> }
> With this setup our performance has taken a horrendous hit as soon as we 
> started this one thread that just polls Kafka in a loop.
> I profiled the application using Java Mission Control and have a few insights.
> 1. There doesn't seem to be a single hotspot. The consumer just ends up using 
> a lot of CPU for handing such a low number of messages. Our process was using 
> 16% CPU before we added a single consumer and it went to 25% and above 

[jira] [Work started] (KAFKA-2673) Log JmxTool output to logger

[chen zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2673 started by chen zhu.
---
> Log JmxTool output to logger
> 
>
> Key: KAFKA-2673
> URL: https://issues.apache.org/jira/browse/KAFKA-2673
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.8.2.1
>Reporter: Eno Thereska
>Assignee: chen zhu
>Priority: Trivial
>  Labels: newbie
> Fix For: 0.8.1.2
>
>
> Currently JmxTool outputs the data into a CSV file. It could be of value to 
> have the data sent to a logger specified in a log4j configuration file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[Rajiv Kurian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajiv Kurian updated KAFKA-3159:

Attachment: Memory-profile-patched-client.png

Memory profile of the patched client. Notes:

1.A lot of it is in clients.consumer.internals.Fetcher.createFetchRequests(). 
Again quite a bit of hash map allocations.
2. The majority of the rest of allocations seems to be in NetworkClient.poll().

> Kafka consumer 0.9.0.0  client poll is very CPU intensive under certain 
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
> Attachments: Memory-profile-patched-client.png, Screen Shot 
> 2016-02-01 at 11.09.32 AM.png
>
>
> We are using the new kafka consumer with the following config (as logged by 
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id = 
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call 
> poll in a loop.  We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get 
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this 
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread 
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all 
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here timeout is about 10 seconds, so it is pretty big.
> records = consumer.poll(timeout);
> } catch (Exception e) {
>// This never hits for us
> logger.error("Exception polling Kafka ", e);
> records = null;
> }
> if (records != null) {
> for (ConsumerRecord record : records) {
>// The handler puts the byte array on a very fast ring buffer 
> so it barely takes any time.
> handler.handleMessage(ByteBuffer.wrap(record.value()));
> }
> }
> }
> With this setup our performance has taken a horrendous hit as soon as we 
> started this one thread that just polls Kafka in a loop.
> I profiled the application using Java Mission Control and have a few insights.
> 1. There doesn't seem to be a single hotspot. The consumer just ends up using 
> a lot of CPU for handing such a low number of messages. Our process was using 
> 16% CPU before we added a single consumer and it went to 25% and above 

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[Rajiv Kurian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126877#comment-15126877
 ] 

Rajiv Kurian commented on KAFKA-3159:
-

[~hachikuji] I tried your patch. The Exceptions are now gone, but the CPU has 
remained high (25% + from 17% before the new client was added). I have attached 
the CPU breakdown and the allocation break down screen shots and comments.
Some notes:
1. The exceptions seem to be gone completely. The overall CPU has gone down to 
25% odd from the 27% before. So it has gotten a bit better. But the percentage 
of CPU used by the Kafka part of the code has gone up to 40.58% of the total 
used by my process. Most of the CPU is now spent on hash map code. Again I 
don't understand why there is so much CPU being used to get single digit 60 
byte messages per second (64 partitions striped across 3 brokers).

2. The allocations % has believe it or not gone up even more at about 31.26% of 
my entire processes allocation. Again it is baffling that it allocates so much 
to get so few messages. The total sum allocations from the TLAB in the 5 minute 
period has gone up to 14.05 GB from the 6.95 GB done by the client which threw 
a lot of exceptions. Again that seems to be a staggering amount of allocations 
for something that does 1 message odd a second.

My poll timings are done with a 5 second timeout which seems high enough.

Let me know if I can do more profiling or provide other info.

> Kafka consumer 0.9.0.0  client poll is very CPU intensive under certain 
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
> Attachments: Memory-profile-patched-client.png, Screen Shot 
> 2016-02-01 at 11.09.32 AM.png
>
>
> We are using the new kafka consumer with the following config (as logged by 
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id = 
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call 
> poll in a loop.  We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get 
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this 
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread 
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all 
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here 

[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126875#comment-15126875
 ] 

Gwen Shapira commented on KAFKA-3172:
-

Your description says "consumer" but your jstack dump says "producer"... can 
you double check if you are dumping the right process?

> Consumer threads stay in 'Watiting' status and are blocked at consumer poll 
> method
> --
>
> Key: KAFKA-3172
> URL: https://issues.apache.org/jira/browse/KAFKA-3172
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: linux
>Reporter: Dany Benjamin
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.9.0.0
>
> Attachments: jmx_info.png, jstack.png, lagSample.png
>
>
> When running multiple consumers on same group (400 - for a 400 partitioned 
> topic), the application for all threads blocks at consumer.poll() method. The 
> timeout parameter sent in is 1.
> Stack dump:
> "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 
> waiting on condition [0x7f63df8f7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000605812710> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 
> tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0006058283e8> (a sun.nio.ch.Util$2)
> - locked <0x0006058283d8> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:425)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[VOTE] KIP-42: Add Producer and Consumer Interceptors

[Anna Povzner]

Hi All,

I am opening the voting thread for KIP-42: Add Producer and Consumer
Interceptors.

For reference, here's the KIP wiki:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors

And the mailing list threads:
January:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
February:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e


Thanks
Anna

[VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Anna Povzner]

Hi All,

I am opening the voting thread for KIP-42: Add Producer and Consumer
Interceptors.

For reference, here's the KIP wiki:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors

And the mailing list threads:
January:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
February:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e


Thanks,
Anna

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126959#comment-15126959
 ] 

Jiangjie Qin commented on KAFKA-3174:
-

[~ijuma] My test code was as below. Initially I only tested 1MB size. I added 
testing on different size after saw your comments above.

{code}
public static void main(String[] args) {
int[] sizes = {8, 16, 32, 128, 1024, 65536, 1048576};
for (int size : sizes) {
byte[] bytes = new byte[size];
Random random = new Random();
random.nextBytes(bytes);
int loop = 5000 * (1048576 / size);

long start = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
Crc32 crc32 = new Crc32();
crc32.update(bytes, 0, bytes.length);
}
System.out.println(String.format("KCrc32: Size = %d\t, time = %d", 
size, (System.currentTimeMillis() - start)));

start = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
CRC32 crc32 = new CRC32();
crc32.update(bytes, 0, bytes.length);
}
System.out.println(String.format("JCrc32: Size = %d\t, time = 
%d\n", size, (System.currentTimeMillis() - start)));

}
}
{code}

And here is the output:
{code}
KCrc32: Size = 8, time = 10400
JCrc32: Size = 8, time = 9907

KCrc32: Size = 16   , time = 6959
JCrc32: Size = 16   , time = 8419

KCrc32: Size = 32   , time = 5596
JCrc32: Size = 32   , time = 5587

KCrc32: Size = 128  , time = 4397
JCrc32: Size = 128  , time = 3305

KCrc32: Size = 1024 , time = 4115
JCrc32: Size = 1024 , time = 2392

KCrc32: Size = 65536, time = 4087
JCrc32: Size = 65536, time = 2296

KCrc32: Size = 1048576  , time = 4078
JCrc32: Size = 1048576  , time = 2298
{code}

>From the output above, it looks for size < 32 bytes KCrc32 and JCrc32 is 
>comparable (except 16 bytes). After size >= 64, JCrc32 is faster. My ~2x 
>result came from 1MB size.

In this Intel paper, they mentioned that the CRC32 instruction is actually in 
SSE4.2, which was introduced in Nov 2008. 
http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
Wikipedia says the same thing.
https://en.wikipedia.org/wiki/SSE4#SSE4.2
AMD started to support SSE4.2 in Oct 2011.

I ran the above test on both my desktop (Intel(R) Xeon(R) CPU E5-2620 v2 @ 
2.10GHz) and macbook (Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz), both of them 
have SSE4.2 support.

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Becket Qin]

+1 (non-binding).

Nice wiki. Thanks Anna.

On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:

> Hi All,
>
> I am opening the voting thread for KIP-42: Add Producer and Consumer
> Interceptors.
>
> For reference, here's the KIP wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
>
> And the mailing list threads:
> January:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> February:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
>
>
> Thanks,
> Anna
>

[GitHub] kafka pull request: MINOR: some javadocs for kstream public api

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/844

MINOR: some javadocs for kstream public api

@guozhangwang 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/844.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #844


commit e935112ba0c9750c412f72d89cdbdd9755fd4eb2
Author: Yasuhiro Matsuda 
Date:   2016-02-01T20:49:52Z

MINOR: some javadocs for kstream public api




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127124#comment-15127124
 ] 

ASF GitHub Bot commented on KAFKA-3175:
---

GitHub user MayureshGharat opened a pull request:

https://github.com/apache/kafka/pull/846

KAFKA-3175 : Topic not accessible after deletion even when 
delete.topic.enable is disabled

Remove topics under /admin/delete_topics path in zk if deleteTopic is 
disabled. The topic should never be enqueued for deletion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MayureshGharat/kafka kafka-3175

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/846.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #846


commit 48676b68df0f33c5c095c73002b7cca10b8a1402
Author: MayureshGharat 
Date:   2016-02-01T21:51:57Z

Remove topics under /admin/delete_topics path in zk if deleteTopic is 
disabled. The topic should never be enqueued for deletion




> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127127#comment-15127127
 ] 

Ismael Juma commented on KAFKA-3174:


Unfortunately, the micro-benchmark above is not reliable. That's why I used JMH 
(which is maintained by the Oracle JVM team for their own Java benchmarks) for 
my benchmarks. See 
https://groups.google.com/d/msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ 
for why you should use JMH for nano and micro-benchmarks.

The Intel CRC32 instruction is for CRC32C, which uses a different polynomial 
than the one we use for Kafka (again, see KAFKA-1449 for more details on this). 
I actually checked the HotSpot code 
(http://hg.openjdk.java.net/hsx/hsx25/hotspot/rev/b800986664f4) and the 
assembly generated by the HotSpot JIT to verify my assertion before I posted my 
previous comment with regards to SSE 2, SSE 4.1, AVX and CLMUL being used.

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Jay Kreps]

+1

-Jay

On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:

> Hi All,
>
> I am opening the voting thread for KIP-42: Add Producer and Consumer
> Interceptors.
>
> For reference, here's the KIP wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
>
> And the mailing list threads:
> January:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> February:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
>
>
> Thanks,
> Anna
>

[jira] [Created] (KAFKA-3184) Add Checkpoint for In-memory State Store

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3184:


 Summary: Add Checkpoint for In-memory State Store
 Key: KAFKA-3184
 URL: https://issues.apache.org/jira/browse/KAFKA-3184
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang


Currently Kafka Streams does not make a checkpoint of the persistent state 
store upon committing, which would be expensive since it is "stopping the 
world" and write on disks: for example, RocksDB would require you to copy the 
file directory to make a copy naively. 

However, for in-memory stores checkpointing maybe doable in an asynchronous 
manner hence it can be done quickly. And the benefit of having intermediate 
checkpoint is to avoid restoring from scratch if standby tasks are not present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127201#comment-15127201
 ] 

Guozhang Wang commented on KAFKA-3177:
--

[~hachikuji] Just checking, this infinite loop's pattern is not the same as the 
scenario when there is no broker up and running, which will fall in the loop of 
finding the coordinator and disconnected from the given socket, right?

It seems today we have a couple of patterns that could result it "not 
respecting timeout in poll" and "unexpected blocking for unblocking functions", 
it would be better to fix these two in a more generic way rather tackling them 
one-by-one?

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3186) Kafka authorizer should be aware of principal types it supports.

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127213#comment-15127213
 ] 

Ashish K Singh commented on KAFKA-3186:
---

[~gwenshap] [~ijuma] I would like to know your thoughts on this, before I 
create a PR.

> Kafka authorizer should be aware of principal types it supports.
> 
>
> Key: KAFKA-3186
> URL: https://issues.apache.org/jira/browse/KAFKA-3186
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently, Kafka authorizer is agnostic of principal types it supports, so 
> are the acls CRUD methods in {{kafka.security.auth.Authorizer}}. The intent 
> behind is to keep Kafka authorization pluggable, which is really great. 
> However, this leads to following issues.
> 1. {{kafka-acls.sh}} supports pluggable authorizer and custom principals, 
> however is some what integrated with {{SimpleAclsAuthorizer}}. The help 
> messages has details which might not be true for a custom authorizer. For 
> instance, assuming User is a supported PrincipalType.
> 2. Acls CRUD methods perform no check on validity of acls, as they are not 
> aware of what principal types the support. This opens up space for lots of 
> user errors, KAFKA-3097 is an instance.
> I suggest we add a {{getSupportedPrincipalTypes}} method to authorizer and 
> use that for acls verification during acls CRUD, and make {{kafka-acls.sh}} 
> help messages more generic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #333

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: removed obsolete class

[wangguoz] KAFKA-1860: Fix issue that file system errors are not detected unless

--
[...truncated 6827 lines...]
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnector PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testReloadOnStart PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testStartStop PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testConsumerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testProducerError PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > 
testReloadOnStart PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testStartStop 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testSetFailure 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testMissingTopic 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testGetSet PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testRestore PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testStartStop PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testPutTaskConfigs 
PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutTaskConfigsDoesNotResolveAllInconsistencies PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelAfterAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testAlreadyFlushing 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testWriteFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullKeyFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullValueFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelBeforeAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testFlushFailureReplacesOffsets PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testNoOffsetsToFlush 
PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testSaveRestore 
PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testGetSet PASSED
:testAll

BUILD SUCCESSFUL

Total time: 1 hrs 4 mins 25.345 secs
+ ./gradlew --stacktrace docsJarAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:docsJar_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava UP-TO-DATE
:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes UP-TO-DATE
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar UP-TO-DATE
:kafka-trunk-jdk8:clients:javadoc
:docsJar_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Could not add entry 
'
 to cache fileHashes.bin 
(
> Corrupted FreeListBlock 677161 found in cache 
> '

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.UncheckedIOException: Could not add entry 
'
 to cache fileHashes.bin 
(
at 
org.gradle.cache.internal.btree.BTreePersistentIndexedCache.put(BTreePersistentIndexedCache.java:155)
at 
org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache$2.run(DefaultMultiProcessSafePersistentIndexedCache.java:51)
at 

[GitHub] kafka pull request: MINOR: removed obsolete class

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/843


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Improve error message for inconsistent ...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/847

MINOR: Improve error message for inconsistent broker ids

Provides a more actionable and descriptive error message. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka broker-id-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/847.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #847


commit ab9e5926c64a32a51bcfdbe8b330815745abfe42
Author: Grant Henke 
Date:   2016-02-01T22:49:36Z

MINOR: Improve error message for inconsistent broker ids




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3175 started by Mayuresh Gharat.
--
> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-3175:
---
Status: Patch Available  (was: In Progress)

> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3175 : Topic not accessible after deleti...

[MayureshGharat]

GitHub user MayureshGharat opened a pull request:

https://github.com/apache/kafka/pull/846

KAFKA-3175 : Topic not accessible after deletion even when 
delete.topic.enable is disabled

Remove topics under /admin/delete_topics path in zk if deleteTopic is 
disabled. The topic should never be enqueued for deletion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MayureshGharat/kafka kafka-3175

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/846.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #846


commit 48676b68df0f33c5c095c73002b7cca10b8a1402
Author: MayureshGharat 
Date:   2016-02-01T21:51:57Z

Remove topics under /admin/delete_topics path in zk if deleteTopic is 
disabled. The topic should never be enqueued for deletion




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127255#comment-15127255
 ] 

Jiangjie Qin commented on KAFKA-3177:
-

[~guozhang]  In this particular case, the issue is different from the actual no 
broker available scenario. Good point about verify the expected behavior of 
methods calls.

Also, should we make some improvements on the Exception definition, too? 
Currently some exception definition is a little confusing. e.g. 
NetworkException is a subclass of InvalidMatadataException; There is a 
StaleMetadataException while we already have more specific exception such as 
UnknownTopicOrPartitionException, NotLeaderForPartitionException, etc.

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3183) Add metrics for persistent store caching layer

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3183:


 Summary: Add metrics for persistent store caching layer
 Key: KAFKA-3183
 URL: https://issues.apache.org/jira/browse/KAFKA-3183
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang


We need to add the metrics collection such as cache hits / misses, cache size, 
dirty key size, etc for the RocksDBStore. However this may need to refactor the 
RocksDBStore a little bit since currently caching is not exposed to the 
MeteredKeyValueStore, and it uses an LRUCacheStore as the cache that does not 
keep the dirty key set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3185) Allow users to cleanup internal data

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3185:


 Summary: Allow users to cleanup internal data
 Key: KAFKA-3185
 URL: https://issues.apache.org/jira/browse/KAFKA-3185
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang


Currently the internal data is managed completely by Kafka Streams framework 
and users cannot clean them up actively. This results in a bad out-of-the-box 
user experience especially for running demo programs since it results internal 
data (changelog topics, RocksDB files, etc) that need to be cleaned manually. 
It will be better to add a

{code}
KafkaStreams.cleanup()
{code}

function call to clean up these internal data programmatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1860) File system errors are not detected unless Kafka tries to write

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1860:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> File system errors are not detected unless Kafka tries to write
> ---
>
> Key: KAFKA-1860
> URL: https://issues.apache.org/jira/browse/KAFKA-1860
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
> Fix For: 0.9.1.0
>
> Attachments: KAFKA-1860.patch
>
>
> When the disk (raid with caches dir) dies on a Kafka broker, typically the 
> filesystem gets mounted into read-only mode, and hence when Kafka tries to 
> read the disk, they'll get a FileNotFoundException with the read-only errno 
> set (EROFS).
> However, as long as there is no produce request received, hence no writes 
> attempted on the disks, Kafka will not exit on such FATAL error (when the 
> disk starts working again, Kafka might think some files are gone while they 
> will reappear later as raid comes back online). Instead it keeps spilling 
> exceptions like:
> {code}
> 2015/01/07 09:47:41.543 ERROR [KafkaScheduler] [kafka-scheduler-1] 
> [kafka-server] [] Uncaught exception in scheduled task 
> 'kafka-recovery-point-checkpoint'
> java.io.FileNotFoundException: 
> /export/content/kafka/i001_caches/recovery-point-offset-checkpoint.tmp 
> (Read-only file system)
>   at java.io.FileOutputStream.open(Native Method)
>   at java.io.FileOutputStream.(FileOutputStream.java:206)
>   at java.io.FileOutputStream.(FileOutputStream.java:156)
>   at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:37)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3187) Make kafka-acls.sh help messages more generic.

[Ashish K Singh (JIRA)]

Ashish K Singh created KAFKA-3187:
-

 Summary: Make kafka-acls.sh help messages more generic.
 Key: KAFKA-3187
 URL: https://issues.apache.org/jira/browse/KAFKA-3187
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ashish K Singh
Assignee: Ashish K Singh


Make kafka-acls.sh help messages generic, agnostic of {{SimpleAclsAuthorizer}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3186) Kafka authorizer should be aware of principal types it supports.

[Ashish K Singh (JIRA)]

Ashish K Singh created KAFKA-3186:
-

 Summary: Kafka authorizer should be aware of principal types it 
supports.
 Key: KAFKA-3186
 URL: https://issues.apache.org/jira/browse/KAFKA-3186
 Project: Kafka
  Issue Type: Improvement
Reporter: Ashish K Singh
Assignee: Ashish K Singh


Currently, Kafka authorizer is agnostic of principal types it supports, so are 
the acls CRUD methods in {{kafka.security.auth.Authorizer}}. The intent behind 
is to keep Kafka authorization pluggable, which is really great. However, this 
leads to following issues.

1. {{kafka-acls.sh}} supports pluggable authorizer and custom principals, 
however is some what integrated with {{SimpleAclsAuthorizer}}. The help 
messages has details which might not be true for a custom authorizer. For 
instance, assuming User is a supported PrincipalType.
2. Acls CRUD methods perform no check on validity of acls, as they are not 
aware of what principal types the support. This opens up space for lots of user 
errors, KAFKA-3097 is an instance.

I suggest we add a {{getSupportedPrincipalTypes}} method to authorizer and use 
that for acls verification during acls CRUD, and make {{kafka-acls.sh}} help 
messages more generic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Reorder StreamThread shutdown sequence

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/845

MINOR: Reorder StreamThread shutdown sequence

We need to close producer first before closing tasks to make sure all 
messages are acked and hence checkpoint offsets are updated before closing 
tasks and their state. It was re-ordered mistakenly before.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka KStreamState

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/845.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #845


commit 5acf1d526a4a2fdcf8d91c582fae8ca014bbadfc
Author: Guozhang Wang 
Date:   2016-02-01T21:44:59Z

Re-order close sequence




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Ismael Juma]

On Mon, Feb 1, 2016 at 7:04 PM, Gwen Shapira  wrote:

> Looking at "existing solutions", it looks like Zookeeper allows plugging in
> any SASL mechanism, but the server will only support one mechanism at a
> time.
>

This was the original proposal from Rajini as that is enough for their
needs.


> If this is good enough for our use-case (do we actually need to support
> multiple mechanisms at once?), it will simplify life a lot for us (
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL)
>

The current thinking is that it would be useful to support multiple SASL
mechanisms simultaneously. In the KIP meeting, Jun mentioned that companies
sometimes support additional authentication mechanisms for partners, for
example. It does make things more complex, as you say, so we need to be
sure the complexity is worth it.

Two more points:

1. It has been suggested that custom security protocol support is needed by
some (KIP-44). Rajini enhanced KIP-43 so that a SASL mechanism with a
custom provider can be used for this purpose instead. Given this, it seems
a bit inconsistent and restrictive not to allow multiple SASL mechanisms
simultaneously (we do allow SSL and SASL authentication simultaneously,
after all).

2. The other option would be to support a single SASL mechanism
simultaneously to start with and then extend this to multiple mechanisms
simultaneously later (if and when needed). It seems like it would be harder
to support the latter in the future if we go down this route, but maybe
there are ways around this.

Thoughts?

Ismael

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Ismael Juma]

+1 (non-binding)

Ismael

On Mon, Feb 1, 2016 at 10:23 PM, Jay Kreps  wrote:

> +1
>
> -Jay
>
> On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:
>
> > Hi All,
> >
> > I am opening the voting thread for KIP-42: Add Producer and Consumer
> > Interceptors.
> >
> > For reference, here's the KIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
> >
> > And the mailing list threads:
> > January:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> > February:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
> >
> >
> > Thanks,
> > Anna
> >
>

[jira] [Assigned] (KAFKA-3097) Acls for PrincipalType User are case sensitive

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh reassigned KAFKA-3097:
-

Assignee: Ashish K Singh

> Acls for PrincipalType User are case sensitive
> --
>
> Key: KAFKA-3097
> URL: https://issues.apache.org/jira/browse/KAFKA-3097
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>Assignee: Ashish K Singh
>
> I gave  a user acls for READ/WRITE but when I went to actually write to the 
> topic failed with auth exception. I figured out it was due to me specifying 
> the user as:  user:tgraves rather then User:tgraves.
> Seems like It should either fail on assign or be case insensitive.
> The principal type of User should also probably be documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1860 File system errors are not detected...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/698


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1860) File system errors are not detected unless Kafka tries to write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127177#comment-15127177
 ] 

ASF GitHub Bot commented on KAFKA-1860:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/698


> File system errors are not detected unless Kafka tries to write
> ---
>
> Key: KAFKA-1860
> URL: https://issues.apache.org/jira/browse/KAFKA-1860
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
> Fix For: 0.10.0.0
>
> Attachments: KAFKA-1860.patch
>
>
> When the disk (raid with caches dir) dies on a Kafka broker, typically the 
> filesystem gets mounted into read-only mode, and hence when Kafka tries to 
> read the disk, they'll get a FileNotFoundException with the read-only errno 
> set (EROFS).
> However, as long as there is no produce request received, hence no writes 
> attempted on the disks, Kafka will not exit on such FATAL error (when the 
> disk starts working again, Kafka might think some files are gone while they 
> will reappear later as raid comes back online). Instead it keeps spilling 
> exceptions like:
> {code}
> 2015/01/07 09:47:41.543 ERROR [KafkaScheduler] [kafka-scheduler-1] 
> [kafka-server] [] Uncaught exception in scheduled task 
> 'kafka-recovery-point-checkpoint'
> java.io.FileNotFoundException: 
> /export/content/kafka/i001_caches/recovery-point-offset-checkpoint.tmp 
> (Read-only file system)
>   at java.io.FileOutputStream.open(Native Method)
>   at java.io.FileOutputStream.(FileOutputStream.java:206)
>   at java.io.FileOutputStream.(FileOutputStream.java:156)
>   at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:37)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1006

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: removed obsolete class

[wangguoz] KAFKA-1860: Fix issue that file system errors are not detected unless

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-4 (docker Ubuntu ubuntu4 ubuntu yahoo-not-h2) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 66ecf3f08d7ce19c45fc16bf679df46cab349a8a 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 66ecf3f08d7ce19c45fc16bf679df46cab349a8a
 > git rev-list eaa1433a2f2a9e9d528f5a2620edb63304067f62 # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson7403151971489810901.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 13.244 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson8189474146882698500.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk7:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 17.325 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Neha Narkhede]

+1

On Mon, Feb 1, 2016 at 2:39 PM, Ismael Juma  wrote:

> +1 (non-binding)
>
> Ismael
>
> On Mon, Feb 1, 2016 at 10:23 PM, Jay Kreps  wrote:
>
> > +1
> >
> > -Jay
> >
> > On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:
> >
> > > Hi All,
> > >
> > > I am opening the voting thread for KIP-42: Add Producer and Consumer
> > > Interceptors.
> > >
> > > For reference, here's the KIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
> > >
> > > And the mailing list threads:
> > > January:
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> > > February:
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
> > >
> > >
> > > Thanks,
> > > Anna
> > >
> >
>



-- 
Thanks,
Neha

Build failed in Jenkins: kafka-trunk-jdk8 #335

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: Reorder StreamThread shutdown sequence

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-2 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 4adfd7960c4bef187454d1ff5186f3be690abbf5 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4adfd7960c4bef187454d1ff5186f3be690abbf5
 > git rev-list 57da044a991ebf8913d44dfcfa6a27729f54a4d5 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5502606352992513141.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 15.54 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson9189924083479446903.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 16.794 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

Build failed in Jenkins: kafka-trunk-jdk8 #334

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3060: Refactor MeteredStore and RockDBStore Impl

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 57da044a991ebf8913d44dfcfa6a27729f54a4d5 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 57da044a991ebf8913d44dfcfa6a27729f54a4d5
 > git rev-list 66ecf3f08d7ce19c45fc16bf679df46cab349a8a # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7538201926404582003.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 38.926 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7614613850353091518.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 42.39 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

[jira] [Created] (KAFKA-3188) Add integration test for KIP-31 and KIP-32

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3188:
---

 Summary: Add integration test for KIP-31 and KIP-32
 Key: KAFKA-3188
 URL: https://issues.apache.org/jira/browse/KAFKA-3188
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 0.9.0.0
Reporter: Jiangjie Qin
 Fix For: 0.10.0.0


The integration test should cover the followings:
1. Compatibility test.
2. Upgrade test
3. Changing message format type on the fly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3189) Kafka server always return UnknownServerException.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127475#comment-15127475
 ] 

Jiangjie Qin commented on KAFKA-3189:
-

[~junrao] Would like to see if you have any concern on moving all the 
exceptions to client exception class? It looks a simple replacement.

> Kafka server always return UnknownServerException.
> --
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while all exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> We should either switch back to Errors.forException() to 
> ErrorMapping.codeFor() or migrate all the exceptions to Java exception.
> I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3060) Refactor MeteredXXStore

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3060:
-
Assignee: Guozhang Wang

> Refactor MeteredXXStore
> ---
>
> Key: KAFKA-3060
> URL: https://issues.apache.org/jira/browse/KAFKA-3060
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
>Assignee: Guozhang Wang
>Priority: Minor
> Fix For: 0.9.1.0
>
>
> ** copied from a github comment by Guozhang Wang **
> The original motivation of having the MeteredXXStore is to wrap all metrics / 
> logging semantics into one place so they do not need to be re-implemented 
> again, but this seems to be an obstacle with the current pattern now, for 
> example MeteredWindowStore.putAndReturnInternalKey is only used for logging, 
> and MeteredWindowStore.putInternal / MeteredWindowStore.getInternal are never 
> used since only its inner will trigger this function. So how about 
> refactoring this piece as follows:
> 1. WindowStore only expose two APIs: put(K, V) and get(K, long).
> 2. Add a RollingRocksDBStores that does not extend any interface, but only 
> implements putInternal, getInternal and putAndReturnInternalKey that uses 
> underlying RocksDBStore as Segments.
> 3. RocksDBWindowStore implements WindowStore with an RollingRocksDBStores 
> inner.
> 4. Let MeteredXXStore only maintain the metrics recording logic, and let 
> different stores implement their own logging logic, since this is now 
> different across different types and are better handled separately. Also some 
> types of stores may not even have a loggingEnabled flag, if it will always 
> log, or will never log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3060) Refactor MeteredXXStore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127306#comment-15127306
 ] 

ASF GitHub Bot commented on KAFKA-3060:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/826


> Refactor MeteredXXStore
> ---
>
> Key: KAFKA-3060
> URL: https://issues.apache.org/jira/browse/KAFKA-3060
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
>Priority: Minor
> Fix For: 0.9.1.0
>
>
> ** copied from a github comment by Guozhang Wang **
> The original motivation of having the MeteredXXStore is to wrap all metrics / 
> logging semantics into one place so they do not need to be re-implemented 
> again, but this seems to be an obstacle with the current pattern now, for 
> example MeteredWindowStore.putAndReturnInternalKey is only used for logging, 
> and MeteredWindowStore.putInternal / MeteredWindowStore.getInternal are never 
> used since only its inner will trigger this function. So how about 
> refactoring this piece as follows:
> 1. WindowStore only expose two APIs: put(K, V) and get(K, long).
> 2. Add a RollingRocksDBStores that does not extend any interface, but only 
> implements putInternal, getInternal and putAndReturnInternalKey that uses 
> underlying RocksDBStore as Segments.
> 3. RocksDBWindowStore implements WindowStore with an RollingRocksDBStores 
> inner.
> 4. Let MeteredXXStore only maintain the metrics recording logic, and let 
> different stores implement their own logging logic, since this is now 
> different across different types and are better handled separately. Also some 
> types of stores may not even have a loggingEnabled flag, if it will always 
> log, or will never log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3060) Refactor MeteredXXStore

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3060.
--
   Resolution: Fixed
Fix Version/s: 0.9.1.0

Issue resolved by pull request 826
[https://github.com/apache/kafka/pull/826]

> Refactor MeteredXXStore
> ---
>
> Key: KAFKA-3060
> URL: https://issues.apache.org/jira/browse/KAFKA-3060
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
>Priority: Minor
> Fix For: 0.9.1.0
>
>
> ** copied from a github comment by Guozhang Wang **
> The original motivation of having the MeteredXXStore is to wrap all metrics / 
> logging semantics into one place so they do not need to be re-implemented 
> again, but this seems to be an obstacle with the current pattern now, for 
> example MeteredWindowStore.putAndReturnInternalKey is only used for logging, 
> and MeteredWindowStore.putInternal / MeteredWindowStore.getInternal are never 
> used since only its inner will trigger this function. So how about 
> refactoring this piece as follows:
> 1. WindowStore only expose two APIs: put(K, V) and get(K, long).
> 2. Add a RollingRocksDBStores that does not extend any interface, but only 
> implements putInternal, getInternal and putAndReturnInternalKey that uses 
> underlying RocksDBStore as Segments.
> 3. RocksDBWindowStore implements WindowStore with an RollingRocksDBStores 
> inner.
> 4. Let MeteredXXStore only maintain the metrics recording logic, and let 
> different stores implement their own logging logic, since this is now 
> different across different types and are better handled separately. Also some 
> types of stores may not even have a loggingEnabled flag, if it will always 
> log, or will never log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3060: Refactor MeteredStore and RockDBSt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/826


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Reorder StreamThread shutdown sequence

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/845


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk7 #1007

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-3189) Kafka server always return UnknownServerException.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3189:
---

 Summary: Kafka server always return UnknownServerException.
 Key: KAFKA-3189
 URL: https://issues.apache.org/jira/browse/KAFKA-3189
 Project: Kafka
  Issue Type: Bug
Reporter: Jiangjie Qin


This issue was introduced in KAFKA-2929. The problem is that we are using 
o.a.k.common.protocol.Errors.forException() while all exceptions thrown by the 
broker are still using old scala exception. This cause Errors.forException() 
always return UnknownServerException.

We should either switch back to Errors.forException() to ErrorMapping.codeFor() 
or migrate all the exceptions to Java exception.

I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3189) Kafka server always return UnknownServerException.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-3189:

Fix Version/s: 0.9.0.1
  Component/s: core

> Kafka server always return UnknownServerException.
> --
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while all exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> We should either switch back to Errors.forException() to 
> ErrorMapping.codeFor() or migrate all the exceptions to Java exception.
> I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

Gwen,

Yes, we should consider the support for different SASL mechanisms via
separate ports. At the moment, this is included under "*Rejected
alternatives*".  It was slightly lost in the text, so I have updated it. It
will be worth exploring this further.

The KIP considers three requirements:

   1. The addition of a non-Kerberos protocol into Kafka, at the moment
   this is SASL/PLAIN
   2. Enable pluggable SASL mechanisms including custom mechanisms
   3. Support for multiple mechanisms in a broker

1) and 2) work well on the existing SASL ports without any mechanism
exchange.
1) and 3) work well on multiple SASL ports as different security protocols

To combine 1), 2) and 3), we have two options

   - Add mechanism exchange to the wire protocol
   - Enable custom security protocols to be defined, change security
   protocol definitions into a String rather than enum and make ChannelBuilder
   configurable

I feel that mechanism exchange makes it easier for users to implement new
SASL mechanisms. But agree that different ports fit in better with the
existing Kafka endpoint selection design.

Thoughts?


On Mon, Feb 1, 2016 at 6:06 AM, Gwen Shapira  wrote:

> Could we support separate SASL mechanisms via separate ports?
>
> This was the main reason we separated the ports in first place, to avoid
> adding protocol negotiation protocol and allow the server to expect the
> correct packets when they arrive to the correct listener. It will be nice
> if new additions could use existing design and code.
>
> On Sun, Jan 31, 2016 at 6:48 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Harsha/Gwen,
> >
> > Thank you both for reviewing the KIP. SASL mechanism negotiation was
> added
> > specifically to support multiple SASL mechanisms within a broker. This
> was
> > brought up at the last KIP meeting and the consensus was that it would be
> > useful to support this. It would become harder to add multiple mechanism
> > support later on after support for other mechanisms are added in this
> KIP.
> >
> > I may have misunderstood Harsha's suggestion. This would use a
> > configuration option to specify one mechanism on the client-side and (one
> > or more?) mechanisms on the server-side. The server still needs to know
> the
> > mechanism that the client is using if it supports multiple mechanisms.
> And
> > this requires a message from the client to the server to send the
> mechanism
> > to the server. Since 0.9.0.0 servers expect to see GSSAPI tokens as soon
> as
> > the connection is established, the additional exchange is required to
> > ensure interoperability with 0.9.0.0. Since an exchange is necessary
> > anyway, sending the server mechanisms to the client seemed to make sense.
> > The current GSSAPI->GSSAPI flow remains exactly as it is now and is not
> > impacted by this KIP. Please let me know if I have missed something.
> >
> > The PR for Kafka-3149 (https://github.com/apache/kafka/pull/812)
> contains
> > the changes described in this KIP. It doesn't have tests for the new
> > features, just the code changes to help with the review.
> >
> >
> >
> > On Sun, Jan 31, 2016 at 12:11 AM, Gwen Shapira 
> wrote:
> >
> > > I want to second Harsha's approach. Configuration is usually much
> easier
> > to
> > > configure and troubleshoot than protocol-negotiation-protocols, which
> > we've
> > > been doing our best to generally avoid in Kafka due to operational
> > > complexity.
> > >
> > > Gwen
> > >
> > > On Sat, Jan 30, 2016 at 3:32 AM, Harsha  wrote:
> > >
> > > > Rajini,
> > > >   Whats the need for server returning acceptable mechs as
> > > >   network package. Why not drive it through the JAAS file
> > > >   itself. I don't see handshake any different than what it is
> > > >   now and only login will change and it can be configured
> based
> > > >   on the JAAS file.
> > > >
> > > > -Harsha
> > > >
> > > > On Fri, Jan 29, 2016, at 02:34 AM, Rajini Sivaram wrote:
> > > > > Ismael,
> > > > >
> > > > > The first packet from the client is deliberately empty to
> distinguish
> > > > > between non-negotiating GSSAPI packet and a negotiation packet. If
> > this
> > > > > packet contained mechanisms, then the code in the broker to
> > distinguish
> > > > > this from a GSSAPI packet would be a bit messy. I was thinking that
> > the
> > > > > client needs to wait for server response anyway before it can
> select
> > > > > a mechanism and start the actual SASL auth process. And once the
> > > > > client gets the server response, it would send the selected
> mechanism
> > > > > followed immediately by the first packet of the SASL auth. So
> perhaps
> > > the
> > > > > overhead is not that bad. Did you have a different flow in mind?
> > > > >
> > > > > On Fri, Jan 29, 2016 at 10:12 AM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Thanks Rajini. One question: would it make sense for the 

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Ismael Juma]

Hi Gwen,

On Mon, Feb 1, 2016 at 6:06 AM, Gwen Shapira  wrote:

> Could we support separate SASL mechanisms via separate ports?
>

This option was also discussed in the KIP and there are some advantages as
you say. However, there are some drawbacks as well.

This was the main reason we separated the ports in first place, to avoid
> adding protocol negotiation protocol and allow the server to expect the
> correct packets when they arrive to the correct listener. It will be nice
> if new additions could use existing design and code.
>

Yes, but the current design and code are not actually extensible as we use
an enum for `SecurityProtocol` and the creation of `ChannelBuilder` is
hardcoded based on that enum. Mapping different SASL mechanisms to
different ports is actually a bigger change to how the existing code works
than the option Rajini favoured in his KIP. This is not to say that we
should not do it, just clarifying that it's not necessarily an advantage in
terms of code reuse.

I would also like to point out that we do protocol negotiation in Kafka in
SSL/TLS today, so it is not totally without precedent. The server and
client negotiate an appropriate SSL/TLS version depending on options set by
both the client and server, not unlike what Rajini has proposed. Of course,
this is a more limited version as the protocols are related and the
negotiation is handled by the SSL/TLS libraries.

Ismael

[jira] [Comment Edited] (KAFKA-2607) Review `Time` interface and its usage

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941066#comment-14941066
 ] 

Ismael Juma edited comment on KAFKA-2607 at 2/1/16 10:11 AM:
-

We should probably do this after KAFKA-2606 (closed in favour of KAFKA-2247) to 
make the change simpler.


was (Author: ijuma):
We should probably do this after KAFKA-2606 (a duplicate of KAFKA-2247) to make 
the change simpler.

> Review `Time` interface and its usage
> -
>
> Key: KAFKA-2607
> URL: https://issues.apache.org/jira/browse/KAFKA-2607
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> Two of `Time` interface's methods are `milliseconds` and `nanoseconds` which 
> are implemented in `SystemTime` as follows:
> {code}
> @Override
> public long milliseconds() {
> return System.currentTimeMillis();
> }
> @Override
> public long nanoseconds() {
> return System.nanoTime();
> }
> {code}
> The issue with this interface is that it makes it seem that the difference is 
> about the unit (`ms` versus `ns`) whereas it's much more than that:
> https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> We should probably change the names of the methods and review our usage to 
> see if we're using the right one in the various places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125984#comment-15125984
 ] 

Ismael Juma commented on KAFKA-3086:


[~gwenshap], you said the following in the PR when I asked the compatibility 
question:

"The custom handler does not exist in 0.8.2, so this will only affect users who 
implemented it on 0.9.0.

I would prefer to remove it now (with an upgrade note). Those who use it will 
understand, since they must have noticed this method is never called."

https://github.com/apache/kafka/pull/758#issuecomment-171099641

And indeed, an upgrade note is included (although this was only committed to 
trunk in the end, so we should move the note to reflect that if it's what we 
want):

https://github.com/apache/kafka/pull/758/files

> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Ismael Juma]

Hi Gwen,

A few comments below.

On Mon, Feb 1, 2016 at 4:30 PM, Gwen Shapira  wrote:

> Thanks for clarifying, Ismael and Rajini. And I'm sorry for reopenning a
> point that was clearly discussed already.


Your input is definitely welcome. :) There was an initial discussion in the
KIP meeting, but it was understood that a wider and more detailed
discussion was needed before we could agree on the right solution.

1) Having two different ways to decide on the protocol (port + negotiation)
> is needlessly confusing for administrators. Its just one more complexity to
> figure out when authentication using SASL is already the most complex
> administrative task one has in Kafka (judging by the number of questions we
> get).
>

Yes, this is definitely a concern. It would be good to contrast this with
how the multiple ports option could look like (given pluggable mechanisms,
it won't be as simple as the existing config for multiple ports).

2) Troubleshooting. Especially in customer-support (and mailing-list
> support) environment. Asking for two configuration files and two netstat
> results is a completely different story than using tcp dump (which requires
> root privileges), catching the correct negotiation packets and decoding
> them to figure out what went wrong.
>

Right. Could we mitigate this somewhat with appropriate logging since we
control the negotiation process?

Ismael

P.S. It is interesting that Cassandra went in the other direction and made
it possible to use a single port for both encrypted and non-encrypted
traffic a few months ago:

https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=commit;h=535c3ac7

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Gwen Shapira]

Thanks for clarifying, Ismael and Rajini. And I'm sorry for reopenning a
point that was clearly discussed already. I wouldn't be doing this if I
didn't think my concerns are important and that the administrative aspect
may have been ignored (or de-prioritize). In my view, we should prioritize
simplicity for users over simplicity for developers, and I'm afraid we are
doing the reverse here.

I agree that supporting extensible ports is a more significant change.
There are two good reasons (and one weak reason) to put this effort in my
opinion:

1) Having two different ways to decide on the protocol (port + negotiation)
is needlessly confusing for administrators. Its just one more complexity to
figure out when authentication using SASL is already the most complex
administrative task one has in Kafka (judging by the number of questions we
get).

2) Troubleshooting. Especially in customer-support (and mailing-list
support) environment. Asking for two configuration files and two netstat
results is a completely different story than using tcp dump (which requires
root privileges), catching the correct negotiation packets and decoding
them to figure out what went wrong.

3) (weak reason): When we first added the multiple ports, people wanted to
make them more extensible for reasons other than security. For example,
allowing the use of different networks for clients and inter-broker
communication. So it may have other good uses.

Can you point to some examples of how other systems (preferably distributed
systems) support multiple SASL mechanisms?

Gwen

On Mon, Feb 1, 2016 at 1:25 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> On Mon, Feb 1, 2016 at 6:06 AM, Gwen Shapira  wrote:
>
> > Could we support separate SASL mechanisms via separate ports?
> >
>
> This option was also discussed in the KIP and there are some advantages as
> you say. However, there are some drawbacks as well.
>
> This was the main reason we separated the ports in first place, to avoid
> > adding protocol negotiation protocol and allow the server to expect the
> > correct packets when they arrive to the correct listener. It will be nice
> > if new additions could use existing design and code.
> >
>
> Yes, but the current design and code are not actually extensible as we use
> an enum for `SecurityProtocol` and the creation of `ChannelBuilder` is
> hardcoded based on that enum. Mapping different SASL mechanisms to
> different ports is actually a bigger change to how the existing code works
> than the option Rajini favoured in his KIP. This is not to say that we
> should not do it, just clarifying that it's not necessarily an advantage in
> terms of code reuse.
>
> I would also like to point out that we do protocol negotiation in Kafka in
> SSL/TLS today, so it is not totally without precedent. The server and
> client negotiate an appropriate SSL/TLS version depending on options set by
> both the client and server, not unlike what Rajini has proposed. Of course,
> this is a more limited version as the protocols are related and the
> negotiation is handled by the SSL/TLS libraries.
>
> Ismael
>

[jira] [Updated] (KAFKA-3181) Check occurrences of Runtime.halt() in the codebase and try to handle it at a single place

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-3181:
---
Description: Currently runtime.halt() is called at multiple places in the 
code. In an ideal case, we would throw an appropriate exception from classes 
and halt the runtime in a centralised place somewhere (so that we could 
override the behaviour during tests). For example : Right now test suit would 
just die halfway due to a halt call in ReplicaManager which, in turn, was due 
to an exception being thrown from this class 
(https://issues.apache.org/jira/browse/KAFKA-3063).

> Check occurrences of Runtime.halt() in the codebase and try to handle it at a 
> single place
> --
>
> Key: KAFKA-3181
> URL: https://issues.apache.org/jira/browse/KAFKA-3181
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>
> Currently runtime.halt() is called at multiple places in the code. In an 
> ideal case, we would throw an appropriate exception from classes and halt the 
> runtime in a centralised place somewhere (so that we could override the 
> behaviour during tests). For example : Right now test suit would just die 
> halfway due to a halt call in ReplicaManager which, in turn, was due to an 
> exception being thrown from this class 
> (https://issues.apache.org/jira/browse/KAFKA-3063).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126609#comment-15126609
 ] 

Jason Gustafson commented on KAFKA-3177:


[~becket_qin] Nice find. Your suggestion seems reasonable to me. As a user, I 
would probably expect to see an exception if a partition doesn't exist.

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin reassigned KAFKA-3177:
---

Assignee: Jiangjie Qin

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3177 started by Jiangjie Qin.
---
> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3180) issue for extracting JSON from https web page

[swayam (JIRA)]

swayam created KAFKA-3180:
-

 Summary: issue for extracting JSON from https web page
 Key: KAFKA-3180
 URL: https://issues.apache.org/jira/browse/KAFKA-3180
 Project: Kafka
  Issue Type: Test
  Components: clients
Affects Versions: 0.9.0.0
 Environment: cloudera 5.4.2.0
Reporter: swayam
Priority: Critical


Hi Team,

Could you help me how to extract JSON info from https web page by help of kafka 
into HDFS . 

here is the json available URL : 
https://affiliate-api.flipkart.net/affiliate/api/8924b177d4c64fcab4db860b94fbcea2.json

Please help me to get the info ..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Pluggable Log Compaction Policy

[Becket Qin]

Hi Bill,

The PR is still under review. It might take some more time because it
touches a bunch of files. You can watch KAFKA-3025 so once it gets closed
you will get email notification.
Looking forward to your tool.

Thanks,

Jiangjie (Becket) Qin

On Mon, Feb 1, 2016 at 6:54 AM, Bill Warshaw 
wrote:

> Becket,
>
> I took a look at KIP-32 and your PR for it.  This looks like something that
> would be great to build off of; I'm envisioning a timestamp-based policy
> where the client application sets a minimum timestamp, before which
> everything can be deleted / compacted.  How far along is this pull request?
>
> Bill Warshaw
>
> On Fri, Jan 22, 2016 at 12:41 AM, Becket Qin  wrote:
>
> > I agree with Guozhang that this seems better to be a separate tool.
> >
> > Also, I am wondering if KIP-32 can be used here. We can have a timestamp
> > based compaction policy if needed, for example, keep any message whose
> > timestamp is greater than (MaxTimestamp - 24 hours).
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Thu, Jan 21, 2016 at 4:35 PM, Guozhang Wang 
> wrote:
> >
> > > Bill,
> > >
> > > For your case since once the log is cleaned up to the given offset
> > > watermark (or threshold, whatever the name is), future cleaning with
> the
> > > same watermark will effectively be a no-op, so I feel your scenario
> will
> > be
> > > better fit as a one-time admin tool to cleanup the logs rather than
> > > customizing the periodic cleaning policy. Does this sound reasonable to
> > > you?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, Jan 20, 2016 at 7:09 PM, Bill Warshaw  >
> > > wrote:
> > >
> > > > For our particular use case, we would need to.  This proposal is
> really
> > > two
> > > > separate pieces:  custom log compaction policy, and the ability to
> set
> > > > arbitrary key-value pairs in a Topic configuration.
> > > >
> > > > I believe that Kafka's current behavior of throwing errors when it
> > > > encounters configuration keys that aren't defined is meant to help
> > users
> > > > not misconfigure their configuration files.  If that is the sole
> > > motivation
> > > > for it, I would propose adding a property namespace, and allow users
> to
> > > > configure arbitrary properties behind that particular namespace,
> while
> > > > still enforcing strict parsing for all other properties.
> > > >
> > > > On Wed, Jan 20, 2016 at 9:23 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > So do you need to periodically update the key-value pairs to
> "advance
> > > the
> > > > > threshold for each topic"?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Wed, Jan 20, 2016 at 5:51 PM, Bill Warshaw <
> > bill.wars...@appian.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Compaction would be performed in the same manner as it is
> > currently.
> > > > > There
> > > > > > is a predicate applied in the "shouldRetainMessage" function in
> > > > > LogCleaner;
> > > > > > ultimately we just want to be able to swap a custom
> implementation
> > of
> > > > > that
> > > > > > particular method in.  Nothing else in the compaction codepath
> > would
> > > > need
> > > > > > to change.
> > > > > >
> > > > > > For advancing the "threshold transaction_id", ideally we would be
> > > able
> > > > to
> > > > > > set arbitrary key-value pairs on the topic configuration.  We
> have
> > > > access
> > > > > > to the topic configuration during log compaction, so a custom
> > policy
> > > > > class
> > > > > > would also have access to that config, and could read anything we
> > > > stored
> > > > > in
> > > > > > there.
> > > > > >
> > > > > > On Wed, Jan 20, 2016 at 8:14 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hello Bill,
> > > > > > >
> > > > > > > Just to clarify your use case, is your "log compaction"
> executed
> > > > > > manually,
> > > > > > > or it is triggered periodically like the current log cleaning
> > > by-key
> > > > > > does?
> > > > > > > If it is the latter case, how will you advance the "threshold
> > > > > > > transaction_id" each time when it executes?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jan 20, 2016 at 1:50 PM, Bill Warshaw <
> > > > bill.wars...@appian.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Damian, I appreciate your quick response.
> > > > > > > >
> > > > > > > > Our transaction_id is incrementing for each transaction, so
> we
> > > will
> > > > > > only
> > > > > > > > ever have one message in Kafka with a given transaction_id.
> We
> > > > > thought
> > > > > > > > about using a rolling counter that is incremented on each
> > > > checkpoint
> > > > > as
> > > > > > > the
> > > > > > > > key, and manually triggering compaction after the checkpoint
> is
> > > > > > complete,
> > > > > > > > but our checkpoints are asynchronous.  This means that we
> would
> > 

[jira] [Updated] (KAFKA-3189) Kafka server always return UnknownServerException.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3189:
---
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0

> Kafka server always return UnknownServerException.
> --
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while all exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> We should either switch back to Errors.forException() to 
> ErrorMapping.codeFor() or migrate all the exceptions to Java exception.
> I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: 0.9.0

[quanrs]

GitHub user quanrs opened a pull request:

https://github.com/apache/kafka/pull/848

0.9.0



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/kafka 0.9.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/848.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #848


commit 7710b367fd26a0c41565f35200748c23616b4477
Author: Gwen Shapira 
Date:   2015-11-07T03:46:30Z

Changing version to 0.9.0.0

commit 27d44afe664bff45d62f72335fdbb56671561512
Author: Jason Gustafson 
Date:   2015-11-08T19:38:50Z

KAFKA-2723: new consumer exception cleanup (0.9.0)

Author: Jason Gustafson 

Reviewers: Guozhang Wang

Closes #452 from hachikuji/KAFKA-2723

commit 32cd3e35f1ea8251a51860cc48a44fb2fbfd7c0e
Author: Jason Gustafson 
Date:   2015-11-08T20:36:42Z

HOTFIX: fix group coordinator edge cases around metadata storage callback 
(0.9.0)

Author: Jason Gustafson 

Reviewers: Guozhang Wang

Closes #453 from hachikuji/hotfix-group-coordinator-0.9

commit 1fd79f57b4a73308c59b797974086ca09af19b98
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T04:41:35Z

KAFKA-2480: Handle retriable and non-retriable exceptions thrown by sink 
tasks.

Author: Ewen Cheslack-Postava 

Reviewers: Gwen Shapira

Closes #450 from ewencp/kafka-2480-unrecoverable-task-errors

(cherry picked from commit f4b87deefecf4902992a84d4a3fe3b99a94ff72b)
Signed-off-by: Gwen Shapira 

commit 48013222fd426685d2907a760290d2e7c7d25aea
Author: Geoff Anderson 
Date:   2015-11-09T04:52:16Z

KAFKA-2773; 0.9.0 branch)Fixed broken vagrant provision scripts for static 
zk/broker cluster

Author: Geoff Anderson 

Reviewers: Gwen Shapira

Closes #455 from granders/KAFKA-2773-0.9.0-vagrant-fix

commit 417e283d643d8865aa3e79dffa373c8cc853d78f
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T06:11:03Z

KAFKA-2774: Rename Copycat to Kafka Connect

Author: Ewen Cheslack-Postava 

Reviewers: Gwen Shapira

Closes #456 from ewencp/kafka-2774-rename-copycat

(cherry picked from commit f2031d40639ef34c1591c22971394ef41c87652c)
Signed-off-by: Gwen Shapira 

commit 02fbdaa4475fd12a0fdccaa103bf27cbc1bfd077
Author: Rajini Sivaram 
Date:   2015-11-09T15:23:47Z

KAFKA-2779; Close SSL socket channel on remote connection close

Close socket channel in finally block to avoid file descriptor leak when 
remote end closes the connection

Author: Rajini Sivaram 

Reviewers: Ismael Juma , Jun Rao 

Closes #460 from rajinisivaram/KAFKA-2779

(cherry picked from commit efbebc6e843850b7ed9a1d015413c99f114a7d92)
Signed-off-by: Jun Rao 

commit fdefef9536acf8569607a980a25237ef4794f645
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T17:10:20Z

KAFKA-2781; Only require signing artifacts when uploading archives.

Author: Ewen Cheslack-Postava 

Reviewers: Jun Rao 

Closes #461 from ewencp/kafka-2781-no-signing-for-install

(cherry picked from commit a24f9a23a6d8759538e91072e8d96d158d03bb63)
Signed-off-by: Jun Rao 

commit 7471394c5485a2114d35c6345d95e161a0ee6586
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T18:19:27Z

KAFKA-2776: Fix lookup of schema conversion cache size in JsonConverter.

Author: Ewen Cheslack-Postava 

Reviewers: Gwen Shapira

Closes #458 from ewencp/kafka-2776-json-converter-cache-config-fix

(cherry picked from commit e9fc7b8c84908ae642339a2522a79f8bb5155728)
Signed-off-by: Gwen Shapira 

commit 3aa3e85d942b514cbe842a6b3c3fe214c0ecf401
Author: Jason Gustafson 
Date:   2015-11-09T18:26:17Z

HOTFIX: bug updating cache when loading group metadata

The bug causes only the first instance of group metadata in the topic to be 
written to the cache (because of the putIfNotExists in addGroup). Coordinator 
fail-over won't work properly unless the cache is loaded with the right 
metadata.

Author: Jason Gustafson 

Reviewers: Guozhang Wang

Closes #462 from hachikuji/hotfix-group-loading

(cherry picked from commit 2b04004de878823fe631af1f3f85108c0b38caec)
Signed-off-by: Guozhang Wang 

[jira] [Created] (KAFKA-3191) Improve offset committing JavaDoc in KafkaConsumer

[Adam Kunicki (JIRA)]

Adam Kunicki created KAFKA-3191:
---

 Summary: Improve offset committing JavaDoc in KafkaConsumer
 Key: KAFKA-3191
 URL: https://issues.apache.org/jira/browse/KAFKA-3191
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Affects Versions: 0.9.0.0
Reporter: Adam Kunicki
Assignee: Neha Narkhede
Priority: Minor


http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
 includes some great examples of how to manage offset commits in Kafka 0.9.

It is currently unclear from the JavaDoc that the following is expected:

{code}
consumer.commitSync(Collections.singletonMap(record.partition(), new
OffsetAndMetadata(record.offset() + 1)));
{code}

The committed offset should always be the offset of the next message that
your application will read.

This information should be incorporated into the existing JavaDoc to make it 
clearer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3189) Kafka server always return UnknownServerException.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127813#comment-15127813
 ] 

Ismael Juma commented on KAFKA-3189:


Changed the fix version to 0.9.1.0 as it doesn't affect the 0.9.0 branch. The 
aim of KAFKA-2929 was to switch the broker to the Java exceptions. cc 
[~granthenke]

> Kafka server always return UnknownServerException.
> --
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while all exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> We should either switch back to Errors.forException() to 
> ErrorMapping.codeFor() or migrate all the exceptions to Java exception.
> I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1696) Kafka should be able to generate Hadoop delegation tokens

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127639#comment-15127639
 ] 

Gwen Shapira commented on KAFKA-1696:
-

[~harsha_ch], you are following [~rsivaram]'s KIP on new SASL mechanisms, 
right? 

Hadoop implements Delegation Tokens as a SASL mechanism and I was wondering if 
you were planning on doing the same? If so, you may have an opinion on how the 
mechanism is determined. If you are not planning on hooking into SASL, then 
never mind :)





> Kafka should be able to generate Hadoop delegation tokens
> -
>
> Key: KAFKA-1696
> URL: https://issues.apache.org/jira/browse/KAFKA-1696
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Parth Brahmbhatt
>
> For access from MapReduce/etc jobs run on behalf of a user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Dana Powers (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127701#comment-15127701
 ] 

Dana Powers commented on KAFKA-3177:


A similar infinite loop happens when the partition exists but has no leader b/c 
it is under-replicated. In that case, Fetcher.listOffset infinitely retries on 
the leaderNotAvailableError returned by sendListOffsetRequest.

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3190) KafkaProducer should not invoke callback in send()

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3190:
---

 Summary: KafkaProducer should not invoke callback in send()
 Key: KAFKA-3190
 URL: https://issues.apache.org/jira/browse/KAFKA-3190
 Project: Kafka
  Issue Type: Bug
  Components: clients, producer 
Affects Versions: 0.9.0.0
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin
 Fix For: 0.9.0.1


Currently KafkaProducer will invoke callback.onComplete() if it receives an 
ApiException during send(). This breaks the guarantee that callback will be 
invoked in order. It seems ApiException in send() only comes from metadata 
refresh. If so, we can probably simply throw it instead of invoking callback().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3186) Kafka authorizer should be aware of principal types it supports.

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127632#comment-15127632
 ] 

Gwen Shapira commented on KAFKA-3186:
-

I assume you are referring to the fact that Sentry also manages Groups and 
Roles, so a Principal may be a group?

Is this specifically for the CLI, or are there other areas that are a problem? 
I'm asking because unlike the authorizer API which is pluggable, the CLI is 
specific to the defaultAuthorizer. We are assuming that Sentry and Ranger users 
will use whatever GUI / CLI is provided by Sentry and Ranger. Does that make 
sense? Or are you talking about something else entirely?

> Kafka authorizer should be aware of principal types it supports.
> 
>
> Key: KAFKA-3186
> URL: https://issues.apache.org/jira/browse/KAFKA-3186
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Currently, Kafka authorizer is agnostic of principal types it supports, so 
> are the acls CRUD methods in {{kafka.security.auth.Authorizer}}. The intent 
> behind is to keep Kafka authorization pluggable, which is really great. 
> However, this leads to following issues.
> 1. {{kafka-acls.sh}} supports pluggable authorizer and custom principals, 
> however is some what integrated with {{SimpleAclsAuthorizer}}. The help 
> messages has details which might not be true for a custom authorizer. For 
> instance, assuming User is a supported PrincipalType.
> 2. Acls CRUD methods perform no check on validity of acls, as they are not 
> aware of what principal types the support. This opens up space for lots of 
> user errors, KAFKA-3097 is an instance.
> I suggest we add a {{getSupportedPrincipalTypes}} method to authorizer and 
> use that for acls verification during acls CRUD, and make {{kafka-acls.sh}} 
> help messages more generic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Ewen Cheslack-Postava]

+1

-Ewen

On Mon, Feb 1, 2016 at 8:06 PM, Gwen Shapira  wrote:

> +1
>
> And seconding Becket, the wiki is great - comprehensive and readable.
>
> On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:
>
> > Hi All,
> >
> > I am opening the voting thread for KIP-42: Add Producer and Consumer
> > Interceptors.
> >
> > For reference, here's the KIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
> >
> > And the mailing list threads:
> > January:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> > February:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
> >
> >
> > Thanks,
> > Anna
> >
>



-- 
Thanks,
Ewen

Re: [VOTE] KIP-42 -- Add Producer and Consumer Interceptors.

[Gwen Shapira]

+1

And seconding Becket, the wiki is great - comprehensive and readable.

On Mon, Feb 1, 2016 at 12:00 PM, Anna Povzner  wrote:

> Hi All,
>
> I am opening the voting thread for KIP-42: Add Producer and Consumer
> Interceptors.
>
> For reference, here's the KIP wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
>
> And the mailing list threads:
> January:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3ccabkn0ivlwdayuzwdkc_ebiplzqqcajcmavmb1utznovmuts...@mail.gmail.com%3e
> February:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3cCABkN0iWFJoPZ29zbAE9RkrFjCk_=ZA=o3zv-2rteut+mbwo...@mail.gmail.com%3e
>
>
> Thanks,
> Anna
>

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[tao xiao]

I am the author of KIP-44. I hope my use case will add some values to this
discussion. The reason I raised KIP44 is that I want to be able to
implement a custom security protocol that can fulfill the need of my
company. As pointed out by Ismael KIP-43 now supports a pluggable way to
inject custom security provider to SASL I think it is enough to cover the
use case I have and address the concerns raised in KIP-44.

For multiple security protocols support simultaneously it is not needed in
my use case and I don't foresee it is needed in the future but as i said
this is my use case only there may be other use cases that need it. But if
we want to support it in the future I prefer to get it right at the first
place given the fact that security protocol is an ENUM and if we stick to
that implementation it is very hard to extend in the future when we decide
multiple security protocols is needed.

Protocol negotiation is a very common usage pattern in security domain. As
suggested in Java SASL doc
http://docs.oracle.com/javase/7/docs/technotes/guides/security/sasl/sasl-refguide.html
client
first sends out a packet to server and server responds with a list of
mechanisms it supports. This is very similar to SSL/TLS negotiation.

On Tue, 2 Feb 2016 at 06:39 Ismael Juma  wrote:

> On Mon, Feb 1, 2016 at 7:04 PM, Gwen Shapira  wrote:
>
> > Looking at "existing solutions", it looks like Zookeeper allows plugging
> in
> > any SASL mechanism, but the server will only support one mechanism at a
> > time.
> >
>
> This was the original proposal from Rajini as that is enough for their
> needs.
>
>
> > If this is good enough for our use-case (do we actually need to support
> > multiple mechanisms at once?), it will simplify life a lot for us (
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL
> )
> >
>
> The current thinking is that it would be useful to support multiple SASL
> mechanisms simultaneously. In the KIP meeting, Jun mentioned that companies
> sometimes support additional authentication mechanisms for partners, for
> example. It does make things more complex, as you say, so we need to be
> sure the complexity is worth it.
>
> Two more points:
>
> 1. It has been suggested that custom security protocol support is needed by
> some (KIP-44). Rajini enhanced KIP-43 so that a SASL mechanism with a
> custom provider can be used for this purpose instead. Given this, it seems
> a bit inconsistent and restrictive not to allow multiple SASL mechanisms
> simultaneously (we do allow SSL and SASL authentication simultaneously,
> after all).
>
> 2. The other option would be to support a single SASL mechanism
> simultaneously to start with and then extend this to multiple mechanisms
> simultaneously later (if and when needed). It seems like it would be harder
> to support the latter in the future if we go down this route, but maybe
> there are ways around this.
>
> Thoughts?
>
> Ismael
>

[GitHub] kafka pull request: MINOR: removed obsolete class

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/843

MINOR: removed obsolete class

@guozhangwang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka remove_unused

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #843


commit 74aeda08c64f67db79c0f7845d3ff6ea83890d50
Author: Yasuhiro Matsuda 
Date:   2016-02-01T18:47:36Z

MINOR: removed obsolete class




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Pluggable Log Compaction Policy

[Bill Warshaw]

Becket,

I took a look at KIP-32 and your PR for it.  This looks like something that
would be great to build off of; I'm envisioning a timestamp-based policy
where the client application sets a minimum timestamp, before which
everything can be deleted / compacted.  How far along is this pull request?

Bill Warshaw

On Fri, Jan 22, 2016 at 12:41 AM, Becket Qin  wrote:

> I agree with Guozhang that this seems better to be a separate tool.
>
> Also, I am wondering if KIP-32 can be used here. We can have a timestamp
> based compaction policy if needed, for example, keep any message whose
> timestamp is greater than (MaxTimestamp - 24 hours).
>
> Jiangjie (Becket) Qin
>
>
>
> On Thu, Jan 21, 2016 at 4:35 PM, Guozhang Wang  wrote:
>
> > Bill,
> >
> > For your case since once the log is cleaned up to the given offset
> > watermark (or threshold, whatever the name is), future cleaning with the
> > same watermark will effectively be a no-op, so I feel your scenario will
> be
> > better fit as a one-time admin tool to cleanup the logs rather than
> > customizing the periodic cleaning policy. Does this sound reasonable to
> > you?
> >
> >
> > Guozhang
> >
> >
> > On Wed, Jan 20, 2016 at 7:09 PM, Bill Warshaw 
> > wrote:
> >
> > > For our particular use case, we would need to.  This proposal is really
> > two
> > > separate pieces:  custom log compaction policy, and the ability to set
> > > arbitrary key-value pairs in a Topic configuration.
> > >
> > > I believe that Kafka's current behavior of throwing errors when it
> > > encounters configuration keys that aren't defined is meant to help
> users
> > > not misconfigure their configuration files.  If that is the sole
> > motivation
> > > for it, I would propose adding a property namespace, and allow users to
> > > configure arbitrary properties behind that particular namespace, while
> > > still enforcing strict parsing for all other properties.
> > >
> > > On Wed, Jan 20, 2016 at 9:23 PM, Guozhang Wang 
> > wrote:
> > >
> > > > So do you need to periodically update the key-value pairs to "advance
> > the
> > > > threshold for each topic"?
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Jan 20, 2016 at 5:51 PM, Bill Warshaw <
> bill.wars...@appian.com
> > >
> > > > wrote:
> > > >
> > > > > Compaction would be performed in the same manner as it is
> currently.
> > > > There
> > > > > is a predicate applied in the "shouldRetainMessage" function in
> > > > LogCleaner;
> > > > > ultimately we just want to be able to swap a custom implementation
> of
> > > > that
> > > > > particular method in.  Nothing else in the compaction codepath
> would
> > > need
> > > > > to change.
> > > > >
> > > > > For advancing the "threshold transaction_id", ideally we would be
> > able
> > > to
> > > > > set arbitrary key-value pairs on the topic configuration.  We have
> > > access
> > > > > to the topic configuration during log compaction, so a custom
> policy
> > > > class
> > > > > would also have access to that config, and could read anything we
> > > stored
> > > > in
> > > > > there.
> > > > >
> > > > > On Wed, Jan 20, 2016 at 8:14 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Hello Bill,
> > > > > >
> > > > > > Just to clarify your use case, is your "log compaction" executed
> > > > > manually,
> > > > > > or it is triggered periodically like the current log cleaning
> > by-key
> > > > > does?
> > > > > > If it is the latter case, how will you advance the "threshold
> > > > > > transaction_id" each time when it executes?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 20, 2016 at 1:50 PM, Bill Warshaw <
> > > bill.wars...@appian.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Damian, I appreciate your quick response.
> > > > > > >
> > > > > > > Our transaction_id is incrementing for each transaction, so we
> > will
> > > > > only
> > > > > > > ever have one message in Kafka with a given transaction_id.  We
> > > > thought
> > > > > > > about using a rolling counter that is incremented on each
> > > checkpoint
> > > > as
> > > > > > the
> > > > > > > key, and manually triggering compaction after the checkpoint is
> > > > > complete,
> > > > > > > but our checkpoints are asynchronous.  This means that we would
> > > have
> > > > a
> > > > > > set
> > > > > > > of messages appended to the log after the checkpoint started,
> > with
> > > > > value
> > > > > > of
> > > > > > > the previous key + 1, that would also be compacted down to a
> > single
> > > > > > entry.
> > > > > > >
> > > > > > > Our particular custom policy would delete all messages whose
> key
> > > was
> > > > > less
> > > > > > > than a given transaction_id that we passed in.  I can imagine a
> > > wide
> > > > > > > variety of other custom policies that could be used for
> retention
> > > > based
> > > > > > on
> > > > > > > the key and value of the 

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Todd Palino (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126292#comment-15126292
 ] 

Todd Palino commented on KAFKA-3174:


Yeah, definitely no problems with Java 1.8. We've been running 1.8 u5 for quite 
some time, and we're in the process of updating to u40. It's worth noting that 
we have been running into a number of SEGVs with mirror maker (but not the 
broker) under u5, but the problem is supposedly fixed in u40.

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3175:
---
Fix Version/s: 0.9.0.1

> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126407#comment-15126407
 ] 

Jun Rao commented on KAFKA-3175:


Also, another thing is that if topic deletion is not enabled, the controller 
doesn't subscribe to the topic deletion path. This means that those to be 
deleted topics in the topic deletion path never get removed. If people enable 
topic deletion, those topics will be suddenly removed, which may be surprising. 
Perhaps we should let controller always subscribe to the topic deletion path, 
if topic deletion is not enabled, the controller can simply remove those to be 
deleted topics without doing any real work.

> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3159:
---
Fix Version/s: 0.9.0.1

> Kafka consumer 0.9.0.0  client poll is very CPU intensive under certain 
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
>
> We are using the new kafka consumer with the following config (as logged by 
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id = 
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call 
> poll in a loop.  We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get 
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this 
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread 
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all 
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here timeout is about 10 seconds, so it is pretty big.
> records = consumer.poll(timeout);
> } catch (Exception e) {
>// This never hits for us
> logger.error("Exception polling Kafka ", e);
> records = null;
> }
> if (records != null) {
> for (ConsumerRecord record : records) {
>// The handler puts the byte array on a very fast ring buffer 
> so it barely takes any time.
> handler.handleMessage(ByteBuffer.wrap(record.value()));
> }
> }
> }
> With this setup our performance has taken a horrendous hit as soon as we 
> started this one thread that just polls Kafka in a loop.
> I profiled the application using Java Mission Control and have a few insights.
> 1. There doesn't seem to be a single hotspot. The consumer just ends up using 
> a lot of CPU for handing such a low number of messages. Our process was using 
> 16% CPU before we added a single consumer and it went to 25% and above after. 
> That's an increase of over 50% from a single consumer getting a single digit 
> number of small messages per second. Here is an attachment of the cpu usage 
> breakdown in the consumer (the namespace is different because we shade the 
> kafka jar before using it) - http://imgur.com/BxWs9Q0 So 20.54% of our entire 
> process CPU is used on polling these 64 partitions 

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126346#comment-15126346
 ] 

Ismael Juma commented on KAFKA-3174:


[~becket_qin] We have started recommending Java 8 around the same time we 
released 0.9.0.0 (we also mention that LinkedIn is using Java 8 there):

http://kafka.apache.org/documentation.html#java

I did some investigation so that we understand the specifics of the improvement 
to CRC32 in the JDK. It relies on SSE 2, SSE 4.1, AVX and CLMUL. SSE has been 
available for a long time, CLMUL since Intel Westmere (2010) and AVX since 
Intel Sandy Bridge (2011). It's probably OK to assume that these instructions 
will be available for those who are constrained by CPU performance.

Note that this is not using CRC32 CPU instruction as we would have to use 
CRC32C for that (see KAFKA-1449 for more details on what is possible if we are 
willing to support CRC32C).

I wrote a simple JMH benchmark:

https://gist.github.com/ijuma/f86ad935715cfd4e258e

I tested it on my Ivy Bridge MacBook on JDK 7 update 80 and JDK 8 update 76, 
configuring JMH to use 10 one second measurement iterations, 10 one second 
warmup iterations and 1 fork.

JDK 8 update 76 results:

{code}
[info] Benchmark  (bytesSize)  Mode  Cnt   Score   Error  
Units
[info] Crc32Bench.jdkCrc32  8  avgt   10  24.902 ± 0.728  
ns/op
[info] Crc32Bench.jdkCrc32 16  avgt   10  48.819 ± 2.550  
ns/op
[info] Crc32Bench.jdkCrc32 32  avgt   10  83.434 ± 2.668  
ns/op
[info] Crc32Bench.jdkCrc32128  avgt   10 127.679 ± 5.185  
ns/op
[info] Crc32Bench.jdkCrc32   1024  avgt   10 450.105 ±18.943  
ns/op
[info] Crc32Bench.jdkCrc32  65536  avgt   10   25579.406 ±   683.017  
ns/op
[info] Crc32Bench.jdkCrc321048576  avgt   10  408708.242 ± 12183.543  
ns/op
[info] Crc32Bench.kafkaCrc328  avgt   10  14.761 ± 0.647  
ns/op
[info] Crc32Bench.kafkaCrc32   16  avgt   10  19.114 ± 0.423  
ns/op
[info] Crc32Bench.kafkaCrc32   32  avgt   10  34.243 ± 1.066  
ns/op
[info] Crc32Bench.kafkaCrc32  128  avgt   10 114.481 ± 2.812  
ns/op
[info] Crc32Bench.kafkaCrc32 1024  avgt   10 835.630 ±22.412  
ns/op
[info] Crc32Bench.kafkaCrc3265536  avgt   10   52234.713 ±  2229.624  
ns/op
[info] Crc32Bench.kafkaCrc32  1048576  avgt   10  822903.613 ± 20950.560  
ns/op
{code}

JDK 7 update 80 results:

{code}
[info] Benchmark  (bytesSize)  Mode  Cnt   Score   Error  
Units
[info] Crc32Bench.jdkCrc32  8  avgt   10 114.802 ± 8.289  
ns/op
[info] Crc32Bench.jdkCrc32 16  avgt   10 122.030 ± 3.153  
ns/op
[info] Crc32Bench.jdkCrc32 32  avgt   10 131.082 ± 5.501  
ns/op
[info] Crc32Bench.jdkCrc32128  avgt   10 154.116 ± 6.164  
ns/op
[info] Crc32Bench.jdkCrc32   1024  avgt   10 512.151 ±15.934  
ns/op
[info] Crc32Bench.jdkCrc32  65536  avgt   10   25460.014 ±  1532.627  
ns/op
[info] Crc32Bench.jdkCrc321048576  avgt   10  401996.290 ± 18606.012  
ns/op
[info] Crc32Bench.kafkaCrc328  avgt   10  14.493 ± 0.494  
ns/op
[info] Crc32Bench.kafkaCrc32   16  avgt   10  20.329 ± 2.019  
ns/op
[info] Crc32Bench.kafkaCrc32   32  avgt   10  37.706 ± 0.338  
ns/op
[info] Crc32Bench.kafkaCrc32  128  avgt   10 124.197 ± 6.368  
ns/op
[info] Crc32Bench.kafkaCrc32 1024  avgt   10 908.327 ±32.487  
ns/op
[info] Crc32Bench.kafkaCrc3265536  avgt   10   57000.705 ±  2976.852  
ns/op
[info] Crc32Bench.kafkaCrc32  1048576  avgt   10  940433.528 ± 26257.962  
ns/op
{code}

Using a VM intrinsic avoids JNI set-up costs making JDK 8 much faster than JDK 
7 for small byte arrays. Having said that, Kafka's pure Java implementation is 
still faster for byte arrays of up to 128 bytes according to this benchmark. 
Surprisingly, the results are similar for JDK 7 and JDK 8 for larger byte 
arrays. I had a quick look at the assembly generated for JDK 8 and it seems to 
use AVX and CLMUL as per the OpenJDK commit I linked to. Unfortunately, it's a 
bit more work to look at the assembly generated by JDK 7 on a Mac and so I 
didn't. More investigation would be required to understand why this is so (and 
to be able to trust the numbers).

Looking at how we compute CRCs in `Record`, there are two different code paths 
depending on whether we call it from `Compressor` or not. The former invokes 
Crc32 update methods several times (both the byte array and int versions) while 
the latter invokes the byte array version once only.

To really understand the impact of this change, I think we need to benchmark 
the producer with varying message sizes with both 

[jira] [Comment Edited] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126346#comment-15126346
 ] 

Ismael Juma edited comment on KAFKA-3174 at 2/1/16 3:02 PM:


[~becket_qin] We have started recommending Java 8 around the same time we 
released 0.9.0.0 (we also mention that LinkedIn is using Java 8 there):

http://kafka.apache.org/documentation.html#java

I did some investigation so that we understand the specifics of the improvement 
to CRC32 in the JDK. It relies on SSE 2, SSE 4.1, AVX and CLMUL. SSE has been 
available for a long time, CLMUL since Intel Westmere (2010) and AVX since 
Intel Sandy Bridge (2011). It's probably OK to assume that these instructions 
will be available for those who are constrained by CPU performance.

Note that this is not using CRC32 CPU instruction as we would have to use 
CRC32C for that (see KAFKA-1449 for more details on what is possible if we are 
willing to support CRC32C).

I wrote a simple JMH benchmark:

https://gist.github.com/ijuma/f86ad935715cfd4e258e

I tested it on my Ivy Bridge MacBook on JDK 7 update 80 and JDK 8 update 76, 
configuring JMH to use 10 one second measurement iterations, 10 one second 
warmup iterations and 1 fork.

JDK 8 update 76 results:

{code}
[info] Benchmark  (bytesSize)  Mode  Cnt   Score   Error  
Units
[info] Crc32Bench.jdkCrc32  8  avgt   10  24.902 ± 0.728  
ns/op
[info] Crc32Bench.jdkCrc32 16  avgt   10  48.819 ± 2.550  
ns/op
[info] Crc32Bench.jdkCrc32 32  avgt   10  83.434 ± 2.668  
ns/op
[info] Crc32Bench.jdkCrc32128  avgt   10 127.679 ± 5.185  
ns/op
[info] Crc32Bench.jdkCrc32   1024  avgt   10 450.105 ±18.943  
ns/op
[info] Crc32Bench.jdkCrc32  65536  avgt   10   25579.406 ±   683.017  
ns/op
[info] Crc32Bench.jdkCrc321048576  avgt   10  408708.242 ± 12183.543  
ns/op
[info] Crc32Bench.kafkaCrc328  avgt   10  14.761 ± 0.647  
ns/op
[info] Crc32Bench.kafkaCrc32   16  avgt   10  19.114 ± 0.423  
ns/op
[info] Crc32Bench.kafkaCrc32   32  avgt   10  34.243 ± 1.066  
ns/op
[info] Crc32Bench.kafkaCrc32  128  avgt   10 114.481 ± 2.812  
ns/op
[info] Crc32Bench.kafkaCrc32 1024  avgt   10 835.630 ±22.412  
ns/op
[info] Crc32Bench.kafkaCrc3265536  avgt   10   52234.713 ±  2229.624  
ns/op
[info] Crc32Bench.kafkaCrc32  1048576  avgt   10  822903.613 ± 20950.560  
ns/op
{code}

JDK 7 update 80 results:

{code}
[info] Benchmark  (bytesSize)  Mode  Cnt   Score   Error  
Units
[info] Crc32Bench.jdkCrc32  8  avgt   10 114.802 ± 8.289  
ns/op
[info] Crc32Bench.jdkCrc32 16  avgt   10 122.030 ± 3.153  
ns/op
[info] Crc32Bench.jdkCrc32 32  avgt   10 131.082 ± 5.501  
ns/op
[info] Crc32Bench.jdkCrc32128  avgt   10 154.116 ± 6.164  
ns/op
[info] Crc32Bench.jdkCrc32   1024  avgt   10 512.151 ±15.934  
ns/op
[info] Crc32Bench.jdkCrc32  65536  avgt   10   25460.014 ±  1532.627  
ns/op
[info] Crc32Bench.jdkCrc321048576  avgt   10  401996.290 ± 18606.012  
ns/op
[info] Crc32Bench.kafkaCrc328  avgt   10  14.493 ± 0.494  
ns/op
[info] Crc32Bench.kafkaCrc32   16  avgt   10  20.329 ± 2.019  
ns/op
[info] Crc32Bench.kafkaCrc32   32  avgt   10  37.706 ± 0.338  
ns/op
[info] Crc32Bench.kafkaCrc32  128  avgt   10 124.197 ± 6.368  
ns/op
[info] Crc32Bench.kafkaCrc32 1024  avgt   10 908.327 ±32.487  
ns/op
[info] Crc32Bench.kafkaCrc3265536  avgt   10   57000.705 ±  2976.852  
ns/op
[info] Crc32Bench.kafkaCrc32  1048576  avgt   10  940433.528 ± 26257.962  
ns/op
{code}

Using a VM intrinsic avoids JNI set-up costs making JDK 8 much faster than JDK 
7 for small byte arrays. Having said that, Kafka's pure Java implementation is 
still faster for byte arrays of up to 128 bytes according to this benchmark. 
Surprisingly, the results are similar for JDK 7 and JDK 8 for larger byte 
arrays. I had a quick look at the assembly generated for JDK 8 and it seems to 
use AVX and CLMUL as per the OpenJDK commit I linked to. Unfortunately, it's a 
bit more work to look at the assembly generated by JDK 7 on a Mac and so I 
didn't. More investigation would be required to understand why this is so (and 
to be able to trust the numbers).

Looking at how we compute CRCs in `Record`, there are two different code paths 
depending on whether we call it from `Compressor` or not. The former invokes 
Crc32 update methods several times (both the byte array and int versions) while 
the latter invokes the byte array version once only.

To really understand the impact of this change, I think we need to benchmark 
the producer 

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126349#comment-15126349
 ] 

Ismael Juma commented on KAFKA-3174:


[~toddpalino], let us know when you switch to u40 so that we update the docs to 
say that (particularly important if you are getting SEGVs with Mirror Maker!).

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126438#comment-15126438
 ] 

Gwen Shapira commented on KAFKA-3086:
-

Ick. Sorry for the confusion and inconsistency. All good then :)

> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126711#comment-15126711
 ] 

Mayuresh Gharat commented on KAFKA-3175:


I am thinking that once we detect that the delete topic is disabled, 
onController restart/shift we can remove the entries under /admin/delete_topic/.

> topic not accessible after deletion even when delete.topic.enable is disabled
> -
>
> Key: KAFKA-3175
> URL: https://issues.apache.org/jira/browse/KAFKA-3175
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> The can be reproduced with the following steps.
> 1. start ZK and 1 broker (with default delete.topic.enable=false)
> 2. create a topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test 
> --partition 1 --replication-factor 1
> 3. delete topic test
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
> 4. restart the broker
> Now topic test still shows up during topic description.
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:testPartitionCount:1ReplicationFactor:1 Configs:
>   Topic: test Partition: 0Leader: 0   Replicas: 0 Isr: 0
> However, one can't produce to this topic any more.
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> [2016-01-29 17:55:24,527] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,725] WARN Error while fetching metadata with correlation 
> id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> [2016-01-29 17:55:24,828] WARN Error while fetching metadata with correlation 
> id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3182) Failure in kafka.network.SocketServerTest.testSocketsCloseOnShutdown

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-3182:


 Summary: Failure in 
kafka.network.SocketServerTest.testSocketsCloseOnShutdown
 Key: KAFKA-3182
 URL: https://issues.apache.org/jira/browse/KAFKA-3182
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang


{code}
Stacktrace

org.scalatest.junit.JUnitTestFailedError: expected exception when writing to 
closed trace socket
at 
org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:102)
at 
org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:79)
at org.scalatest.Assertions$class.fail(Assertions.scala:1328)
at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:79)
at 
kafka.network.SocketServerTest.testSocketsCloseOnShutdown(SocketServerTest.scala:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at 
org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}

Example:

https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/2181/testReport/junit/kafka.network/SocketServerTest/testSocketsCloseOnShutdown/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3181) Check occurrences of Runtime.halt() in the codebase and try to handle it at a single place

[Mayuresh Gharat (JIRA)]

Mayuresh Gharat created KAFKA-3181:
--

 Summary: Check occurrences of Runtime.halt() in the codebase and 
try to handle it at a single place
 Key: KAFKA-3181
 URL: https://issues.apache.org/jira/browse/KAFKA-3181
 Project: Kafka
  Issue Type: Bug
Reporter: Mayuresh Gharat






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126619#comment-15126619
 ] 

Jiangjie Qin commented on KAFKA-3177:
-

Thanks [~hachikuji], I'll submit patch for the ticket then.

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)