[jira] [Commented] (KAFKA-4335) FileStreamSource Connector not working for large files (~ 1GB)

[Rahul Shukla (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604115#comment-15604115
 ] 

Rahul Shukla commented on KAFKA-4335:
-

It did not throw any exception but not producing content to the topic as well. 
I looked into source code and find that it's trying to read the file in memory 
and then produce the record. Which I believe it's difficult for hold entire 
file in memory. Below is source code snippet which tries to do ...  

int nread = 0;
while (readerCopy.ready()) {
nread = readerCopy.read(buffer, offset, buffer.length - offset);
log.trace("Read {} bytes from {}", nread, logFilename());

if (nread > 0) {
offset += nread;
if (offset == buffer.length) {
char[] newbuf = new char[buffer.length * 2];
System.arraycopy(buffer, 0, newbuf, 0, buffer.length);
buffer = newbuf;
}

String line;
do {
line = extractLine();
if (line != null) {
log.trace("Read a line from {}", logFilename());
if (records == null)
records = new ArrayList<>();
records.add(new SourceRecord(offsetKey(filename), 
offsetValue(streamOffset), topic, VALUE_SCHEMA, line));
}
} while (line != null);
}
}

> FileStreamSource Connector not working for large files (~ 1GB)
> --
>
> Key: KAFKA-4335
> URL: https://issues.apache.org/jira/browse/KAFKA-4335
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Rahul Shukla
>Assignee: Ewen Cheslack-Postava
>
> I was trying to sink large file about (1gb). FileStreamSource connector is 
> not working for that it's working fine for small files.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604027#comment-15604027
 ] 

ASF GitHub Bot commented on KAFKA-4339:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2061

KAFKA-4339: Update system tests to accommodate the new consumer group 
describe output



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2061


commit afba6d104f9aa37465907a96ed1874dbe01ba4f3
Author: Vahid Hashemian 
Date:   2016-10-25T03:18:01Z

KAFKA-4339: Update system tests to accommodate the new consumer group 
describe output




> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2061: KAFKA-4339: Update system tests to accommodate the...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2061

KAFKA-4339: Update system tests to accommodate the new consumer group 
describe output



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2061


commit afba6d104f9aa37465907a96ed1874dbe01ba4f3
Author: Vahid Hashemian 
Date:   2016-10-25T03:18:01Z

KAFKA-4339: Update system tests to accommodate the new consumer group 
describe output




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604019#comment-15604019
 ] 

Vahid Hashemian commented on KAFKA-4339:


Sure, I'll submit a PR shortly, and then kick off the system tests to validate.

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1002

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2089; Increase metadata wait time in unit test

--
[...truncated 6388 lines...]

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionWithTopicDescribeOnlyAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionWithTopicDescribeOnlyAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > testDeleteWithWildCardAuth STARTED

kafka.api.AuthorizerIntegrationTest > testDeleteWithWildCardAuth PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead PASSED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopicWithDescribeOnlyPermission STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopicWithDescribeOnlyPermission PASSED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicNotExisting 
STARTED

kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicNotExisting 
PASSED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionNotMatchingInternalTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testPatternSubscriptionNotMatchingInternalTopic PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithoutTopicDescribeAccess 
STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithoutTopicDescribeAccess 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicExisting STARTED

kafka.api.AuthorizerIntegrationTest > 

[GitHub] kafka pull request #1985: MINOR: A bunch of clean-ups related to usage of un...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1985


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookeeper)

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603917#comment-15603917
 ] 

Ismael Juma commented on KAFKA-4337:


Yes, this is not a valid bug. You can configure multiple host:port pairs for 
the case when one or more are down. You are supposed to pass host:port pairs 
for servers in the _same_ ZooKeeper ensemble.

> Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA 
> API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the 
> specified zookeeper)
> ---
>
> Key: KAFKA-4337
> URL: https://issues.apache.org/jira/browse/KAFKA-4337
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Reporter: Bharat Patel
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  Want to use below code snippet to create topic in multiple zookeepers with 
> Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable 
> it only creates topic in anyone of zookeeper. Both the zookeeper are 2 
> different kafka clusters.
>  String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
>int sessionTimeoutMs = 10 * 1000;
>int connectionTimeoutMs = 8 * 1000;
>try {
> ZkClient zkClient = new ZkClient(
> zookeeperConnect,
> sessionTimeoutMs,
> connectionTimeoutMs,
> ZKStringSerializer$.MODULE$);
> boolean isSecureKafkaCluster = false;
> ZkUtils zkUtils = new ZkUtils(zkClient, 
> new ZkConnection(zookeeperConnect), isSecureKafkaCluster);
>  String topic1 = "nameofTopictobeCreated";
>  int partitions = 1;
>  int replication = 1;
>  Properties topicConfig = new Properties(); // add per-topic 
> configurations settings here
>  AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
> topicConfig);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookee

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603886#comment-15603886
 ] 

huxi edited comment on KAFKA-4337 at 10/25/16 1:57 AM:
---

Zookeeper client will try to connect to the specified ZK servers in a 
robin-round manner, so it will connect to the first zookeeper server in the 
list. If it failed, it will try to connect to the second one.


was (Author: huxi_2b):
After checking the source code, zookeeper client will try to connect to the 
specified ZK servers in a robin-round manner, so it will connect to the first 
zookeeper server in the list. If it failed, it will try to connect to the 
second one.

> Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA 
> API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the 
> specified zookeeper)
> ---
>
> Key: KAFKA-4337
> URL: https://issues.apache.org/jira/browse/KAFKA-4337
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Reporter: Bharat Patel
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  Want to use below code snippet to create topic in multiple zookeepers with 
> Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable 
> it only creates topic in anyone of zookeeper. Both the zookeeper are 2 
> different kafka clusters.
>  String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
>int sessionTimeoutMs = 10 * 1000;
>int connectionTimeoutMs = 8 * 1000;
>try {
> ZkClient zkClient = new ZkClient(
> zookeeperConnect,
> sessionTimeoutMs,
> connectionTimeoutMs,
> ZKStringSerializer$.MODULE$);
> boolean isSecureKafkaCluster = false;
> ZkUtils zkUtils = new ZkUtils(zkClient, 
> new ZkConnection(zookeeperConnect), isSecureKafkaCluster);
>  String topic1 = "nameofTopictobeCreated";
>  int partitions = 1;
>  int replication = 1;
>  Properties topicConfig = new Properties(); // add per-topic 
> configurations settings here
>  AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
> topicConfig);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookeeper)

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603886#comment-15603886
 ] 

huxi commented on KAFKA-4337:
-

After checking the source code, zookeeper client will try to connect to the 
specified ZK servers in a robin-round manner, so it will connect to the first 
zookeeper server in the list. If it failed, it will try to connect to the 
second one.

> Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA 
> API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the 
> specified zookeeper)
> ---
>
> Key: KAFKA-4337
> URL: https://issues.apache.org/jira/browse/KAFKA-4337
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Reporter: Bharat Patel
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  Want to use below code snippet to create topic in multiple zookeepers with 
> Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable 
> it only creates topic in anyone of zookeeper. Both the zookeeper are 2 
> different kafka clusters.
>  String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
>int sessionTimeoutMs = 10 * 1000;
>int connectionTimeoutMs = 8 * 1000;
>try {
> ZkClient zkClient = new ZkClient(
> zookeeperConnect,
> sessionTimeoutMs,
> connectionTimeoutMs,
> ZKStringSerializer$.MODULE$);
> boolean isSecureKafkaCluster = false;
> ZkUtils zkUtils = new ZkUtils(zkClient, 
> new ZkConnection(zookeeperConnect), isSecureKafkaCluster);
>  String topic1 = "nameofTopictobeCreated";
>  int partitions = 1;
>  int replication = 1;
>  Properties topicConfig = new Properties(); // add per-topic 
> configurations settings here
>  AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
> topicConfig);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-4340:
---

 Summary: Change the default value of 
log.message.timestamp.difference.max.ms to the same as log.retention.ms
 Key: KAFKA-4340
 URL: https://issues.apache.org/jira/browse/KAFKA-4340
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.10.1.0
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin
 Fix For: 0.10.1.1


[~junrao] brought up the following scenario: 

If users are pumping data with timestamp already passed log.retention.ms into 
Kafka, the messages will be appended to the log but will be immediately rolled 
out by log retention thread when it kicks in and the messages will be deleted. 

To avoid this produce-and-deleted scenario, we can set the default value of 
log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603857#comment-15603857
 ] 

Jiangjie Qin commented on KAFKA-4099:
-

[~junrao] Good point about making log.message.timestamp.difference.max.ms the 
same as log.retention.ms. That makes sense. This change may have some impact on 
the users who is currently pumping data into Kafka and consume it immediately 
before the log retention thread kicks in. But it should be rare and seems 
fragile anyway.

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above. If the first message in the segment does not 
> have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4339:
---
Fix Version/s: 0.10.2.0

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka clients

[Jun Rao]

Hi, Rajini,

Thanks for the proposal. It looks very useful.

Jun

On Mon, Oct 24, 2016 at 6:27 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> If there are no more comments or suggestions, I will start voting process
> for this KIP later this week.
>
> Please feel free to add any comments in the meantime.
>
> Thank you,
>
> Rajini
>
> On Fri, Oct 7, 2016 at 11:32 PM, Gwen Shapira  wrote:
>
> > Huge +1.
> >
> > Thank you Rajini for all the hard work on security improvements. With
> > more Docker deployments of micro-services, getting rid of mandatory
> > config file will be super important.
> >
> > Gwen
> >
> > On Thu, Oct 6, 2016 at 5:43 AM, Edoardo Comar  wrote:
> > > Hi Rajini
> > > great improvement and I see you had the code ready ... !
> > >
> > > I now think it'd easier to implement a PR for KIP-83 (multiple logins
> per
> > > JVM) on top of yours,
> > > as you have now identified a client property that can be used for
> caching
> > > logins.
> > >
> > > I'm actually wondering if I caused you to cut down your KIP !!
> > >
> > > Also, I think your JIRA encompasses
> > > https://issues.apache.org/jira/browse/KAFKA-3302
> > >
> > > thanks,
> > > Edo
> > > --
> > > Edoardo Comar
> > > IBM MessageHub
> > > eco...@uk.ibm.com
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > > IBM United Kingdom Limited Registered in England and Wales with number
> > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> > PO6
> > > 3AU
> > >
> > >
> > >
> > > From:   Rajini Sivaram 
> > > To: dev@kafka.apache.org
> > > Date:   06/10/2016 10:49
> > > Subject:[DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka
> > > clients
> > >
> > >
> > >
> > > Hi all,
> > >
> > > I have just created KIP-85 to enable JAAS login context for Kafka
> clients
> > > to be configured without a physical file:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 85%3A+Dynamic+JAAS+configuration+for+Kafka+clients
> > >
> > >
> > >
> > > Comments and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > >
> > > Unless stated otherwise above:
> > > IBM United Kingdom Limited - Registered in England and Wales with
> number
> > > 741598.
> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> > 3AU
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
>
> --
> Regards,
>
> Rajini
>

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603853#comment-15603853
 ] 

Ismael Juma commented on KAFKA-4339:


Expecting the topic name sounds good. It would be great to have a PR for this 
soon as it's causing the system tests builds to fail. Generally, if a fix for a 
consistent failure is not available quickly, the PR that introduced the failure 
needs to be reverted.

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookeeper)

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603845#comment-15603845
 ] 

huxi commented on KAFKA-4337:
-

could you paste how you specify multiple zookeeper IPs? Those should be comma 
separated host:port pairs. Besides, could you offer the error log also?

> Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA 
> API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the 
> specified zookeeper)
> ---
>
> Key: KAFKA-4337
> URL: https://issues.apache.org/jira/browse/KAFKA-4337
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Reporter: Bharat Patel
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  Want to use below code snippet to create topic in multiple zookeepers with 
> Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable 
> it only creates topic in anyone of zookeeper. Both the zookeeper are 2 
> different kafka clusters.
>  String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
>int sessionTimeoutMs = 10 * 1000;
>int connectionTimeoutMs = 8 * 1000;
>try {
> ZkClient zkClient = new ZkClient(
> zookeeperConnect,
> sessionTimeoutMs,
> connectionTimeoutMs,
> ZKStringSerializer$.MODULE$);
> boolean isSecureKafkaCluster = false;
> ZkUtils zkUtils = new ZkUtils(zkClient, 
> new ZkConnection(zookeeperConnect), isSecureKafkaCluster);
>  String topic1 = "nameofTopictobeCreated";
>  int partitions = 1;
>  int replication = 1;
>  Properties topicConfig = new Properties(); // add per-topic 
> configurations settings here
>  AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
> topicConfig);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4295) kafka-console-consumer.sh does not delete the temporary group in zookeeper

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603797#comment-15603797
 ] 

huxi commented on KAFKA-4295:
-

[~guozhang] Seems that pull request always got wrong due to failing to pass 
testReprocessingFromScratchAfterReset although the commit should have been no 
relevant to this error. Please advice how to handle such a situation.

> kafka-console-consumer.sh does not delete the temporary group in zookeeper
> --
>
> Key: KAFKA-4295
> URL: https://issues.apache.org/jira/browse/KAFKA-4295
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Sswater Shi
>Assignee: huxi
>Priority: Minor
>
> I'm not sure it is a bug or you guys designed it.
> Since 0.9.x.x, the kafka-console-consumer.sh will not delete the group 
> information in zookeeper/consumers on exit when without "--new-consumer". 
> There will be a lot of abandoned zookeeper/consumers/console-consumer-xxx if 
> kafka-console-consumer.sh runs a lot of times.
> When 0.8.x.x,  the kafka-console-consumer.sh can be followed by an argument 
> "group". If not specified, the kafka-console-consumer.sh will create a 
> temporary group name like 'console-consumer-'. If the group name is 
> specified by "group", the information in the zookeeper/consumers will be kept 
> on exit. If the group name is a temporary one, the information in the 
> zookeeper will be deleted when kafka-console-consumer.sh is quitted by 
> Ctrl+C. Why this is changed from 0.9.x.x.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1652

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-47 - Add timestamp-based log deletion policy

[Bill Warshaw]

Hi Jun,

Those are valid concerns.  For our particular use case, application events
triggering the timestamp update will never occur more than once an hour,
and we maintain a sliding window so that we don't delete messages too close
to what our consumers may be reading.
For more general use cases, developers will need to be aware of these
issues, and would need to write their application code with that in mind.


To your second point: I initially wanted to just have a trim() admin api.
I started implementing it, but ran into difficulties with synchronously
acknowledging to the calling code that all brokers had truncated the given
partitions.  It seemed like we would have to do something similar to how
topic deletion is implemented, where the initial broker uses Zookeeper to
coordinate the deletion on the other brokers.  If you have a simpler idea
in mind, I'd be happy to update this KIP to provide a trim() api instead.

On Mon, Oct 24, 2016 at 8:15 PM Jun Rao  wrote:

> Hi, Bill,
>
> Thanks for the proposal. Sorry for the late reply.
>
> The motivation of the proposal makes sense: don't delete the messages until
> the application tells you so.
>
> I am wondering if the current proposal is the best way to address the need
> though. There are couple of issues that I saw with the proposal. (1)
> Messages in the log may not always be stored in increasing timestamp order.
> Suppose that the application sets log.retention.min.timestamp to T and
> after that messages with timestamp older than T ((either due to delay or
> reprocessing) are published to that topic. Those newly published messages
> are likely going to be deleted immediately before the application gets a
> chance to read them, which is probably not what the application wants. (2)
> The configuration for the topic has to be changed continuously to implement
> the use case. Intuitively, one probably shouldn't be changing a
> configuration all the time.
>
> Another way to achieve the goal is what Jay mentioned earlier. We could add
> a trim() api like the following that will trim the log up to the specified
> offsets. This addresses both of the above issues that I mentioned. Will
> that work for you?
>
> void trim(Map offsetsToTruncate)
>
> Thanks,
>
> Jun
>
> On Wed, Oct 5, 2016 at 1:55 PM, Bill Warshaw  wrote:
>
> > Bumping for visibility.  KIP is here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 47+-+Add+timestamp-based+log+deletion+policy
> >
> > On Wed, Aug 24, 2016 at 2:32 PM Bill Warshaw 
> wrote:
> >
> > > Hello Guozhang,
> > >
> > > KIP-71 seems unrelated to this KIP.  KIP-47 is just adding a new
> deletion
> > > policy (minimum timestamp), while KIP-71 is allowing deletion and
> > > compaction to coexist.
> > >
> > > They both will touch LogManager, but the change for KIP-47 is very
> > > isolated.
> > >
> > > On Wed, Aug 24, 2016 at 2:21 PM Guozhang Wang 
> > wrote:
> > >
> > > Hi Bill,
> > >
> > > I would like to reason if there is any correlation between this KIP and
> > > KIP-71
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 71%3A+Enable+log+compaction+and+deletion+to+co-exist
> > >
> > > I feel they are orthogonal but would like to double check with you.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Wed, Aug 24, 2016 at 11:05 AM, Bill Warshaw 
> > > wrote:
> > >
> > > > I'd like to re-awaken this voting thread now that KIP-33 has merged.
> > > This
> > > > KIP is now completely unblocked.  I have a working branch off of
> trunk
> > > with
> > > > my proposed fix, including testing.
> > > >
> > > > On Mon, May 9, 2016 at 8:30 PM Guozhang Wang 
> > wrote:
> > > >
> > > > > Jay, Bill:
> > > > >
> > > > > I'm thinking of one general use case of using timestamp rather than
> > > > offset
> > > > > for log deletion, which is that for expiration handling in data
> > > > > replication, when the source data store decides to expire some data
> > > > records
> > > > > based on their timestamps, today we need to configure the
> > corresponding
> > > > > Kafka changelog topic for compaction, and actively send a tombstone
> > for
> > > > > each expired record. Since expiration usually happens with a bunch
> of
> > > > > records, this could generate large tombstone traffic. For example I
> > > think
> > > > > LI's data replication for Espresso is seeing similar issues and
> they
> > > are
> > > > > just not sending tombstone at all.
> > > > >
> > > > > With timestamp based log deletion policy, this can be easily
> handled
> > by
> > > > > simply setting the current expiration timestamp; but ideally one
> > would
> > > > > prefer to configure this topic to be both log compaction enabled as
> > > well
> > > > as
> > > > > log deletion enabled. From that point of view, I feel that current
> > KIP
> > > > > still has value to be accepted.
> > > > >
> > > > > 

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603683#comment-15603683
 ] 

Vahid Hashemian commented on KAFKA-4339:


[~hachikuji] This issue is very likely because we no longer print out the 
repeating (and redundant) consumer group name in the output when using the 
{{--describe}} option. This system test fails because it looks for the consumer 
group name followed by the topic name in the output. I could change the regular 
expression to expect the topic name followed by a 0 (partition), or perhaps 
just the topic name; as either of these two indicates that the command ran 
successfully and info was returned on the consumer group.

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-47 - Add timestamp-based log deletion policy

[Jun Rao]

Hi, Bill,

Thanks for the proposal. Sorry for the late reply.

The motivation of the proposal makes sense: don't delete the messages until
the application tells you so.

I am wondering if the current proposal is the best way to address the need
though. There are couple of issues that I saw with the proposal. (1)
Messages in the log may not always be stored in increasing timestamp order.
Suppose that the application sets log.retention.min.timestamp to T and
after that messages with timestamp older than T ((either due to delay or
reprocessing) are published to that topic. Those newly published messages
are likely going to be deleted immediately before the application gets a
chance to read them, which is probably not what the application wants. (2)
The configuration for the topic has to be changed continuously to implement
the use case. Intuitively, one probably shouldn't be changing a
configuration all the time.

Another way to achieve the goal is what Jay mentioned earlier. We could add
a trim() api like the following that will trim the log up to the specified
offsets. This addresses both of the above issues that I mentioned. Will
that work for you?

void trim(Map offsetsToTruncate)

Thanks,

Jun

On Wed, Oct 5, 2016 at 1:55 PM, Bill Warshaw  wrote:

> Bumping for visibility.  KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 47+-+Add+timestamp-based+log+deletion+policy
>
> On Wed, Aug 24, 2016 at 2:32 PM Bill Warshaw  wrote:
>
> > Hello Guozhang,
> >
> > KIP-71 seems unrelated to this KIP.  KIP-47 is just adding a new deletion
> > policy (minimum timestamp), while KIP-71 is allowing deletion and
> > compaction to coexist.
> >
> > They both will touch LogManager, but the change for KIP-47 is very
> > isolated.
> >
> > On Wed, Aug 24, 2016 at 2:21 PM Guozhang Wang 
> wrote:
> >
> > Hi Bill,
> >
> > I would like to reason if there is any correlation between this KIP and
> > KIP-71
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 71%3A+Enable+log+compaction+and+deletion+to+co-exist
> >
> > I feel they are orthogonal but would like to double check with you.
> >
> >
> > Guozhang
> >
> >
> > On Wed, Aug 24, 2016 at 11:05 AM, Bill Warshaw 
> > wrote:
> >
> > > I'd like to re-awaken this voting thread now that KIP-33 has merged.
> > This
> > > KIP is now completely unblocked.  I have a working branch off of trunk
> > with
> > > my proposed fix, including testing.
> > >
> > > On Mon, May 9, 2016 at 8:30 PM Guozhang Wang 
> wrote:
> > >
> > > > Jay, Bill:
> > > >
> > > > I'm thinking of one general use case of using timestamp rather than
> > > offset
> > > > for log deletion, which is that for expiration handling in data
> > > > replication, when the source data store decides to expire some data
> > > records
> > > > based on their timestamps, today we need to configure the
> corresponding
> > > > Kafka changelog topic for compaction, and actively send a tombstone
> for
> > > > each expired record. Since expiration usually happens with a bunch of
> > > > records, this could generate large tombstone traffic. For example I
> > think
> > > > LI's data replication for Espresso is seeing similar issues and they
> > are
> > > > just not sending tombstone at all.
> > > >
> > > > With timestamp based log deletion policy, this can be easily handled
> by
> > > > simply setting the current expiration timestamp; but ideally one
> would
> > > > prefer to configure this topic to be both log compaction enabled as
> > well
> > > as
> > > > log deletion enabled. From that point of view, I feel that current
> KIP
> > > > still has value to be accepted.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, May 2, 2016 at 2:37 PM, Bill Warshaw 
> > > wrote:
> > > >
> > > > > Yes, I'd agree that offset is a more precise configuration than
> > > > timestamp.
> > > > > If there was a way to set a partition-level configuration, I would
> > > rather
> > > > > use log.retention.min.offset than timestamp.  If you have an
> approach
> > > in
> > > > > mind I'd be open to investigating it.
> > > > >
> > > > > On Mon, May 2, 2016 at 5:33 PM, Jay Kreps 
> wrote:
> > > > >
> > > > > > Gotcha, good point. But barring that limitation, you agree that
> > that
> > > > > makes
> > > > > > more sense?
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Mon, May 2, 2016 at 2:29 PM, Bill Warshaw <
> wdwars...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > The problem with offset as a config option is that offsets are
> > > > > > > partition-specific, so we'd need a per-partition config.  This
> > > would
> > > > > work
> > > > > > > for our particular use case, where we have single-partition
> > topics,
> > > > but
> > > > > > for
> > > > > > > multiple-partition topics it would delete from all partitions
> > based
> > > > on
> > > > > a

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603576#comment-15603576
 ] 

Vahid Hashemian commented on KAFKA-4339:


Thanks [~hachikuji]. I'll take a look.

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4211) Change system tests to use the new consumer by default

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4211:
---
Status: Patch Available  (was: In Progress)

> Change system tests to use the new consumer by default
> --
>
> Key: KAFKA-4211
> URL: https://issues.apache.org/jira/browse/KAFKA-4211
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> We have utility methods like `run_produce_consume_validate` that use the old 
> consumer by default. We should change them to use the new consumer by default 
> while ensuring that we still have coverage for the old consumer (while we 
> still support it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2060: KAFKA-4211: Update system tests to use the new con...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2060

KAFKA-4211: Update system tests to use the new consumer by default

Update system test method signatures and method calls to use the new 
consumer by default.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4211

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2060


commit aca03aaffbb3153a3ce440b57d00540c15570509
Author: Vahid Hashemian 
Date:   2016-10-03T23:17:12Z

KAFKA-4211: Update system tests to use the new consumer by default

Update system test method signatures and method calls to use the new 
consumer by default.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603569#comment-15603569
 ] 

Jason Gustafson commented on KAFKA-4339:


[~vahid] I went ahead and assigned you since you are most familiar with the 
changes in KAFKA-3144. Feel free to unassign if you don't have time.

> Failing ConsumerGroupCommand System Tests
> -
>
> Key: KAFKA-4339
> URL: https://issues.apache.org/jira/browse/KAFKA-4339
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>
> This appears to have been caused by KAFKA-3144, but system tests are now 
> failing. For example:
> {code}
> test_id:
> 2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
> status: FAIL
> run time:   42.521 seconds
> Timed out waiting to list expected consumer groups.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 106, in test_describe_consumer_group
> self.setup_and_verify(security_protocol, group="test-consumer-group")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
>  line 85, in setup_and_verify
> err_msg="Timed out waiting to list expected consumer groups.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to list expected consumer groups.
> {code}
> The error message is misleading (and should be fixed). The error was actually 
> in DescribeGroups. Doesn't seem like there is any problem with the command 
> itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4211) Change system tests to use the new consumer by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603566#comment-15603566
 ] 

ASF GitHub Bot commented on KAFKA-4211:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2060

KAFKA-4211: Update system tests to use the new consumer by default

Update system test method signatures and method calls to use the new 
consumer by default.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4211

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2060


commit aca03aaffbb3153a3ce440b57d00540c15570509
Author: Vahid Hashemian 
Date:   2016-10-03T23:17:12Z

KAFKA-4211: Update system tests to use the new consumer by default

Update system test method signatures and method calls to use the new 
consumer by default.




> Change system tests to use the new consumer by default
> --
>
> Key: KAFKA-4211
> URL: https://issues.apache.org/jira/browse/KAFKA-4211
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> We have utility methods like `run_produce_consume_validate` that use the old 
> consumer by default. We should change them to use the new consumer by default 
> while ensuring that we still have coverage for the old consumer (while we 
> still support it).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4339) Failing ConsumerGroupCommand System Tests

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4339:
--

 Summary: Failing ConsumerGroupCommand System Tests
 Key: KAFKA-4339
 URL: https://issues.apache.org/jira/browse/KAFKA-4339
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Reporter: Jason Gustafson
Assignee: Vahid Hashemian


This appears to have been caused by KAFKA-3144, but system tests are now 
failing. For example:

{code}
test_id:
2016-10-23--001.kafkatest.tests.core.consumer_group_command_test.ConsumerGroupCommandTest.test_describe_consumer_group.security_protocol=PLAINTEXT
status: FAIL
run time:   42.521 seconds


Timed out waiting to list expected consumer groups.
Traceback (most recent call last):
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 106, in run_all_tests
data = self.run_single_test()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
 line 162, in run_single_test
return self.current_test_context.function(self.current_test)
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
 line 331, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
 line 106, in test_describe_consumer_group
self.setup_and_verify(security_protocol, group="test-consumer-group")
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/consumer_group_command_test.py",
 line 85, in setup_and_verify
err_msg="Timed out waiting to list expected consumer groups.")
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/utils/util.py",
 line 36, in wait_until
raise TimeoutError(err_msg)
TimeoutError: Timed out waiting to list expected consumer groups.
{code}

The error message is misleading (and should be fixed). The error was actually 
in DescribeGroups. Doesn't seem like there is any problem with the command 
itself, so probably we just need to update the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-73 - Replication Quotas

[Jun Rao]

Yes, that sounds like a good idea. Updated the wiki for KIP-73.

Thanks,

Jun

On Fri, Oct 21, 2016 at 4:26 PM, Joel Koshy  wrote:

> Thanks for catching that and the fix as well. Makes sense to me.
>
> We should consider adding an "amendments" section to KIPs - perhaps just a
> link to KAFKA-4313 would suffice in this case.
>
> Thanks,
>
> Joel
>
> On Wed, Oct 19, 2016 at 7:12 PM, Jun Rao  wrote:
>
> > Hi,
> >
> > While testing KIP-73, we found an issue described in
> > https://issues.apache.org/jira/browse/KAFKA-4313. Basically, when there
> > are
> > mixed high-volume and low-volume partitions, when replication throttling
> is
> > specified, ISRs for those low volume partitions could thrash. KAFKA-4313
> > fixes this issue by avoiding throttling those replicas in the throttled
> > replica list that are already in sync. Those in-sync replicas traffic
> will
> > still be accounted for the throttled traffic though. Just want to bring
> > this up since it slightly changes the behavior described in the KIP. If
> > anyone has concerns on this, please comment on the jira.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Aug 23, 2016 at 3:25 PM, Ismael Juma  wrote:
> >
> > > For the record, there were 4 binding +1s.
> > >
> > > Ismael
> > >
> > > On Tue, Aug 23, 2016 at 11:16 PM, Ben Stopford 
> wrote:
> > >
> > > > Thanks everyone. It looks like this KIP has now been accepted.
> > > >
> > > > There is a corresponding PR  kafka/pull/1776
> > >
> > > > for the implementation also.
> > > >
> > > > All the best
> > > >
> > > > B
> > > >
> > > >
> > > > > On 23 Aug 2016, at 22:39, Joel Koshy  wrote:
> > > > >
> > > > > +1
> > > > > (sent some very minor edits to you off-thread)
> > > > >
> > > > > On Fri, Aug 19, 2016 at 1:21 AM, Ben Stopford 
> > > wrote:
> > > > >
> > > > >> I’d like to initiate the voting process for KIP-73:
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> 73+Replication+Quotas  > > > >> confluence/display/KAFKA/KIP-73+Replication+Quotas>
> > > > >>
> > > > >> Ben
> > > >
> > > >
> > >
> >
>

[jira] [Commented] (KAFKA-2089) MetadataTest transient failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603531#comment-15603531
 ] 

ASF GitHub Bot commented on KAFKA-2089:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2057


> MetadataTest transient failure
> --
>
> Key: KAFKA-2089
> URL: https://issues.apache.org/jira/browse/KAFKA-2089
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-2089.patch, KAFKA-2089.patch, 
> KAFKA-2089_2015-04-13_18:59:33.patch
>
>
> org.apache.kafka.clients.MetadataTest > testMetadata FAILED
> java.lang.AssertionError:
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertFalse(Assert.java:68)
> at org.junit.Assert.assertFalse(Assert.java:79)
> at 
> org.apache.kafka.clients.MetadataTest.tearDown(MetadataTest.java:34)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4333) Report consumer group coordinator id when '--list' option is used

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4333:
---
Status: Patch Available  (was: In Progress)

> Report consumer group coordinator id when '--list' option is used
> -
>
> Key: KAFKA-4333
> URL: https://issues.apache.org/jira/browse/KAFKA-4333
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> One piece of information missing when extracting information about consumer 
> groups (Java API based) is the coordinator id (broker id of the coordinator). 
> It would be useful to enhance the {{--list}} option of the consumer group 
> command to report the corresponding coordinator id of each consumer group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2057: KAFKA-2089: Increase metadata wait time in unit te...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2057


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-2089) MetadataTest transient failure

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-2089.

   Resolution: Fixed
Fix Version/s: (was: 0.9.0.0)
   0.10.2.0

Issue resolved by pull request 2057
[https://github.com/apache/kafka/pull/2057]

> MetadataTest transient failure
> --
>
> Key: KAFKA-2089
> URL: https://issues.apache.org/jira/browse/KAFKA-2089
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-2089.patch, KAFKA-2089.patch, 
> KAFKA-2089_2015-04-13_18:59:33.patch
>
>
> org.apache.kafka.clients.MetadataTest > testMetadata FAILED
> java.lang.AssertionError:
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertFalse(Assert.java:68)
> at org.junit.Assert.assertFalse(Assert.java:79)
> at 
> org.apache.kafka.clients.MetadataTest.tearDown(MetadataTest.java:34)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2059: KAFKA-4333: Report coordinator id of the consumer ...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2059

KAFKA-4333: Report coordinator id of the consumer group with '--list' option

Enahnce the output of the consumer group commend and report the coordinator 
id of each group (for Java API based consumers) when `--list` option is used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2059


commit 373d66e4531df08cbb9bd56e24b1115e3bd346c1
Author: Vahid Hashemian 
Date:   2016-10-24T23:12:20Z

KAFKA-4333: Report coordinator id of the consumer group with that --list 
option

Enahnce the output of the consumer group commend and report the coordinator 
id of each group (for Java API based consumers) when `--list` option is used.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4333) Report consumer group coordinator id when '--list' option is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603525#comment-15603525
 ] 

ASF GitHub Bot commented on KAFKA-4333:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2059

KAFKA-4333: Report coordinator id of the consumer group with '--list' option

Enahnce the output of the consumer group commend and report the coordinator 
id of each group (for Java API based consumers) when `--list` option is used.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2059


commit 373d66e4531df08cbb9bd56e24b1115e3bd346c1
Author: Vahid Hashemian 
Date:   2016-10-24T23:12:20Z

KAFKA-4333: Report coordinator id of the consumer group with that --list 
option

Enahnce the output of the consumer group commend and report the coordinator 
id of each group (for Java API based consumers) when `--list` option is used.




> Report consumer group coordinator id when '--list' option is used
> -
>
> Key: KAFKA-4333
> URL: https://issues.apache.org/jira/browse/KAFKA-4333
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> One piece of information missing when extracting information about consumer 
> groups (Java API based) is the coordinator id (broker id of the coordinator). 
> It would be useful to enhance the {{--list}} option of the consumer group 
> command to report the corresponding coordinator id of each consumer group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #1001

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2054: kafka-4295: ConsoleConsumer does not delete the te...

[amethystic]

Github user amethystic closed the pull request at:

https://github.com/apache/kafka/pull/2054


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2058: kafka-4295: ConsoleConsumer does not delete the te...

[amethystic]

GitHub user amethystic opened a pull request:

https://github.com/apache/kafka/pull/2058

kafka-4295: ConsoleConsumer does not delete the temporary group in zookeeper

ConsoleConsumer does not delete the temporary group in zookeeper
Author: huxi...@hotmail.com
Since consumer stop logic and zk node removal code are in separate threads, 
so when two threads execute in an interleaving manner, persistent node 
'/consumers/' might not be removed for those console consumer groups which do 
not specify "group.id". This will pollute Zookeeper with lots of inactive 
console consumer offset information.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amethystic/kafka 
kafka-4295_ConsoleCoumer_failed_to_cleanup_zkNodes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2058.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2058


commit db20e7d9fced365cc2a56ebb66b173c2855fe4bd
Author: amethystic 
Date:   2016-10-24T23:03:28Z

kafka-4295: ConsoleConsumer does not delete the temporary group in zookeeper
Author: huxi...@hotmail.com
Since consumer stop logic and zk node removal code are in separate threads, 
so when two threads execute in an interleaving manner, persistent node 
'/consumers/' might not be removed for those console consumer groups which do 
not specify "group.id". This will pollute Zookeeper with lots of inactive 
console consumer offset information.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4185) Abstract out password verifier in SaslServer as an injectable dependency

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4185:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Abstract out password verifier in SaslServer as an injectable dependency
> 
>
> Key: KAFKA-4185
> URL: https://issues.apache.org/jira/browse/KAFKA-4185
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.0.1
>Reporter: Piyush Vijay
> Fix For: 0.10.2.0
>
>
> Kafka comes with a default SASL/PLAIN implementation which assumes that 
> username and password are present in a JAAS
> config file. People often want to use some other way to provide username and 
> password to SaslServer. Their best bet,
> currently, is to have their own implementation of SaslServer (which would be, 
> in most cases, a copied version of PlainSaslServer
> minus the logic where password verification happens). This is not ideal.
> We believe that there exists a better way to structure the current 
> PlainSaslServer implementation which makes it very
> easy for people to plug-in their custom password verifier without having to 
> rewrite SaslServer or copy any code.
> The idea is to have an injectable dependency interface PasswordVerifier which 
> can be re-implemented based on the
> requirements. There would be no need to re-implement or extend 
> PlainSaslServer class.
> Note that this is commonly asked feature and there have been some attempts in 
> the past to solve this problem:
> https://github.com/apache/kafka/pull/1350
> https://github.com/apache/kafka/pull/1770
> https://issues.apache.org/jira/browse/KAFKA-2629
> https://issues.apache.org/jira/browse/KAFKA-3679
> We believe that this proposed solution does not have the demerits because of 
> previous proposals were rejected.
> I would be happy to discuss more.
> Please find the link to the PR in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3585) Shutdown slow when there is only one broker which is controller

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3585:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Shutdown slow when there is only one broker which is controller
> ---
>
> Key: KAFKA-3585
> URL: https://issues.apache.org/jira/browse/KAFKA-3585
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.1
>Reporter: Pengwei
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> Reproducer Step:
> 1. Install 3 brokers's cluster
> 2. create a topic with 3 partition
> 3. shutdown the broker one by one , you will find the last one shutdown very 
> slow because of error:
> [2016-04-19 20:30:19,168] INFO [Kafka Server 1], Remaining partitions to 
> move: 
> __consumer_offsets-48,__consumer_offsets-13,__consumer_offsets-46,__consumer_offsets-11,__consumer_offsets-44,__consumer_offsets-42,__consumer_offsets-21,__consumer_offsets-19,__consumer_offsets-32,__consumer_offsets-30,__consumer_offsets-28,__consumer_offsets-26,__consumer_offsets-7,__consumer_offsets-40,__consumer_offsets-38,__consumer_offsets-36,__consumer_offsets-1,__consumer_offsets-34,__consumer_offsets-16,__consumer_offsets-45,__consumer_offsets-14,__consumer_offsets-12,__consumer_offsets-41,__consumer_offsets-10,__consumer_offsets-24,__consumer_offsets-22,__consumer_offsets-20,__consumer_offsets-49,__consumer_offsets-18,__consumer_offsets-31,__consumer_offsets-0,test2-0,__consumer_offsets-27,__consumer_offsets-39,__consumer_offsets-8,__consumer_offsets-37,__consumer_offsets-6,__consumer_offsets-4,__consumer_offsets-2
>  (kafka.server.KafkaServer)
> [2016-04-19 20:30:19,169] INFO [Kafka Server 1], Error code from controller: 
> 0 (kafka.server.KafkaServer)
> [2016-04-19 20:30:24,169] WARN [Kafka Server 1], Retrying controlled shutdown 
> after the previous attempt failed... (kafka.server.KafkaServer)
> [2016-04-19 20:30:24,171] WARN [Kafka Server 1], Proceeding to do an unclean 
> shutdown as all the controlled shutdown attempts failed 
> (kafka.server.KafkaServer)
> it is determined by :
> controlled.shutdown.retry.backoff.ms  = 5000
> controlled.shutdown.max.retries=3
> It slow because the last one can not elect the new leader for the remaining 
> partitions , the last one can improve to shutdown quickly, we can skip the 
> shutdown error when it is the last broker



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4116) Specifying 0.0.0.0 in "listeners" doesn't work

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4116:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Specifying 0.0.0.0 in "listeners" doesn't work
> --
>
> Key: KAFKA-4116
> URL: https://issues.apache.org/jira/browse/KAFKA-4116
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1, 0.10.0.1
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
> Fix For: 0.10.2.0
>
>
> The document of {{listeners}} says:
> "Specify hostname as 0.0.0.0 to bind to all interfaces."
> However when I give config such as below, a started kafka broker can't join 
> the cluster due to invalid address advertised on zk.
> {code}
> listeners=PLAINTEXT://0.0.0.0:9092
> # advertised.listeners=
> {code}
> This is because of:
> - {{advertised.listeners}} which is used as an address to publish on zk 
> defaults to {{listeners}}
> - KafkaHealthcheck#register isn't considering the host "0.0.0.0" as a special 
> case : 
> https://github.com/apache/kafka/blob/8f3462552fa4d6a6d70a837c2ef7439bba512657/core/src/main/scala/kafka/server/KafkaHealthcheck.scala#L60-L61
> h3. Proof
> Test environment:
> - kafka-broker version 0.10.1.0-SNAPSHOT(build from trunk)
> - Brokers HOST-A, HOST-B, HOST-C
> - Controller: HOST-A
> - topic-A has 3 replicas, 3 partitions
> Update HOST-B's server.properties with updating listeners to below and 
> restart the broker.
> {code}
> listeners=PLAINTEXT://0.0.0.0:9092
> {code}
> Then HOST-B registeres it's broker info to ZK path {{/brokers/ids/2}}, but 
> "0.0.0.0" is used as it's host:
> {code}
> [zk: ZKHOST1:2181,ZKHOST2:2181,ZKHOST3:2181/kafka-test(CONNECTED) 8] get 
> /brokers/ids/2
> {"jmx_port":12345,"timestamp":"1472796372181","endpoints":["PLAINTEXT://0.0.0.0:9092"],"host":"0.0.0.0","version":3,"port":9092}
> {code}
> Controller tries to send an request to the above address but of course it 
> will never reach to the HOST-B.
> controller.log:
> {code}
> [2016-09-02 15:06:12,206] INFO [Controller-1-to-broker-2-send-thread], 
> Controller 1 connected to 0.0.0.0:9092 (id: 2 rack: null) for sending state 
> change requests (kafka.controller.RequestSendThread)
> {code}
> I'm guessing maybe controller sending a request to itself(kafka broker 
> working on the same instance), as calling connect("0.0.0.0") results to 
> connect to localhost, which sounds scary but havn't digged into.
> So the ISR won't recovered even through a broker starts up.
> {code}
> ./kafka-topics.sh ... --describe --topic topic-A
> Topic:topic-A   PartitionCount:3ReplicationFactor:3 
> Configs:retention.ms=8640,min.insync.replicas=2
> Topic: topic-A  Partition: 0Leader: 3   Replicas: 3,2,1 Isr: 
> 1,3
> Topic: topic-A  Partition: 1Leader: 1   Replicas: 1,3,2 Isr: 
> 1,3
> Topic: topic-A  Partition: 2Leader: 1   Replicas: 2,1,3 Isr: 
> 1,3
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3994:
---
Fix Version/s: (was: 0.10.1.1)
   (was: 0.10.0.2)
   0.10.2.0

> Deadlock between consumer heartbeat expiration and offset commit.
> -
>
> Key: KAFKA-3994
> URL: https://issues.apache.org/jira/browse/KAFKA-3994
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
> Fix For: 0.10.2.0
>
>
> I got the following stacktraces from ConsumerBounceTest
> {code}
> ...
> "Test worker" #12 prio=5 os_prio=0 tid=0x7fbb28b7f000 nid=0x427c runnable 
> [0x7fbb06445000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0003d48bcbc0> (a sun.nio.ch.Util$2)
> - locked <0x0003d48bcbb0> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x0003d48bbd28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:454)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:277)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:411)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1086)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1054)
> at 
> kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:103)
> at 
> kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at 

[jira] [Updated] (KAFKA-3986) completedReceives can contain closed channels

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3986:
---
Fix Version/s: 0.10.1.1

> completedReceives can contain closed channels 
> --
>
> Key: KAFKA-3986
> URL: https://issues.apache.org/jira/browse/KAFKA-3986
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Reporter: Ryan P
> Fix For: 0.10.0.2, 0.10.1.1
>
>
> I'm not entirely sure why at this point but it is possible to throw a Null 
> Pointer Exception when processingCompletedReceives. This happens when a 
> fairly substantial number of simultaneously initiated connections are 
> initiated with the server. 
> The processor thread does carry on but it may be worth investigating how the 
> channel could be both closed and  completedReceives. 
> The NPE in question is thrown here:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L490
> It can not be consistently reproduced either. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3493) Replica fetcher load is not balanced over fetcher threads

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3493:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Replica fetcher load is not balanced over fetcher threads
> -
>
> Key: KAFKA-3493
> URL: https://issues.apache.org/jira/browse/KAFKA-3493
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.1
>Reporter: Maysam Yabandeh
> Fix For: 0.10.2.0
>
>
> The replicas are not evenly distributed among the fetcher threads. This has 
> caused some fetcher threads get overloaded and hence their requests time out 
> frequently. This is especially a big issue when a new node is added to the 
> cluster and the fetch traffic is high. 
> Here is an example run in a test cluster with 10 brokers and 6 fetcher 
> threads (per source broker). A single topic consisting of 500+ partitions was 
> assigned to have a replica for each parition on the newly added broker.
> {code}[kafka-jetstream.canary]myabandeh@sjc8c-rl17-23b:~$ for i in `seq 0 5`; 
> do grep ReplicaFetcherThread-$i- /var/log/kafka/server.log | grep "reset its 
> fetch offset from 0" | wc -l; done
> 85
> 83
> 85
> 83
> 85
> 85
> [kafka-jetstream.canary]myabandeh@sjc8c-rl17-23b:~$ for i in `seq 0 5`; do 
> grep ReplicaFetcherThread-$i-22 /var/log/kafka/server.log | grep "reset its 
> fetch offset from 0" | wc -l; done
> 15
> 1
> 13
> 1
> 14
> 1
> {code}
> The problem is that AbstractFetcherManager::getFetcherId method does not take 
> the broker id into account:
> {code}
>   private def getFetcherId(topic: String, partitionId: Int) : Int = {
> Utils.abs(31 * topic.hashCode() + partitionId) % numFetchers
>   }
> {code}
> Hence although the replicas are evenly distributed among the fetcher ids 
> across all source brokers, this is not necessarily the case for each broker 
> separately. 
> I think a random function would do a much better job in distributing the load 
> over the fetcher threads from each source broker.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3135) Unexpected delay before fetch response transmission

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3135:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Unexpected delay before fetch response transmission
> ---
>
> Key: KAFKA-3135
> URL: https://issues.apache.org/jira/browse/KAFKA-3135
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.2.0
>
>
> From the user list, Krzysztof Ciesielski reports the following:
> {quote}
> Scenario description:
> First, a producer writes 50 elements into a topic
> Then, a consumer starts to read, polling in a loop.
> When "max.partition.fetch.bytes" is set to a relatively small value, each
> "consumer.poll()" returns a batch of messages.
> If this value is left as default, the output tends to look like this:
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> As we can see, there are weird "gaps" when poll returns 0 elements for some
> time. What is the reason for that? Maybe there are some good practices
> about setting "max.partition.fetch.bytes" which I don't follow?
> {quote}
> The gist to reproduce this problem is here: 
> https://gist.github.com/kciesielski/054bb4359a318aa17561.
> After some initial investigation, the delay appears to be in the server's 
> networking layer. Basically I see a delay of 5 seconds from the time that 
> Selector.send() is invoked in SocketServer.Processor with the fetch response 
> to the time that the send is completed. Using netstat in the middle of the 
> delay shows the following output:
> {code}
> tcp4   0  0  10.191.0.30.55455  10.191.0.30.9092   ESTABLISHED
> tcp4   0 102400  10.191.0.30.9092   10.191.0.30.55454  ESTABLISHED
> {code}
> From this, it looks like the data reaches the send buffer, but needs to be 
> flushed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3190:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> KafkaProducer should not invoke callback in send()
> --
>
> Key: KAFKA-3190
> URL: https://issues.apache.org/jira/browse/KAFKA-3190
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Critical
> Fix For: 0.10.2.0
>
>
> Currently KafkaProducer will invoke callback.onComplete() if it receives an 
> ApiException during send(). This breaks the guarantee that callback will be 
> invoked in order. It seems ApiException in send() only comes from metadata 
> refresh. If so, we can probably simply throw it instead of invoking 
> callback().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3123:
---
Fix Version/s: (was: 0.10.0.2)
   0.10.2.0

> Follower Broker cannot start if offsets are already out of range
> 
>
> Key: KAFKA-3123
> URL: https://issues.apache.org/jira/browse/KAFKA-3123
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Affects Versions: 0.9.0.0
>Reporter: Soumyajit Sahu
>Assignee: Soumyajit Sahu
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.2.0
>
> Attachments: 
> 0001-Fix-Follower-crashes-when-offset-out-of-range-during.patch
>
>
> I was trying to upgrade our test Windows cluster from 0.8.1.1 to 0.9.0 one 
> machine at a time. Our logs have just 2 hours of retention. I had re-imaged 
> the test machine under consideration, and got the following error in loop 
> after starting afresh with 0.9.0 broker:
> [2016-01-19 13:57:28,809] WARN [ReplicaFetcherThread-1-169595708], Replica 
> 15588 for partition [EventLogs4,1] reset its fetch offset from 0 to 
> current leader 169595708's start offset 334086 
> (kafka.server.ReplicaFetcherThread)
> [2016-01-19 13:57:28,809] ERROR [ReplicaFetcherThread-1-169595708], Error 
> getting offset for partition [EventLogs4,1] to broker 169595708 
> (kafka.server.ReplicaFetcherThread)
> java.lang.IllegalStateException: Compaction for partition [EXO_EventLogs4,1] 
> cannot be aborted and paused since it is in LogCleaningPaused state.
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply$mcV$sp(LogCleanerManager.scala:149)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.log.LogCleanerManager.abortAndPauseCleaning(LogCleanerManager.scala:140)
>   at kafka.log.LogCleaner.abortAndPauseCleaning(LogCleaner.scala:141)
>   at kafka.log.LogManager.truncateFullyAndStartAt(LogManager.scala:304)
>   at 
> kafka.server.ReplicaFetcherThread.handleOffsetOutOfRange(ReplicaFetcherThread.scala:185)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:152)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:122)
>   at scala.Option.foreach(Option.scala:236)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:122)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:120)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> I could unblock myself with a code change. I deleted the action for "case s 
> =>" in the LogCleanerManager.scala's abortAndPauseCleaning(). I think we 
> should not throw exception if the state is already LogCleaningAborted or 
> LogCleaningPaused in this function, but instead just let it roll.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603377#comment-15603377
 ] 

Jun Rao commented on KAFKA-4099:


[~becket_qin], thanks the explanation. What you described makes sense. So the 
issue is probably not that bad since the log won't be rolled as frequently as I 
thought. In the worse case, if we hit this issue, we may create twice as many 
segments as we ideally want to have in the interim. However, since this is 
relatively rare, we can probably just leave the current implementation as it is.

A related issue is on log retention. Suppose that an app reprocesses data from 
more than 7 days ago. What will happen is that those data will be written to a 
log segment only to be deleted when the log retention thread kicks in, at which 
point, a new segment will be rolled. So, in this case, a log will be rolled as 
frequently as log.retention.check.interval.ms, which defaults to 5 mins. I am 
wondering if we should improve this by configuring 
log.message.timestamp.difference.max.ms to match log.retention.ms. This will 
avoid older data to be unnecessarily written to the log. It will help 
time-based log rolling as well.

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above. If the first message in the segment does not 
> have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ismael Juma]

If we want to start a vote on this, I suggest starting a [VOTE] thread
instead of mentioning it in the middle of a very long [DISCUSS] thread. :)

Ismael

On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison 
wrote:

> This KIP has got muddy on differing opinions of what should be part of
> Kafka etc. I agree with Suresh, let's have a vote on if we actually want a
> REST client in Kafka core, and then we can work out what it actually looks
> like (personally I would like to reuse Confluents, great REST client if
> they donate it to ASF)
>
> For me +1 on REST client, this is a fundamental feature that's missing from
> Kafka.
>
> On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps  wrote:
>
> > Hey Suresh,
> >
> > I think we agree that REST apis are useful. I don't think we agree that
> > they need to be part of the Kafka project. We've had this discussion a
> > dozen odd times for different protocols---AMQP, REST, MQTT. Going back
> the
> > last five years we've always rejected these. That doesn't mean they
> aren't
> > useful, I think having these as separate is fine, they don't have any
> > complex interaction with Kafka, they just use the vanilla APIs like any
> of
> > the dozens of things on the ecosystem page.
> >
> > In terms of how they're developed. I think there are two discussions
> here:
> > 1. Separate project or not
> > 2. Standalone Apache project or github
> >
> > The first of those I just talked about---I think this makes sense as an
> > independent project.
> >
> > For the second of these, I actually don't think that spawning off a bunch
> > of itty-bitty independent Apache projects is a good thing. I think you
> can
> > see this in the Hadoop ecosystem where the parts kind of all evolve off
> in
> > different directions and don't really work together as a whole. I think
> > that makes sense for deep infrastructure projects, but not for these
> little
> > helper projects. I think a hub and spoke model where you have a central
> > project (Kafka) and then a bunch of helper tools that strive to fit in
> with
> > it (in terms of config, monitoring, apis, and every other conventions) is
> > actually much better. In any case there are already so many of these
> tools
> > (we capture maybe 20% of them on the ecosystem page), made by so many
> > people, and virtually all developed in this style, it is a bit late to
> put
> > the cat back in the bag.
> >
> > Perhaps it's just a difference in background. For my part I had many
> years
> > successfully running github-style projects, and i think that model can
> work
> > quite well for small things. I do think it is important for these
> projects
> > to clarify governance, which we should absolutely do, but realistically
> it
> > is a pretty simple tool, there isn't a huge governance challenge for
> > something like this because its scope is so limited ("take http requests,
> > make Kafka requests").
> >
> > More specifically, I don't think there is an actual problem being solved
> > here. I haven't heard any complaint about direction or patches not
> getting
> > added. The only complaint I've heard is missing features where the normal
> > "patches accepted" rule applies. I would urge people to actually get
> > involved in contribution here. In the future if there is a situation
> where
> > people don't like the direction of a given tool, they can fork it and
> > either turn it into an independent Apache project or develop it
> > independently, trying to do that preemptively seems a bit hostile.
> >
> > -Jay
> >
> > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> sur...@hortonworks.com>
> > wrote:
> >
> > > I am dividing this discussion into two parts:
> > > 1. REST APIs as core Apache Kafka capability
> > > This should be a core Kafka functionality. Same view has been reflected
> > by
> > > others (users and developers) as well. While we can debate whether
> other
> > > capabilities are core Kafka (Streams, Connect), it would be good
> separate
> > > that out and to keep this discussion focussed on REST APIs as proposed
> by
> > > this KIP. If there is ambivalence about the need for this in core
> kafka,
> > > we could have voting in the mailing list.
> > >
> > > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> > >
> > > 2. Community where Kafka REST APIs need to be collaborated on
> > > There is already a GitHub project that caters to Kafka REST APIs. But a
> > > company owned Github is less than ideal for collaboration due to lack
> of
> > > governance, open community and roadmap. This is reflected by many
> people
> > > interested in this functionality and still not contributing to and
> > > adopting the code base in the GitHub. I think trying overlay the ASF
> > > governance model on GitHub project, which is what the need is, seems
> > > unnecessary, if the code can be part of Apache Kafka.
> > >
> > > The question is, would Confluent be okay with licensing/contributing
> the
> > > code to 

[jira] [Resolved] (KAFKA-4338) Release Kafka 0.10.1.0 on Maven Central

[Emanuele Cesena (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Emanuele Cesena resolved KAFKA-4338.

Resolution: Invalid

Omg, you're right! Sorry about that.

> Release Kafka 0.10.1.0 on Maven Central
> ---
>
> Key: KAFKA-4338
> URL: https://issues.apache.org/jira/browse/KAFKA-4338
> Project: Kafka
>  Issue Type: Task
>Reporter: Emanuele Cesena
>
> Unless I'm missing something, Kafka 0.10.1.0 doesn't seem to be on maven 
> central yet:
> https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4338) Release Kafka 0.10.1.0 on Maven Central

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603250#comment-15603250
 ] 

Stevo Slavic commented on KAFKA-4338:
-

It's already there. See 
http://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.10.1.0/ and 
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.kafka%22

> Release Kafka 0.10.1.0 on Maven Central
> ---
>
> Key: KAFKA-4338
> URL: https://issues.apache.org/jira/browse/KAFKA-4338
> Project: Kafka
>  Issue Type: Task
>Reporter: Emanuele Cesena
>
> Unless I'm missing something, Kafka 0.10.1.0 doesn't seem to be on maven 
> central yet:
> https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4338) Release Kafka 0.10.1.0 on Maven Central

[Emanuele Cesena (JIRA)]

Emanuele Cesena created KAFKA-4338:
--

 Summary: Release Kafka 0.10.1.0 on Maven Central
 Key: KAFKA-4338
 URL: https://issues.apache.org/jira/browse/KAFKA-4338
 Project: Kafka
  Issue Type: Task
Reporter: Emanuele Cesena


Unless I'm missing something, Kafka 0.10.1.0 doesn't seem to be on maven 
central yet:
https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2015: Small refactoring to improve readability and reduc...

[picadoh]

Github user picadoh closed the pull request at:

https://github.com/apache/kafka/pull/2015


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ben Davison]

This KIP has got muddy on differing opinions of what should be part of
Kafka etc. I agree with Suresh, let's have a vote on if we actually want a
REST client in Kafka core, and then we can work out what it actually looks
like (personally I would like to reuse Confluents, great REST client if
they donate it to ASF)

For me +1 on REST client, this is a fundamental feature that's missing from
Kafka.

On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps  wrote:

> Hey Suresh,
>
> I think we agree that REST apis are useful. I don't think we agree that
> they need to be part of the Kafka project. We've had this discussion a
> dozen odd times for different protocols---AMQP, REST, MQTT. Going back the
> last five years we've always rejected these. That doesn't mean they aren't
> useful, I think having these as separate is fine, they don't have any
> complex interaction with Kafka, they just use the vanilla APIs like any of
> the dozens of things on the ecosystem page.
>
> In terms of how they're developed. I think there are two discussions here:
> 1. Separate project or not
> 2. Standalone Apache project or github
>
> The first of those I just talked about---I think this makes sense as an
> independent project.
>
> For the second of these, I actually don't think that spawning off a bunch
> of itty-bitty independent Apache projects is a good thing. I think you can
> see this in the Hadoop ecosystem where the parts kind of all evolve off in
> different directions and don't really work together as a whole. I think
> that makes sense for deep infrastructure projects, but not for these little
> helper projects. I think a hub and spoke model where you have a central
> project (Kafka) and then a bunch of helper tools that strive to fit in with
> it (in terms of config, monitoring, apis, and every other conventions) is
> actually much better. In any case there are already so many of these tools
> (we capture maybe 20% of them on the ecosystem page), made by so many
> people, and virtually all developed in this style, it is a bit late to put
> the cat back in the bag.
>
> Perhaps it's just a difference in background. For my part I had many years
> successfully running github-style projects, and i think that model can work
> quite well for small things. I do think it is important for these projects
> to clarify governance, which we should absolutely do, but realistically it
> is a pretty simple tool, there isn't a huge governance challenge for
> something like this because its scope is so limited ("take http requests,
> make Kafka requests").
>
> More specifically, I don't think there is an actual problem being solved
> here. I haven't heard any complaint about direction or patches not getting
> added. The only complaint I've heard is missing features where the normal
> "patches accepted" rule applies. I would urge people to actually get
> involved in contribution here. In the future if there is a situation where
> people don't like the direction of a given tool, they can fork it and
> either turn it into an independent Apache project or develop it
> independently, trying to do that preemptively seems a bit hostile.
>
> -Jay
>
> On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas 
> wrote:
>
> > I am dividing this discussion into two parts:
> > 1. REST APIs as core Apache Kafka capability
> > This should be a core Kafka functionality. Same view has been reflected
> by
> > others (users and developers) as well. While we can debate whether other
> > capabilities are core Kafka (Streams, Connect), it would be good separate
> > that out and to keep this discussion focussed on REST APIs as proposed by
> > this KIP. If there is ambivalence about the need for this in core kafka,
> > we could have voting in the mailing list.
> >
> > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> >
> > 2. Community where Kafka REST APIs need to be collaborated on
> > There is already a GitHub project that caters to Kafka REST APIs. But a
> > company owned Github is less than ideal for collaboration due to lack of
> > governance, open community and roadmap. This is reflected by many people
> > interested in this functionality and still not contributing to and
> > adopting the code base in the GitHub. I think trying overlay the ASF
> > governance model on GitHub project, which is what the need is, seems
> > unnecessary, if the code can be part of Apache Kafka.
> >
> > The question is, would Confluent be okay with licensing/contributing the
> > code to Kafka project (assuming either Confluent or another contributor
> > can work on it)? I see clear intent in collaborating with others on REST
> > APIs from confluent. Why not do it in Kafka project under ASF? This takes
> > away all the barrier to collaboration? Alternatively, if Confluent is not
> > willing to contribute the code from GitHub, would anyone veto building a
> > separate REST API functionality in ASF Kafka? There are enough people who

[jira] [Commented] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603156#comment-15603156
 ] 

ASF GitHub Bot commented on KAFKA-4331:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2049


> Kafka Streams resetter is slow because it joins the same group for each topic
> -
>
> Key: KAFKA-4331
> URL: https://issues.apache.org/jira/browse/KAFKA-4331
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Roger Hoover
>Assignee: Matthias J. Sax
> Fix For: 0.10.2.0
>
>
> The resetter is joining the same group for each topic which takes ~10secs in 
> my testing.  This makes the reset very slow when you have a lot of topics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2049: KAFKA-4331: Kafka Streams resetter is slow because...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2049


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4331:
-
   Resolution: Fixed
Fix Version/s: 0.10.2.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2049
[https://github.com/apache/kafka/pull/2049]

> Kafka Streams resetter is slow because it joins the same group for each topic
> -
>
> Key: KAFKA-4331
> URL: https://issues.apache.org/jira/browse/KAFKA-4331
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Roger Hoover
>Assignee: Matthias J. Sax
> Fix For: 0.10.2.0
>
>
> The resetter is joining the same group for each topic which takes ~10secs in 
> my testing.  This makes the reset very slow when you have a lot of topics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

Hey Suresh,

I think we agree that REST apis are useful. I don't think we agree that
they need to be part of the Kafka project. We've had this discussion a
dozen odd times for different protocols---AMQP, REST, MQTT. Going back the
last five years we've always rejected these. That doesn't mean they aren't
useful, I think having these as separate is fine, they don't have any
complex interaction with Kafka, they just use the vanilla APIs like any of
the dozens of things on the ecosystem page.

In terms of how they're developed. I think there are two discussions here:
1. Separate project or not
2. Standalone Apache project or github

The first of those I just talked about---I think this makes sense as an
independent project.

For the second of these, I actually don't think that spawning off a bunch
of itty-bitty independent Apache projects is a good thing. I think you can
see this in the Hadoop ecosystem where the parts kind of all evolve off in
different directions and don't really work together as a whole. I think
that makes sense for deep infrastructure projects, but not for these little
helper projects. I think a hub and spoke model where you have a central
project (Kafka) and then a bunch of helper tools that strive to fit in with
it (in terms of config, monitoring, apis, and every other conventions) is
actually much better. In any case there are already so many of these tools
(we capture maybe 20% of them on the ecosystem page), made by so many
people, and virtually all developed in this style, it is a bit late to put
the cat back in the bag.

Perhaps it's just a difference in background. For my part I had many years
successfully running github-style projects, and i think that model can work
quite well for small things. I do think it is important for these projects
to clarify governance, which we should absolutely do, but realistically it
is a pretty simple tool, there isn't a huge governance challenge for
something like this because its scope is so limited ("take http requests,
make Kafka requests").

More specifically, I don't think there is an actual problem being solved
here. I haven't heard any complaint about direction or patches not getting
added. The only complaint I've heard is missing features where the normal
"patches accepted" rule applies. I would urge people to actually get
involved in contribution here. In the future if there is a situation where
people don't like the direction of a given tool, they can fork it and
either turn it into an independent Apache project or develop it
independently, trying to do that preemptively seems a bit hostile.

-Jay

On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas 
wrote:

> I am dividing this discussion into two parts:
> 1. REST APIs as core Apache Kafka capability
> This should be a core Kafka functionality. Same view has been reflected by
> others (users and developers) as well. While we can debate whether other
> capabilities are core Kafka (Streams, Connect), it would be good separate
> that out and to keep this discussion focussed on REST APIs as proposed by
> this KIP. If there is ambivalence about the need for this in core kafka,
> we could have voting in the mailing list.
>
> Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
>
> 2. Community where Kafka REST APIs need to be collaborated on
> There is already a GitHub project that caters to Kafka REST APIs. But a
> company owned Github is less than ideal for collaboration due to lack of
> governance, open community and roadmap. This is reflected by many people
> interested in this functionality and still not contributing to and
> adopting the code base in the GitHub. I think trying overlay the ASF
> governance model on GitHub project, which is what the need is, seems
> unnecessary, if the code can be part of Apache Kafka.
>
> The question is, would Confluent be okay with licensing/contributing the
> code to Kafka project (assuming either Confluent or another contributor
> can work on it)? I see clear intent in collaborating with others on REST
> APIs from confluent. Why not do it in Kafka project under ASF? This takes
> away all the barrier to collaboration? Alternatively, if Confluent is not
> willing to contribute the code from GitHub, would anyone veto building a
> separate REST API functionality in ASF Kafka? There are enough people who
> want to work on this and maintain it.
>
> Regards,
> Suresh
>
>
>
> On 10/21/16, 9:41 PM, "Harsha Chintalapani"  wrote:
>
> >Sriram,
> >   "Can the streaming platform exist without stream processing? - No.
> >Processing stream data again is a core part of streaming platform."
> >
> >Yes, it can. There are no.of Stream processing frameworks out there, and
> >they all have integration into Kafka.
> >It doesn't need to be developed within Kafka.
> >
> >
> >"Can the platform exist without the rest proxy? - Yes. The proxy does not
> >complete the platform vision in anyway. It is just a good to have 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Suresh Srinivas]

I am dividing this discussion into two parts:
1. REST APIs as core Apache Kafka capability
This should be a core Kafka functionality. Same view has been reflected by
others (users and developers) as well. While we can debate whether other
capabilities are core Kafka (Streams, Connect), it would be good separate
that out and to keep this discussion focussed on REST APIs as proposed by
this KIP. If there is ambivalence about the need for this in core kafka,
we could have voting in the mailing list.

Can we get an agreement on this? I am +1 on REST API in Apache Kafka.

2. Community where Kafka REST APIs need to be collaborated on
There is already a GitHub project that caters to Kafka REST APIs. But a
company owned Github is less than ideal for collaboration due to lack of
governance, open community and roadmap. This is reflected by many people
interested in this functionality and still not contributing to and
adopting the code base in the GitHub. I think trying overlay the ASF
governance model on GitHub project, which is what the need is, seems
unnecessary, if the code can be part of Apache Kafka.

The question is, would Confluent be okay with licensing/contributing the
code to Kafka project (assuming either Confluent or another contributor
can work on it)? I see clear intent in collaborating with others on REST
APIs from confluent. Why not do it in Kafka project under ASF? This takes
away all the barrier to collaboration? Alternatively, if Confluent is not
willing to contribute the code from GitHub, would anyone veto building a
separate REST API functionality in ASF Kafka? There are enough people who
want to work on this and maintain it.

Regards,
Suresh



On 10/21/16, 9:41 PM, "Harsha Chintalapani"  wrote:

>Sriram,
>   "Can the streaming platform exist without stream processing? - No.
>Processing stream data again is a core part of streaming platform."
>
>Yes, it can. There are no.of Stream processing frameworks out there, and
>they all have integration into Kafka.
>It doesn't need to be developed within Kafka.
>
>
>"Can the platform exist without the rest proxy? - Yes. The proxy does not
>complete the platform vision in anyway. It is just a good to have tool
>that
>might be required by quite a few users and there is an active project that
>works on this - https://github.com/confluentinc/kafka-rest;
>
>The rest proxy is as important as any API. The vision that shown here
>http://kafka.apache.org/intro
>require users to write the producers and consumers to get their data into
>and out of Kafka, without which having Streams or Connect won't help
>anyone.
>The rest proxy makes easier for users get their data into Kafka.
>Adding the rest proxy to the project doesn't invalidate the current
>vision,
>it only strengthens it.
>
>Thanks,
>Harsha
>
>
>
>
>On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian 
>wrote:
>
>FWIW, Apache Kafka has evolved a lot from where it started. It did start
>as
>a messaging system. Over time we realized that that the vision for Kafka
>is
>to build a streaming platform and not just a messaging system. You can
>take
>a look at the site for more description about what comprises the streaming
>platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
>
>Can the streaming platform exist without Connect? - No. Data integration
>is
>fundamental to building an end to end platform
>
>Can the streaming platform exist without stream processing? - No.
>Processing stream data again is a core part of streaming platform.
>
>Can the streaming platform exist without clients? - We at least need one
>client library to complete the platform. Our Java clients help us to
>complete the platform story. The rest of the clients are built and
>maintained outside the project.
>
>Can the platform exist without the rest proxy? - Yes. The proxy does not
>complete the platform vision in anyway. It is just a good to have tool
>that
>might be required by quite a few users and there is an active project that
>works on this - https://github.com/confluentinc/kafka-rest
>
>
>
>
>On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
>
>wrote:
>
>> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
>Connect
>> are not subjective?
>>
>> > "there are likely places that can live without a rest proxy"
>>
>> There are also places that can live without Kafka Streams and Kafka
>> Connect.
>>
>> Nacho
>>
>> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao  wrote:
>>
>> > At the high level, I think ideally it makes sense to add a component
>>to
>> > Apache Kafka if (1) it's widely needed and (2) it needs tight
>integration
>> > with Kafka core. For Kafka Stream, we do expect stream processing will
>be
>> > used widely in the future. Implementation wise, Kafka Stream only
>> supports
>> > getting data from Kafka and leverages quite a few of the core
>> > functionalities in Kafka core. For example, it uses customized
>>rebalance
>> > 

[jira] [Commented] (KAFKA-4335) FileStreamSource Connector not working for large files (~ 1GB)

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602851#comment-15602851
 ] 

Ewen Cheslack-Postava commented on KAFKA-4335:
--

Can you be more specific about what isn't working? Does it throw an exception 
or some other error?

> FileStreamSource Connector not working for large files (~ 1GB)
> --
>
> Key: KAFKA-4335
> URL: https://issues.apache.org/jira/browse/KAFKA-4335
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Rahul Shukla
>Assignee: Ewen Cheslack-Postava
>
> I was trying to sink large file about (1gb). FileStreamSource connector is 
> not working for that it's working fine for small files.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Guozhang Wang]

Overall I think the motivation is common and of interests to lots of users.
Would like to throw my two cents on this discussion:

1. Kafka topics can be used in different ways. For some categories of
topics (think: "pageView" event topics), it is a shared topic among
different teams / apps within the organization and lots of temporary
consumers (for debugging, trouble shooting, prototype development, etc) can
come and go dynamically, in which case it is hard to track all of such
consumer and maintain the minimum committed offsets; on the other hand,
there are another category of topics (think: stream-app owned intermediate
topics like "pricing-enriched-bid-activity", as Becket mentioned above)
which are particularly own but only one or a few apps, and hence the
consumer groups for those topics are pre-defined and roughly static. In
this case I think it makes sense to allow such consumer-drive log retention
features.

2. In this case, my question is then whether this bookkeeping of
min-committed-offsets should be done at the brokers side or it should be on
the app side. My gut feeling is that it could be better bookkept on the app
(i.e. client) side which has the full information of the "registered
consumer groups" for certain topics, and then knows the
min-committed-offsets. And a slightly-modified KIP-47 mentioned by Dong
could a better fit, where a) app side bookkeep the consumer-driven min
offset based on their committed offsets, by either talking to the consumer
clients directly or query broker for the committed offsets of those
registered consumer groups, and then b) write
*log.retention.min.offset* periodically
to broker to let it delete old segments before that offset (NOTE that the
semantics is exactly the same as to KIP-47, while the only difference is
that we use offset instead of timestamp to indicate, which can be honor by
the same implementation of KIP-47 on broker side).

My arguments for letting the app side to bookkeep such min-offsets and only
let brokers to take requests to delete segments accordingly are 1) keeping
the broker simple without any querying each other about such offsets and
does the min() calculation, rather only keeping / deleting messages from
client admin requests, and 2) allowing more generalized client-driven log
retention policies with KIP-47 (i.e. broker is brainless and only take
requests while client-app can apply any customized logic to determine the
config values of *og.retention.min.offset or **og.retention.min.timestamp* that
they send to the brokers).



Guozhang


On Sat, Oct 22, 2016 at 5:46 PM, Becket Qin  wrote:

> Hi David,
>
> > 1. What scenario is used to this configuration?
>
> One scenario is stream processing pipeline. In a stream processing DAG,
> there will be a bunch of intermediate result, we only care about the
> consumer group that is in the downstream of the DAG, but not other groups.
> Ideally we want to delete the log of the intermediate topics right after
> all the downstream processing jobs has successfully processed the messages.
> In that case, we only care about the downstream processing jobs, but not
> other groups. That means if a down stream job did not commit offset for
> some reason, we want to wait for that job. Without the predefined
> interested group, it is hard to achieve this.
>
>
> 2. Yes, the configuration should be at topic level and set dynamically.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Oct 21, 2016 at 7:40 AM, 东方甲乙 <254479...@qq.com> wrote:
>
> > Hi Mayuresh,
> > Thanks for the reply:
> > 1.  In the log retention check schedule, the broker first find the all
> the
> > consumed group which are consuming this topic, and query the commit
> offset
> > of this consumed group for the topic
> > using the OffsetFetch API. And the min commit offset is the minimal
> commit
> > offset between these commit offsets.
> >
> >
> > 2.  If the console consumer reading and commit, its commit offset will be
> > used to calculate the min commit offset for this topic.
> > We can avoid the random consumer using the method Becket suggested.
> >
> >
> > 3. It will not delete the log immediately, the log will stay some time (
> > retention.commitoffset.ms), and after that we only delete
> > the log segments whose offsets are less than the min commit offset.  So
> > the user can rewind its offset in the log.retention.ms.
> >
> >
> > Thanks,
> > David
> >
> >
> >
> >
> > -- 原始邮件 --
> > 发件人: "Mayuresh Gharat";;
> > 发送时间: 2016年10月19日(星期三) 上午10:25
> > 收件人: "dev";
> >
> > 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log
> retention
> >
> >
> >
> > Hi David,
> >
> > Thanks for the KIP.
> >
> > I had some questions/suggestions :
> >
> > It would be great if you can explain with an example about how the min
> > offset for all the consumers will be calculated, in the KIP.
> > What I meant was, it would be great to 

Re: Kafka KIP meeting Oct 19 at 11:00am PST

[Mayuresh Gharat]

I agree with Nacho.
+1 for the KIP.

Thanks,

Mayuresh

On Fri, Oct 21, 2016 at 11:46 AM, Nacho Solis 
wrote:

> I think a separate KIP is a good idea as well.  Note however that potential
> decisions in this KIP could affect the other KIP.
>
> Nacho
>
> On Fri, Oct 21, 2016 at 10:23 AM, Jun Rao  wrote:
>
> > Michael,
> >
> > Yes, doing a separate KIP to address the null payload issue for compacted
> > topics is a good idea.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Oct 21, 2016 at 12:57 AM, Michael Pearce 
> > wrote:
> >
> > > I had noted that what ever the solution having compaction based on null
> > > payload was agreed isn't elegant.
> > >
> > > Shall we raise another kip to : as discussed propose using a attribute
> > bit
> > > for delete/compaction flag as well/or instead of null value and
> updating
> > > compaction logic to look at that delelete/compaction attribute
> > >
> > > I believe this is less contentious, so that at least we get that done
> > > alleviating some concerns whilst the below gets discussed further?
> > >
> > > 
> > > From: Jun Rao 
> > > Sent: Wednesday, October 19, 2016 8:56:52 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: Kafka KIP meeting Oct 19 at 11:00am PST
> > >
> > > The following are the notes from today's KIP discussion.
> > >
> > >
> > >- KIP-82 - add record header: We agreed that there are use cases for
> > >third-party vendors building tools around Kafka. We haven't reached
> > the
> > >conclusion whether the added complexity justifies the use cases. We
> > will
> > >follow up on the mailing list with use cases, container format
> people
> > > have
> > >been using, and details on the proposal.
> > >
> > >
> > > The video will be uploaded soon in https://cwiki.apache.org/
> > > confluence/display/KAFKA/Kafka+Improvement+Proposals .
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Oct 17, 2016 at 10:49 AM, Jun Rao  wrote:
> > >
> > > > Hi, Everyone.,
> > > >
> > > > We plan to have a Kafka KIP meeting this coming Wednesday at 11:00am
> > PST.
> > > > If you plan to attend but haven't received an invite, please let me
> > know.
> > > > The following is the tentative agenda.
> > > >
> > > > Agenda:
> > > > KIP-82: add record header
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
>
>
> --
> Nacho (Ignacio) Solis
> Kafka
> nso...@linkedin.com
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[jira] [Created] (KAFKA-4337) Create topic in multiple zookeepers with Kafka AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets created in only one of the specified zookeeper)

[Bharat Patel (JIRA)]

Bharat Patel created KAFKA-4337:
---

 Summary: Create topic in multiple zookeepers with Kafka 
AdminUtils.CreateTopic JAVA API in Kafka 0.9.0.1 gives Error (Topic gets 
created in only one of the specified zookeeper)
 Key: KAFKA-4337
 URL: https://issues.apache.org/jira/browse/KAFKA-4337
 Project: Kafka
  Issue Type: Bug
  Components: admin
Reporter: Bharat Patel


 Want to use below code snippet to create topic in multiple zookeepers with 
Kafka Java APIS. when I specify 2 zookeeprs IPs in zookeeperConnect variable it 
only creates topic in anyone of zookeeper. Both the zookeeper are 2 different 
kafka clusters.

 String zookeeperConnect = zookeeperIPs; // Multiple zookeeper IPs
   int sessionTimeoutMs = 10 * 1000;
   int connectionTimeoutMs = 8 * 1000;

   try {
ZkClient zkClient = new ZkClient(
zookeeperConnect,
sessionTimeoutMs,
connectionTimeoutMs,
ZKStringSerializer$.MODULE$);

boolean isSecureKafkaCluster = false;
ZkUtils zkUtils = new ZkUtils(zkClient, 
new ZkConnection(zookeeperConnect), isSecureKafkaCluster);

 String topic1 = "nameofTopictobeCreated";
 int partitions = 1;
 int replication = 1;
 Properties topicConfig = new Properties(); // add per-topic configurations 
settings here
 AdminUtils.createTopic(zkUtils, topic1, partitions, replication, 
topicConfig);




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602189#comment-15602189
 ] 

Ismael Juma commented on KAFKA-4099:


I filed KAFKA-4336 to make sure we don't lose track of this. It would be good 
to continue the discussion there.

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above. If the first message in the segment does not 
> have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4335) FileStreamSource Connector not working for large files (~ 1GB)

[Rahul Shukla (JIRA)]

Rahul Shukla created KAFKA-4335:
---

 Summary: FileStreamSource Connector not working for large files (~ 
1GB)
 Key: KAFKA-4335
 URL: https://issues.apache.org/jira/browse/KAFKA-4335
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 0.10.0.0
Reporter: Rahul Shukla
Assignee: Ewen Cheslack-Postava


I was trying to sink large file about (1gb). FileStreamSource connector is not 
working for that it's working fine for small files.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4336) Frequent log rolling when there's a mix of delayed and current data

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602187#comment-15602187
 ] 

Ismael Juma commented on KAFKA-4336:


cc [~junrao] [~becket_qin] [~ewencp] [~norwood]

> Frequent log rolling when there's a mix of delayed and current data
> ---
>
> Key: KAFKA-4336
> URL: https://issues.apache.org/jira/browse/KAFKA-4336
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Ismael Juma
> Fix For: 0.10.2.0
>
>
> Jun said in KAFKA-4099: suppose that you have 2 producers, one producing data 
> with the current timestamp and another producing data with timestamp 7 days 
> old (e.g., if some data is delayed or some old data is replayed), this will 
> still cause the log segments to roll frequently. This may not be common, but 
> can definitely happen. So, it seems we will still need to improve on how log 
> rolls.
> I am creating a new JIRA because KAFKA-4099 is closed so easy to lose track.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4336) Frequent log rolling when there's a mix of delayed and current data

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4336:
--

 Summary: Frequent log rolling when there's a mix of delayed and 
current data
 Key: KAFKA-4336
 URL: https://issues.apache.org/jira/browse/KAFKA-4336
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.1.0
Reporter: Ismael Juma
 Fix For: 0.10.2.0


Jun said in KAFKA-4099: suppose that you have 2 producers, one producing data 
with the current timestamp and another producing data with timestamp 7 days old 
(e.g., if some data is delayed or some old data is replayed), this will still 
cause the log segments to roll frequently. This may not be common, but can 
definitely happen. So, it seems we will still need to improve on how log rolls.

I am creating a new JIRA because KAFKA-4099 is closed so easy to lose track.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602159#comment-15602159
 ] 

Davor Poldrugo edited comment on KAFKA-4273 at 10/24/16 2:36 PM:
-

Is this possibly resolved in 0.10.1.0 (KAFKA-3740) with streams config: 
*rocksdb.config.setter* ?
https://kafka.apache.org/0101/documentation.html#streamsconfigs

Description of the config: _A Rocks DB config setter class that implements the 
RocksDBConfigSetter interface_

Interface is here: 
https://github.com/apache/kafka/blob/0.10.1.0/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java


was (Author: dpoldrugo):
Is this possibly resolved in 0.10.1.0 with streams config: 
*rocksdb.config.setter* ?
https://kafka.apache.org/0101/documentation.html#streamsconfigs

Description of the config: _A Rocks DB config setter class that implements the 
RocksDBConfigSetter interface_

Interface is here: 
https://github.com/apache/kafka/blob/0.10.1.0/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java

> Streams DSL - Add TTL / retention period support for intermediate topics and 
> state stores
> -
>
> Key: KAFKA-4273
> URL: https://issues.apache.org/jira/browse/KAFKA-4273
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Davor Poldrugo
>Assignee: Guozhang Wang
>
> Hi!
> I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state 
> as far as I know - it's not configurable.
> In my use case my data has TTL / retnetion period. It's 48 hours. After that 
> - data can be discarded.
> I join two topics: "messages" and "prices" using windowed inner join.
> The two intermediate Kafka topics for this join are named:
>  * messages-prices-join-this-changelog
>  * messages-prices-join-other-changelog
> Since these topics are created as compacted by Kafka Streams, and I don't 
> wan't to keep data forever, I have altered them to not use compaction. Right 
> now my RocksDB state stores grow indefinitely, and I don't have any options 
> to define TTL, or somehow periodically clean the older data.
> A "hack" that I use to keep my disk usage low - I have schedulled a job to 
> periodically stop Kafka Streams instances - one at the time. This triggers a 
> rebalance, and partitions migrate to other instances. When the instance is 
> started again, there's another rebalance, and sometimes this instance starts 
> processing partitions that wasn't processing before the stop - which leads to 
> deletion of the RocksDB state store for those partitions 
> (state.cleanup.delay.ms). In the next rebalance the local store is recreated 
> with a restore consumer - which reads data from - as previously mentioned - a 
> non compacted topic. And this effectively leads to a "hacked TTL support" in 
> Kafka Streams DSL.
> Questions:
>  * Do you think would be reasonable to add support in the DSL api to define 
> TTL for local store?
>  * Which opens another question - there are use cases which don't need the 
> intermediate topics to be created as "compact". Could also this be added to 
> the DSL api? Maybe only this could be added, and this flag should also be 
> used for the RocksDB TTL. Of course in this case another config would be 
> mandatory - the retention period or TTL for the intermediate topics and the 
> state stores. I saw there is a new cleanup.policy - compact_and_delete - 
> added with KAFKA-4015.
>  * Which also leads to another question, maybe some intermediate topics / 
> state stores need different TTL, so a it's not as simple as that. But after 
> KAFKA-3870, it will be easier.
> RocksDB supports TTL:
>  * 
> https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
>  * https://github.com/facebook/rocksdb/wiki/Time-to-Live
>  * 
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java
> A somehow similar issue: KAFKA-4212



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602159#comment-15602159
 ] 

Davor Poldrugo edited comment on KAFKA-4273 at 10/24/16 2:34 PM:
-

Is this possibly resolved in 0.10.1.0 with streams config: 
*rocksdb.config.setter* ?
https://kafka.apache.org/0101/documentation.html#streamsconfigs

Description of the config: _A Rocks DB config setter class that implements the 
RocksDBConfigSetter interface_

Interface is here: 
https://github.com/apache/kafka/blob/0.10.1.0/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java


was (Author: dpoldrugo):
Is this possibly resolved in 0.10.1.0 with streams config: 
*rocksdb.config.setter* ?
http://kafka.apache.org/documentation#streamsconfigs

Description of the config: _A Rocks DB config setter class that implements the 
RocksDBConfigSetter interface_

Interface is here: 
https://github.com/apache/kafka/blob/0.10.1.0/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java

> Streams DSL - Add TTL / retention period support for intermediate topics and 
> state stores
> -
>
> Key: KAFKA-4273
> URL: https://issues.apache.org/jira/browse/KAFKA-4273
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Davor Poldrugo
>Assignee: Guozhang Wang
>
> Hi!
> I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state 
> as far as I know - it's not configurable.
> In my use case my data has TTL / retnetion period. It's 48 hours. After that 
> - data can be discarded.
> I join two topics: "messages" and "prices" using windowed inner join.
> The two intermediate Kafka topics for this join are named:
>  * messages-prices-join-this-changelog
>  * messages-prices-join-other-changelog
> Since these topics are created as compacted by Kafka Streams, and I don't 
> wan't to keep data forever, I have altered them to not use compaction. Right 
> now my RocksDB state stores grow indefinitely, and I don't have any options 
> to define TTL, or somehow periodically clean the older data.
> A "hack" that I use to keep my disk usage low - I have schedulled a job to 
> periodically stop Kafka Streams instances - one at the time. This triggers a 
> rebalance, and partitions migrate to other instances. When the instance is 
> started again, there's another rebalance, and sometimes this instance starts 
> processing partitions that wasn't processing before the stop - which leads to 
> deletion of the RocksDB state store for those partitions 
> (state.cleanup.delay.ms). In the next rebalance the local store is recreated 
> with a restore consumer - which reads data from - as previously mentioned - a 
> non compacted topic. And this effectively leads to a "hacked TTL support" in 
> Kafka Streams DSL.
> Questions:
>  * Do you think would be reasonable to add support in the DSL api to define 
> TTL for local store?
>  * Which opens another question - there are use cases which don't need the 
> intermediate topics to be created as "compact". Could also this be added to 
> the DSL api? Maybe only this could be added, and this flag should also be 
> used for the RocksDB TTL. Of course in this case another config would be 
> mandatory - the retention period or TTL for the intermediate topics and the 
> state stores. I saw there is a new cleanup.policy - compact_and_delete - 
> added with KAFKA-4015.
>  * Which also leads to another question, maybe some intermediate topics / 
> state stores need different TTL, so a it's not as simple as that. But after 
> KAFKA-3870, it will be easier.
> RocksDB supports TTL:
>  * 
> https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
>  * https://github.com/facebook/rocksdb/wiki/Time-to-Live
>  * 
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java
> A somehow similar issue: KAFKA-4212



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602159#comment-15602159
 ] 

Davor Poldrugo commented on KAFKA-4273:
---

Is this possibly resolved in 0.10.1.0 with streams config: 
*rocksdb.config.setter* ?
http://kafka.apache.org/documentation#streamsconfigs

Description of the config: _A Rocks DB config setter class that implements the 
RocksDBConfigSetter interface_

Interface is here: 
https://github.com/apache/kafka/blob/0.10.1.0/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java

> Streams DSL - Add TTL / retention period support for intermediate topics and 
> state stores
> -
>
> Key: KAFKA-4273
> URL: https://issues.apache.org/jira/browse/KAFKA-4273
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Davor Poldrugo
>Assignee: Guozhang Wang
>
> Hi!
> I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state 
> as far as I know - it's not configurable.
> In my use case my data has TTL / retnetion period. It's 48 hours. After that 
> - data can be discarded.
> I join two topics: "messages" and "prices" using windowed inner join.
> The two intermediate Kafka topics for this join are named:
>  * messages-prices-join-this-changelog
>  * messages-prices-join-other-changelog
> Since these topics are created as compacted by Kafka Streams, and I don't 
> wan't to keep data forever, I have altered them to not use compaction. Right 
> now my RocksDB state stores grow indefinitely, and I don't have any options 
> to define TTL, or somehow periodically clean the older data.
> A "hack" that I use to keep my disk usage low - I have schedulled a job to 
> periodically stop Kafka Streams instances - one at the time. This triggers a 
> rebalance, and partitions migrate to other instances. When the instance is 
> started again, there's another rebalance, and sometimes this instance starts 
> processing partitions that wasn't processing before the stop - which leads to 
> deletion of the RocksDB state store for those partitions 
> (state.cleanup.delay.ms). In the next rebalance the local store is recreated 
> with a restore consumer - which reads data from - as previously mentioned - a 
> non compacted topic. And this effectively leads to a "hacked TTL support" in 
> Kafka Streams DSL.
> Questions:
>  * Do you think would be reasonable to add support in the DSL api to define 
> TTL for local store?
>  * Which opens another question - there are use cases which don't need the 
> intermediate topics to be created as "compact". Could also this be added to 
> the DSL api? Maybe only this could be added, and this flag should also be 
> used for the RocksDB TTL. Of course in this case another config would be 
> mandatory - the retention period or TTL for the intermediate topics and the 
> state stores. I saw there is a new cleanup.policy - compact_and_delete - 
> added with KAFKA-4015.
>  * Which also leads to another question, maybe some intermediate topics / 
> state stores need different TTL, so a it's not as simple as that. But after 
> KAFKA-3870, it will be easier.
> RocksDB supports TTL:
>  * 
> https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
>  * https://github.com/facebook/rocksdb/wiki/Time-to-Live
>  * 
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java
> A somehow similar issue: KAFKA-4212



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2089) MetadataTest transient failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602091#comment-15602091
 ] 

ASF GitHub Bot commented on KAFKA-2089:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2057

KAFKA-2089: Increase metadata wait time in unit test

Increase timeout in test to avoid transient failures due to long GC or slow 
machine.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2089

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2057.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2057


commit 1b00341724f59ff545c5f808316df64538963bf0
Author: Rajini Sivaram 
Date:   2016-10-24T13:47:02Z

KAFKA-2089: Increase metadata wait time in unit test




> MetadataTest transient failure
> --
>
> Key: KAFKA-2089
> URL: https://issues.apache.org/jira/browse/KAFKA-2089
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2089.patch, KAFKA-2089.patch, 
> KAFKA-2089_2015-04-13_18:59:33.patch
>
>
> org.apache.kafka.clients.MetadataTest > testMetadata FAILED
> java.lang.AssertionError:
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertFalse(Assert.java:68)
> at org.junit.Assert.assertFalse(Assert.java:79)
> at 
> org.apache.kafka.clients.MetadataTest.tearDown(MetadataTest.java:34)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2057: KAFKA-2089: Increase metadata wait time in unit te...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2057

KAFKA-2089: Increase metadata wait time in unit test

Increase timeout in test to avoid transient failures due to long GC or slow 
machine.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2089

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2057.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2057


commit 1b00341724f59ff545c5f808316df64538963bf0
Author: Rajini Sivaram 
Date:   2016-10-24T13:47:02Z

KAFKA-2089: Increase metadata wait time in unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka clients

[Rajini Sivaram]

If there are no more comments or suggestions, I will start voting process
for this KIP later this week.

Please feel free to add any comments in the meantime.

Thank you,

Rajini

On Fri, Oct 7, 2016 at 11:32 PM, Gwen Shapira  wrote:

> Huge +1.
>
> Thank you Rajini for all the hard work on security improvements. With
> more Docker deployments of micro-services, getting rid of mandatory
> config file will be super important.
>
> Gwen
>
> On Thu, Oct 6, 2016 at 5:43 AM, Edoardo Comar  wrote:
> > Hi Rajini
> > great improvement and I see you had the code ready ... !
> >
> > I now think it'd easier to implement a PR for KIP-83 (multiple logins per
> > JVM) on top of yours,
> > as you have now identified a client property that can be used for caching
> > logins.
> >
> > I'm actually wondering if I caused you to cut down your KIP !!
> >
> > Also, I think your JIRA encompasses
> > https://issues.apache.org/jira/browse/KAFKA-3302
> >
> > thanks,
> > Edo
> > --
> > Edoardo Comar
> > IBM MessageHub
> > eco...@uk.ibm.com
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> > IBM United Kingdom Limited Registered in England and Wales with number
> > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> PO6
> > 3AU
> >
> >
> >
> > From:   Rajini Sivaram 
> > To: dev@kafka.apache.org
> > Date:   06/10/2016 10:49
> > Subject:[DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka
> > clients
> >
> >
> >
> > Hi all,
> >
> > I have just created KIP-85 to enable JAAS login context for Kafka clients
> > to be configured without a physical file:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 85%3A+Dynamic+JAAS+configuration+for+Kafka+clients
> >
> >
> >
> > Comments and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Regards,

Rajini

[jira] [Commented] (KAFKA-4311) Multi layer cache eviction causes forwarding to incorrect ProcessorNode

[Frank Lyaruu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601431#comment-15601431
 ] 

Frank Lyaruu commented on KAFKA-4311:
-

I'm also out at a conference most of this week. If I can find reliable WiFi, 
I'll have a look, otherwise it will be next week.

> Multi layer cache eviction causes forwarding to incorrect ProcessorNode 
> 
>
> Key: KAFKA-4311
> URL: https://issues.apache.org/jira/browse/KAFKA-4311
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.1
>
>
> The two exceptions below were reported by Frank on the dev mailing list. 
> After investigation, the root cause is multiple cache evictions happening in 
> the same topology. 
> Given a topology like the one below. If a record arriving in `tableOne` 
> causes a cache eviction, it will trigger the `leftJoin` that will do a `get` 
> from `reducer-store`. If the key is not currently cached in `reducer-store`, 
> but is in the backing store, it will be put into the cache, and it may also 
> trigger an eviction. If it does trigger an eviction and the eldest entry is 
> dirty it will flush the dirty keys. It is at this point that the exception in 
> the comment happens (ClassCastException). This occurs because the 
> ProcessorContext is still set to the context of the `leftJoin` and the next 
> child in the topology is `mapValues`.
> We need to set the correct `ProcessorNode`, on the context,  in the 
> `ForwardingCacheFlushListener` prior to calling `context.forward`. We also 
> need to set remember to reset the `ProcessorNode` to the previous node once 
> `context.forward` has completed.
> {code}
> final KTable one = builder.table(Serdes.String(), 
> Serdes.String(), tableOne, tableOne);
> final KTable two = builder.table(Serdes.Long(), 
> Serdes.String(), tableTwo, tableTwo);
> final KTable reduce = two.groupBy(new 
> KeyValueMapper>() {
> @Override
> public KeyValue apply(final Long key, final String 
> value) {
> return new KeyValue<>(value, key);
> }
> }, Serdes.String(), Serdes.Long())
> .reduce(new Reducer() {
> @Override
> public Long apply(final Long value1, final Long value2) {
> return value1 + value2;
> }
> }, new Reducer() {
> @Override
> public Long apply(final Long value1, final Long value2) {
> return value1 - value2;
> }
> }, "reducer-store");
> one.leftJoin(reduce, new ValueJoiner() {
> @Override
> public String apply(final String value1, final Long value2) {
> return value1 + ":" + value2;
> }
> })
> .mapValues(new ValueMapper() {
> @Override
> public String apply(final String value) {
> return value;
> }
> });
> {code}
> This exception is actually a symptom of the exception reported below in the 
> comment. After the first exception is thrown, the StreamThread triggers a 
> shutdown that then throws this exception.
> [StreamThread-1] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [StreamThread-1] Failed to close state manager for StreamTask 0_0:
> org.apache.kafka.streams.errors.ProcessorStateException: task [0_0] Failed
> to close state store addr-organization
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:342)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:121)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:341)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:322)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.closeAllStateManagers(StreamThread.java:338)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:299)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:262)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:245)
> Caused by: java.lang.IllegalStateException: Key found in dirty key set, but
> entry is null
> at
> 

[jira] [Updated] (KAFKA-4128) Kafka broker losses messages when zookeeper session times out

[Mazhar Shaikh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mazhar Shaikh updated KAFKA-4128:
-
Attachment: log.tar.gz

Attaching logs for this scenario:

Total Brokers : 6 [b1-b6]
Total Zookeepers : 3 [b0,b1,b2]



> Kafka broker losses messages when zookeeper session times out
> -
>
> Key: KAFKA-4128
> URL: https://issues.apache.org/jira/browse/KAFKA-4128
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1, 0.9.0.1
>Reporter: Mazhar Shaikh
>Priority: Critical
> Attachments: log.tar.gz
>
>
> Pumping 30k msgs/second after some 6-8 hrs of run below logs are printed and 
> the messages are lost.
> [More than 5k messages are lost on every partitions]
> Below are few logs:
> [2016-09-06 05:00:42,595] INFO Client session timed out, have not heard from 
> server in 20903ms for sessionid 0x256fabec47c0003, closing socket connection 
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:42,696] INFO zookeeper state changed (Disconnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-09-06 05:00:42,753] INFO Partition [topic,62] on broker 4: Shrinking 
> ISR for partition [topic,62] from 4,2 to 4 (kafka.cluster.Partition)
> [2016-09-06 05:00:43,585] INFO Opening socket connection to server 
> b0/169.254.2.1:2182. Will not attempt to authenticate using SASL (unknown 
> error) (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:43,586] INFO Socket connection established to 
> b0/169.254.2.1:2182, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:43,587] INFO Unable to read additional data from server 
> sessionid 0x256fabec47c0003, likely server has closed socket, closing socket 
> connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,644] INFO Opening socket connection to server 
> b1/169.254.2.116:2181. Will not attempt to authenticate using SASL (unknown 
> error) (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,651] INFO Socket connection established to 
> b1/169.254.2.116:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,658] INFO zookeeper state changed (Expired) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-09-06 05:00:44,659] INFO Initiating client connection, 
> connectString=b2.broker.com:2181,b1.broker.com:2181,zoo3.broker.com:2182 
> sessionTimeout=15000 watcher=org.I0Itec.zkclient.ZkClient@37b8e86a 
> (org.apache.zookeeper.ZooKeeper)
> [2016-09-06 05:00:44,659] INFO Unable to reconnect to ZooKeeper service, 
> session 0x256fabec47c0003 has expired, closing socket connection 
> (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,661] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,662] INFO Opening socket connection to server 
> b2/169.254.2.216:2181. Will not attempt to authenticate using SASL (unknown 
> error) (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,662] INFO Socket connection established to 
> b2/169.254.2.216:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,665] ERROR Error handling event ZkEvent[New session 
> event sent to 
> kafka.controller.KafkaController$SessionExpirationListener@33b7dedc] 
> (org.I0Itec.zkclient.ZkEventThread)
> java.lang.IllegalStateException: Kafka scheduler has not been started
> at kafka.utils.KafkaScheduler.ensureStarted(KafkaScheduler.scala:114)
> at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:86)
> at 
> kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:350)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1108)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1107)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1107)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at 
> kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1107)
> at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> [2016-09-06 05:00:44,666] INFO re-registering broker info in ZK for broker 4 
> (kafka.server.KafkaHealthcheck)
> [2016-09-06 05:00:44,801] INFO Session establishment complete on server 
> b2/169.254.2.216:2181, sessionid = 0x256fabec47c0005, negotiated timeout = 
> 15000 (org.apache.zookeeper.ClientCnxn)
> [2016-09-06 05:00:44,802] INFO zookeeper state changed (SyncConnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-09-06 05:00:44,812] INFO Registered 

Build failed in Jenkins: kafka-trunk-jdk7 #1651

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: Check for null timestamp rather than value in hashcode

--
[...truncated 14166 lines...]
org.apache.kafka.streams.kstream.JoinWindowsTest > 
timeDifferenceMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.JoinWindowsTest > 
timeDifferenceMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStream STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStream PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStreamWithProcessors STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldHaveCorrectSourceTopicsForTableFromMergedStreamWithProcessors PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenTopicNamesAreNull PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldThrowExceptionWhenNoTopicPresent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > advanceIntervalMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows STARTED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #999

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFAK-4058: Failure in

--
[...truncated 14341 lines...]

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[0] PASSED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] STARTED

org.apache.kafka.streams.integration.KStreamKTableJoinIntegrationTest > 
shouldCountClicksPerRegion[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[0] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryState[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance[1] PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses[1] PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testMultipleConsumersCanReadFromPartitionedTopic PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenDeleted PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testNoMessagesSentExceptionFromOverlappingPatterns PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperations[0] STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperations[0] PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperations[1] STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperations[1] PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldCompactTopicsForStateChangelogs PASSED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs STARTED

org.apache.kafka.streams.integration.InternalTopicIntegrationTest > 
shouldUseCompactAndDeleteForWindowStoreChangelogs PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce[0] STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldReduce[0] PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate[0] STARTED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
shouldAggregate[0] PASSED

org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 

Jenkins build is back to normal : kafka-trunk-jdk7 #1650

[Apache Jenkins Server]

See