[jira] [Commented] (KAFKA-4862) Kafka client connect to a shutdown node will block for a long time
[ https://issues.apache.org/jira/browse/KAFKA-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177574#comment-16177574 ] Pengwei commented on KAFKA-4862: [~guozhang] This jira is for the KIP-148, I think I can merge it into the 1.0.0 if this KIP is voted, thanks > Kafka client connect to a shutdown node will block for a long time > -- > > Key: KAFKA-4862 > URL: https://issues.apache.org/jira/browse/KAFKA-4862 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.0, 0.10.2.0 >Reporter: Pengwei >Assignee: Pengwei > Fix For: 1.0.0 > > > Currently in our test env, we found after one of the broker node crash(reboot > or os crash), the client maybe still connecting to the crash node to send > metadata request or other request, and it need about several minutes to > aware the connection is timeout then try another node to connect to send the > request. Then the client may still not aware the metadata change after > several minutes. > We don't have a connection timeout for the network client, we should add a > connection timeout for the client -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
[ https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3465: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode > -- > > Key: KAFKA-3465 > URL: https://issues.apache.org/jira/browse/KAFKA-3465 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.9.0.0, 0.11.0.0 >Reporter: BrianLing >Assignee: Vahid Hashemian >Priority: Minor > Fix For: 1.0.0 > > > 1. When we enable mirrorMake to migrate Kafka event from one to other with > "new.consumer" mode: > java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > -Djava.awt.headless=true -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dkafka.logs.dir=/kafka/kafka-app-logs > -Dlog4j.configuration=file:/kafka/kafka_2.10-0.9.0.0/bin/../config/tools-log4j.properties > -cp :/kafka/kafka_2.10-0.9.0.0/bin/../libs/* > -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker lvs-slca-mm.log > --consumer.config ../config/consumer.properties --new.consumer --num.streams > 4 --producer.config ../config/producer-slca.properties --whitelist risk.* > 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and > the owner is none. > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info > --group lvs.slca.mirrormaker --zookeeper lvsdmetlvm01.lvs.paypal.com:2181 > --topic > Group Topic Pid Offset logSize > Lag Owner > lvs.slca.mirrormaker 0 418578332 418678347 100015 > none > lvs.slca.mirrormaker 1 418598026 418698338 100312 > none > [Root Cause] > I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka > internal to store offset & consumer owner information. > Does it mean we can not use the below command to check new consumer’s > lag since current lag formula: lag= logSize – offset > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80 > > https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182 > => offSet Fetch from zookeeper instead of from Kafka -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5759) Allow user to specify relative path as log directory
[ https://issues.apache.org/jira/browse/KAFKA-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5759: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Allow user to specify relative path as log directory > > > Key: KAFKA-5759 > URL: https://issues.apache.org/jira/browse/KAFKA-5759 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin >Priority: Critical > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5692: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 1.0.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5746) Add new metrics to support health checks
[ https://issues.apache.org/jira/browse/KAFKA-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5746: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Add new metrics to support health checks > > > Key: KAFKA-5746 > URL: https://issues.apache.org/jira/browse/KAFKA-5746 > Project: Kafka > Issue Type: New Feature > Components: metrics >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram > Fix For: 1.0.0 > > > It will be useful to have some additional metrics to support health checks. > Details are in > [KIP-188|https://cwiki.apache.org/confluence/display/KAFKA/KIP-188+-+Add+new+metrics+to+support+health+checks] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5764) KafkaShortnamer should allow for case-insensitive matches
[ https://issues.apache.org/jira/browse/KAFKA-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5764: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > KafkaShortnamer should allow for case-insensitive matches > -- > > Key: KAFKA-5764 > URL: https://issues.apache.org/jira/browse/KAFKA-5764 > Project: Kafka > Issue Type: Improvement > Components: security >Affects Versions: 0.11.0.0 >Reporter: Ryan P >Assignee: Manikumar > Fix For: 1.0.0 > > > Currently it does not appear that the KafkaShortnamer allows for case > insensitive search and replace rules. > It would be good to match the functionality provided by HDFS as operators are > familiar with this. This also makes it easier to port auth_to_local rules > from your existing hdfs configurations to your new kafka configuration. > HWX auth_to_local guide for reference > https://community.hortonworks.com/articles/14463/auth-to-local-rules-syntax.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5302) Improve exception handling on streams client (communication with brokers)
[ https://issues.apache.org/jira/browse/KAFKA-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5302: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Improve exception handling on streams client (communication with brokers) > - > > Key: KAFKA-5302 > URL: https://issues.apache.org/jira/browse/KAFKA-5302 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > These are exceptions in StreamsKafkaClient.java. > Currently throws either StreamsException or BrokerNotFoundException. > Used by InternalTopicManager to create topics and get their metadata. > Used by StreamPartitionAssignor. > Currently InternalTopicManager retries a few times after catching an > exception. > A failure here is sent all the way up to the stream thread and will stop the > streams pipeline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-2651) Remove deprecated config alteration from TopicCommand in 0.9.1.0
[ https://issues.apache.org/jira/browse/KAFKA-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2651: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Remove deprecated config alteration from TopicCommand in 0.9.1.0 > > > Key: KAFKA-2651 > URL: https://issues.apache.org/jira/browse/KAFKA-2651 > Project: Kafka > Issue Type: Task >Affects Versions: 0.9.0.0 >Reporter: Grant Henke >Assignee: Manikumar > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3300) Calculate the initial/max size of offset index files and reduce the memory footprint for memory mapped index files.
[
https://issues.apache.org/jira/browse/KAFKA-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3300:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Calculate the initial/max size of offset index files and reduce the memory
> footprint for memory mapped index files.
> ---
>
> Key: KAFKA-3300
> URL: https://issues.apache.org/jira/browse/KAFKA-3300
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 0.9.0.1
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 1.0.0
>
>
> Currently the initial/max size of offset index file is configured by
> {{log.index.max.bytes}}. This will be the offset index file size for active
> log segment until it rolls out.
> Theoretically, we can calculate the upper bound of offset index size using
> the following formula:
> {noformat}
> log.segment.bytes / index.interval.bytes * 8
> {noformat}
> With default setting the bytes needed for an offset index size is 1GB / 4K *
> 8 = 2MB. And the default log.index.max.bytes is 10MB.
> This means we are over-allocating at least 8MB on disk and mapping it to
> memory.
> We can probably do the following:
> 1. When creating a new offset index, calculate the size using the above
> formula,
> 2. If the result in (1) is greater than log.index.max.bytes, we allocate
> log.index.max.bytes instead.
> This should be able to significantly save memory if a broker has a lot of
> partitions on it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-3733) Avoid long command lines by setting CLASSPATH in environment
[
https://issues.apache.org/jira/browse/KAFKA-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3733:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Avoid long command lines by setting CLASSPATH in environment
>
>
> Key: KAFKA-3733
> URL: https://issues.apache.org/jira/browse/KAFKA-3733
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Reporter: Adrian Muraru
>Assignee: Adrian Muraru
>Priority: Minor
> Fix For: 1.0.0
>
>
> {{kafka-run-class.sh}} sets the JVM classpath in the command line via {{-cp}}.
> This generates long command lines that gets trimmed by the shell in commands
> like ps, pgrep,etc.
> An alternative is to set the CLASSPATH in environment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4862) Kafka client connect to a shutdown node will block for a long time
[ https://issues.apache.org/jira/browse/KAFKA-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4862: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Kafka client connect to a shutdown node will block for a long time > -- > > Key: KAFKA-4862 > URL: https://issues.apache.org/jira/browse/KAFKA-4862 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.0, 0.10.2.0 >Reporter: Pengwei >Assignee: Pengwei > Fix For: 1.0.0 > > > Currently in our test env, we found after one of the broker node crash(reboot > or os crash), the client maybe still connecting to the crash node to send > metadata request or other request, and it need about several minutes to > aware the connection is timeout then try another node to connect to send the > request. Then the client may still not aware the metadata change after > several minutes. > We don't have a connection timeout for the network client, we should add a > connection timeout for the client -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-2435) More optimally balanced partition assignment strategy
[ https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2435: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > More optimally balanced partition assignment strategy > - > > Key: KAFKA-2435 > URL: https://issues.apache.org/jira/browse/KAFKA-2435 > Project: Kafka > Issue Type: Improvement >Reporter: Andrew Olson >Assignee: Andrew Olson > Fix For: 1.0.0 > > Attachments: KAFKA-2435.patch > > > While the roundrobin partition assignment strategy is an improvement over the > range strategy, when the consumer topic subscriptions are not identical > (previously disallowed but will be possible as of KAFKA-2172) it can produce > heavily skewed assignments. As suggested > [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767] > it would be nice to have a strategy that attempts to assign an equal number > of partitions to each consumer in a group, regardless of how similar their > individual topic subscriptions are. We can accomplish this by tracking the > number of partitions assigned to each consumer, and having the partition > assignment loop assign each partition to a consumer interested in that topic > with the least number of partitions assigned. > Additionally, we can optimize the distribution fairness by adjusting the > partition assignment order: > * Topics with fewer consumers are assigned first. > * In the event of a tie for least consumers, the topic with more partitions > is assigned first. > The general idea behind these two rules is to keep the most flexible > assignment choices available as long as possible by starting with the most > constrained partitions/consumers. > This JIRA addresses the original high-level consumer. For the new consumer, > see KAFKA-3297. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5156) Options for handling exceptions in streams
[ https://issues.apache.org/jira/browse/KAFKA-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5156: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Options for handling exceptions in streams > -- > > Key: KAFKA-5156 > URL: https://issues.apache.org/jira/browse/KAFKA-5156 > Project: Kafka > Issue Type: Task > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Labels: user-experience > Fix For: 1.0.0 > > > This is a task around options for handling exceptions in streams. It focuses > around options for dealing with corrupt data (keep going, stop streams, log, > retry, etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.
[
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3177:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 1.0.0
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset
> strategy. in sendListOffsetRequest(), if the consumer does not know the
> TopicPartition, it will refresh its metadata and retry. In this case, because
> the partition does not exist, we fall in to the infinite loop of refreshing
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does
> not exist, position() call will actually create the topic due to the fact
> that currently topic metadata request could automatically create the topic.
> This is a known separate issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5569) Document any changes from this task
[ https://issues.apache.org/jira/browse/KAFKA-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5569: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Document any changes from this task > --- > > Key: KAFKA-5569 > URL: https://issues.apache.org/jira/browse/KAFKA-5569 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > After fixing the exceptions, document what was done, e.g., KIP-161 at a > minimum. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5123) Refactor ZkUtils readData* methods
[ https://issues.apache.org/jira/browse/KAFKA-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5123: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Refactor ZkUtils readData* methods > --- > > Key: KAFKA-5123 > URL: https://issues.apache.org/jira/browse/KAFKA-5123 > Project: Kafka > Issue Type: Bug >Reporter: Balint Molnar >Assignee: Balint Molnar >Priority: Minor > Fix For: 1.0.0 > > > Usually only the data value is required but every readData method in the > ZkUtils returns a Tuple with the data and the stat. > https://github.com/apache/kafka/pull/2888 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4064) Add support for infinite endpoints for range queries in Kafka Streams KV stores
[ https://issues.apache.org/jira/browse/KAFKA-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4064: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Add support for infinite endpoints for range queries in Kafka Streams KV > stores > --- > > Key: KAFKA-4064 > URL: https://issues.apache.org/jira/browse/KAFKA-4064 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.1.0, 0.10.2.0 >Reporter: Roger Hoover >Assignee: Xavier Léauté >Priority: Minor > Labels: needs-kip > Fix For: 1.0.0 > > > In some applications, it's useful to iterate over the key-value store either: > 1. from the beginning up to a certain key > 2. from a certain key to the end > We can add two new methods rangeUtil() and rangeFrom() easily to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3866) KerberosLogin refresh time bug and other improvements
[ https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3866: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > KerberosLogin refresh time bug and other improvements > - > > Key: KAFKA-3866 > URL: https://issues.apache.org/jira/browse/KAFKA-3866 > Project: Kafka > Issue Type: Bug > Components: security >Affects Versions: 0.10.0.0 >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 1.0.0, 0.11.0.2 > > > ZOOKEEPER-2295 describes a bug in the Kerberos refresh time logic that is > also present in our KerberosLogin class. While looking at the code, I found a > number of things that could be improved. More details in the PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties
[ https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2394: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Use RollingFileAppender by default in log4j.properties > -- > > Key: KAFKA-2394 > URL: https://issues.apache.org/jira/browse/KAFKA-2394 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: Dustin Cote >Priority: Minor > Labels: newbie > Fix For: 1.0.0 > > Attachments: log4j.properties.patch > > > The default log4j.properties bundled with Kafka uses ConsoleAppender and > DailyRollingFileAppender, which offer no protection to users from spammy > logging. In extreme cases (such as when issues like KAFKA-1461 are > encountered), the logs can exhaust the local disk space. This could be a > problem for Kafka adoption since new users are less likely to adjust the > logging properties themselves, and are more likely to have configuration > problems which result in log spam. > To fix this, we can use RollingFileAppender, which offers two settings for > controlling the maximum space that log files will use. > maxBackupIndex: how many backup files to retain > maxFileSize: the max size of each log file > One question is whether this change is a compatibility concern? The backup > strategy and filenames used by RollingFileAppender are different from those > used by DailyRollingFileAppender, so any tools which depend on the old format > will break. If we think this is a serious problem, one solution would be to > provide two versions of log4j.properties and add a flag to enable the new > one. Another solution would be to include the RollingFileAppender > configuration in the default log4j.properties, but commented out. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin
[
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-2875:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Class path contains multiple SLF4J bindings warnings when using scripts under
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
> Labels: patch
> Fix For: 1.0.0
>
>
> This adds a lot of noise when running the scripts, see example when running
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5679) Add logging to distinguish between internally and externally initiated shutdown of Kafka
[
https://issues.apache.org/jira/browse/KAFKA-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5679:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Add logging to distinguish between internally and externally initiated
> shutdown of Kafka
>
>
> Key: KAFKA-5679
> URL: https://issues.apache.org/jira/browse/KAFKA-5679
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> Currently, if there is an internal error that triggers a shutdown of the
> Kafka server, the {{Exit}} class is used, which begins the shutdown
> procedure. The other way a shutdown is triggered is by {{SIGTERM}} or some
> other signal.
> We would like to distinguish between shutdown due to internal errors and
> external signals. This helps when debugging. Particularly, a natural question
> when a broker shuts down unexpectedly is: "did the deployment system send
> the signal or is there some un logged fatal error in the broker"?
> Today, we rely on callers of {{Exit}} to log the error before making the
> call. However, this won't always have 100% coverage. It would be good to add
> a log message in {{Exit}} to record that an exit method was invoked
> explicitly.
> We could also add a signal handler to log when {{SIGTERM}}, {{SIGKILL}} etc.
> are received.
> This would make operating Kafka a bit easier.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4440) Make producer RecordMetadata non-final
[ https://issues.apache.org/jira/browse/KAFKA-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4440: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Make producer RecordMetadata non-final > -- > > Key: KAFKA-4440 > URL: https://issues.apache.org/jira/browse/KAFKA-4440 > Project: Kafka > Issue Type: Improvement > Components: producer >Affects Versions: 0.10.0.1 >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram > Fix For: 1.0.0 > > > ProducerRecord and ConsumerRecord were made non-final in KAFKA-4250. It will > be good to make RecordMetadata also non-final for the same reason of > extensibility of Producer/Consumer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5495) Replace the deprecated 'ConsumerOffsetChecker' in documentation
[
https://issues.apache.org/jira/browse/KAFKA-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5495:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Replace the deprecated 'ConsumerOffsetChecker' in documentation
> ---
>
> Key: KAFKA-5495
> URL: https://issues.apache.org/jira/browse/KAFKA-5495
> Project: Kafka
> Issue Type: Improvement
> Components: documentation
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 1.0.0
>
>
> Use {{kafka-consumer-groups.sh}} instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4850) RocksDb cannot use Bloom Filters
[ https://issues.apache.org/jira/browse/KAFKA-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4850: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > RocksDb cannot use Bloom Filters > > > Key: KAFKA-4850 > URL: https://issues.apache.org/jira/browse/KAFKA-4850 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska >Assignee: Bharat Viswanadham > Fix For: 1.0.0 > > > Bloom Filters would speed up RocksDb lookups. However they currently do not > work in RocksDb 5.0.2. This has been fixed in trunk, but we'll have to wait > until that is released and tested. > Then we can add the line in RocksDbStore.java in openDb: > tableConfig.setFilter(new BloomFilter(10)); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3096) Leader is not set to -1 when it is shutdown if followers are down
[ https://issues.apache.org/jira/browse/KAFKA-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3096: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Leader is not set to -1 when it is shutdown if followers are down > - > > Key: KAFKA-3096 > URL: https://issues.apache.org/jira/browse/KAFKA-3096 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.0 >Reporter: Ismael Juma >Assignee: Ismael Juma > Labels: reliability > Fix For: 1.0.0 > > > Assuming a cluster with 2 brokers with unclear leader election disabled: > 1. Start brokers 0 and 1 > 2. Perform partition assignment > 3. Broker 0 is elected leader > 4. Produce message and wait until metadata is propagated > 6. Shutdown follower > 7. Produce message > 8. Shutdown leader > 9. Start follower > 10. Wait for leader election > Expected: leader is -1 > Actual: leader is 0 > We have a test for this, but a bug in `waitUntilLeaderIsElectedOrChanged` > means that `newLeaderOpt` is not being checked. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5750) Elevate log messages for denials to INFO in SimpleAclAuthorizer class
[ https://issues.apache.org/jira/browse/KAFKA-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5750: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Elevate log messages for denials to INFO in SimpleAclAuthorizer class > - > > Key: KAFKA-5750 > URL: https://issues.apache.org/jira/browse/KAFKA-5750 > Project: Kafka > Issue Type: Improvement > Components: security >Reporter: Phillip Walker >Assignee: Manikumar > Fix For: 1.0.0 > > > Currently, the authorizer logs all messages at DEBUG level and logs every > single authorization attempt, which can greatly decrease cluster performance, > especially when Mirrormaker also produces to that cluster. Many InfoSec > requirements, though, require that authorization denials be logged. The > proposed solution is to elevate any denial in SimpleAclAuthorizer and any > other relevant class to WARN while leaving approvals at their currently > logging levels. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4928) Add integration test for DumpLogSegments
[ https://issues.apache.org/jira/browse/KAFKA-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4928: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Add integration test for DumpLogSegments > > > Key: KAFKA-4928 > URL: https://issues.apache.org/jira/browse/KAFKA-4928 > Project: Kafka > Issue Type: Test >Reporter: Ismael Juma > Labels: newbie > Fix For: 1.0.0 > > > DumpLogSegments is an important tool to analyse log files, but we have no > JUnit tests for it. It would be good to have some tests that verify that the > output is sane for a populated log. > Our system tests call DumpLogSegments, but we should be able to detect > regressions via the JUnit test suite. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5499) Double check how we handle exceptions when commits fail
[ https://issues.apache.org/jira/browse/KAFKA-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5499: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Double check how we handle exceptions when commits fail > --- > > Key: KAFKA-5499 > URL: https://issues.apache.org/jira/browse/KAFKA-5499 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > When a task does a lot of processing in-between calls to poll() it happens > that it might miss a rebalance. It can find that out once it tries to > commit() since it will get an exception. Double check what is supposed to > happen on such an exception, e.g., should the thread fail, or should it > continue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4914) Partition re-assignment tool should check types before persisting state in ZooKeeper
[
https://issues.apache.org/jira/browse/KAFKA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4914:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Partition re-assignment tool should check types before persisting state in
> ZooKeeper
>
>
> Key: KAFKA-4914
> URL: https://issues.apache.org/jira/browse/KAFKA-4914
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Affects Versions: 0.10.1.1
>Reporter: Nick Travers
>Priority: Minor
> Fix For: 1.0.0
>
>
> The partition-reassignment too currently allows non-type-safe information to
> be persisted into ZooKeeper, which can result in a ClassCastException at
> runtime for brokers.
> Specifically, this occurred when the broker assignment field was a List of
> Strings, instead of a List of Integers.
> {code}
> 2017-03-15 01:44:04,572 ERROR
> [ZkClient-EventThread-36-samsa-zkserver.stage.sjc1.square:26101/samsa]
> controller.ReplicaStateMachine$BrokerChangeListener - [BrokerChangeListener
> on Controller 10]: Error while handling broker changes
> java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Integer
> at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101)
> at
> kafka.controller.KafkaController$$anonfun$8$$anonfun$apply$2.apply(KafkaController.scala:436)
> at
> scala.collection.LinearSeqOptimized$class.exists(LinearSeqOptimized.scala:93)
> at scala.collection.immutable.List.exists(List.scala:84)
> at
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:436)
> at
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:435)
> at
> scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at
> scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
> at
> scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
> at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
> at
> kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:435)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:374)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:357)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:355)
> at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:843)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4108) Improve DumpLogSegments offsets-decoder output format
[
https://issues.apache.org/jira/browse/KAFKA-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4108:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Improve DumpLogSegments offsets-decoder output format
> -
>
> Key: KAFKA-4108
> URL: https://issues.apache.org/jira/browse/KAFKA-4108
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> When using the DumpLogSegments with the "--offsets-decoder" option (for
> consuming __consumer_offsets), the encoding of group metadata makes it a
> little difficult to identify individual fields. In particular, we use the
> following formatted string for group metadata:
> {code}
> ${protocolType}:${groupMetadata.protocol}:${groupMetadata.generationId}:${assignment}
> {code}
> Keys have a similar formatting. Most users are probably not going to know
> which field is which based only on the output, so it would be helpful to
> include field names. Maybe we could just output a JSON object?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5276) Support derived and prefixed configs in DescribeConfigs (KIP-133)
[ https://issues.apache.org/jira/browse/KAFKA-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5276: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Support derived and prefixed configs in DescribeConfigs (KIP-133) > - > > Key: KAFKA-5276 > URL: https://issues.apache.org/jira/browse/KAFKA-5276 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 1.0.0 > > > The broker supports config overrides per listener. The way we do that is by > prefixing the configs with the listener name. These configs are not defined > by ConfigDef and they don't appear in `values()`. They do appear in > `originals()`. We should change the code so that we return these configs. > Because these configs are read-only, nothing needs to be done for > AlterConfigs. > With regards to derived configs, an example is advertised.listeners, which > falls back to listeners. This is currently done outside AbstractConfig. We > should look into including these into AbstractConfig so that the fallback > happens for the returned configs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3186) KIP-50: Move Authorizer and related classes to separate package.
[ https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3186: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > KIP-50: Move Authorizer and related classes to separate package. > > > Key: KAFKA-3186 > URL: https://issues.apache.org/jira/browse/KAFKA-3186 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.0 >Reporter: Ashish Singh >Assignee: Ashish Singh > Fix For: 1.0.0 > > > [KIP-50|https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+a+separate+package] > has more details. > Kafka supports pluggable authorization. Third party authorizer > implementations allow existing authorization systems like, Apache Sentry, > Apache Ranger, etc to extend authorization to Kafka as well. Implementing > Kafka's authorizer interface requires depending on kafka's core, which is > huge. This has been already raised as a concern by Sentry, Ranger and Kafka > community. Even Kafka clients require duplication of authorization related > classes, like Resource, Operation, etc, for adding ACLs CRUD APIs. > Kafka authorizer is agnostic of principal types it supports, so are the acls > CRUD methods in Authorizer interface. The intent behind is to keep Kafka > principal types pluggable, which is really great. However, this leads to Acls > CRUD methods not performing any check on validity of acls, as they are not > aware of what principal types Authorizer implementation supports. This opens > up space for lots of user errors, KAFKA-3097 is an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5018) LogCleaner tests to verify behaviour of message format v2
[ https://issues.apache.org/jira/browse/KAFKA-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5018: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > LogCleaner tests to verify behaviour of message format v2 > - > > Key: KAFKA-5018 > URL: https://issues.apache.org/jira/browse/KAFKA-5018 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Ismael Juma > Fix For: 1.0.0 > > > It would be good to add LogCleaner tests to verify the behaviour of fields > like baseOffset after compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4201) Add an --assignment-strategy option to new-consumer-based Mirror Maker
[
https://issues.apache.org/jira/browse/KAFKA-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4201:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Add an --assignment-strategy option to new-consumer-based Mirror Maker
> --
>
> Key: KAFKA-4201
> URL: https://issues.apache.org/jira/browse/KAFKA-4201
> Project: Kafka
> Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> The default assignment strategy in mirror maker will be changed from range to
> round robin in an upcoming release ([see
> KAFKA-3818|https://issues.apache.org/jira/browse/KAFKA-3818]). In order to
> make it easier for users to change the assignment strategy, add an
> {{--assignment-strategy}} option to Mirror Maker command line tool.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5637) Document compatibility and release policies
[ https://issues.apache.org/jira/browse/KAFKA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5637: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Document compatibility and release policies > --- > > Key: KAFKA-5637 > URL: https://issues.apache.org/jira/browse/KAFKA-5637 > Project: Kafka > Issue Type: Improvement > Components: documentation >Reporter: Ismael Juma >Assignee: Sönke Liebau > Fix For: 1.0.0 > > > We should document our compatibility and release policies in one place so > that people have the correct expectations. This is generally important, but > more so now that we are releasing 1.0.0. > I extracted the following topics from the mailing list thread as the ones > that should be documented as a minimum: > *Code stability* > * Explanation of stability annotations and their implications > * Explanation of what public apis are > * *Discussion point: * Do we want to keep the _unstable_ annotation or is > _evolving_ sufficient going forward? > *Support duration* > * How long are versions supported? > * How far are bugfixes backported? > * How far are security fixes backported? > * How long are protocol versions supported by subsequent code versions? > * How long are older clients supported? > * How long are older brokers supported? > I will create an initial pull request to add a section to the documentation > as basis for further discussion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4307) Inconsistent parameters between console producer and consumer
[ https://issues.apache.org/jira/browse/KAFKA-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4307: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Inconsistent parameters between console producer and consumer > - > > Key: KAFKA-4307 > URL: https://issues.apache.org/jira/browse/KAFKA-4307 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.10.1.0 >Reporter: Gwen Shapira >Assignee: Balint Molnar > Labels: newbie > Fix For: 1.0.0 > > > kafka-console-producer uses --broker-list while kafka-console-consumer uses > --bootstrap-server. > Let's add --bootstrap-server to the producer for some consistency? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5163) Support replicas movement between log directories (KIP-113)
[ https://issues.apache.org/jira/browse/KAFKA-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5163: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Support replicas movement between log directories (KIP-113) > --- > > Key: KAFKA-5163 > URL: https://issues.apache.org/jira/browse/KAFKA-5163 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.0.0 > > > See > https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories > for motivation and design. > Note that part 1 was merged via KAFKA-5694. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3331) Refactor TopicCommand to make it testable and add unit tests
[ https://issues.apache.org/jira/browse/KAFKA-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3331: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Refactor TopicCommand to make it testable and add unit tests > > > Key: KAFKA-3331 > URL: https://issues.apache.org/jira/browse/KAFKA-3331 > Project: Kafka > Issue Type: Wish >Affects Versions: 0.9.0.1 >Reporter: Ashish Singh >Assignee: Ashish Singh > Fix For: 1.0.0 > > > TopicCommand has become a functionality packed, hard to read, class. Adding > or changing it with confidence requires some unit tests around it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5301) Improve exception handling on consumer path
[ https://issues.apache.org/jira/browse/KAFKA-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5301: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Improve exception handling on consumer path > --- > > Key: KAFKA-5301 > URL: https://issues.apache.org/jira/browse/KAFKA-5301 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > Used in StreamsThread.java, mostly to .poll() but also to restore data. > Used in StreamsTask.java, mostly to .pause(), .resume() > All exceptions here are currently caught all the way up to the main running > loop in a broad catch(Exception e) statement in StreamThread.run(). > One main concern on the consumer path is handling deserialization errors that > happen before streams has even had a chance to look at the data: > https://issues.apache.org/jira/browse/KAFKA-5157 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4416) Add a '--group' option to the console consumer
[
https://issues.apache.org/jira/browse/KAFKA-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4416:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Add a '--group' option to the console consumer
> --
>
> Key: KAFKA-4416
> URL: https://issues.apache.org/jira/browse/KAFKA-4416
> Project: Kafka
> Issue Type: Improvement
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 1.0.0
>
>
> Add a {{--group}} option to the console consumer to simplify associating
> consumers to consumer groups. The command line option would overwrite any
> {{group.id}} property that may be specified in the consumer config.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5403) Transactions system test should dedup consumed messages by offset
[ https://issues.apache.org/jira/browse/KAFKA-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5403: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Transactions system test should dedup consumed messages by offset > - > > Key: KAFKA-5403 > URL: https://issues.apache.org/jira/browse/KAFKA-5403 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.11.0.0 >Reporter: Apurva Mehta >Assignee: Apurva Mehta > Fix For: 1.0.0 > > > In KAFKA-5396, we saw that the consumers which verify the data in multiple > topics could read the same offsets multiple times, for instance when a > rebalance happens. > This would detect spurious duplicates, causing the test to fail. We should > dedup the consumed messages by offset and only fail the test if we have > duplicate values for a if for a unique set of offsets. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4665) Inconsistent handling of non-existing topics in offset fetch handling
[
https://issues.apache.org/jira/browse/KAFKA-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4665:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Inconsistent handling of non-existing topics in offset fetch handling
> -
>
> Key: KAFKA-4665
> URL: https://issues.apache.org/jira/browse/KAFKA-4665
> Project: Kafka
> Issue Type: Bug
> Components: consumer, core
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 1.0.0
>
>
> For version 0 of the offset fetch API, the broker returns
> UNKNOWN_TOPIC_OR_PARTITION for any topics/partitions which do not exist at
> the time of fetching. In later versions, we skip this check. We do, however,
> continue to return UNKNOWN_TOPIC_OR_PARTITION for authorization errors (i.e.
> if the principal does not have Describe access to the corresponding topic).
> We should probably make this behavior consistent across versions.
> Note also that currently the consumer raises {{KafkaException}} when it
> encounters an UNKNOWN_TOPIC_OR_PARTITION error in the offset fetch response,
> which is inconsistent with how we usually handle this error. This probably
> doesn't cause any problems currently only because of the inconsistency
> mentioned in the first paragraph above.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4126) No relevant log when the topic is non-existent
[ https://issues.apache.org/jira/browse/KAFKA-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4126: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > No relevant log when the topic is non-existent > -- > > Key: KAFKA-4126 > URL: https://issues.apache.org/jira/browse/KAFKA-4126 > Project: Kafka > Issue Type: Bug >Reporter: Balázs Barnabás >Assignee: Vahid Hashemian >Priority: Minor > Fix For: 1.0.0 > > > When a producer sends a ProducerRecord into a Kafka topic that doesn't > existst, there is no relevant debug/error log that points out the error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3356) Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11
[ https://issues.apache.org/jira/browse/KAFKA-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3356: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11 > > > Key: KAFKA-3356 > URL: https://issues.apache.org/jira/browse/KAFKA-3356 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.10.2.0 >Reporter: Ashish Singh >Assignee: Mickael Maison >Priority: Blocker > Fix For: 1.0.0 > > > ConsumerOffsetChecker is marked deprecated as of 0.9, should be removed in > 0.11. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic
[
https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4879:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> KafkaConsumer.position may hang forever when deleting a topic
> -
>
> Key: KAFKA-4879
> URL: https://issues.apache.org/jira/browse/KAFKA-4879
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.10.2.0
>Reporter: Shixiong Zhu
>Assignee: Balint Molnar
> Fix For: 1.0.0
>
>
> KafkaConsumer.position may hang forever when deleting a topic. The problem is
> this line
> https://github.com/apache/kafka/blob/022bf129518e33e165f9ceefc4ab9e622952d3bd/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L374
> The timeout is "Long.MAX_VALUE", and it will just retry forever for
> UnknownTopicOrPartitionException.
> Here is a reproducer
> {code}
> import org.apache.kafka.clients.consumer.KafkaConsumer;
> import org.apache.kafka.common.TopicPartition;
> import org.apache.kafka.common.serialization.StringDeserializer;
> import java.util.Collections;
> import java.util.Properties;
> import java.util.Set;
> public class KafkaReproducer {
> public static void main(String[] args) {
> // Make sure "delete.topic.enable" is set to true.
> // Please create the topic test with "3" partitions manually.
> // The issue is gone when there is only one partition.
> String topic = "test";
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", "testgroup");
> props.put("value.deserializer", StringDeserializer.class.getName());
> props.put("key.deserializer", StringDeserializer.class.getName());
> props.put("enable.auto.commit", "false");
> KafkaConsumer kc = new KafkaConsumer(props);
> kc.subscribe(Collections.singletonList(topic));
> kc.poll(0);
> Set partitions = kc.assignment();
> System.out.println("partitions: " + partitions);
> kc.pause(partitions);
> kc.seekToEnd(partitions);
> System.out.println("please delete the topic in 30 seconds");
> try {
> // Sleep 30 seconds to give us enough time to delete the topic.
> Thread.sleep(3);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> System.out.println("sleep end");
> for (TopicPartition p : partitions) {
> System.out.println(p + " offset: " + kc.position(p));
> }
> System.out.println("cannot reach here");
> kc.close();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-3999) Consumer bytes-fetched metric uses decompressed message size
[ https://issues.apache.org/jira/browse/KAFKA-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3999: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Consumer bytes-fetched metric uses decompressed message size > > > Key: KAFKA-3999 > URL: https://issues.apache.org/jira/browse/KAFKA-3999 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.1, 0.10.0.0 >Reporter: Jason Gustafson >Assignee: Vahid Hashemian >Priority: Minor > Fix For: 1.0.0 > > > It looks like the computation for the bytes-fetched metrics uses the size of > the decompressed message set. I would have expected it to be based off of the > raw size of the fetch responses. Perhaps it would be helpful to expose both > the raw and decompressed fetch sizes? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5949) User Callback Exceptions need to be handled properly
[ https://issues.apache.org/jira/browse/KAFKA-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5949: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > User Callback Exceptions need to be handled properly > > > Key: KAFKA-5949 > URL: https://issues.apache.org/jira/browse/KAFKA-5949 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Matthias J. Sax >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > In Streams, we allow to register multiple user callbacks. We need to handle > those exceptions gracefully, by catching and wrapping with a StreamsException. > - TimestampExtractor > - DeserializationHandler > - StateRestoreListener -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5185) Adding the RecordMetadata that is returned by the producer to the commitRecord method for SourceTask
[ https://issues.apache.org/jira/browse/KAFKA-5185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5185: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Adding the RecordMetadata that is returned by the producer to the > commitRecord method for SourceTask > > > Key: KAFKA-5185 > URL: https://issues.apache.org/jira/browse/KAFKA-5185 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: George Smith > Fix For: 1.0.0 > > > An improvement request I thought would be useful. > Added the producers record metadata object to the commitRecord method on the > SourceTask class so more data is provided from the producer and it allows > anyone overriding and hooking into the commitRecord method to receive more > information about where the record was procuded to. > Left the old commitRecord method with just the sourcerecord for backwards > compatbility even though this would technically be included in a new version > of kafka, it would intoduce a breaking change without it. > Opened up PR here: https://github.com/apache/kafka/pull/2989 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3438) Rack Aware Replica Reassignment should warn of overloaded brokers
[ https://issues.apache.org/jira/browse/KAFKA-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3438: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Rack Aware Replica Reassignment should warn of overloaded brokers > - > > Key: KAFKA-3438 > URL: https://issues.apache.org/jira/browse/KAFKA-3438 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.10.0.0 >Reporter: Ben Stopford >Assignee: Vahid Hashemian > Fix For: 1.0.0 > > > We've changed the replica reassignment code to be rack aware. > One problem that might catch users out would be that they rebalance the > cluster using kafka-reassign-partitions.sh but their rack configuration means > that some high proportion of replicas are pushed onto a single, or small > number of, brokers. > This should be an easy problem to avoid, by changing the rack assignment > information, but we should probably warn users if they are going to create > something that is unbalanced. > So imagine I have a Kafka cluster of 12 nodes spread over two racks with rack > awareness enabled. If I add a 13th machine, on a new rack, and run the > rebalance tool, that new machine will get ~6x as many replicas as the least > loaded broker. > Suggest a warning be added to the tool output when --generate is called. > "The most loaded broker has 2.3x as many replicas as the the least loaded > broker. This is likely due to an uneven distribution of brokers across racks. > You're advised to alter the rack config so there are approximately the same > number of brokers per rack" and displays the individual rack→#brokers and > broker→#replicas data for the proposed move. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5767) Kafka server should halt if IBP < 1.0.0 and there is log directory failure
[ https://issues.apache.org/jira/browse/KAFKA-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5767: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Kafka server should halt if IBP < 1.0.0 and there is log directory failure > -- > > Key: KAFKA-5767 > URL: https://issues.apache.org/jira/browse/KAFKA-5767 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin >Priority: Critical > Fix For: 1.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3554) Generate actual data with specific compression ratio and add multi-thread support in the ProducerPerformance tool.
[ https://issues.apache.org/jira/browse/KAFKA-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3554: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Generate actual data with specific compression ratio and add multi-thread > support in the ProducerPerformance tool. > -- > > Key: KAFKA-3554 > URL: https://issues.apache.org/jira/browse/KAFKA-3554 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.1 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin > Fix For: 1.0.0 > > > Currently the ProducerPerformance always generate the payload with same > bytes. This does not quite well to test the compressed data because the > payload is extremely compressible no matter how big the payload is. > We can make some changes to make it more useful for compressed messages. > Currently I am generating the payload containing integer from a given range. > By adjusting the range of the integers, we can get different compression > ratios. > API wise, we can either let user to specify the integer range or the expected > compression ratio (we will do some probing to get the corresponding range for > the users) > Besides that, in many cases, it is useful to have multiple producer threads > when the producer threads themselves are bottleneck. Admittedly people can > run multiple ProducerPerformance to achieve similar result, but it is still > different from the real case when people actually use the producer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3252) compression type for a topic should be used during log compaction
[ https://issues.apache.org/jira/browse/KAFKA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3252: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > compression type for a topic should be used during log compaction > -- > > Key: KAFKA-3252 > URL: https://issues.apache.org/jira/browse/KAFKA-3252 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: Ashish Singh > Fix For: 1.0.0 > > > Currently, the broker uses the specified compression type in a topic for > newly published messages. However, during log compaction, it still uses the > compression codec in the original message. To be consistent, it seems that we > should use the compression type in a topic when copying the messages to new > log segments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3436) Speed up controlled shutdown.
[
https://issues.apache.org/jira/browse/KAFKA-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3436:
-
*Reminder to the contributor / reviewer of the PR*: please note that the code
deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate
your JIRA and see if it still makes sense to be merged into 1.0.0 or it could
be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid
any more, or re-assign yourself as contributor / committer if you are no longer
working on the JIRA.
> Speed up controlled shutdown.
> -
>
> Key: KAFKA-3436
> URL: https://issues.apache.org/jira/browse/KAFKA-3436
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 1.0.0
>
>
> Currently rolling bounce a Kafka cluster with tens of thousands of partitions
> can take very long (~2 min for each broker with ~5000 partitions/broker in
> our environment). The majority of the time is spent on shutting down a
> broker. The time of shutting down a broker usually includes the following
> parts:
> T1: During controlled shutdown, people usually want to make sure there is no
> under replicated partitions. So shutting down a broker during a rolling
> bounce will have to wait for the previous restarted broker to catch up. This
> is T1.
> T2: The time to send controlled shutdown request and receive controlled
> shutdown response. Currently the a controlled shutdown request will trigger
> many LeaderAndIsrRequest and UpdateMetadataRequest. And also involving many
> zookeeper update in serial.
> T3: The actual time to shutdown all the components. It is usually small
> compared with T1 and T2.
> T1 is related to:
> A) the inbound throughput on the cluster, and
> B) the "down" time of the broker (time between replica fetchers stop and
> replica fetchers restart)
> The larger the traffic is, or the longer the broker stopped fetching, the
> longer it will take for the broker to catch up and get back into ISR.
> Therefore the longer T1 will be. Assume:
> * the in bound network traffic is X bytes/second on a broker
> * the time T1.B ("down" time) mentioned above is T
> Theoretically it will take (X * T) / (NetworkBandwidth - X) =
> InBoundNetworkUtilization * T / (1 - InboundNetworkUtilization) for a the
> broker to catch up after the restart. While X is out of our control, T is
> largely related to T2.
> The purpose of this ticket is to reduce T2 by:
> 1. Batching the LeaderAndIsrRequest and UpdateMetadataRequest during
> controlled shutdown.
> 2. Use async zookeeper write to pipeline zookeeper writes. According to
> Zookeeper wiki(https://wiki.apache.org/hadoop/ZooKeeper/Performance), a 3
> node ZK cluster should be able to handle 20K writes (1K size). So if we use
> async write, likely we will be able to reduce zookeeper update time to lower
> seconds or even sub-second level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()
[ https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3190: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > KafkaProducer should not invoke callback in send() > -- > > Key: KAFKA-3190 > URL: https://issues.apache.org/jira/browse/KAFKA-3190 > Project: Kafka > Issue Type: Bug > Components: clients, producer >Affects Versions: 0.9.0.0 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin >Priority: Critical > Fix For: 1.0.0 > > > Currently KafkaProducer will invoke callback.onComplete() if it receives an > ApiException during send(). This breaks the guarantee that callback will be > invoked in order. It seems ApiException in send() only comes from metadata > refresh. If so, we can probably simply throw it instead of invoking > callback(). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5158) Options for handling exceptions during processing
[ https://issues.apache.org/jira/browse/KAFKA-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5158: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Options for handling exceptions during processing > - > > Key: KAFKA-5158 > URL: https://issues.apache.org/jira/browse/KAFKA-5158 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.0.0 > > > Imagine the app-level processing of a (non-corrupted) record fails (e.g. the > user attempted to do a RPC to an external system, and this call failed). How > can you process such failed records in a scalable way? For example, imagine > you need to implement a retry policy such as "retry with exponential > backoff". Here, you have the problem that 1. you can't really pause > processing a single record because this will pause the processing of the full > stream (bottleneck!) and 2. there is no straight-forward way to "sort" failed > records based on their "next retry time". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5886) Introduce delivery.timeout.ms producer config (KIP-91)
[ https://issues.apache.org/jira/browse/KAFKA-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5886: - *Reminder to the contributor / reviewer of the PR*: please note that the code deadline for 1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer if you are no longer working on the JIRA. > Introduce delivery.timeout.ms producer config (KIP-91) > -- > > Key: KAFKA-5886 > URL: https://issues.apache.org/jira/browse/KAFKA-5886 > Project: Kafka > Issue Type: Improvement > Components: producer >Reporter: Sumant Tambe >Assignee: Sumant Tambe > Fix For: 1.0.0 > > > We propose adding a new timeout delivery.timeout.ms. The window of > enforcement includes batching in the accumulator, retries, and the inflight > segments of the batch. With this config, the user has a guaranteed upper > bound on when a record will either get sent, fail or expire from the point > when send returns. In other words we no longer overload request.timeout.ms to > act as a weak proxy for accumulator timeout and instead introduce an explicit > timeout that users can rely on without exposing any internals of the producer > such as the accumulator. > See > [KIP-91|https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer] > for more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3514) Stream timestamp computation needs some further thoughts
[
https://issues.apache.org/jira/browse/KAFKA-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3514:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Stream timestamp computation needs some further thoughts
>
>
> Key: KAFKA-3514
> URL: https://issues.apache.org/jira/browse/KAFKA-3514
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
> Labels: architecture
> Fix For: 1.1.0
>
>
> Our current stream task's timestamp is used for punctuate function as well as
> selecting which stream to process next (i.e. best effort stream
> synchronization). And it is defined as the smallest timestamp over all
> partitions in the task's partition group. This results in two unintuitive
> corner cases:
> 1) observing a late arrived record would keep that stream's timestamp low for
> a period of time, and hence keep being process until that late record. For
> example take two partitions within the same task annotated by their
> timestamps:
> {code}
> Stream A: 5, 6, 7, 8, 9, 1, 10
> {code}
> {code}
> Stream B: 2, 3, 4, 5
> {code}
> The late arrived record with timestamp "1" will cause stream A to be selected
> continuously in the thread loop, i.e. messages with timestamp 5, 6, 7, 8, 9
> until the record itself is dequeued and processed, then stream B will be
> selected starting with timestamp 2.
> 2) an empty buffered partition will cause its timestamp to be not advanced,
> and hence the task timestamp as well since it is the smallest among all
> partitions. This may not be a severe problem compared with 1) above though.
> *Update*
> There is one more thing to consider (full discussion found here:
> http://search-hadoop.com/m/Kafka/uyzND1iKZJN1yz0E5?subj=Order+of+punctuate+and+process+in+a+stream+processor)
> {quote}
> Let's assume the following case.
> - a stream processor that uses the Processor API
> - context.schedule(1000) is called in the init()
> - the processor reads only one topic that has one partition
> - using custom timestamp extractor, but that timestamp is just a wall
> clock time
> Image the following events:
> 1., for 10 seconds I send in 5 messages / second
> 2., does not send any messages for 3 seconds
> 3., starts the 5 messages / second again
> I see that punctuate() is not called during the 3 seconds when I do not
> send any messages. This is ok according to the documentation, because
> there is not any new messages to trigger the punctuate() call. When the
> first few messages arrives after a restart the sending (point 3. above) I
> see the following sequence of method calls:
> 1., process() on the 1st message
> 2., punctuate() is called 3 times
> 3., process() on the 2nd message
> 4., process() on each following message
> What I would expect instead is that punctuate() is called first and then
> process() is called on the messages, because the first message's timestamp
> is already 3 seconds older then the last punctuate() was called, so the
> first message belongs after the 3 punctuate() calls.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4218) Enable access to key in ValueTransformer
[
https://issues.apache.org/jira/browse/KAFKA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4218:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Enable access to key in ValueTransformer
>
>
> Key: KAFKA-4218
> URL: https://issues.apache.org/jira/browse/KAFKA-4218
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
> Labels: api, kip
> Fix For: 1.1.0
>
>
> While transforming values via {{KStream.transformValues}} and
> {{ValueTransformer}}, the key associated with the value may be needed, even
> if it is not changed. For instance, it may be used to access stores.
> As of now, the key is not available within these methods and interfaces,
> leading to the use of {{KStream.transform}} and {{Transformer}}, and the
> unnecessary creation of new {{KeyValue}} objects.
> KIP-149:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5228) Revisit Streams DSL JavaDocs
[
https://issues.apache.org/jira/browse/KAFKA-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5228:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Revisit Streams DSL JavaDocs
>
>
> Key: KAFKA-5228
> URL: https://issues.apache.org/jira/browse/KAFKA-5228
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>Priority: Trivial
> Labels: beginner, documentation, newbie
> Fix For: 1.1.0
>
>
> We got some user feedback, that is it sometimes not clear from the JavaDocs,
> if provides {{Serdes}} are for input or output records.
> For example:
> {noformat}
> ...
> * @param keySerde key serdes for materializing this stream.
> * If not specified the default serdes defined in the
> configs will be used
> * @param valSerde value serdes for materializing this stream,
> * if not specified the default serdes defined in the
> configs will be used
> ...
> KStream join(final KTable table,
> final ValueJoiner extends VR> joiner,
> final Serde keySerde,
> final Serde valSerde);
> {noformat}
> The phrase "for this stream" means the input stream. But it is rather subtle.
> We should revisit the complete JavaDocs and rephrase the Serde parameter
> description if required. We should also rename the parameter names (in the
> example about, maybe from {{keySerde}} to {{inputKStreamKeySerde}})
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3186) KIP-50: Move Authorizer and related classes to separate package.
[ https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177495#comment-16177495 ] Guozhang Wang commented on KAFKA-3186: -- I saw that KIP-50 has been merged to in 0.10.1.0, which information is true? cc [~ijuma] > KIP-50: Move Authorizer and related classes to separate package. > > > Key: KAFKA-3186 > URL: https://issues.apache.org/jira/browse/KAFKA-3186 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.0 >Reporter: Ashish Singh >Assignee: Ashish Singh > Fix For: 1.0.0 > > > [KIP-50|https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+a+separate+package] > has more details. > Kafka supports pluggable authorization. Third party authorizer > implementations allow existing authorization systems like, Apache Sentry, > Apache Ranger, etc to extend authorization to Kafka as well. Implementing > Kafka's authorizer interface requires depending on kafka's core, which is > huge. This has been already raised as a concern by Sentry, Ranger and Kafka > community. Even Kafka clients require duplication of authorization related > classes, like Resource, Operation, etc, for adding ACLs CRUD APIs. > Kafka authorizer is agnostic of principal types it supports, so are the acls > CRUD methods in Authorizer interface. The intent behind is to keep Kafka > principal types pluggable, which is really great. However, this leads to Acls > CRUD methods not performing any check on validity of acls, as they are not > aware of what principal types Authorizer implementation supports. This opens > up space for lots of user errors, KAFKA-3097 is an instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5347) OutOfSequence error should be fatal
[
https://issues.apache.org/jira/browse/KAFKA-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5347:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> OutOfSequence error should be fatal
> ---
>
> Key: KAFKA-5347
> URL: https://issues.apache.org/jira/browse/KAFKA-5347
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
> Labels: exactly-once
> Fix For: 1.1.0
>
>
> If the producer sees an OutOfSequence error for a given partition, we
> currently treat it as an abortable error. This makes some sense because
> OutOfSequence won't prevent us from being able to send the EndTxn to abort
> the transaction. The problem is that the producer, even after aborting, still
> won't be able to send to the topic with an OutOfSequence. One way to deal
> with this is to ask the user to call {{initTransactions()}} again to bump the
> epoch, but this is a bit difficult to explain and could be dangerous since it
> renders zombie checking less effective. Probably we should just consider
> OutOfSequence fatal for the transactional producer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-1120) Controller could miss a broker state change
[ https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1120: - Fix Version/s: (was: 1.0.0) 1.1.0 > Controller could miss a broker state change > > > Key: KAFKA-1120 > URL: https://issues.apache.org/jira/browse/KAFKA-1120 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.1 >Reporter: Jun Rao > Labels: reliability > Fix For: 1.1.0 > > > When the controller is in the middle of processing a task (e.g., preferred > leader election, broker change), it holds a controller lock. During this > time, a broker could have de-registered and re-registered itself in ZK. After > the controller finishes processing the current task, it will start processing > the logic in the broker change listener. However, it will see no broker > change and therefore won't do anything to the restarted broker. This broker > will be in a weird state since the controller doesn't inform it to become the > leader of any partition. Yet, the cached metadata in other brokers could > still list that broker as the leader for some partitions. Client requests > routed to that broker will then get a TopicOrPartitionNotExistException. This > broker will continue to be in this bad state until it's restarted again. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5022) Improve CRC tests so that we verify which fields are included in the CRC
[ https://issues.apache.org/jira/browse/KAFKA-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5022: - Fix Version/s: (was: 1.0.0) 1.1.0 > Improve CRC tests so that we verify which fields are included in the CRC > > > Key: KAFKA-5022 > URL: https://issues.apache.org/jira/browse/KAFKA-5022 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Ismael Juma > Fix For: 1.1.0 > > > It would be good to have a test that updates each field in the record/record > batch, recomputes the CRC and then asserts that the CRC changes or not > depending on whether that field is meant to be part of the CRC or not. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4628) Support KTable/GlobalKTable Joins
[ https://issues.apache.org/jira/browse/KAFKA-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4628: - Fix Version/s: (was: 1.0.0) 1.1.0 > Support KTable/GlobalKTable Joins > - > > Key: KAFKA-4628 > URL: https://issues.apache.org/jira/browse/KAFKA-4628 > Project: Kafka > Issue Type: Sub-task > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Damian Guy > Fix For: 1.1.0 > > > In KIP-99 we have added support for GlobalKTables, however we don't currently > support KTable/GlobalKTable joins as they require materializing a state store > for the join. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5255) Auto generate request/response classes
[ https://issues.apache.org/jira/browse/KAFKA-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5255: - Fix Version/s: (was: 1.0.0) 1.1.0 > Auto generate request/response classes > -- > > Key: KAFKA-5255 > URL: https://issues.apache.org/jira/browse/KAFKA-5255 > Project: Kafka > Issue Type: New Feature >Reporter: Ismael Juma >Assignee: Tom Bentley > Fix For: 1.1.0 > > > We should automatically generate the request/response classes from the > protocol definition. This is a major source of boilerplate, development > effort and inconsistency at the moment. If we auto-generate the classes, we > may also be able to avoid the intermediate `Struct` representation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes
[ https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3806: - Fix Version/s: (was: 1.0.0) 1.1.0 > Adjust default values of log.retention.hours and offsets.retention.minutes > -- > > Key: KAFKA-3806 > URL: https://issues.apache.org/jira/browse/KAFKA-3806 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 0.9.0.1, 0.10.0.0 >Reporter: Michal Turek >Priority: Minor > Fix For: 1.1.0 > > > Combination of default values of log.retention.hours (168 hours = 7 days) and > offsets.retention.minutes (1440 minutes = 1 day) may be dangerous in special > cases. Offset retention should be always greater than log retention. > We have observed the following scenario and issue: > - Producing of data to a topic was disabled two days ago by producer update, > topic wasn't deleted. > - Consumer consumed all data and properly committed offsets to Kafka. > - Consumer made no more offset commits for that topic because there was no > more incoming data and there was nothing to confirm. (We have auto-commit > disabled, I'm not sure how behaves enabled auto-commit.) > - After one day: Kafka cleared too old offsets according to > offsets.retention.minutes. > - After two days: Long-term running consumer was restarted after update, it > didn't find any committed offsets for that topic since they were deleted by > offsets.retention.minutes so it started consuming from the beginning. > - The messages were still in Kafka due to larger log.retention.hours, about 5 > days of messages were read again. > Known workaround to solve this issue: > - Explicitly configure log.retention.hours and offsets.retention.minutes, > don't use defaults. > Proposals: > - Prolong default value of offsets.retention.minutes to be at least twice > larger than log.retention.hours. > - Check these values during Kafka startup and log a warning if > offsets.retention.minutes is smaller than log.retention.hours. > - Add a note to migration guide about differences between storing of offsets > in ZooKeeper and Kafka (http://kafka.apache.org/documentation.html#upgrade). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4487) Tests should be run in Jenkins with INFO or DEBUG level
[ https://issues.apache.org/jira/browse/KAFKA-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4487: - Fix Version/s: (was: 1.0.0) 1.1.0 > Tests should be run in Jenkins with INFO or DEBUG level > --- > > Key: KAFKA-4487 > URL: https://issues.apache.org/jira/browse/KAFKA-4487 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma > Fix For: 1.1.0 > > > KAFKA-4483 is an example of what can be missed by running them at ERROR > level. Worse than that would be subtle issues that would escape detection > altogether. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5356) Producer with transactionalId should be able to send outside of a transaction
[ https://issues.apache.org/jira/browse/KAFKA-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5356: - Fix Version/s: (was: 1.0.0) 1.1.0 > Producer with transactionalId should be able to send outside of a transaction > - > > Key: KAFKA-5356 > URL: https://issues.apache.org/jira/browse/KAFKA-5356 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Jason Gustafson > Fix For: 1.1.0 > > > This is debatable, but it may be useful to allow a producer to send > non-trasactional data even if they have a transactionalId set. At the moment, > this is explicitly forbidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5025) FetchRequestTest should use batches with more than one message
[ https://issues.apache.org/jira/browse/KAFKA-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5025: - Fix Version/s: (was: 1.0.0) 1.1.0 > FetchRequestTest should use batches with more than one message > -- > > Key: KAFKA-5025 > URL: https://issues.apache.org/jira/browse/KAFKA-5025 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Ismael Juma >Assignee: Umesh Chaudhary > Fix For: 1.1.0 > > > As part of the message format changes for KIP-98, > FetchRequestTest.produceData was changed to always use record batches > containing a single message. We should restructure the test so that it's more > realistic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5286) Producer should await transaction completion in close
[ https://issues.apache.org/jira/browse/KAFKA-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5286: - Fix Version/s: (was: 1.0.0) 1.1.0 > Producer should await transaction completion in close > - > > Key: KAFKA-5286 > URL: https://issues.apache.org/jira/browse/KAFKA-5286 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Affects Versions: 0.11.0.0 >Reporter: Jason Gustafson > Fix For: 1.1.0 > > > We should wait at least as long as the timeout for a transaction which has > begun completion (commit or abort) to be finished. Tricky thing is whether we > should abort a transaction which is in progress. It seems reasonable since > that's the coordinator will either timeout and abort the transaction or the > next producer using the same transactionalId will fence the producer and > abort the transaction. In any case, the transaction will be aborted, so > perhaps we should do it proactively. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5903) Create Connect metrics for workers
[ https://issues.apache.org/jira/browse/KAFKA-5903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5903: - Fix Version/s: (was: 1.0.0) 1.1.0 > Create Connect metrics for workers > -- > > Key: KAFKA-5903 > URL: https://issues.apache.org/jira/browse/KAFKA-5903 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Randall Hauch >Assignee: Randall Hauch >Priority: Critical > Fix For: 1.1.0 > > > See KAFKA-2376 for parent task and > [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework] > for the details on the metrics. This subtask is to create the "Worker > Metrics". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4650) Improve test coverage org.apache.kafka.streams.kstream.internals
[
https://issues.apache.org/jira/browse/KAFKA-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4650:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Improve test coverage org.apache.kafka.streams.kstream.internals
>
>
> Key: KAFKA-4650
> URL: https://issues.apache.org/jira/browse/KAFKA-4650
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Minor
> Fix For: 1.1.0
>
>
> Lots of classes have little or no coverage at all, i.e.,
> {{KTableAggregate.KTableAggregateValueGetter}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetterSupplier}}
> {{KStreamAggregate.KStreamAggregateValueGetter}}
> {{KStreamReduce.KStreamReduceValueGetter}}
> {{KStreamWindowReduce.new KTableValueGetterSupplier}}
> {{KTableAggregate.new KTableValueGetterSupplier}}
> {{KTableRepartitionMap.new KTableValueGetterSupplier}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetter}}
> {{KTableKTableLeftJoinValueGetter}}
> {{KStreamWindowReduce.KStreamWindowReduceValueGetter}}
> {{TimeWindow}}
> {{ChangedSerializer}}
> {{UnlimitedWindow}}
> {{WindowedDeserializer}}
> {{KStreamSessionWindowAggregate.KTableSessionWindowValueGetter}}
> {{KTableRepartitionMap}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-3986) completedReceives can contain closed channels
[ https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3986: - Fix Version/s: (was: 1.0.0) 1.1.0 > completedReceives can contain closed channels > -- > > Key: KAFKA-3986 > URL: https://issues.apache.org/jira/browse/KAFKA-3986 > Project: Kafka > Issue Type: Bug > Components: network >Reporter: Ryan P > Fix For: 1.1.0 > > > I'm not entirely sure why at this point but it is possible to throw a Null > Pointer Exception when processingCompletedReceives. This happens when a > fairly substantial number of simultaneously initiated connections are > initiated with the server. > The processor thread does carry on but it may be worth investigating how the > channel could be both closed and completedReceives. > The NPE in question is thrown here: > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L490 > It can not be consistently reproduced either. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5904) Create Connect metrics for worker rebalances
[ https://issues.apache.org/jira/browse/KAFKA-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5904: - Fix Version/s: (was: 1.0.0) 1.1.0 > Create Connect metrics for worker rebalances > > > Key: KAFKA-5904 > URL: https://issues.apache.org/jira/browse/KAFKA-5904 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Randall Hauch >Assignee: Randall Hauch >Priority: Critical > Fix For: 1.1.0 > > > See KAFKA-2376 for parent task and > [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework] > for the details on the metrics. This subtask is to create the "Worker > Rebalance Metrics". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4875) Kafka Streams: topic groups and builder.stream API
[ https://issues.apache.org/jira/browse/KAFKA-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4875: - Fix Version/s: (was: 1.0.0) 1.1.0 > Kafka Streams: topic groups and builder.stream API > -- > > Key: KAFKA-4875 > URL: https://issues.apache.org/jira/browse/KAFKA-4875 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska > Fix For: 1.1.0 > > > One thing that has come up in mailing list is that the notion of topic groups > is unclear. So if you have 2 topics, each with 3 partitions, you'd expect 6 > tasks. However, if you do `builder.stream(topic1, topic2)` you actually get > only 3 tasks created. If you do `builder.stream(topic1); > builder.stream(topic2)` you get 6 tasks, i.e., parallelism is increased. So > the same application, calling builder.stream() in two different ways, might > see different performance. > In the Kafka Streams documentations we mention partitions and tasks, but not > topic groups. We also do not document the effects of using builder.stream > with a topic array. We also need to revisit whether the API and its effects > are confusing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5383) Additional Test Cases for ReplicaManager
[ https://issues.apache.org/jira/browse/KAFKA-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5383: - Fix Version/s: (was: 1.0.0) 1.1.0 > Additional Test Cases for ReplicaManager > > > Key: KAFKA-5383 > URL: https://issues.apache.org/jira/browse/KAFKA-5383 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Jason Gustafson > Fix For: 1.1.0 > > > KAFKA-5355 and KAFKA-5376 have shown that current testing of ReplicaManager > is inadequate. This is definitely the case when it comes to KIP-98 and is > likely true in general. We should improve this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-1696) Kafka should be able to generate Hadoop delegation tokens
[ https://issues.apache.org/jira/browse/KAFKA-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1696: - Fix Version/s: (was: 1.0.0) 1.1.0 > Kafka should be able to generate Hadoop delegation tokens > - > > Key: KAFKA-1696 > URL: https://issues.apache.org/jira/browse/KAFKA-1696 > Project: Kafka > Issue Type: New Feature > Components: security >Reporter: Jay Kreps >Assignee: Parth Brahmbhatt > Fix For: 1.1.0 > > > For access from MapReduce/etc jobs run on behalf of a user. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5631) Use Jackson for serialising to JSON
[ https://issues.apache.org/jira/browse/KAFKA-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5631: - Fix Version/s: (was: 1.0.0) 1.1.0 > Use Jackson for serialising to JSON > --- > > Key: KAFKA-5631 > URL: https://issues.apache.org/jira/browse/KAFKA-5631 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Umesh Chaudhary > Labels: newbie > Fix For: 1.1.0 > > > We currently serialise to JSON via a manually written method `Json.encode`. > The implementation is naive: it does a lot of unnecessary String > concatenation and it doesn't handle escaping well. > KAFKA-1595 switches to Jackson for parsing, so it would make sense to do this > after that one is merged. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5092) KIP 141 - ProducerRecord Interface Improvements
[ https://issues.apache.org/jira/browse/KAFKA-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5092: - Fix Version/s: (was: 1.0.0) 1.1.0 > KIP 141 - ProducerRecord Interface Improvements > --- > > Key: KAFKA-5092 > URL: https://issues.apache.org/jira/browse/KAFKA-5092 > Project: Kafka > Issue Type: Improvement >Reporter: Stephane Maarek > Labels: kip > Fix For: 1.1.0 > > > See KIP here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP+141+-+ProducerRecord+Interface+Improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5792) Transient failure in KafkaAdminClientTest.testHandleTimeout
[
https://issues.apache.org/jira/browse/KAFKA-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5792:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Transient failure in KafkaAdminClientTest.testHandleTimeout
> ---
>
> Key: KAFKA-5792
> URL: https://issues.apache.org/jira/browse/KAFKA-5792
> Project: Kafka
> Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Colin P. McCabe
> Labels: transient-unit-test-failure
> Fix For: 1.1.0
>
>
> The {{KafkaAdminClientTest.testHandleTimeout}} test occasionally fails with
> the following:
> {noformat}
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node
> assignment.
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213)
> at
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testHandleTimeout(KafkaAdminClientTest.java:356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting
> for a node assignment.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5493) Optimize calls to flush for tasks and standby tasks
[
https://issues.apache.org/jira/browse/KAFKA-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5493:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Optimize calls to flush for tasks and standby tasks
> ---
>
> Key: KAFKA-5493
> URL: https://issues.apache.org/jira/browse/KAFKA-5493
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 1.1.0
>
>
> With EOS enabled we don't checkpoint on {{commit}} so there is no need to
> call {{flush}} when committing _top level tasks_ . However for _standby
> tasks_ we still checkpoint thus need to still flush when committing. We need
> to develop an approach where we can optimize for top level tasks by avoid
> flushing on commit, while still preserving flush on commit for standby tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5834) AbstractConfig.logUnused() may log confusing warning information.
[
https://issues.apache.org/jira/browse/KAFKA-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5834:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> AbstractConfig.logUnused() may log confusing warning information.
> -
>
> Key: KAFKA-5834
> URL: https://issues.apache.org/jira/browse/KAFKA-5834
> Project: Kafka
> Issue Type: Bug
> Components: log
>Reporter: Jiangjie Qin
> Fix For: 1.1.0
>
>
> Currently {{AbstractConfig.logUnused()}} logs unused configurations in at
> WARN level. It is a little weird because as long as there is a configurable
> class taking a configuration, that configuration will be logged as unused at
> WARN level even if it is actually used. It seems better to make it an INFO
> level logging instead, or maybe it can take a log level argument to allow
> caller to decide which log level should be used.
> [~hachikuji] [~ijuma] what do you think?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5876) IQ should throw different exceptions for different errors
[
https://issues.apache.org/jira/browse/KAFKA-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5876:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> IQ should throw different exceptions for different errors
> -
>
> Key: KAFKA-5876
> URL: https://issues.apache.org/jira/browse/KAFKA-5876
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
> Labels: needs-kip, newbie++
> Fix For: 1.1.0
>
>
> Currently, IQ does only throws {{InvalidStateStoreException}} for all errors
> that occur. However, we have different types of errors and should throw
> different exceptions for those types.
> For example, if a store was migrated it must be rediscovered while if a store
> cannot be queried yet, because it is still re-created after a rebalance, the
> user just needs to wait until store recreation is finished.
> There might be other examples, too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5552) testTransactionalProducerTopicAuthorizationExceptionInCommit fails
[ https://issues.apache.org/jira/browse/KAFKA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5552: - Fix Version/s: (was: 1.0.0) 1.1.0 > testTransactionalProducerTopicAuthorizationExceptionInCommit fails > --- > > Key: KAFKA-5552 > URL: https://issues.apache.org/jira/browse/KAFKA-5552 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska > Fix For: 1.1.0 > > > Got a unit test error: > https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5877/ > Error Message > org.apache.kafka.common.KafkaException: Cannot execute transactional method > because we are in an error state > Stacktrace > org.apache.kafka.common.KafkaException: Cannot execute transactional method > because we are in an error state > at > org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524) > at > org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:190) > at > org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:583) > at > kafka.api.AuthorizerIntegrationTest.testTransactionalProducerTopicAuthorizationExceptionInCommit(AuthorizerIntegrationTest.scala:1027) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109) > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at >
[jira] [Updated] (KAFKA-3135) Unexpected delay before fetch response transmission
[
https://issues.apache.org/jira/browse/KAFKA-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3135:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Unexpected delay before fetch response transmission
> ---
>
> Key: KAFKA-3135
> URL: https://issues.apache.org/jira/browse/KAFKA-3135
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 1.1.0
>
>
> From the user list, Krzysztof Ciesielski reports the following:
> {quote}
> Scenario description:
> First, a producer writes 50 elements into a topic
> Then, a consumer starts to read, polling in a loop.
> When "max.partition.fetch.bytes" is set to a relatively small value, each
> "consumer.poll()" returns a batch of messages.
> If this value is left as default, the output tends to look like this:
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 0 elements
> Poll returned 13793 elements
> Poll returned 13793 elements
> As we can see, there are weird "gaps" when poll returns 0 elements for some
> time. What is the reason for that? Maybe there are some good practices
> about setting "max.partition.fetch.bytes" which I don't follow?
> {quote}
> The gist to reproduce this problem is here:
> https://gist.github.com/kciesielski/054bb4359a318aa17561.
> After some initial investigation, the delay appears to be in the server's
> networking layer. Basically I see a delay of 5 seconds from the time that
> Selector.send() is invoked in SocketServer.Processor with the fetch response
> to the time that the send is completed. Using netstat in the middle of the
> delay shows the following output:
> {code}
> tcp4 0 0 10.191.0.30.55455 10.191.0.30.9092 ESTABLISHED
> tcp4 0 102400 10.191.0.30.9092 10.191.0.30.55454 ESTABLISHED
> {code}
> From this, it looks like the data reaches the send buffer, but needs to be
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4125) Provide low-level Processor API meta data in DSL layer
[
https://issues.apache.org/jira/browse/KAFKA-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4125:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Provide low-level Processor API meta data in DSL layer
> --
>
> Key: KAFKA-4125
> URL: https://issues.apache.org/jira/browse/KAFKA-4125
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>Priority: Minor
> Labels: kip
> Fix For: 1.1.0
>
>
> For Processor API, user can get meta data like record offset, timestamp etc
> via the provided {{Context}} object. It might be useful to allow uses to
> access this information in DSL layer, too.
> The idea would be, to do it "the Flink way", ie, by providing
> RichFunctions; {{mapValue()}} for example.
> Is takes a {{ValueMapper}} that only has method
> {noformat}
> V2 apply(V1 value);
> {noformat}
> Thus, you cannot get any meta data within apply (it's completely "blind").
> We would add two more interfaces: {{RichFunction}} with a method
> {{open(Context context)}} and
> {noformat}
> RichValueMapper extends ValueMapper, RichFunction
> {noformat}
> This way, the user can chose to implement Rich- or Standard-function and
> we do not need to change existing APIs. Both can be handed into
> {{KStream.mapValues()}} for example. Internally, we check if a Rich
> function is provided, and if yes, hand in the {{Context}} object once, to
> make it available to the user who can now access it within {{apply()}} -- or
> course, the user must set a member variable in {{open()}} to hold the
> reference to the Context object.
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-159%3A+Introducing+Rich+functions+to+Streams
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-4904) Performance of RocksDb with state record cache
[ https://issues.apache.org/jira/browse/KAFKA-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4904: - Fix Version/s: (was: 1.0.0) 1.1.0 > Performance of RocksDb with state record cache > -- > > Key: KAFKA-4904 > URL: https://issues.apache.org/jira/browse/KAFKA-4904 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.10.2.0 >Reporter: Eno Thereska > Fix For: 1.1.0 > > > The performance of RocksDb without any record cache is slightly better than > with the default of 10MB of the default cache. This needs investigating. It > is likely to be the case that this is not an entirely apples-to-apples > comparison since the record cache holds records for a maximum of > 'commit.time.ms', which by default is 30 seconds. So the record cache is > adding quite a bit of latency, and we know that, however documenting the > tradeoff and looking if there is any other bugs needs to be done. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5944) Add unit tests for handling of authentication failures in clients
[ https://issues.apache.org/jira/browse/KAFKA-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5944: - Fix Version/s: (was: 1.0.0) 1.1.0 > Add unit tests for handling of authentication failures in clients > - > > Key: KAFKA-5944 > URL: https://issues.apache.org/jira/browse/KAFKA-5944 > Project: Kafka > Issue Type: Test > Components: clients >Reporter: Rajini Sivaram >Assignee: Vahid Hashemian > Fix For: 1.1.0 > > > KAFKA-5854 improves authentication failures in clients and has added > integration tests and some basic client-side tests that create actual > connections to a mock server. It will be good to add a set of tests for > producers, consumers etc. that use MockClient to add more extensive tests for > various scenarios. > cc [~hachikuji] [~vahid] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5877) Controller should only update reassignment znode if there is change in the reassignment data
[ https://issues.apache.org/jira/browse/KAFKA-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5877: - Fix Version/s: (was: 1.0.0) 1.1.0 > Controller should only update reassignment znode if there is change in the > reassignment data > > > Key: KAFKA-5877 > URL: https://issues.apache.org/jira/browse/KAFKA-5877 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.1.0 > > > I encountered a scenario where controller keeps printing the following stack > trace repeatedly for a finite set of partitions. Although I have not fully > figured out the cause of this event, it seems that controller will update the > reassignment znode even if the new data is same as existing data. This patch > optimizes the controller behavior by only updating reassignment znode if it > needs to change the reassignment znode data. > 2017/09/12 20:34:05.842 [KafkaController] [Controller 1376005]: Error > completing reassignment of partition [FederatorResultEvent,202] > kafka.common.KafkaException: Partition [FederatorResultEvent,202] to be > reassigned is already assigned to replicas 1367001,1384010,1386010. Ignoring > request for partition reassignment > at > kafka.controller.KafkaController.initiateReassignReplicasForTopicPartition(KafkaController.scala:608) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1327) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment$$anonfun$process$14.apply(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > ~[scala-library-2.10.4.jar:?] > at > kafka.controller.KafkaController$PartitionReassignment.process(KafkaController.scala:1320) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) > ~[kafka_2.10-0.11.0.9.jar:?] > at > kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:52) > ~[kafka_2.10-0.11.0.9.jar:?] > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) > ~[kafka_2.10-0.11.0.9.jar:?] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5219) Move transaction expiration logic and scheduling to the Transaction Manager
[
https://issues.apache.org/jira/browse/KAFKA-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5219:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Move transaction expiration logic and scheduling to the Transaction Manager
> ---
>
> Key: KAFKA-5219
> URL: https://issues.apache.org/jira/browse/KAFKA-5219
> Project: Kafka
> Issue Type: Sub-task
> Components: core
>Reporter: Damian Guy
> Fix For: 1.1.0
>
>
> Presently the transaction expiration logic is spread between the
> {{TransactionStateManager}} and the {{TransactionCoordinator}}. It would be
> best if it was all in the {{TransactionStateManager}}. This requires moving
> the bulk of the commit/abort logic, too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5901) Create Connect metrics for source tasks
[ https://issues.apache.org/jira/browse/KAFKA-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5901: - Fix Version/s: (was: 1.0.0) 1.1.0 > Create Connect metrics for source tasks > --- > > Key: KAFKA-5901 > URL: https://issues.apache.org/jira/browse/KAFKA-5901 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Randall Hauch >Assignee: Randall Hauch >Priority: Critical > Fix For: 1.1.0 > > > See KAFKA-2376 for parent task and > [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework] > for the details on the metrics. This subtask is to create the "Source Task > Metrics". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-4696) Streams standby task assignment should be state-store aware
[
https://issues.apache.org/jira/browse/KAFKA-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-4696:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Streams standby task assignment should be state-store aware
> ---
>
> Key: KAFKA-4696
> URL: https://issues.apache.org/jira/browse/KAFKA-4696
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0, 0.11.0.0
>Reporter: Damian Guy
> Fix For: 1.1.0
>
>
> Task Assignment is currently not aware of which tasks have State Stores. This
> can result in uneven balance of standby task assignment as all tasks are
> assigned, but only those tasks with state-stores are ever created by
> {{StreamThread}}. So what seems like an optimal strategy during assignment
> time could be sub-optimal post-assignment.
> For example, lets say we have 4 tasks (2 with state-stores), 2 clients,
> numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks.
> One of the clients may end up with both state-store tasks, while the other
> has none.
> Further to this, standby task configuration is currently "all or nothing". It
> might make sense to allow more fine grained configuration, i.e., the ability
> to specify the number of standby replicas individually for each stateful
> operator.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5270) TransactionManager should send and `AddOffsetsToTxn` request only once per group per transaction
[ https://issues.apache.org/jira/browse/KAFKA-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5270: - Fix Version/s: (was: 1.0.0) 1.1.0 > TransactionManager should send and `AddOffsetsToTxn` request only once per > group per transaction > > > Key: KAFKA-5270 > URL: https://issues.apache.org/jira/browse/KAFKA-5270 > Project: Kafka > Issue Type: Sub-task > Components: clients, core, producer >Reporter: Apurva Mehta > Labels: exactly-once > Fix For: 1.1.0 > > > Currently, we send the `AddOffsetsToTxn` request unconditionally every time, > even if we receive multiple sendOffsets for the same group. We could keep > track of the added groups in the TransactionManager and not resend this RPC > multiple times for the same transaction as the subsequent instances add no > new information. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5864) ReplicaFetcherThread should not die due to replica in offline log directory
[ https://issues.apache.org/jira/browse/KAFKA-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5864: - Fix Version/s: (was: 1.0.0) 1.1.0 > ReplicaFetcherThread should not die due to replica in offline log directory > --- > > Key: KAFKA-5864 > URL: https://issues.apache.org/jira/browse/KAFKA-5864 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5524) Streams systems tests should be with EoS
[ https://issues.apache.org/jira/browse/KAFKA-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5524: - Fix Version/s: (was: 1.0.0) 1.1.0 > Streams systems tests should be with EoS > > > Key: KAFKA-5524 > URL: https://issues.apache.org/jira/browse/KAFKA-5524 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.11.0.0 >Reporter: Eno Thereska >Assignee: Matthias J. Sax > Fix For: 1.1.0 > > > Several system tests in streams should be using exactly-once. These include: > - all streams performance benchmarks in tests/kafkatest/benchmarks/streams/ > - tests/kafkatest/tests/streams/streams_broker_bounce_test.py especially > should be modified to strengthen the requirements for success. Currently it's > declared a success if streams doesn't crash, but it should also check that no > records have been delivered more than once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5695) Test DeleteRecordsRequest in AuthorizerIntegrationTest
[ https://issues.apache.org/jira/browse/KAFKA-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5695: - Fix Version/s: (was: 1.0.0) 1.1.0 > Test DeleteRecordsRequest in AuthorizerIntegrationTest > -- > > Key: KAFKA-5695 > URL: https://issues.apache.org/jira/browse/KAFKA-5695 > Project: Kafka > Issue Type: Improvement >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 1.1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5720) In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException
[
https://issues.apache.org/jira/browse/KAFKA-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5720:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with
> org.apache.kafka.common.errors.TimeoutException
> ---
>
> Key: KAFKA-5720
> URL: https://issues.apache.org/jira/browse/KAFKA-5720
> Project: Kafka
> Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Minor
> Fix For: 1.1.0
>
>
> In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with
> org.apache.kafka.common.errors.TimeoutException.
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.TimeoutException: Aborted due to timeout.
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213)
> at
> kafka.api.AdminClientIntegrationTest.testCallInFlightTimeouts(AdminClientIntegrationTest.scala:399)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Aborted due to
> timeout.
> {code}
> It's unclear whether this was an environment error or test bug.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5234) GetOffsetShell: retrieve offsets for multiple topics with single request
[ https://issues.apache.org/jira/browse/KAFKA-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-5234: - Fix Version/s: (was: 1.0.0) 1.1.0 > GetOffsetShell: retrieve offsets for multiple topics with single request > > > Key: KAFKA-5234 > URL: https://issues.apache.org/jira/browse/KAFKA-5234 > Project: Kafka > Issue Type: Improvement > Components: tools >Reporter: Arseniy Tashoyan > Labels: tool > Fix For: 1.1.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > At present, GetOffsetShell is able to retrieve offsets for one topic only: > --topic REQUIRED: The topic to get offsets from. > If user wants to get offsets for several topics, he has to call > GetOffsetShell as many times as the number of topics to explore. Some > solutions may have dozens of topics. Monitoring of a large Kafka cluster with > GetOffsetShell requires additional scripting efforts and produces visible > performance drawback due to multiple requests to the broker. > Instead, GetOffsetShell should support multiple topics, for example: > --topics topic1,topic2,topic3 > Moreover, GetOffsetShell should be able to retrieve offsets for _all_ topics, > when user specified none topics in command line. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5204) Connect needs to validate Connector type during instantiation
[
https://issues.apache.org/jira/browse/KAFKA-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5204:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Connect needs to validate Connector type during instantiation
> -
>
> Key: KAFKA-5204
> URL: https://issues.apache.org/jira/browse/KAFKA-5204
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
> Fix For: 1.1.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently Connect will accept to instantiate connectors that extend the
> {{Connector}} abstract class but not one of its subclasses,
> {{SourceConnector}} or {{SinkConnector}}.
> However, in distributed mode as well as in REST, Connect assumes in a few
> places that there are only two types of connectors, sinks or sources. Based
> on this assumption it checks the type dynamically, and if it is not a sink it
> treats it as a source (by constructing the corresponding configs).
> A connector that implements only the {{Connector}} abstract class does not
> fit into this classification. Therefore a validation needs to take place
> early, during the instantiation of the {{Connector}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5037) Infinite loop if all input topics are unknown at startup
[
https://issues.apache.org/jira/browse/KAFKA-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-5037:
-
Fix Version/s: (was: 1.0.0)
1.1.0
> Infinite loop if all input topics are unknown at startup
>
>
> Key: KAFKA-5037
> URL: https://issues.apache.org/jira/browse/KAFKA-5037
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Labels: user-experience
> Fix For: 1.1.0
>
>
> See discusion: https://github.com/apache/kafka/pull/2815
> We will need some rewrite on {{StreamPartitionsAssignor}} and to add much
> more test for all kind of corner cases, including pattern subscriptions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
