[jira] [Created] (KAFKA-6697) JBOD configured broker should not die if log directory is invalid
Dong Lin created KAFKA-6697: --- Summary: JBOD configured broker should not die if log directory is invalid Key: KAFKA-6697 URL: https://issues.apache.org/jira/browse/KAFKA-6697 Project: Kafka Issue Type: Improvement Reporter: Dong Lin Assignee: Dong Lin Currently JBOD configured broker will still die on startup if dir.getCanonicalPath() throws IOException. We should mark such log directory as offline and broker should still run if there is good disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6685) Connect deserialization log message should distinguish key from value
[ https://issues.apache.org/jira/browse/KAFKA-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407436#comment-16407436 ] Jagadesh Adireddi commented on KAFKA-6685: -- Hi , I am unable to assign this issue to myself. I made code changes. Waiting to get clarity on Error messages. Could you please assign this ticket to me. > Connect deserialization log message should distinguish key from value > - > > Key: KAFKA-6685 > URL: https://issues.apache.org/jira/browse/KAFKA-6685 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Yeva Byzek >Priority: Minor > Labels: newbie > > Connect was configured for Avro key and value but data had String key and > Avro value. The resulting error message was misleading because it didn't > distinguish key from value, and so I was chasing problems with the value > instead of the key. > tl;dr Connect should at least tell you whether the problem is with > deserializing the key or value of a record > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group
[
https://issues.apache.org/jira/browse/KAFKA-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407378#comment-16407378
]
Narayan Periwal commented on KAFKA-6681:
[~yuzhih...@gmail.com]
The partition read by the two consumers did not appear in any of the kafka
broker logs. Partition number 1 of the topic renderCpmAms1 was being consumed
by two consumer instances within a consumer group
The following are the log lines in the server logs related to this topic
{noformat}
[2018-03-14 05:01:53,456] INFO Partition [renderCpmAms1,10] on broker 1:
Shrinking ISR for partition [renderCpmAms1,10] from 1,2,3 to 1,3
(kafka.cluster.Partition)
[2018-03-14 05:02:14,122] INFO Partition [renderCpmAms1,10] on broker 1:
Expanding ISR for partition renderCpmAms1-10 from 1,3 to 1,3,2
(kafka.cluster.Partition)
[2018-03-14 05:01:52,376] INFO Partition [renderCpmAms1,9] on broker 15:
Shrinking ISR for partition [renderCpmAms1,9] from 2,15,1 to 15,1
(kafka.cluster.Partition)
[2018-03-14 05:02:14,193] INFO Partition [renderCpmAms1,9] on broker 15:
Expanding ISR for partition renderCpmAms1-9 from 15,1 to 15,1,2
(kafka.cluster.Partition)
[2018-03-14 05:02:17,510] INFO Partition [renderCpmAms1,11] on broker 2:
Shrinking ISR for partition [renderCpmAms1,11] from 2,4,3 to 2,4
(kafka.cluster.Partition)
[2018-03-14 05:02:17,530] INFO Partition [renderCpmAms1,11] on broker 2: Cached
zkVersion [171] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
{noformat}
Wondering, if this error or log line has any corelation with the issue.
{noformat}
[2018-03-14 05:02:17,530] INFO Partition [renderCpmAms1,11] on broker 2: Cached
zkVersion [171] not equal to that in zookeeper, skip updating ISR
(kafka.cluster.Partition)
{noformat}
> Two instances of kafka consumer reading the same partition within a consumer
> group
> --
>
> Key: KAFKA-6681
> URL: https://issues.apache.org/jira/browse/KAFKA-6681
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.10.2.1
>Reporter: Narayan Periwal
>Priority: Critical
> Attachments: server-1.log, server-2.log
>
>
> We have seen this issue with the Kafka consumer, the new library that got
> introduced in 0.9
> With this new client, the group management is done by kafka coordinator,
> which is one of the kafka broker.
> We are using Kafka broker 0.10.2.1 and consumer client version is also
> 0.10.2.1
> The issue that we have faced is that, after rebalancing, some of the
> partitions gets consumed by 2 instances within a consumer group, leading to
> duplication of the entire partition data. Both the instances continue to read
> until the next rebalancing, or the restart of those clients.
> It looks like that a particular consumer goes on fetching the data from a
> partition, but the broker is not able to identify this "stale" consumer
> instance.
> During this time, we also see the underreplicated partition metrics spiking.
> We have hit this twice in production. Please look at it the earliest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics
[
https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407352#comment-16407352
]
Ewen Cheslack-Postava commented on KAFKA-3821:
--
So a couple of points --
* That new SourceRecordReceiver interface doesn't necessarily avoid
allocations, it likely just moves them around. The framework would still want
to track the collection generated from a single poll() call because in the case
of a connector throwing an error mid-poll(), we'd want to not actually write
any of the messages generated thus far. So we basically just move the list into
SourceRecordReceiver. In fact, doing so could make things worse because whereas
a Connector may be able to figure out how large the list needs to be and do a
single allocation, the framework has no idea and would probably do multiple
rounds of expanding the list.
* In any case, I really think that one allocation isn't worth worrying about
unless someone has profiling to show it. We allocate so many objects just for a
single SourceRecord, *especially* in pretty common cases of complex object
structure, that this doesn't seem worth optimizing.
* I think we should focus on optimizing for the common case, which is that
there will be multiple messages. When that's not the case, the performance
impact seems unimportant since it would probably mean you have relatively
infrequent events.
* Adding more interfaces adds to cognitive load and makes it harder to learn
how to write connectors. Context objects already provide a place to request the
framework do things outside the common workflow, so it seems like a natural
place to add this functionality if we decided to. Same deal for the EOS stuff,
which could potentially just be transaction APIs in the Context.
* Just stylistically, Kafka's public APIs tend to try to keep things simple
and straightforward (for some definition of those words that I honestly am not
sure I could give a clear explanation of).
I don't want to discourage the discussion of how to solve this problem for
Debezium's use case, but I do want to make sure we're taking into account
broader goals for the framework when figuring out how to solve it (e.g.
SourceRecordReceiver may work, but I would argue returning a list of records
makes things easier since it's obvious from one line of code how to implement
it).
It might also help to explain better why the general idea here just rubs me the
wrong way. Mostly it boils down to sort of breaking the abstraction. The way
offsets are handled for source connectors was supposed to mirror Kafka's log
abstraction such that an offset really is unique. I guess maybe a gap in
understanding for me is why offsets *wouldn't* be unique during snapshots and
how rewind works for these snapshots. I get that not all systems or use cases
can map perfectly, but a) would another option be to change how those offsets
are handled by Debezium, b) would another option be having Debezium read 1
record forward to determine before returning the record and/or constructing its
offset whether this is the final record of the snapshot, and c) do we have
other use cases the demonstrate an impedance mismatch that would be well
answered by solutions being proposed here?
> Allow Kafka Connect source tasks to produce offset without writing to topics
>
>
> Key: KAFKA-3821
> URL: https://issues.apache.org/jira/browse/KAFKA-3821
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>Priority: Major
> Labels: needs-kip
> Fix For: 1.2.0
>
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for
> a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown
> number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a
> database). Once the task completes those records, the connector wants to
> update the offsets (e.g., the snapshot is complete) but has no more records
> to be written to a topic. With this change, the task could simply supply an
> updated offset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6673) Segment and Stamped implement Comparable, but don't override equals.
[
https://issues.apache.org/jira/browse/KAFKA-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asutosh Pandya updated KAFKA-6673:
--
Attachment: KAFKA_6673.patch
> Segment and Stamped implement Comparable, but don't override equals.
>
>
> Key: KAFKA-6673
> URL: https://issues.apache.org/jira/browse/KAFKA-6673
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Koen De Groote
>Priority: Minor
> Attachments: KAFKA_6673.patch
>
>
> The classes in question:
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/Segment.java
> and
> https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/Stamped.java
> This came up while doing static analysis on the codebase on the trunk branch.
> As described by the analysis tool built into Intellij:
> {quote}
> Reports classes which implement java.lang.Comparable which do not override
> equals(). If equals() is not overridden, the equals() implementation is not
> consistent with the compareTo() implementation. If an object of such a class
> is added to a collection such as java.util.SortedSet, this collection will
> violate the contract of java.util.Set, which is defined in terms of equals().
> {quote}
>
> Implementing an equals for an object is generally a best practice, especially
> considering this caveat, where it's not the compareTo that will be used but
> the equals method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6696) Trogdor should support destroying tasks
Colin P. McCabe created KAFKA-6696: -- Summary: Trogdor should support destroying tasks Key: KAFKA-6696 URL: https://issues.apache.org/jira/browse/KAFKA-6696 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Colin P. McCabe Assignee: Colin P. McCabe Trogdor should support destroying tasks. This will make it more practical to have very long running Trogdor instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6695) Add a system test for dynamic broker config update
Rajini Sivaram created KAFKA-6695: - Summary: Add a system test for dynamic broker config update Key: KAFKA-6695 URL: https://issues.apache.org/jira/browse/KAFKA-6695 Project: Kafka Issue Type: Test Components: core Reporter: Rajini Sivaram Assignee: Rajini Sivaram Add a system test that does some basic validation of dynamic broker configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6694) The Trogdor Coordinator should support filtering task responses
Colin P. McCabe created KAFKA-6694:
--
Summary: The Trogdor Coordinator should support filtering task
responses
Key: KAFKA-6694
URL: https://issues.apache.org/jira/browse/KAFKA-6694
Project: Kafka
Issue Type: Improvement
Components: system tests
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe
Currently, a user must get the status of all tasks when hitting the
{{/coordinator/tasks}} endpoint on the Trogdor coordinator. To make the
responses smaller, the Trogdor coordinator should support filtering task
responses.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-5943) Reduce dependency on mock in connector tests
[ https://issues.apache.org/jira/browse/KAFKA-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KAFKA-5943: -- Description: Currently connector tests make heavy use of mock (easymock, power mock). This may hide the real logic behind operations and makes finding bugs difficult. We should reduce the use of mocks so that developers can debug connector code using unit tests. This would shorten the development cycle for connector. was: Currently connector tests make heavy use of mock (easymock, power mock). This may hide the real logic behind operations and makes finding bugs difficult. We should reduce the use of mocks so that developers can debug connector code using unit tests. This would shorten the development cycle for connector. > Reduce dependency on mock in connector tests > > > Key: KAFKA-5943 > URL: https://issues.apache.org/jira/browse/KAFKA-5943 > Project: Kafka > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Labels: connector > > Currently connector tests make heavy use of mock (easymock, power mock). > This may hide the real logic behind operations and makes finding bugs > difficult. > We should reduce the use of mocks so that developers can debug connector code > using unit tests. > This would shorten the development cycle for connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group
[
https://issues.apache.org/jira/browse/KAFKA-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407036#comment-16407036
]
Ted Yu commented on KAFKA-6681:
---
{code}
[2018-03-14 05:04:13,231] ERROR [ReplicaFetcherThread-0-15], Error for
partition [adhoc_network_click_ams1_ams1,5] to broker
15:org.apache.kafka.common.errors.
NotLeaderForPartitionException: This server is not the leader for that
topic-partition. (kafka.server.ReplicaFetcherThread)
[2018-03-14 05:04:13,233] ERROR [ReplicaFetcherThread-7-17], Error for
partition [cas_rr,21] to broker
17:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
isnot the leader for that topic-partition.
(kafka.server.ReplicaFetcherThread)
{code}
Did the partition being read by two consumers appear in the ERROR log ?
Thanks
> Two instances of kafka consumer reading the same partition within a consumer
> group
> --
>
> Key: KAFKA-6681
> URL: https://issues.apache.org/jira/browse/KAFKA-6681
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.10.2.1
>Reporter: Narayan Periwal
>Priority: Critical
> Attachments: server-1.log, server-2.log
>
>
> We have seen this issue with the Kafka consumer, the new library that got
> introduced in 0.9
> With this new client, the group management is done by kafka coordinator,
> which is one of the kafka broker.
> We are using Kafka broker 0.10.2.1 and consumer client version is also
> 0.10.2.1
> The issue that we have faced is that, after rebalancing, some of the
> partitions gets consumed by 2 instances within a consumer group, leading to
> duplication of the entire partition data. Both the instances continue to read
> until the next rebalancing, or the restart of those clients.
> It looks like that a particular consumer goes on fetching the data from a
> partition, but the broker is not able to identify this "stale" consumer
> instance.
> During this time, we also see the underreplicated partition metrics spiking.
> We have hit this twice in production. Please look at it the earliest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6693) Add Consumer-only benchmark workload to Trogdor
Anna Povzner created KAFKA-6693: --- Summary: Add Consumer-only benchmark workload to Trogdor Key: KAFKA-6693 URL: https://issues.apache.org/jira/browse/KAFKA-6693 Project: Kafka Issue Type: Improvement Components: system tests Reporter: Anna Povzner Assignee: Anna Povzner Consumer-only benchmark workload that uses existing pre-populated topic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6683) ReplicaFetcher crashes with "Attempted to complete a transaction which was not started"
[
https://issues.apache.org/jira/browse/KAFKA-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406835#comment-16406835
]
Chema Sanchez commented on KAFKA-6683:
--
Thanks to you for the response. Sadly I had no luck trying to reproduce the
issue with a single partition topic.
This is the full data log dir, and application logs from broker 10 I got today
at debug level. The issue raised at 2018-03-20 15:17:17,148
https://drive.google.com/file/d/19mL1drSEwDqKbkyX4MfDupr3LDIJHwfG/view?usp=sharing
This is the same from broker 11, casually it experienced the same issue some
minutes after broker 10, at 2018-03-20 15:19:28,132
https://drive.google.com/file/d/1KEJvk73x1DxXQe2ob6F8V09rdASg_2FI/view?usp=sharing
> ReplicaFetcher crashes with "Attempted to complete a transaction which was
> not started"
>
>
> Key: KAFKA-6683
> URL: https://issues.apache.org/jira/browse/KAFKA-6683
> Project: Kafka
> Issue Type: Bug
> Components: replication
>Affects Versions: 1.0.0
> Environment: os: GNU/Linux
> arch: x86_64
> Kernel: 4.9.77
> jvm: OpenJDK 1.8.0
>Reporter: Chema Sanchez
>Priority: Critical
> Attachments: server.properties
>
>
> We have been experiencing this issue lately when restarting or replacing
> brokers of our Kafka clusters during maintenance operations.
> Having restarted or replaced a broker, after some minutes performing normally
> it may suddenly throw the following exception and stop replicating some
> partitions:
> {code:none}
> 2018-03-15 17:23:01,482] ERROR [ReplicaFetcher replicaId=12, leaderId=10,
> fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread)
> java.lang.IllegalArgumentException: Attempted to complete a transaction which
> was not started
> at
> kafka.log.ProducerStateManager.completeTxn(ProducerStateManager.scala:720)
> at kafka.log.Log.$anonfun$loadProducersFromLog$4(Log.scala:540)
> at
> kafka.log.Log.$anonfun$loadProducersFromLog$4$adapted(Log.scala:540)
> at scala.collection.immutable.List.foreach(List.scala:389)
> at
> scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:35)
> at
> scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:35)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:44)
> at kafka.log.Log.loadProducersFromLog(Log.scala:540)
> at kafka.log.Log.$anonfun$loadProducerState$5(Log.scala:521)
> at kafka.log.Log.$anonfun$loadProducerState$5$adapted(Log.scala:514)
> at scala.collection.Iterator.foreach(Iterator.scala:929)
> at scala.collection.Iterator.foreach$(Iterator.scala:929)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
> at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.log.Log.loadProducerState(Log.scala:514)
> at kafka.log.Log.$anonfun$truncateTo$2(Log.scala:1487)
> at
> scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:12)
> at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> at kafka.log.Log.truncateTo(Log.scala:1467)
> at kafka.log.LogManager.$anonfun$truncateTo$2(LogManager.scala:454)
> at
> kafka.log.LogManager.$anonfun$truncateTo$2$adapted(LogManager.scala:445)
> at
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:120)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
> at kafka.log.LogManager.truncateTo(LogManager.scala:445)
> at
> kafka.server.ReplicaFetcherThread.$anonfun$maybeTruncate$1(ReplicaFetcherThread.scala:281)
> at scala.collection.Iterator.foreach(Iterator.scala:929)
> at scala.collection.Iterator.foreach$(Iterator.scala:929)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
> at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.server.ReplicaFetcherThread.maybeTruncate(ReplicaFetcherThread.scala:265)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeTruncate$2(AbstractFetcherThread.scala:135)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
> at
>
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406832#comment-16406832
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:29 PM:
-
The prefix of those topics is the user specified {{application.id}} that is
passed in as a config. Thus, users can also add a "{{__}}"-prefix to the
application ID to get the same naming pattern.
was (Author: mjsax):
The prefix of those topics is the user specified {{application.id}} that is
passed in as a config. Thus, users can also add the {{__}} to the application
ID to get the same naming pattern.
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406832#comment-16406832
]
Matthias J. Sax commented on KAFKA-6692:
The prefix of those topics is the user specified {{application.id}} that is
passed in as a config. Thus, users can also add the {{__}} to the application
ID to get the same naming pattern.
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406801#comment-16406801
]
Guozhang Wang commented on KAFKA-6692:
--
Maybe this can better be handled in
https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-6692:
---
Comment: was deleted
(was: All internal topics are prefixed with the user specified
{{application.id}}. It's {{"DASH-repartition"}} and
{{"DASH-changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough))
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:04 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"DASH-repartition"}} and
{{"DASH-changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough)
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough)
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:03 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough)
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"---repartition"-}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough)
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:02 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:02 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"---repartition"-}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
(Sorry, the mark up is terrible... can get rid or "dash" triggering
strikethough)
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6376) Improve Streams metrics for skipped records
[
https://issues.apache.org/jira/browse/KAFKA-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406785#comment-16406785
]
Guozhang Wang commented on KAFKA-6376:
--
Just to summarize places where Streams library could skip record on top of my
head:
1) deser errors on consumer.
2) produce errors on producer.
3) null key records on aggregate operator.
4) too old records received past the window retention on windowing operator.
> Improve Streams metrics for skipped records
> ---
>
> Key: KAFKA-6376
> URL: https://issues.apache.org/jira/browse/KAFKA-6376
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: John Roesler
>Priority: Major
> Labels: needs-kip
>
> Copy this from KIP-210 discussion thread:
> {quote}
> Note that currently we have two metrics for `skipped-records` on different
> levels:
> 1) on the highest level, the thread-level, we have a `skipped-records`,
> that records all the skipped records due to deserialization errors.
> 2) on the lower processor-node level, we have a
> `skippedDueToDeserializationError`, that records the skipped records on
> that specific source node due to deserialization errors.
> So you can see that 1) does not cover any other scenarios and can just be
> thought of as an aggregate of 2) across all the tasks' source nodes.
> However, there are other places that can cause a record to be dropped, for
> example:
> 1) https://issues.apache.org/jira/browse/KAFKA-5784: records could be
> dropped due to window elapsed.
> 2) KIP-210: records could be dropped on the producer side.
> 3) records could be dropped during user-customized processing on errors.
> {quote}
> [~guozhang] Not sure what you mean by "3) records could be dropped during
> user-customized processing on errors."
> Btw: we also drop record with {{null}} key and/or value for certain DSL
> operations. This should be included as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:01 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 6:00 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's {{"--repartition"}} and
{{"--changelog"}}. I am wondering it this is
sufficient or not? What does adding two underscored improve?
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's "--repartition" and
"--changelog". I am wondering it this is sufficient
or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax edited comment on KAFKA-6692 at 3/20/18 5:59 PM:
-
All internal topics are prefixed with the user specified {{application.id}}.
It's "--repartition" and
"--changelog". I am wondering it this is sufficient
or not? What does adding two underscored improve?
was (Author: mjsax):
All internal topics are prefixed with the user specified {{application.id}}.
It's "--repartition" and
"--changelog". I am wondering it this is sufficient
or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406777#comment-16406777
]
Matthias J. Sax commented on KAFKA-6692:
All internal topics are prefixed with the user specified {{application.id}}.
It's "--repartition" and
"--changelog". I am wondering it this is sufficient
or not? What does adding two underscored improve?
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6690: --- Component/s: streams > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bala Prassanna I >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6690: --- Issue Type: New Feature (was: Improvement) > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6682) Kafka reconnection after broker restart
[
https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406717#comment-16406717
]
Manikumar edited comment on KAFKA-6682 at 3/20/18 5:14 PM:
---
Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260
You can try using 1.0.1 client library.
was (Author: omkreddy):
Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260
> Kafka reconnection after broker restart
> ---
>
> Key: KAFKA-6682
> URL: https://issues.apache.org/jira/browse/KAFKA-6682
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 1.0.0
>Reporter: madi
>Priority: Major
>
> I am using kafka producer plugin for logback (danielwegener) with the clients
> library 1.0.0 and after restart of broker all my JVMs connected to it get
> tons of the exceptions:
> {code:java}
> 11:22:48.738 [kafka-producer-network-thread | app-logback-relaxed] cid:
> clid: E [ @] a: o.a.k.c.p.internals.Sender - [Producer
> clientId=id-id-logback-relaxed] Uncaught error in kafka producer I/O
> thread: ex:java.lang.NullPointerException: null
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:399)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:798){code}
> During restart there are still other brokers available behind LB.
> Dosen't matter kafka is up again, only restarting JVM helps
> {code:java}
> class="com.github.danielwegener.logback.kafka.KafkaAppender">
>
>
>
> %date{"-MM-dd'T'HH:mm:ss.SSS'Z'"} ${HOSTNAME}
> [%thread] %logger{32} - %message ex:%exf%n
>
> mytopichere
>
> class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy"
> />
>
> class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"
> />
>
>
>
>
> bootstrap.servers=10.99.99.1:9092
>
> acks=0
>
> block.on.buffer.full=false
>
>
> client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed
>
>
> compression.type=none
>
>
> max.block.ms=0
> {code}
> I provide loadbalancer address in bootstrap servers here. There are three
> kafka brokers behind.
> {code:java}
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build pap6470sr9fp60ifix-20161110_01(SR9
> FP60)+IV90630+IV90578))
> IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References
> 20161005_321282 (JIT enabled, AOT enabled)
> J9VM - R26_Java726_SR9_20161005_1259_B321282
> JIT - tr.r11_20161001_125404
> GC - R26_Java726_SR9_20161005_1259_B321282_CMPRSS
> J9CL - 20161005_321282)
> JCL - 20161021_01 based on Oracle jdk7u121-b15{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6682) Kafka reconnection after broker restart
[
https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406717#comment-16406717
]
Manikumar commented on KAFKA-6682:
--
Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260
> Kafka reconnection after broker restart
> ---
>
> Key: KAFKA-6682
> URL: https://issues.apache.org/jira/browse/KAFKA-6682
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 1.0.0
>Reporter: madi
>Priority: Major
>
> I am using kafka producer plugin for logback (danielwegener) with the clients
> library 1.0.0 and after restart of broker all my JVMs connected to it get
> tons of the exceptions:
> {code:java}
> 11:22:48.738 [kafka-producer-network-thread | app-logback-relaxed] cid:
> clid: E [ @] a: o.a.k.c.p.internals.Sender - [Producer
> clientId=id-id-logback-relaxed] Uncaught error in kafka producer I/O
> thread: ex:java.lang.NullPointerException: null
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:399)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:798){code}
> During restart there are still other brokers available behind LB.
> Dosen't matter kafka is up again, only restarting JVM helps
> {code:java}
> class="com.github.danielwegener.logback.kafka.KafkaAppender">
>
>
>
> %date{"-MM-dd'T'HH:mm:ss.SSS'Z'"} ${HOSTNAME}
> [%thread] %logger{32} - %message ex:%exf%n
>
> mytopichere
>
> class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy"
> />
>
> class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"
> />
>
>
>
>
> bootstrap.servers=10.99.99.1:9092
>
> acks=0
>
> block.on.buffer.full=false
>
>
> client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed
>
>
> compression.type=none
>
>
> max.block.ms=0
> {code}
> I provide loadbalancer address in bootstrap servers here. There are three
> kafka brokers behind.
> {code:java}
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build pap6470sr9fp60ifix-20161110_01(SR9
> FP60)+IV90630+IV90578))
> IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References
> 20161005_321282 (JIT enabled, AOT enabled)
> J9VM - R26_Java726_SR9_20161005_1259_B321282
> JIT - tr.r11_20161001_125404
> GC - R26_Java726_SR9_20161005_1259_B321282_CMPRSS
> J9CL - 20161005_321282)
> JCL - 20161021_01 based on Oracle jdk7u121-b15{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6084) ReassignPartitionsCommand should propagate JSON parsing failures
[
https://issues.apache.org/jira/browse/KAFKA-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406696#comment-16406696
]
Viktor Somogyi commented on KAFKA-6084:
---
[~nickt] it seems the linked PR doesn't fix my general problem. That is when I
(in a very simple case) enter an invalid json which is just as simple as having
"dgdfgdfhdfgfgd" in it, I get a "kafka.common.AdminCommandFailedException:
Partition reassignment data file is empty" exception. This is true for more
complex json parsing cases too as mentioned in the description. Do you think
it's possible once your commit is merged I rebase on it and add my amendments?
> ReassignPartitionsCommand should propagate JSON parsing failures
>
>
> Key: KAFKA-6084
> URL: https://issues.apache.org/jira/browse/KAFKA-6084
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Affects Versions: 0.11.0.0
>Reporter: Viktor Somogyi
>Assignee: Viktor Somogyi
>Priority: Minor
> Labels: easyfix, newbie
> Attachments: Screen Shot 2017-10-18 at 23.31.22.png
>
>
> Basically looking at Json.scala it will always swallow any parsing errors:
> {code}
> def parseFull(input: String): Option[JsonValue] =
> try Option(mapper.readTree(input)).map(JsonValue(_))
> catch { case _: JsonProcessingException => None }
> {code}
> However sometimes it is easy to figure out the problem by simply looking at
> the JSON, in some cases it is not very trivial, such as some invisible
> characters (like byte order mark) won't be displayed by most of the text
> editors and can people spend time on figuring out what's the problem.
> As Jackson provides a really detailed exception about what failed and how, it
> is easy to propagate the failure to the user.
> As an example I attached a BOM prefixed JSON which fails with the following
> error which is very counterintuitive:
> {noformat}
> [root@localhost ~]# kafka-reassign-partitions --zookeeper localhost:2181
> --reassignment-json-file /root/increase-replication-factor.json --execute
> Partitions reassignment failed due to Partition reassignment data file
> /root/increase-replication-factor.json is empty
> kafka.common.AdminCommandFailedException: Partition reassignment data file
> /root/increase-replication-factor.json is empty
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:120)
> at
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:52)
> at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> ...
> {noformat}
> In case of the above error it would be much better to see what fails exactly:
> {noformat}
> kafka.common.AdminCommandFailedException: Admin command failed
> at
> kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:267)
> at
> kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:275)
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:197)
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:193)
> at
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:64)
> at
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
> Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected
> character ('' (code 65279 / 0xfeff)): expected a valid value (number,
> String, array, object, 'true', 'false' or 'null')
> at [Source: (String)"{"version":1,
> "partitions":[
>{"topic": "test1", "partition": 0, "replicas": [1,2]},
>{"topic": "test2", "partition": 1, "replicas": [2,3]}
> ]}"; line: 1, column: 2]
> at
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
> at
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
> at
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
> at
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1892)
> at
> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:747)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4030)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2539)
> at kafka.utils.Json$.kafka$utils$Json$$doParseFull(Json.scala:46)
> at kafka.utils.Json$$anonfun$tryParseFull$1.apply(Json.scala:44)
> at kafka.utils.Json$$anonfun$tryParseFull$1.apply(Json.scala:44)
> at scala.util.Try$.apply(Try.scala:192)
>
[jira] [Updated] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
[
https://issues.apache.org/jira/browse/KAFKA-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yeva Byzek updated KAFKA-6692:
--
Description:
Issue: users cannot quickly/easily differentiate between their explicitly
created topics from internal topics that Kafka Streams uses, e.g.
{{*-changelog}} and {{*-repartition}}
Proposed solution: Kafka Streams internal topics should be prefixed with an
underscore
Value add: ability to introspect kstreams and be able to *know* a set of topics
are associated with a topology. This can help downstream tooling to do cool
visualizations and tracking
was:
Issue: users cannot quickly/easily differentiate between their explicitly
created topics from internal topics that Kafka Streams uses, e.g.
{{*-changelog}} and {{*-repartition}}
Proposed solution: Kafka Streams internal topics should be prefixed with an
underscore
> Kafka Streams internal topics should be prefixed with an underscore
> ---
>
> Key: KAFKA-6692
> URL: https://issues.apache.org/jira/browse/KAFKA-6692
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Yeva Byzek
>Priority: Major
>
> Issue: users cannot quickly/easily differentiate between their explicitly
> created topics from internal topics that Kafka Streams uses, e.g.
> {{*-changelog}} and {{*-repartition}}
>
> Proposed solution: Kafka Streams internal topics should be prefixed with an
> underscore
> Value add: ability to introspect kstreams and be able to *know* a set of
> topics are associated with a topology. This can help downstream tooling to
> do cool visualizations and tracking
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6692) Kafka Streams internal topics should be prefixed with an underscore
Yeva Byzek created KAFKA-6692:
-
Summary: Kafka Streams internal topics should be prefixed with an
underscore
Key: KAFKA-6692
URL: https://issues.apache.org/jira/browse/KAFKA-6692
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 1.0.0
Reporter: Yeva Byzek
Issue: users cannot quickly/easily differentiate between their explicitly
created topics from internal topics that Kafka Streams uses, e.g.
{{*-changelog}} and {{*-repartition}}
Proposed solution: Kafka Streams internal topics should be prefixed with an
underscore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group
[ https://issues.apache.org/jira/browse/KAFKA-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406681#comment-16406681 ] Narayan Periwal commented on KAFKA-6681: [~tedyu], Attached the server side logs. Could not retrieve the consumer side logs as it has hit the retention, I am again trying to reproduce this in our QA setup. See if the server side logs is of any help. The server side logs correspond to those node in which the under replicated metrics spiked during this time There is no logs in the controller.log file during this time. > Two instances of kafka consumer reading the same partition within a consumer > group > -- > > Key: KAFKA-6681 > URL: https://issues.apache.org/jira/browse/KAFKA-6681 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.10.2.1 >Reporter: Narayan Periwal >Priority: Critical > Attachments: server-1.log, server-2.log > > > We have seen this issue with the Kafka consumer, the new library that got > introduced in 0.9 > With this new client, the group management is done by kafka coordinator, > which is one of the kafka broker. > We are using Kafka broker 0.10.2.1 and consumer client version is also > 0.10.2.1 > The issue that we have faced is that, after rebalancing, some of the > partitions gets consumed by 2 instances within a consumer group, leading to > duplication of the entire partition data. Both the instances continue to read > until the next rebalancing, or the restart of those clients. > It looks like that a particular consumer goes on fetching the data from a > partition, but the broker is not able to identify this "stale" consumer > instance. > During this time, we also see the underreplicated partition metrics spiking. > We have hit this twice in production. Please look at it the earliest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5891) Cast transformation fails if record schema contains timestamp field
[
https://issues.apache.org/jira/browse/KAFKA-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406667#comment-16406667
]
Randall Hauch commented on KAFKA-5891:
--
KAFKA-6290 is another issue related to the CAST transformation, but deals with
actually being able to CAST fields that use logical types.
This issue, OTOH, is simply that the record's Struct can only contain fields
that have a castable (i.e., currently primitive) type.
> Cast transformation fails if record schema contains timestamp field
> ---
>
> Key: KAFKA-5891
> URL: https://issues.apache.org/jira/browse/KAFKA-5891
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Artem Plotnikov
>Priority: Major
>
> I have the following simple type cast transformation:
> {code}
> name=postgresql-source-simple
> connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
> tasks.max=1
> connection.url=jdbc:postgresql://localhost:5432/testdb?user=postgres=mysecretpassword
> query=SELECT 1::INT as a, '2017-09-14 10:23:54'::TIMESTAMP as b
> transforms=Cast
> transforms.Cast.type=org.apache.kafka.connect.transforms.Cast$Value
> transforms.Cast.spec=a:boolean
> mode=bulk
> topic.prefix=clients
> {code}
> Which fails with the following exception in runtime:
> {code}
> [2017-09-14 16:51:01,885] ERROR Task postgresql-source-simple-0 threw an
> uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask:148)
> org.apache.kafka.connect.errors.DataException: Invalid Java object for schema
> type INT64: class java.sql.Timestamp for field: "null"
> at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:239)
> at
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:209)
> at org.apache.kafka.connect.data.Struct.put(Struct.java:214)
> at
> org.apache.kafka.connect.transforms.Cast.applyWithSchema(Cast.java:152)
> at org.apache.kafka.connect.transforms.Cast.apply(Cast.java:108)
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:190)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:168)
> at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> If I remove the transforms.* part of the connector it will work correctly.
> Actually, it doesn't really matter which types I use in the transformation
> for field 'a', just the existence of a timestamp field brings the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group
[ https://issues.apache.org/jira/browse/KAFKA-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narayan Periwal updated KAFKA-6681: --- Attachment: server-2.log server-1.log > Two instances of kafka consumer reading the same partition within a consumer > group > -- > > Key: KAFKA-6681 > URL: https://issues.apache.org/jira/browse/KAFKA-6681 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.10.2.1 >Reporter: Narayan Periwal >Priority: Critical > Attachments: server-1.log, server-2.log > > > We have seen this issue with the Kafka consumer, the new library that got > introduced in 0.9 > With this new client, the group management is done by kafka coordinator, > which is one of the kafka broker. > We are using Kafka broker 0.10.2.1 and consumer client version is also > 0.10.2.1 > The issue that we have faced is that, after rebalancing, some of the > partitions gets consumed by 2 instances within a consumer group, leading to > duplication of the entire partition data. Both the instances continue to read > until the next rebalancing, or the restart of those clients. > It looks like that a particular consumer goes on fetching the data from a > partition, but the broker is not able to identify this "stale" consumer > instance. > During this time, we also see the underreplicated partition metrics spiking. > We have hit this twice in production. Please look at it the earliest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6684) Support casting values with bytes schema to string
[
https://issues.apache.org/jira/browse/KAFKA-6684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406470#comment-16406470
]
Amit Sela commented on KAFKA-6684:
--
[~hachikuji] mind taking a look?
> Support casting values with bytes schema to string
> ---
>
> Key: KAFKA-6684
> URL: https://issues.apache.org/jira/browse/KAFKA-6684
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Amit Sela
>Priority: Major
>
> Casting from BYTES is not supported, which means that casting LogicalTypes is
> not supported.
> This proposes to allow casting anything to a string, kind of like Java's
> {{toString()}}, such that if the object is actually a LogicalType it can be
> "serialized" as string instead of bytes+schema.
> Worst case, bytes are "casted" to whatever the {{toString()}} returns - its
> up to the user to know the data.
> This would help when using a JSON sink, or anything that's not Avro.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6680) Fix config initialization in DynamicBrokerConfig
[
https://issues.apache.org/jira/browse/KAFKA-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406439#comment-16406439
]
ASF GitHub Bot commented on KAFKA-6680:
---
rajinisivaram opened a new pull request #4740: MINOR: Document workaround for
KAFKA-6680 for 1.1
URL: https://github.com/apache/kafka/pull/4740
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix config initialization in DynamicBrokerConfig
>
>
> Key: KAFKA-6680
> URL: https://issues.apache.org/jira/browse/KAFKA-6680
> Project: Kafka
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Major
> Fix For: 1.2.0
>
>
> Below issues observed while testing dynamic config update feature
> 1. {{kafkaConfig}} doesn't get updated during {{initialize}} if there are no
> dynamic configs defined in ZK.
> 2. update DynamicListenerConfig.validateReconfiguration() to check new
> Listeners must be subset of listener map
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6691) Requesting High Watermark and Truncation documentation be added to Docs
Justin Manchester created KAFKA-6691: Summary: Requesting High Watermark and Truncation documentation be added to Docs Key: KAFKA-6691 URL: https://issues.apache.org/jira/browse/KAFKA-6691 Project: Kafka Issue Type: Improvement Components: documentation Affects Versions: 1.0.1 Reporter: Justin Manchester Requesting that we add documentation for what is outlined in KIP-101, specifically: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation#KIP-101-AlterReplicationProtocoltouseLeaderEpochratherthanHighWatermarkforTruncation-Scenario1:HighWatermarkTruncationfollowedbyImmediateLeaderElection -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Prassanna I updated KAFKA-6690: Description: We often encounter use cases where we need to prioritise source topics. If a consumer is listening more than one topic, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well. (was: We often encounter use cases where we need to prioritise source topics. If a consumer is listening more than one topics, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well.) > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: Improvement >Reporter: Bala Prassanna I >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Prassanna I updated KAFKA-6690: Description: We often encounter use cases where we need to prioritise source topics. If a consumer is listening more than one topics, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well. (was: We often encounter use cases where we need to prioritise source topics. Such that if a consumer is listening more than one topics, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well.) > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: Improvement >Reporter: Bala Prassanna I >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topics, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6690) Priorities for Source Topics
Bala Prassanna I created KAFKA-6690: --- Summary: Priorities for Source Topics Key: KAFKA-6690 URL: https://issues.apache.org/jira/browse/KAFKA-6690 Project: Kafka Issue Type: Improvement Reporter: Bala Prassanna I We often encounter use cases where we need to prioritise source topics. Such that if a consumer is listening more than one topics, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6685) Connect deserialization log message should distinguish key from value
[ https://issues.apache.org/jira/browse/KAFKA-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406081#comment-16406081 ] Jagadesh Adireddi edited comment on KAFKA-6685 at 3/20/18 10:25 AM: Hi [~rhauch], I would like to pick this Jira up. Wanted some advice on the error message .I am thinking , if the exception occurred during: a) keyAndSchema conversion, then error msg logged as : Failed to convert message Key And Schema to Kafka Connect format. b) valueAndSchema conversion, then error msg logged as : Failed to convert message Value And Schema to Kafka Connect format. And Main ConnectException thrown as : Exiting WorkerSinkTask due to unconvertedmessage exception. Any help on the hint message to fix the issue would be great? was (Author: adireddijagad...@gmail.com): Hi [~rhauch], I would like to pick this Jira up. Wanted some advice on the error message . I am thinking , if the exception occurred during: a) keyAndSchema conversion, then error msg logged as : Failed to convert message Key And Schema to Kafka Connect format. b) valueAndSchema conversion, then error msg logged as : Failed to convert message Value And Schema to Kafka Connect format. And Main ConnectException thrown as : Exiting WorkerSinkTask due to unconvertedmessage exception. Any help on the hint message to fix the issue would be great? > Connect deserialization log message should distinguish key from value > - > > Key: KAFKA-6685 > URL: https://issues.apache.org/jira/browse/KAFKA-6685 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Yeva Byzek >Priority: Minor > Labels: newbie > > Connect was configured for Avro key and value but data had String key and > Avro value. The resulting error message was misleading because it didn't > distinguish key from value, and so I was chasing problems with the value > instead of the key. > tl;dr Connect should at least tell you whether the problem is with > deserializing the key or value of a record > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-6613) The controller shouldn't stop partition reassignment after an exception is being thrown
[
https://issues.apache.org/jira/browse/KAFKA-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chandra kasiraju resolved KAFKA-6613.
-
Resolution: Fixed
Fix Version/s: (was: 0.11.0.2)
1.0.0
> The controller shouldn't stop partition reassignment after an exception is
> being thrown
> ---
>
> Key: KAFKA-6613
> URL: https://issues.apache.org/jira/browse/KAFKA-6613
> Project: Kafka
> Issue Type: Bug
> Components: admin, config, controller, core
>Affects Versions: 0.11.0.2
>Reporter: chandra kasiraju
>Priority: Major
> Fix For: 1.0.0
>
>
> I issued a partition reassignment command . It created the following entries
> in the zookeeper .
> But the entry never gets deleted because the partition reassigment hangs gets
> some exceptions in kafka logs . After that no matter how many hours the
> movement of partitions to other brokers never happens .
>
> *Path in Zookeeper*
> get /admin/reassign_partitions
> {"version":1,"partitions":[{"topic":"__consumer_offsets","partition":44,"replicas":([1003,1001,1004,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic1","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"__CruiseControlMetrics","partition":0,"replicas":[1002,1001,1004,1003]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"88ec4bd5-e149-4c98-8e8e-952e86ba5fae_topic","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"c8c56723-73a5-4a37-93bf-b8ecaf766429_topic","partition":4,"replicas":[1002,1003,1004,1001]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":4,"replicas":[1003,1001,1004,1002]},\{"topic":"9db0cad2-69f8-4e85-b663-cd3987bd90fe_topic","partition":0,"replicas":[1003,1001,1004]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_topic","partition":1,"replicas":[1003,1004,1001,1002]}]}
> cZxid = 0x552f8
> ctime = Tue Mar 06 01:27:54 UTC 2018
> mZxid = 0x55359
> mtime = Tue Mar 06 01:28:06 UTC 2018
> pZxid = 0x552f8
> cversion = 0
> dataVersion = 13
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 1114
> numChildren = 0
>
>
> *Exception*
>
> ERROR [KafkaApi-1002] Error when handling request
> \{replica_id=1005,max_wait_time=500,min_bytes=1,max_bytes=10485760,isolation_level=0,topics=[{topic=__consumer_offsets,partitions=[{partition=41,fetch_offset=0,log_start_offset=0,max_bytes=1048576}]}]}
> (kafka.server.KafkaApis)
> kafka.common.NotAssignedReplicaException: Leader 1002 failed to record
> follower 1005's position 0 since the replica is not recognized to be one of
> the assigned replicas 1001,1002,1004 for partition __consumer_offsets-41.
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:274)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1092)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1089)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:1089)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:623)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:606)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:98)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:66)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> I was expecting it would be recover from that exception move the partitions
> to other nodes and finally remove the entries in /admin/reassign_partitions
> after the move has happened.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6613) The controller shouldn't stop partition reassignment after an exception is being thrown
[
https://issues.apache.org/jira/browse/KAFKA-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405915#comment-16405915
]
chandra kasiraju commented on KAFKA-6613:
-
thanks [~omkreddy] i will try 1.0 and see if that is fixed. closing the issue
for now .
> The controller shouldn't stop partition reassignment after an exception is
> being thrown
> ---
>
> Key: KAFKA-6613
> URL: https://issues.apache.org/jira/browse/KAFKA-6613
> Project: Kafka
> Issue Type: Bug
> Components: admin, config, controller, core
>Affects Versions: 0.11.0.2
>Reporter: chandra kasiraju
>Priority: Major
> Fix For: 1.0.0
>
>
> I issued a partition reassignment command . It created the following entries
> in the zookeeper .
> But the entry never gets deleted because the partition reassigment hangs gets
> some exceptions in kafka logs . After that no matter how many hours the
> movement of partitions to other brokers never happens .
>
> *Path in Zookeeper*
> get /admin/reassign_partitions
> {"version":1,"partitions":[{"topic":"__consumer_offsets","partition":44,"replicas":([1003,1001,1004,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic1","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"__CruiseControlMetrics","partition":0,"replicas":[1002,1001,1004,1003]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"88ec4bd5-e149-4c98-8e8e-952e86ba5fae_topic","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"c8c56723-73a5-4a37-93bf-b8ecaf766429_topic","partition":4,"replicas":[1002,1003,1004,1001]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":4,"replicas":[1003,1001,1004,1002]},\{"topic":"9db0cad2-69f8-4e85-b663-cd3987bd90fe_topic","partition":0,"replicas":[1003,1001,1004]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_topic","partition":1,"replicas":[1003,1004,1001,1002]}]}
> cZxid = 0x552f8
> ctime = Tue Mar 06 01:27:54 UTC 2018
> mZxid = 0x55359
> mtime = Tue Mar 06 01:28:06 UTC 2018
> pZxid = 0x552f8
> cversion = 0
> dataVersion = 13
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 1114
> numChildren = 0
>
>
> *Exception*
>
> ERROR [KafkaApi-1002] Error when handling request
> \{replica_id=1005,max_wait_time=500,min_bytes=1,max_bytes=10485760,isolation_level=0,topics=[{topic=__consumer_offsets,partitions=[{partition=41,fetch_offset=0,log_start_offset=0,max_bytes=1048576}]}]}
> (kafka.server.KafkaApis)
> kafka.common.NotAssignedReplicaException: Leader 1002 failed to record
> follower 1005's position 0 since the replica is not recognized to be one of
> the assigned replicas 1001,1002,1004 for partition __consumer_offsets-41.
> at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:274)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1092)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1089)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:1089)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:623)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:606)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:98)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:66)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> I was expecting it would be recover from that exception move the partitions
> to other nodes and finally remove the entries in /admin/reassign_partitions
> after the move has happened.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6689) Kafka not release .deleted file.
[ https://issues.apache.org/jira/browse/KAFKA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405870#comment-16405870 ] Manikumar edited comment on KAFKA-6689 at 3/20/18 7:05 AM: --- These temp files will be deleted after file.delete.delay.ms (default: 6) milliseconds after file creation. file count increasing continuously? Are you observing any exceptions in logs? was (Author: omkreddy): There temp files will be deleted after file.delete.delay.ms (default: 6) milliseconds after file creation. file count increasing continuously? Are you observing any exceptions in logs? > Kafka not release .deleted file. > > > Key: KAFKA-6689 > URL: https://issues.apache.org/jira/browse/KAFKA-6689 > Project: Kafka > Issue Type: Bug > Components: config, controller, log >Affects Versions: 0.10.1.1 > Environment: 3 Kafka Broker running on CentOS6.9(rungi3 VMs) >Reporter: A >Priority: Critical > Labels: newbie > Fix For: 0.10.1.1 > > > After Kafka cleaned log .timeindex / .index files based on topic > retention. I can > still lsof a lot of .index.deleted and .timeindex.deleted files. > We have 3 Brokers on 3 VMs , It's happen only 2 brokers. > [brokeer-01 ~]$ lsof -p 24324 | grep deleted | wc -l > 28 > [broker-02 ~]$ lsof -p 12131 | grep deleted | wc -l > 14599 > [broker-03 ~]$ lsof -p 3349 | grep deleted | wc -l > 14523 > > Configuration on 3 broker is same. (Rolllog every hour, Retention time 11 > Hours) > * INFO KafkaConfig values: > advertised.host.name = null > advertised.listeners = PLAINTEXT://Broker-02:9092 > advertised.port = null > authorizer.class.name = > auto.create.topics.enable = true > auto.leader.rebalance.enable = true > background.threads = 10 > broker.id = 2 > broker.id.generation.enable = true > broker.rack = null > compression.type = producer > connections.max.idle.ms = 60 > controlled.shutdown.enable = true > controlled.shutdown.max.retries = 3 > controlled.shutdown.retry.backoff.ms = 5000 > controller.socket.timeout.ms = 3 > default.replication.factor = 3 > delete.topic.enable = true > fetch.purgatory.purge.interval.requests = 1000 > group.max.session.timeout.ms = 30 > group.min.session.timeout.ms = 6000 > host.name = > inter.broker.protocol.version = 0.10.1-IV2 > leader.imbalance.check.interval.seconds = 300 > leader.imbalance.per.broker.percentage = 10 > listeners = null > log.cleaner.backoff.ms = 15000 > log.cleaner.dedupe.buffer.size = 134217728 > log.cleaner.delete.retention.ms = 8640 > log.cleaner.enable = true > log.cleaner.io.buffer.load.factor = 0.9 > log.cleaner.io.buffer.size = 524288 > log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 > log.cleaner.min.cleanable.ratio = 0.5 > log.cleaner.min.compaction.lag.ms = 0 > log.cleaner.threads = 1 > log.cleanup.policy = [delete] > log.dir = /tmp/kafka-logs > log.dirs = /data/appdata/kafka/data > log.flush.interval.messages = 9223372036854775807 > log.flush.interval.ms = null > log.flush.offset.checkpoint.interval.ms = 6 > log.flush.scheduler.interval.ms = 9223372036854775807 > log.index.interval.bytes = 4096 > log.index.size.max.bytes = 10485760 > log.message.format.version = 0.10.1-IV2 > log.message.timestamp.difference.max.ms = 9223372036854775807 > log.message.timestamp.type = CreateTime > log.preallocate = false > log.retention.bytes = -1 > log.retention.check.interval.ms = 30 > log.retention.hours = 11 > log.retention.minutes = 660 > log.retention.ms = 3960 > log.roll.hours = 1 > log.roll.jitter.hours = 0 > log.roll.jitter.ms = null > log.roll.ms = null > log.segment.bytes = 1073741824 > log.segment.delete.delay.ms = 6 > max.connections.per.ip = 2147483647 > max.connections.per.ip.overrides = > message.max.bytes = 112 > metric.reporters = [] > metrics.num.samples = 2 > metrics.sample.window.ms = 3 > min.insync.replicas = 2 > num.io.threads = 16 > num.network.threads = 16 > num.partitions = 10 > num.recovery.threads.per.data.dir = 3 > num.replica.fetchers = 1 > offset.metadata.max.bytes = 4096 > offsets.commit.required.acks = -1 > offsets.commit.timeout.ms = 5000 > offsets.load.buffer.size = 5242880 > offsets.retention.check.interval.ms = 60 > offsets.retention.minutes = 1440 > offsets.topic.compression.codec = 0 > offsets.topic.num.partitions = 50 > offsets.topic.replication.factor = 3 > offsets.topic.segment.bytes = 104857600 > port = 9092 > principal.builder.class = class > org.apache.kafka.common.security.auth.DefaultPrincipalBuilder > producer.purgatory.purge.interval.requests = 1000 > queued.max.requests = 3 > quota.consumer.default = 9223372036854775807 > quota.producer.default = 9223372036854775807 >
[jira] [Commented] (KAFKA-6689) Kafka not release .deleted file.
[ https://issues.apache.org/jira/browse/KAFKA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405892#comment-16405892 ] A commented on KAFKA-6689: -- This parameter affect on broker1 (it cleaning temp file normally) But on the Broker2 and Broker3 , The .deleted continue increasing until the number of .deleted file nearly 64K. then broker is crashed. --- logging when it crash FATAL [Replica Manager on Broker 1]: Halting due to unrecoverable I/O error while handling produce request: (kafka.server.ReplicaManager) kafka.common.KafkaStorageException: I/O exception in append to log 'amgw.topic.notify.update.db-2' at kafka.log.Log.append(Log.scala:349) at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:443) at kafka.cluster.Partition$$anonfun$10.apply(Partition.scala:429) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234) at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:240) at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:429) at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:407) at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:393) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashMap.foreach(HashMap.scala:99) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:393) at kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:330) at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:436) at kafka.server.KafkaApis.handle(KafkaApis.scala:78) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) at kafka.log.AbstractIndex.(AbstractIndex.scala:61) at kafka.log.TimeIndex.(TimeIndex.scala:55) at kafka.log.LogSegment.(LogSegment.scala:68) at kafka.log.Log.roll(Log.scala:778) at kafka.log.Log.maybeRoll(Log.scala:744) at kafka.log.Log.append(Log.scala:405) ... 22 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ... 28 more > Kafka not release .deleted file. > > > Key: KAFKA-6689 > URL: https://issues.apache.org/jira/browse/KAFKA-6689 > Project: Kafka > Issue Type: Bug > Components: config, controller, log >Affects Versions: 0.10.1.1 > Environment: 3 Kafka Broker running on CentOS6.9(rungi3 VMs) >Reporter: A >Priority: Critical > Labels: newbie > Fix For: 0.10.1.1 > > > After Kafka cleaned log .timeindex / .index files based on topic > retention. I can > still lsof a lot of .index.deleted and .timeindex.deleted files. > We have 3 Brokers on 3 VMs , It's happen only 2 brokers. > [brokeer-01 ~]$ lsof -p 24324 | grep deleted | wc -l > 28 > [broker-02 ~]$ lsof -p 12131 | grep deleted | wc -l > 14599 > [broker-03 ~]$ lsof -p 3349 | grep deleted | wc -l > 14523 > > Configuration on 3 broker is same. (Rolllog every hour, Retention time 11 > Hours) > * INFO KafkaConfig values: > advertised.host.name = null > advertised.listeners = PLAINTEXT://Broker-02:9092 > advertised.port = null > authorizer.class.name = > auto.create.topics.enable = true > auto.leader.rebalance.enable = true > background.threads = 10 > broker.id = 2 > broker.id.generation.enable = true > broker.rack = null > compression.type = producer > connections.max.idle.ms = 60 > controlled.shutdown.enable = true > controlled.shutdown.max.retries = 3 > controlled.shutdown.retry.backoff.ms = 5000 > controller.socket.timeout.ms = 3 > default.replication.factor = 3 > delete.topic.enable = true > fetch.purgatory.purge.interval.requests = 1000 > group.max.session.timeout.ms = 30 > group.min.session.timeout.ms = 6000 > host.name = > inter.broker.protocol.version = 0.10.1-IV2 > leader.imbalance.check.interval.seconds = 300 > leader.imbalance.per.broker.percentage = 10 > listeners = null > log.cleaner.backoff.ms = 15000 > log.cleaner.dedupe.buffer.size = 134217728 > log.cleaner.delete.retention.ms = 8640 > log.cleaner.enable = true >
[jira] [Commented] (KAFKA-6689) Kafka not release .deleted file.
[ https://issues.apache.org/jira/browse/KAFKA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405870#comment-16405870 ] Manikumar commented on KAFKA-6689: -- There temp files will be deleted after file.delete.delay.ms (default: 6) milliseconds after file creation. file count increasing continuously? Are you observing any exceptions in logs? > Kafka not release .deleted file. > > > Key: KAFKA-6689 > URL: https://issues.apache.org/jira/browse/KAFKA-6689 > Project: Kafka > Issue Type: Bug > Components: config, controller, log >Affects Versions: 0.10.1.1 > Environment: 3 Kafka Broker running on CentOS6.9(rungi3 VMs) >Reporter: A >Priority: Critical > Labels: newbie > Fix For: 0.10.1.1 > > > After Kafka cleaned log .timeindex / .index files based on topic > retention. I can > still lsof a lot of .index.deleted and .timeindex.deleted files. > We have 3 Brokers on 3 VMs , It's happen only 2 brokers. > [brokeer-01 ~]$ lsof -p 24324 | grep deleted | wc -l > 28 > [broker-02 ~]$ lsof -p 12131 | grep deleted | wc -l > 14599 > [broker-03 ~]$ lsof -p 3349 | grep deleted | wc -l > 14523 > > Configuration on 3 broker is same. (Rolllog every hour, Retention time 11 > Hours) > * INFO KafkaConfig values: > advertised.host.name = null > advertised.listeners = PLAINTEXT://Broker-02:9092 > advertised.port = null > authorizer.class.name = > auto.create.topics.enable = true > auto.leader.rebalance.enable = true > background.threads = 10 > broker.id = 2 > broker.id.generation.enable = true > broker.rack = null > compression.type = producer > connections.max.idle.ms = 60 > controlled.shutdown.enable = true > controlled.shutdown.max.retries = 3 > controlled.shutdown.retry.backoff.ms = 5000 > controller.socket.timeout.ms = 3 > default.replication.factor = 3 > delete.topic.enable = true > fetch.purgatory.purge.interval.requests = 1000 > group.max.session.timeout.ms = 30 > group.min.session.timeout.ms = 6000 > host.name = > inter.broker.protocol.version = 0.10.1-IV2 > leader.imbalance.check.interval.seconds = 300 > leader.imbalance.per.broker.percentage = 10 > listeners = null > log.cleaner.backoff.ms = 15000 > log.cleaner.dedupe.buffer.size = 134217728 > log.cleaner.delete.retention.ms = 8640 > log.cleaner.enable = true > log.cleaner.io.buffer.load.factor = 0.9 > log.cleaner.io.buffer.size = 524288 > log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 > log.cleaner.min.cleanable.ratio = 0.5 > log.cleaner.min.compaction.lag.ms = 0 > log.cleaner.threads = 1 > log.cleanup.policy = [delete] > log.dir = /tmp/kafka-logs > log.dirs = /data/appdata/kafka/data > log.flush.interval.messages = 9223372036854775807 > log.flush.interval.ms = null > log.flush.offset.checkpoint.interval.ms = 6 > log.flush.scheduler.interval.ms = 9223372036854775807 > log.index.interval.bytes = 4096 > log.index.size.max.bytes = 10485760 > log.message.format.version = 0.10.1-IV2 > log.message.timestamp.difference.max.ms = 9223372036854775807 > log.message.timestamp.type = CreateTime > log.preallocate = false > log.retention.bytes = -1 > log.retention.check.interval.ms = 30 > log.retention.hours = 11 > log.retention.minutes = 660 > log.retention.ms = 3960 > log.roll.hours = 1 > log.roll.jitter.hours = 0 > log.roll.jitter.ms = null > log.roll.ms = null > log.segment.bytes = 1073741824 > log.segment.delete.delay.ms = 6 > max.connections.per.ip = 2147483647 > max.connections.per.ip.overrides = > message.max.bytes = 112 > metric.reporters = [] > metrics.num.samples = 2 > metrics.sample.window.ms = 3 > min.insync.replicas = 2 > num.io.threads = 16 > num.network.threads = 16 > num.partitions = 10 > num.recovery.threads.per.data.dir = 3 > num.replica.fetchers = 1 > offset.metadata.max.bytes = 4096 > offsets.commit.required.acks = -1 > offsets.commit.timeout.ms = 5000 > offsets.load.buffer.size = 5242880 > offsets.retention.check.interval.ms = 60 > offsets.retention.minutes = 1440 > offsets.topic.compression.codec = 0 > offsets.topic.num.partitions = 50 > offsets.topic.replication.factor = 3 > offsets.topic.segment.bytes = 104857600 > port = 9092 > principal.builder.class = class > org.apache.kafka.common.security.auth.DefaultPrincipalBuilder > producer.purgatory.purge.interval.requests = 1000 > queued.max.requests = 3 > quota.consumer.default = 9223372036854775807 > quota.producer.default = 9223372036854775807 > quota.window.num = 11 > quota.window.size.seconds = 1 > replica.fetch.backoff.ms = 1000 > replica.fetch.max.bytes = 1048576 > replica.fetch.min.bytes = 1 > replica.fetch.response.max.bytes = 10485760 > replica.fetch.wait.max.ms = 500 >
