Build failed in Jenkins: kafka-2.5-jdk8 #67

See Changes: [wangguoz] HOTFIX: do not rely on file modified time in StateDirectoryTest -- [...truncated 1.55 MB...] kafka.tools.ConsoleProducerTest > testParseKeyProp

Re: Jira Cleanup

FYI: just closed those five tickets On Wed, 11 Mar 2020 at 22:09, Sönke Liebau wrote: > All, > > I left a few comments on some old but still open jiras in an attempt to > clean up a little bit. > > Since probably no one would notice these comments I thought I'd quickly > list them here to give

Build failed in Jenkins: kafka-trunk-jdk8 #4337

See Changes: [github] KAFKA-5604: Remove the redundant TODO marker on the Streams side (#8313) -- [...truncated 1.73 MB...] unit.kafka.cluster.AssignmentStateTest >

Re: Need for histogram type for Kafka Connect JMX Metrics

Hello, Kanupriya Recently, I’ve implemented a histogram metric [1] in Apache Ignite. My implementation can work on a stream load. It just counts events that is fall into the predefined intervals(buckets). I think histogram can be useful to measure consumer lag or similar metrics. So, If the

[jira] [Resolved] (KAFKA-1231) Support partition shrink (delete partition)

[ https://issues.apache.org/jira/browse/KAFKA-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sönke Liebau resolved KAFKA-1231. - Resolution: Won't Fix Closed due to doubts regarding the feasibility of this ticket and long

[jira] [Resolved] (KAFKA-1518) KafkaMetricsReporter prevents Kafka from starting if the custom reporter throws an exception

[ https://issues.apache.org/jira/browse/KAFKA-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sönke Liebau resolved KAFKA-1518. - Resolution: Not A Bug Closed as this seems to be working as designed and long inactivity on

Kafka JMX monitoring

Hi, I want to know what the best practice to collect Kafka JMX metrics is. I haven't found a decent way to collect and parse JMX in Java (because it is too much) and I learn that there are tools like tools like jmxtrans to do this. I wonder if there is more. Thanks. Regards.

Re: Kafka JMX monitoring

Hi, You could start by looking at Jolokia or Prometheus JMX Exporter depending on your needs. Jorge. On Thu, Mar 19, 2020 at 7:54 AM 张祥 wrote: > Hi, > > I want to know what the best practice to collect Kafka JMX metrics is. I

[jira] [Resolved] (KAFKA-1265) SBT and Gradle create jars without expected Maven files

[ https://issues.apache.org/jira/browse/KAFKA-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sönke Liebau resolved KAFKA-1265. - Resolution: Not A Problem Closed as this seems to be covered by current functionality. Also long

[jira] [Resolved] (KAFKA-1336) Create partition compaction analyzer

[ https://issues.apache.org/jira/browse/KAFKA-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sönke Liebau resolved KAFKA-1336. - Resolution: Won't Fix Closed due to long inactivity and no comment upon inquiry. > Create

[jira] [Resolved] (KAFKA-1440) Per-request tracing

[ https://issues.apache.org/jira/browse/KAFKA-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sönke Liebau resolved KAFKA-1440. - Resolution: Won't Fix Closed due to long inactivity and no comment upon inquiry. > Per-request

Jenkins build is back to normal : kafka-trunk-jdk8 #4338

See

[jira] [Created] (KAFKA-9734) Streams IllegalStateException in trunk during rebalance

John Roesler created KAFKA-9734: --- Summary: Streams IllegalStateException in trunk during rebalance Key: KAFKA-9734 URL: https://issues.apache.org/jira/browse/KAFKA-9734 Project: Kafka Issue

[jira] [Resolved] (KAFKA-9722) Kafka stops consuming messages after error

[ https://issues.apache.org/jira/browse/KAFKA-9722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Loginov resolved KAFKA-9722. -- Resolution: Invalid > Kafka stops consuming messages after error >

Request to review the pull request

Hi devs, I created a ticket on Jira and a pull request. https://issues.apache.org/jira/browse/KAFKA-9642 What is the next step? Who can review this? Best regards Soontaek Lim

Re: Kafka JMX monitoring

If you are not aware of Grafana and Prometheus then... You can enable JMx_PORT on kafka broker. Install and run kafka manager. https://github.com/lemastero/kafka-manager There you can see metrics on kafka manager UI. But yes, grafana-prometheus is Better one. On Thu, 19 Mar 2020 at 12:25 PM,

[jira] [Resolved] (KAFKA-9677) Low consume bandwidth quota may cause consumer not being able to fetch data

[ https://issues.apache.org/jira/browse/KAFKA-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-9677. Fix Version/s: 2.6.0 Resolution: Fixed > Low consume bandwidth quota may cause consumer not

[jira] [Resolved] (KAFKA-9723) Consumers ignore fetched records "since it no longer has valid position"

[ https://issues.apache.org/jira/browse/KAFKA-9723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-9723. Resolution: Duplicate Duplicate of KAFKA-9724 > Consumers ignore fetched records "since it no

Re: Regarding segment size config

Hopefully this will help explain the reason behind the 2g limit https://issues.apache.org/jira/browse/KAFKA-1670?focusedCommentId=14161185=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14161185 On Wed, Mar 18, 2020 at 2:10 AM 张祥 wrote: > Hi community, > > I

[jira] [Created] (KAFKA-9735) Kafka Connect REST endpoint doesn't consider keystore/truststore

SledgeHammer created KAFKA-9735: --- Summary: Kafka Connect REST endpoint doesn't consider keystore/truststore Key: KAFKA-9735 URL: https://issues.apache.org/jira/browse/KAFKA-9735 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk11 #1260

See Changes: [github] KAFKA-8618: Replace Txn marker with automated protocol (#7039) [github] MINOR: return unmodifiableMap for PartitionStates.partitionStateMap. [github] KAFKA-9435: DescribeLogDirs

Build failed in Jenkins: kafka-trunk-jdk11 #1261

See Changes: [github] MINOR: Fix javadoc warning in StreamsMetric (#8314) -- [...truncated 2.93 MB...] org.apache.kafka.streams.internals.WindowStoreFacadeTest >

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Thank you for the comments Guozhang. I’ll leave this KIP out for discussion till the end of the week and then start a vote for this early next week. Sanjana On Mar 18, 2020, 3:38 PM -0700, Guozhang Wang , wrote: > Hello Sanjana, > > Thanks for the proposed KIP, I think that makes a lot of

[jira] [Resolved] (KAFKA-9441) Refactor commit logic

[ https://issues.apache.org/jira/browse/KAFKA-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-9441. Fix Version/s: 2.6.0 Resolution: Fixed > Refactor commit logic >

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Hi Sanjana, The KIP looks good to me. I had just one question about the default behavior. As I understand, if the user has specified `retry.backoff.ms` explicitly, then we will not apply the default max backoff. As such, there's no way to get the benefit of this feature if you are providing a `

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Hey Guozhang, I was referring to this: > For users who have not set retry.backoff.ms explicitly, the default behavior will change so that the backoff will grow up to 1000 ms. For users who have set retry.backoff.ms explicitly, the behavior will remain the same as they could have specific

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Ah yes that makes sense. I’ll update the KIP to reflect this. Thanks, Sanjana On Thu, Mar 19, 2020 at 5:48 PM Guozhang Wang wrote: > Following the formula you have in the KIP, if it is simply: > > MIN(retry.backoff.max.ms, (retry.backoff.ms * 2**(failures - 1)) * random( > 0.8, 1.2)) > > then

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Hey Jason and Guozhang, Jason is right, I took this inspiration from KIP-144 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-144%3A+Exponential+backoff+for+broker+reconnect+attempts) which had the same logic in order to preserve the existing behavior. In this case however, if we are

Jenkins build is back to normal : kafka-trunk-jdk8 #4342

See

Re: Regarding segment size config

Thanks, it does make sense. Shailesh Panwar 于2020年3月20日周五 上午12:38写道： > Hopefully this will help explain the reason behind the 2g limit > >

[jira] [Created] (KAFKA-9736) Use internal consumer config to fail for unsupported protocol

Matthias J. Sax created KAFKA-9736: -- Summary: Use internal consumer config to fail for unsupported protocol Key: KAFKA-9736 URL: https://issues.apache.org/jira/browse/KAFKA-9736 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk11 #1262

See Changes: [github] KAFKA-6145: Add new assignment configs [github] KAFKA-9441: Unify committing within TaskManager (#8218) -- [...truncated 2.94 MB...]

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Hey Jason, My understanding is a bit different here: even if user has an explicit overridden "retry.backoff.ms", the exponential mechanism still triggers and the backoff would be increased till "retry.backoff.max.ms"; and if the specified "retry.backoff.ms" is already larger than the "

Build failed in Jenkins: kafka-trunk-jdk8 #4341

See Changes: [github] MINOR: Fix generic types in StreamsBuilder and Topology (#8273) -- [...truncated 2.92 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Thanks Sanjana, I did not capture the part that Jason referred to, so that's my bad :P Regarding your last statement, I actually feel that instead of take the larger of the two, we should respect "retry.backoff.max.ms" even if it is smaller than "retry.backoff.ms". I do not have a very strong

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

If that’s the case then what should we base the starting point as? Currently in the KIP the starting point is retry.backoff.ms and it exponentially goes up to retry.backoff.max.ms. If retry.backoff.max.ms is smaller than retry.backoff.ms then that could pose a bit of a problem there right? On

Re: [DISCUSS] KIP-577: Allow HTTP Response Headers Configured for Kafka Connect

1. Updated KIP adding prefix response.http.headers.{name}. 2. Updated KIP using a table to describe type and validate values. 3. Added meaningful examples for Connect scenarios. 4. We use comma to separate different HTTP headers. But we do allow the header value contain comma, just need use

[jira] [Created] (KAFKA-9737) Describing log dir reassignment times out if broker is offline

Jason Gustafson created KAFKA-9737: -- Summary: Describing log dir reassignment times out if broker is offline Key: KAFKA-9737 URL: https://issues.apache.org/jira/browse/KAFKA-9737 Project: Kafka

JMX expired topic metircs

Hi, I notice that there are jmx metrics for deleted topics when using java code and jmxterm. Has anyone also run into this ? If yes, what is the reason behind this and how can I filter expired metrics ? Thanks.

[jira] [Resolved] (KAFKA-9721) ReassignPartitionsCommand#reassignPartitions doesn't wait for the completion of altering replica folder

[ https://issues.apache.org/jira/browse/KAFKA-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-9721. --- Resolution: Won't Fix ReassignPartitionsCommand does not guarantee to "complete" the replica

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

Following the formula you have in the KIP, if it is simply: MIN(retry.backoff.max.ms, (retry.backoff.ms * 2**(failures - 1)) * random( 0.8, 1.2)) then the behavior would stay consistent at retry.backoff.max.ms. Guozhang On Thu, Mar 19, 2020 at 5:46 PM Sanjana Kaundinya wrote: > If that’s

Jenkins build is back to normal : kafka-trunk-jdk11 #1263

See

Re: Need for histogram type for Kafka Connect JMX Metrics

>So, If the community is interested we can have the same approach in Kafka. +1 On Thu, Mar 19, 2020 at 12:40 PM Nikolay Izhikov wrote: > Hello, Kanupriya > > Recently, I’ve implemented a histogram metric [1] in Apache Ignite. > My implementation can work on a stream load. > > It just counts