Kafka JMX monitoring

2020-03-18 Thread 张祥
Hi, I want to know what the best practice to collect Kafka JMX metrics is. I haven't found a decent way to collect and parse JMX in Java (because it is too much) and I learn that there are tools like tools like jmxtrans to do this. I wonder if there is more. Thanks. Regards.

Build failed in Jenkins: kafka-2.5-jdk8 #67

2020-03-18 Thread Apache Jenkins Server
See Changes: [wangguoz] HOTFIX: do not rely on file modified time in StateDirectoryTest -- [...truncated 1.55 MB...] kafka.tools.ConsoleProducerTest > testParseKeyProp START

Re: 回复:回复:回复:[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-03-18 Thread Boyang Chen
Thanks for the insight Feyman. I personally feel adding another admin client command is redundant, so picking option 1). The memberInfos struct is internal and just used for result reference purposes. I think it could still work even we overload with `removeAll` option, if that makes sense. Boyang

Build failed in Jenkins: kafka-trunk-jdk11 #1259

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] KAFKA-9568: enforce rebalance if client endpoint has changed (#8299) -- [...truncated 2.93 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

2020-03-18 Thread Chia-Ping Tsai
hi Cheng seems the "Next KIP Number" is not synced :( I have updated my KIP number and corrected the "Next KIP Number" thanks for the reminder. On 2020/03/19 02:32:55, "Cheng Pan" <379377...@qq.com> wrote: > Hi Chia-Ping, > > > I'm afraid the KIP number is conflict. I created (KIP-581: Value

回复:回复:回复:回复:[Vote] KIP-571: Add option to force remove members in StreamsResetter

2020-03-18 Thread feyman2009
Hi, team Before going too far on the KIP update, I would like to hear your opinions about how we would change the interface of AdminClient, the two alternatives I could think of are: 1) Extend adminClient.removeMembersFromConsumerGroup to support remove all As Guochang suggested,

Build failed in Jenkins: kafka-trunk-jdk11 #1258

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] KAFKA-5604: Remove the redundant TODO marker on the Streams side (#8313) -- [...truncated 5.92 MB...] org.apache.kafka.streams.TopologyTestDriver

Re: [DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

2020-03-18 Thread Cheng Pan
Hi Chia-Ping, I'm afraid the KIP number is conflict. I created (KIP-581: Value of optional null field which has default value) yesterday early, and insert it into table [KIPs under discussion]. https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+d

[jira] [Resolved] (KAFKA-9636) Simple join of two KTables fails

2020-03-18 Thread John Roesler (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-9636. - Resolution: Not A Bug Closing this ticket as "not a bug" because the test failure turned out to b

Build failed in Jenkins: kafka-trunk-jdk8 #4336

2020-03-18 Thread Apache Jenkins Server
See Changes: [wangguoz] HOTFIX: do not depend on file modified time in StateDirectoryTest [github] KAFKA-9047; AdminClient group operations should respect retries and [github] KAFKA-9656; Return COORDINATOR_NOT

Build failed in Jenkins: kafka-trunk-jdk11 #1257

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] KAFKA-9047; AdminClient group operations should respect retries and [github] KAFKA-9656; Return COORDINATOR_NOT_AVAILABLE for older producer clients -

Re: Add a customized logo for Kafka Streams

2020-03-18 Thread Guozhang Wang
I like the drawings too :) On Fri, Mar 13, 2020 at 3:35 AM Patrik Kleindl wrote: > Great idea, I would definitely buy the book with that on the cover :-) > best regards > Patrik > > On Fri, 13 Mar 2020 at 09:57, Becket Qin wrote: > > > I also like this one! > > > > Jiangjie (Becket) Qin > > > >

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

2020-03-18 Thread Guozhang Wang
Hello Sanjana, Thanks for the proposed KIP, I think that makes a lot of sense -- as you mentioned in the motivation, we've indeed seen many issues with regard to the frequent retries, with bounded exponential backoff in the scenario where there's a long connectivity issue we would effectively redu

[jira] [Resolved] (KAFKA-5604) All producer methods should raise `ProducerFencedException` after the first time.

2020-03-18 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5604. -- Fix Version/s: 2.3.0 Assignee: Guozhang Wang (was: Apurva Mehta) Resolution: Fi

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

2020-03-18 Thread Colin McCabe
+1 (binding). Thanks, Aneel. best, Colin On Wed, Mar 18, 2020, at 00:04, David Jacot wrote: > +1 (non-binding), thanks for the KIP! > > David > > On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth wrote: > > > Hello all, > > > > Thanks to the folks who have given feedback. I've incorporated the

Build failed in Jenkins: kafka-trunk-jdk11 #1256

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] MINOR: Update Scala to 2.12.11 (#8308) [wangguoz] HOTFIX: do not depend on file modified time in StateDirectoryTest -- [...truncated 2.92 MB...]

Build failed in Jenkins: kafka-trunk-jdk8 #4335

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] MINOR: Update Scala to 2.12.11 (#8308) -- [...truncated 2.91 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldNotUpdateStoreForSmall

[Discuss] KIP-582 Add a "continue" option for Kafka Connect error handling

2020-03-18 Thread Zihan Li
Hi all, I'd like to use this thread to discuss KIP-582 Add a "continue" option for Kafka Connect error handling, please see detail at: https://cwiki.apache.org/confluence/x/XRvcC Best, Zihan Li

[jira] [Resolved] (KAFKA-9656) TxnOffsetCommit should not return COORDINATOR_LOADING error for old request versions

2020-03-18 Thread Jason Gustafson (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9656. Fix Version/s: 2.6.0 Resolution: Fixed > TxnOffsetCommit should not return COORDINAT

[jira] [Reopened] (KAFKA-8122) Flaky Test EosIntegrationTest#shouldNotViolateEosIfOneTaskFailsWithState

2020-03-18 Thread Boyang Chen (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen reopened KAFKA-8122: Assignee: (was: Matthias J. Sax) > Flaky Test EosIntegrationTest#shouldNotViolateEosIfOneTask

Jenkins build is back to normal : kafka-trunk-jdk11 #1255

2020-03-18 Thread Apache Jenkins Server
See

Build failed in Jenkins: kafka-trunk-jdk8 #4334

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] KAFKA-9404: Use ArrayList instead of LinkedList in Sensor (#7936) -- [...truncated 5.88 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryT

Re: [DISCUSS] KIP-577: Allow HTTP Response Headers Configured for Kafka Connect

2020-03-18 Thread Randall Hauch
Thanks for the proposal, Jeff. I can see how this proposal will add value. I have a few comments, most of which are asking for more detail in the KIP. 1. I do think the KIP needs to be much more explicit about what the new configuration properties will be and to use them consistently. For e

[DISCUSS] KIP-581: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same broker

2020-03-18 Thread Chia-Ping Tsai
hi this ticket is about to records partition metrics rather than topic metrics. It helps us to observe more precis metrics for specify partition. The downside is that broker needs more space to keep metrics and the origin metrics interface (tags) is broken since this ticket adds new tag "partit

[jira] [Created] (KAFKA-9733) Consider addition to Kafka's replication model

2020-03-18 Thread Richard Yu (Jira)
Richard Yu created KAFKA-9733: - Summary: Consider addition to Kafka's replication model Key: KAFKA-9733 URL: https://issues.apache.org/jira/browse/KAFKA-9733 Project: Kafka Issue Type: New Featur

[jira] [Created] (KAFKA-9732) Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce

2020-03-18 Thread Adam Bellemare (Jira)
Adam Bellemare created KAFKA-9732: - Summary: Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce Key: KAFKA-9732 URL: https://issues.apache.org/jira/br

[jira] [Resolved] (KAFKA-9404) Use ArrayList instead of LinkedList in Sensor Class

2020-03-18 Thread Ismael Juma (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-9404. Fix Version/s: 2.6.0 Resolution: Fixed > Use ArrayList instead of LinkedList in Sensor Class

Conditionally applying SMTs

2020-03-18 Thread Tom Bentley
Hi, I opened https://issues.apache.org/jira/browse/KAFKA-9673 a week or so ago sketching an idea for conditionally applying SMTs, which improve the experience for users of connectors, such as Debezium, which can send records to multiple different topics where SMTs need to be applied to for some to

Jenkins build is back to normal : kafka-2.5-jdk8 #66

2020-03-18 Thread Apache Jenkins Server
See

Need for histogram type for Kafka Connect JMX Metrics

2020-03-18 Thread Kanupriya Batra
Hi, We are using kafka-connect framework in our use-case, and would also like to track metrics for connectors/task. But currently Kafka Connect JMX metrics are not supporting histogram types due to which we are not able to plot percentiles on a lot of important metrics like : kafka_connect_si

Build failed in Jenkins: kafka-trunk-jdk11 #1254

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] KAFKA-9625: Fix altering and describing dynamic broker configurations -- [...truncated 2.93 MB...] org.apache.kafka.streams.TopologyTestDriverTest

Regarding segment size config

2020-03-18 Thread 张祥
Hi community, I understand that there are two configs regarding segment file size, log.segment.bytes for broker and segment.bytes for topic. The default values are both 1G and they are required to be an integer so they cannot be larger than 2G. My question is, assuming I am not making any mistakes

Build failed in Jenkins: kafka-trunk-jdk8 #4333

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] HOTFIX: fix flaky [github] MINOR: clean up required setup for StreamsPartitionAssignorTest (#8306) [github] KAFKA-9625: Fix altering and describing dynamic broker configurations

Build failed in Jenkins: kafka-trunk-jdk11 #1253

2020-03-18 Thread Apache Jenkins Server
See Changes: [github] HOTFIX: fix flaky [github] MINOR: clean up required setup for StreamsPartitionAssignorTest (#8306) -- [...truncated 2.92 MB...] org.apache.kafka.s

Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients

2020-03-18 Thread Sanjana Kaundinya
Thanks for the feedback Boyang. If there’s anyone else who has feedback regarding this KIP, would really appreciate it hearing it! Thanks, Sanjana On Tue, Mar 17, 2020 at 11:38 PM Boyang Chen wrote: > Sounds great! > > Get Outlook for iOS > __

Re: [VOTE] KIP-574: CLI Dynamic Configuration with file input

2020-03-18 Thread David Jacot
+1 (non-binding), thanks for the KIP! David On Mon, Mar 16, 2020 at 4:06 PM Aneel Nazareth wrote: > Hello all, > > Thanks to the folks who have given feedback. I've incorporated the > suggestions, and think that this is now ready for a vote: > > > https://cwiki.apache.org/confluence/display/KAF

[Discuss] KIP-581: Value of optional null field which has default value

2020-03-18 Thread Cheng Pan
Hi all, I'd like to use this thread to discuss KIP-581: Value of optional null field which has default value, please see detail at: https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value There are some previous discussion at: https://g