Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Syed Mudassir Ahmed
Sonke, This issue seems serious. Customers raised bug with our product. And I suspect the bug is in apache-kafka clients library. I executed the kafka reader without any snaplogic-specific code. There were hardly about twenty messages in the topics. The code consumed about 300MB of memory i

Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Sönke Liebau
Hi Syed, Apache Kafka is an open source software that comes as is without any support attached to it. It may well be that this is a bug in the Kafka client library, though tbh I doubt that from what my tests have shown and since I think someone else would have noticed this as well. Even if this is

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

2019-03-05 Thread Jan Filipiak
On 04.03.2019 19:14, Matthias J. Sax wrote: > Thanks Adam, > > *> Q) Range scans work with caching enabled, too. Thus, there is no > functional/correctness requirement to disable caching. I cannot > remember why Jan's proposal added this? It might be an > implementation detail though (maybe ju

Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Syed Mudassir Ahmed
Sonke, I am not blaming apache-kafka for the tickets raised by our customers. I am saying there could be an issue in kafka-clients library causing resource/memory leak. If that issue is resolved, I can resolve my tickets as well automatically. I don't find any issue with the snaplogic code.

Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Sönke Liebau
Hi Syed, the next step would be for someone else to be able to reproduce the issue. If you could give me the values for the input parameters that your code runs with (as per my last mail) then I am happy to run this again and take another look at memory consumption. I'll then upload the exact cod

Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Syed Mudassir Ahmed
I already shared the code snippet earlier. What do you mean by input params? I am just running the consumer in an infinite loop polling for new messages.Polling time is 100 millisecs. If the topic is empty, the memory consumption is gradually increasing over the time and reaching to about 4G

[jira] [Resolved] (KAFKA-7966) Flaky Test DynamicBrokerReconfigurationTest#testLogCleanerConfig

2019-03-05 Thread Rajini Sivaram (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-7966. --- Resolution: Duplicate > Flaky Test DynamicBrokerReconfigurationTest#testLogCleanerConfig > --

Re: [DISCUSS] KIP-436 Add a metric indicating start time

2019-03-05 Thread Stanislav Kozlovski
Thanks for the discussion, folks. I'll be starting a vote thread now. Dongjin, thanks for updating the link to the discussion thread. Best, Stanislav On Sun, Mar 3, 2019 at 1:30 PM Manikumar wrote: > Hi Stanislav, > > I too don't have strong preference between both the metrics. > Adding `start

[VOTE] KIP-436 Add a metric indicating start time

2019-03-05 Thread Stanislav Kozlovski
Hey everybody, I'd like to start a vote thread about the lightweight KIP-436 KIP: KIP-436 JIRA: KAFKA-7992 Pull Request: 6318

Re: Apache Kafka Memory Leakage???

2019-03-05 Thread Sönke Liebau
My questions about input parameters are mostly targeted at finding out whether your code enters the conditional block calling commit at the bottom of your code. Endless loop with 100ms poll duration is fine for everything else. Regarding the screenshot that seem to be a snaplogic gui again. Didn't

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

2019-03-05 Thread Viktor Somogyi-Vass
Hey George, Sorry for the delay. I'll answer point-by-point: 1.2: I think it's fine. As you say we presume that the client knows the state of the cluster before doing the reassignment, so we can presume the same for cancellation. 2.: One follow-up question: if the reassignment cancellation gets in

Fluent Kafka Streams Test + Schema Registry Mock

2019-03-05 Thread Arvid Heise
Dear Kafka devs, we recently published a fluent kafka streams test library. https://github.com/bakdata/fluent-kafka-streams-tests https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec We wanted to ask if there is any interest in us contributing the rather small wrapper around the Te

Fluent Kafka Streams Test + Schema Registry Mock

2019-03-05 Thread Arvid Heise
Dear Kafka devs, we recently published a fluent kafka streams test library. https://github.com/bakdata/fluent-kafka-streams-tests https://medium.com/bakdata/fluent-kafka-streams-tests-e641785171ec We wanted to ask if there is any interest in us contributing the rather small wrapper around the Te

[VOTE] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

2019-03-05 Thread Kevin Lu
Hi All, I would like to start the vote thread for KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option). https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089398 Thanks! Regards, Kevin

[DISCUSS] KIP-438: Expose task, connector IDs in Connect API

2019-03-05 Thread Ryanne Dolan
Hey y'all, please consider the following small KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-438%3A+Expose+task%2C+connector+IDs+in+Connect+API Thanks! Ryanne

[jira] [Created] (KAFKA-8042) Kafka Streams creates many segment stores on state restore

2019-03-05 Thread Adrian McCague (JIRA)
Adrian McCague created KAFKA-8042: - Summary: Kafka Streams creates many segment stores on state restore Key: KAFKA-8042 URL: https://issues.apache.org/jira/browse/KAFKA-8042 Project: Kafka Is

Re: [VOTE] KIP-427: Add AtMinIsr topic partition category (new metric & TopicCommand option)

2019-03-05 Thread Dong Lin
Hey Kevin, Thanks for the KIP! +1 (binding) Thanks, Dong On Tue, Mar 5, 2019 at 9:38 AM Kevin Lu wrote: > Hi All, > > I would like to start the vote thread for KIP-427: Add AtMinIsr topic > partition category (new metric & TopicCommand option). > > https://cwiki.apache.org/confluence/pages/vi

[jira] [Created] (KAFKA-8043) Host

2019-03-05 Thread Bolke de Bruin (JIRA)
Bolke de Bruin created KAFKA-8043: - Summary: Host Key: KAFKA-8043 URL: https://issues.apache.org/jira/browse/KAFKA-8043 Project: Kafka Issue Type: Bug Reporter: Bolke de Bruin

[jira] [Resolved] (KAFKA-8043) Host

2019-03-05 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved KAFKA-8043. --- Resolution: Incomplete > Host > > > Key: KAFKA-8043 > UR

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

2019-03-05 Thread Konstantine Karantasis
Hi Guozhang, thanks for your detailed comments! I thought it'd be a good idea to have some code supporting my replies. The PR for KIP-415 was opened this week. It needs some cleanup but I think it can give some good hints on the things you've touched upon so far. I'm answering to your four points

[jira] [Created] (KAFKA-8044) System Test Failure: ReassignPartitionsTest.test_reassign_partitions

2019-03-05 Thread Manikumar (JIRA)
Manikumar created KAFKA-8044: Summary: System Test Failure: ReassignPartitionsTest.test_reassign_partitions Key: KAFKA-8044 URL: https://issues.apache.org/jira/browse/KAFKA-8044 Project: Kafka I

Re: [VOTE] 2.2.0 RC1

2019-03-05 Thread Mickael Maison
+1 I checked signatures, built from source and ran through quickstart It's probably worth mentioning KIP-377 in the short release summary as in many cloud environments access to zookeeper is disabled so many users will benefit from it. Also "- Added SSL support for custom principle name" -> "- Add

Re: Fluent Kafka Streams Test + Schema Registry Mock

2019-03-05 Thread Guozhang Wang
Hello Arvid, Thanks for sharing! I've read your two blog posts and also the code repo and they are really interesting. Your observation is right that since schema registry is under Confluent schema registry the integration may not be contributed to AK. On the other hand, I think having this as pa

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

2019-03-05 Thread Matthias J. Sax
>> I dont know how to range scan over a caching store, probably one had >> to open 2 iterators and merge them. That happens automatically. If you query a cached KTable, it ranges over the cache and the underlying RocksDB and performs the merging under the hood. >> Other than that, I still think e

[jira] [Created] (KAFKA-8045) Possible Flaky Test LogOffsetTest.testGetOffsetsBeforeNow

2019-03-05 Thread Bill Bejeck (JIRA)
Bill Bejeck created KAFKA-8045: -- Summary: Possible Flaky Test LogOffsetTest.testGetOffsetsBeforeNow Key: KAFKA-8045 URL: https://issues.apache.org/jira/browse/KAFKA-8045 Project: Kafka Issue Typ

[jira] [Created] (KAFKA-8046) Shutdown broker because all log dirs in /tmp/kafka-logs have failed

2019-03-05 Thread jaren (JIRA)
jaren created KAFKA-8046: Summary: Shutdown broker because all log dirs in /tmp/kafka-logs have failed Key: KAFKA-8046 URL: https://issues.apache.org/jira/browse/KAFKA-8046 Project: Kafka Issue Type

Build failed in Jenkins: kafka-2.2-jdk8 #44

2019-03-05 Thread Apache Jenkins Server
See Changes: [jason] MINOR: Log exception thrown by consumer.poll() in VerifiableConsumer -- [...truncated 2.69 MB...] kafka.controller.PartitionStateMachineTest > testInvali

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

2019-03-05 Thread George Li
Hi Viktor, >  2.: One follow-up question: if the reassignment cancellation gets >interrupted and a failover happens after step #2 but before step #3, how will >the new controller continue? At this stage Zookeeper would contain OAR + RAR, >however the brokers will have the updated LeaderAndIsr

[jira] [Created] (KAFKA-8047) remove KafkaMbean when network close

2019-03-05 Thread limeng (JIRA)
limeng created KAFKA-8047: - Summary: remove KafkaMbean when network close Key: KAFKA-8047 URL: https://issues.apache.org/jira/browse/KAFKA-8047 Project: Kafka Issue Type: Bug Components: cl

[jira] [Created] (KAFKA-8048) remove KafkaMbean when network close

2019-03-05 Thread limeng (JIRA)
limeng created KAFKA-8048: - Summary: remove KafkaMbean when network close Key: KAFKA-8048 URL: https://issues.apache.org/jira/browse/KAFKA-8048 Project: Kafka Issue Type: Bug Components: cl

[jira] [Created] (KAFKA-8049) remove KafkaMbean when network close

2019-03-05 Thread limeng (JIRA)
limeng created KAFKA-8049: - Summary: remove KafkaMbean when network close Key: KAFKA-8049 URL: https://issues.apache.org/jira/browse/KAFKA-8049 Project: Kafka Issue Type: Bug Components: cl

[jira] [Created] (KAFKA-8051) remove KafkaMbean when network close

2019-03-05 Thread limeng (JIRA)
limeng created KAFKA-8051: - Summary: remove KafkaMbean when network close Key: KAFKA-8051 URL: https://issues.apache.org/jira/browse/KAFKA-8051 Project: Kafka Issue Type: Bug Components: cl

[jira] [Created] (KAFKA-8050) remove KafkaMbean when network close

2019-03-05 Thread limeng (JIRA)
limeng created KAFKA-8050: - Summary: remove KafkaMbean when network close Key: KAFKA-8050 URL: https://issues.apache.org/jira/browse/KAFKA-8050 Project: Kafka Issue Type: Bug Components: cl