is it possbile to get the last commit time for certain consumer group to target topic on v0.10.1.1

when i using kafka 0.8.2.2, I can use the mtime of consumer group's offset node in zk to detect the last commit time. After i upgrade to kafka 0.10.1.1, the consumer group's offset info is no long stored in zk. how can i find the last commit time for certain consumer group to some topic, thanks

Re: [VOTE] 1.0.0 RC4

+1. Hope this is the last one. On Fri, Oct 27, 2017 at 10:28 AM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the fifth candidate for release of Apache Kafka 1.0.0. The main PRs > that gets merged in after RC3 are the following: > >

Listing Consumer Groups Per Topic

Hi All! I am looking for a way to get a list of all the consumer groups that have offsets stored for a particular topic. I was thinking of making a request to describe all consumer groups, then filtering for the groups with offsets in the topic I care about. I noticed, however, that while

Re: How to choose which topic to replicate in MirrorMaker?

Thank you for helping! Best regards, Alexandr On Mon, Oct 30, 2017 at 7:21 PM, Sam Pegler wrote: > Use a whitelist with one topic. > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330# > Kafkamirroring(MirrorMaker)-Whitelistorblacklist >

Re: Offsets in Kafka producer start with -1 for new topic

No exceptions or dropped messages. I injected/sent 100 messages and could read back all 100 of them. Its just that the offset numbers are not as expected for the producer. On 10/30/17, 2:28 AM, "Manikumar" wrote: Any exception in the callback exception field?

Re: Kafka Connect

Have you tried the HDFS connector here? https://github.com/confluentinc/kafka-connect-hdfs Its master and 4.0.x branches contain export support for JSON in '.json' text files. Konstantine. On Sat, Oct 28, 2017 at 11:51 PM, Alexander Atanasov < alexandaratana...@gmail.com> wrote: > Hello, > >

Re: How to choose which topic to replicate in MirrorMaker?

Use a whitelist with one topic. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330#Kafkamirroring(MirrorMaker)-Whitelistorblacklist __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 3-7 Herbal Hill / London / EC1R 5EJ

Re: Streams changelog topic retention is high

The retention for the joins is as specified above. With until set to 240 and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG set to 1 day (the default) this would be 10080 For plain key value stores, there should be no retention period as the topics are compacted only.

Re: Best practices - Using kafka (with http server) as source-of-truth

can you share your results ?

How to choose which topic to replicate in MirrorMaker?

Hello, Is it possible to choose which topic to replicate in MirrorMaker (to not mirror the whole cluster but the part of it)? Is it possible to choose a partition in a topic to replicate? Best regards, Alexandr

Re: Streams changelog topic retention is high

Actually I am using Key Value store, I do use join as part of my DAG(until for the same has been set at 240 mins). The sink processor is key-value, is there any option to control it. -Sameer. On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy wrote: > The topics in question are

Re: Streams changelog topic retention is high

The topics in question are both changelogs for window stores. The retention period for them is calculated as the Window retention period, which is the value that is passed to `JoinWindows.until(...)` (default is 1 day) plus the value of the config

Streams changelog topic retention is high

Hi, I have configured my settings to be the following:- log.retention.hours=3 delete.topic.enable=true delete.retention.ms=1080 min.cleanable.dirty.ratio=0.20 segment.ms=18 Howsoever, the changelog topic created as part of stream has the rentention.ms to be 10080, the source topic

Re: Experimenting with Kafka and OpenSSL

I haven't yet had a chance to try out Java 9, but that's definitely on my TODO list, maybe sometime this weekend. Thanks for pointing me to KAFKA-2561. I had missed that. -Jaikiran On 30/10/17 4:17 PM, Mickael Maison wrote: Thanks for sharing, very interesting read. Did you get a chance to

Mirror partitions from several clusters into one

Hello, I have a next use case: I need to mirror some partitions in some topics at some cluster to totally different cluster into some topic into some partition. Is it possible to organize with MirrorMaker tool or any other tools? Here is an example: I have 3 data centers: DC1, DC2, DC3 Each data

Re: Experimenting with Kafka and OpenSSL

Thanks for sharing, very interesting read. Did you get a chance to try JDK 9 ? We also considered using OpenSSL instead of JSSE especially since Netty made an easy to re-use package (netty-tcnative). There was KAFKA-2561 (https://issues.apache.org/jira/browse/KAFKA-2561) where people shared a

Re: Servers Getting disconnected

Please take a look at KAFKA-4477 On Mon, Oct 30, 2017 at 2:06 AM, Yuanjia wrote: > Hi, > My cluster version is 0.10.0.0, 10 nodes. One server has many error > log, like that > java.io.IOException: Connection to 6 was disconnected before the response > was read >

Servers Getting disconnected

Hi, My cluster version is 0.10.0.0, 10 nodes. One server has many error log, like that java.io.IOException: Connection to 6 was disconnected before the response was read at

Experimenting with Kafka and OpenSSL

We have been using Kafka in some of our projects for the past couple of years. Our experience with Kafka and SSL had shown some performance issues when we had seriously tested it (which admittedly was around a year back). Our basic tests did show that things had improved over time with newer

Re: Offsets in Kafka producer start with -1 for new topic

Any exception in the callback exception field? may be you can enable client debug logs to check any errors. On Mon, Oct 30, 2017 at 7:25 AM, Thakrar, Jayesh < jthak...@conversantmedia.com> wrote: > I created a new Kafka topic with 1 partition and then sent 10 messages > using the KafkaProducer