Re: Kafka Streams : CommitFailedException

Hi Guozang, Thanks for looking into this. I was using 0.11.0.0 version of the library earlier when I was getting the CommitFailed exception and the tasks were terminating. The application config then was Replication Factor = 2, Num Stream Threads = 1, Consumer Max Poll Records = 1000 & Consumer

JOB | Permanent Sysadmin (Amsterdam, Netherlands OR remote)

Hello, I'm working with an employer that is looking to hire a permanent sysadmin to help maintain their Hadoop and Kafka clusters at their Amsterdam, Netherlands office OR remote. Consequently I had hoped that some members of this mailing list may like to discuss further off-list using

java.lang.OutOfMemoryError memory leak on 1.0.0 with 0.11.0.1 on disk and converting to 0.9 clients

https://issues.apache.org/jira/browse/KAFKA-6185 This was also reported by another person, Ben, attached to the end of this ticket https://issues.apache.org/jira/browse/KAFKA-6042 which is a broker lockup/FD issue, but a new ticket was requested. It was also suggested by Ismael Juma there to

Re: [ANNOUNCE] New committer: Onur Karaman

Congratulations Onur! On Tue, 7 Nov 2017 at 21:44 Jun Rao wrote: > Affan, > > All known problems in the controller are described in the doc linked from > https://issues.apache.org/jira/browse/KAFKA-5027. > > Thanks, > > Jun > > On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed

Re: GlobalKTable never finishes restoring

Did you delete and recreate the topic of the GlobalStore? I did have a look into the code, too, and think there is a bug in `GlobalStateManagerImpl#restoreState()`. I did some initial investigation using an existing test, and the test passed without transactional data but fails if the global

Re: Kafka Streams : CommitFailedException

Hello Tony, You mentioned in 0.11.0.0 the props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1) while in 0.11.0.1 props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2). But from your logs it seems you set this config as 2 in both versions. Right? Anyways, I took a look into your logs and I think

Using Kafka to access APIs.

Hello, I want to build a project that accesses the Gdax websocket in a real time stream, and push that data into Kafka to reformat it and prep it for a ML model. *Can I use Kafka to access GDAX's API and pull data from it in realtime?* or *Should I pull data from their API to a text file in

GlobalKTable never finishes restoring

Hi All, I have a simple kafka streams app that seems to hang when restoring state for a GlobalKTable. We're running in a test environment at the moment and the topic it is loading from only has two messages in it, I don't know if the very low volume of messages would affect the restore? I've

Re: GlobalKTable never finishes restoring

Alex: In the future, please use pastebin if the log is not too large. When people find this thread in mailing list archive, the attachment wouldn't be there. Thanks On Tue, Nov 7, 2017 at 8:32 AM, Matthias J. Sax wrote: > Alex, > > I am not sure, but maybe it's a bug. I

Re: clean leader election on kafka 0.10.2.1

Henry, I am not sure what do you mean by "waits for the leader of partition to start up"? Leader election should not affect leader - follower starting up process. Guozhang On Thu, Nov 2, 2017 at 4:56 PM, Henry Cai wrote: > We were on kafka 0.10.2.1. We tried to

Re: GlobalKTable never finishes restoring

Previously deleting and recreating the topic has solved the problem. Based on what you've said about the offset correction I did a quick test by building kafka streams myself with the following code in `GlobalStateManagerImpl#restoreState()` while (offset < highWatermark) { final

Kafka connect python api

Hi Is there a python API for the source and sink connect implementation? My source is - Filesystem Sink - HDFS Thanks -- Swaapnika

Kafka Start Up and Shutdown

Hi All, I am trying to understand different configurations to apply on my cluster. I find the num.recovery.threads.per.data.dir=1,bit interesting. In most of configuration example I find in the google, the value set as 1. Everyone wants quick start and recovery, to achieve the same can I increase

Client addressable listener

I can't seem to get a listeners and advertised.listeners configuration for server properties figured out so I can connect remotely with my producer and consumers. If I set it like this... listeners=CLIENT://:9090,PLAINTEXT://:9092 advertised.listeners=CLIENT://:9090,PLAINTEXT://:9092 >From my

Re: [ANNOUNCE] New committer: Onur Karaman

Affan, All known problems in the controller are described in the doc linked from https://issues.apache.org/jira/browse/KAFKA-5027. Thanks, Jun On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed wrote: > Congrats Onur, > > Can you also share the document where all known problems are

Re: GlobalKTable never finishes restoring

Disabling transactions doesn't seem to have changed anything. I've had a read through the kafka streams source code, specifically the parts relating to the restoration of the global stores and I can't see anything obvious I should look at. @Ted will do, here's a pastebin of the most recent run

Re: GlobalKTable never finishes restoring

Alex, I am not sure, but maybe it's a bug. I noticed that you read transaction data. Can you try to write to the topic without using transactions and/or set the consumer into READ_UNCOMMITTED mode to verify? It only a guess that it might be related to transactions and it would be great to verify

Re: Offset commit for partitions not owned by consumer

Can you reveal code snippet for BufferedConsumerClientAdapterImpl ? I took a look at the logs. There was no log around 19:46:5x in either server or controller log. Thanks On Tue, Nov 7, 2017 at 8:35 AM, Mana M wrote: > Ted, did you get chance to look at the issue? I am

Re: Offset commit for partitions not owned by consumer

Ted, did you get chance to look at the issue? I am also planning to update to latest version to find out if we still see the same issue. On Fri, Nov 3, 2017 at 8:40 PM, Mana M wrote: > Below are the logs: > > Consumer 1 logs, where issue can be seen:

RE: [EXTERNAL]Cannot list topics

It's almost certainly a typo in your command lines. Not sure how to help without you posting them as requested. Also, post the console output from when you created the topic. Dale -Original Message- From: Donghun Kim [mailto:kimdho...@gmail.com] Sent: Monday, November 6, 2017 7:03 PM

RE: Public IP

Hello, Got a bit further. I registered a DNS using my router IP address and then I can connect to the Kafka server from remote at my house through the ADSL. So the "advertised.host.name" does not refer to an IP address but an actual domain host name (am I correct in saying this). Is there no