Kafka AdminClient

Hi All, My requirement is to automate "Partition Rebalance" through java APIs, rather than .sh scripts. I am in assumption that Kafka AdminClient (0.11.0 v) have these features and hence kindly let me know if it is TRUE. Also, if AdminCLient API are document on net then kindly guide me to the l

Re: Infinite loop trying to connect to coordinator

Is it possible than a max.poll.records too high can cause this instability? On Wed, Jul 12, 2017 at 8:43 AM, Pierre Coquentin < pierre.coquen...@gmail.com> wrote: > It was on our test environment and nothing was running when the incident > occurred. > In the server log we have a bunch of > [2017-

Re: Windows OS platform support

Hi M.Manna, Thanks for the update. If possible, can you please share which admin I/O level functionality will impact? Do you mean something related to log files renaming? In documentation, there is no pointer with respect to I/O. Kindly advice. ThanksHarish On Saturday, July 15, 2017, 1:59:5

Re: Windows OS platform support

Hi, First I apologise and should just correct myself. I should use "Disk/File level operations" instead of using "I/O" since that refers to different things. What I mean to say that file deletion/renaming doesn't work even when you provide full permissions in Windows. But Unix/Linux based system

Re: Behavior when kafka producer is network partitioned

On Sun, Jul 16, 2017 at 8:32 PM, Abhishek Agarwal wrote: > What is the behaviour in 0.10.x if kafka producer is not able to reach any > of the broker for a brief duration? > > From what I understand from the code, all the nodes will be in > disconnected state. Since there is no node to query to,

Re: Kafka AdminClient

Hi Nitin, The AdminClient doesn't support partition reassignment yet. It is something we would like to support in the future. Ismael On Mon, Jul 17, 2017 at 12:16 AM, nitin sharma wrote: > Hi All, > > My requirement is to automate "Partition Rebalance" through java APIs, > rather than .sh scri

Re: Kafka AdminClient

thanks Ismael. However, i believe it does support few admin options at present. Can you kindly highlight options it supports. Regards, Nitin Kumar Sharma. On Mon, Jul 17, 2017 at 4:02 PM, Ismael Juma wrote: > Hi Nitin, > > The AdminClient doesn't support partition reassignment yet. It is some

Re: Kafka AdminClient

You can check the Javadoc: http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html Ismael On Mon, Jul 17, 2017 at 3:49 AM, nitin sharma wrote: > thanks Ismael. > > However, i believe it does support few admin options at present. Can you > kindly highligh

Re: Windows OS platform support

Hello M.Manna, Thanks for your quick and valuable response. Actually I am getting following error - Caused by: java.nio.file.FileSystemException: \tmp\kafka-logs\TopicName-2\7262.log-> \tmp\kafka-logs\TopicName -2\7262.log.deleted: Theprocess cannot access the f

Re: [kafka question] leader of partitions is none

Hi, I had similar problems where new topics were assigned none, or some cases -1 leader. In my case, was due to a very busy ZK cluster. Once I removed load in the ZK cluster, I've restarted (rolling restart) kafka brokers and everything went back to normal. However I was worried about restarting th

Re: Do we have to query localWindowStore in same java instance we are creating the store

Hi Sachin, 1. You can run a remote query and we provide some example code (https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/ ) however by default A

Why Kafka treat the value of TopicCommand's topic parameter as regex expression?

Hi Team, In TopicCommand class, the description for "--topic" parameter is "The topic to be create, alter or describe. Can also accept a regular expression except for --create option". It means the value of topic parameter will be treated as regex expression in the logic, so multiple topics may b

Re: decreasing loglevel for basic creation-roll-delete events

ping? On 13/07/17 16:53, mosto...@gmail.com wrote: Hi While testing Kafka in our environment, we have noticed it creates A LOT of "debug" logs (not to be confused with topic logs!). eg: Jul 13 16:44:26 host kafka[28511]: [2017-07-13 16:44:26,879] INFO Deleting index /data/kafka-1/group

Re: Logs truncated at o'clock

ping? On 13/07/17 17:09, mosto...@gmail.com wrote: Hi With swiss precission, our kafka test environment seems to truncate topics at o'clock hours. This might be confirmed with the following trace, which states "Truncating log ... to offset 0" We are still using Kafka 0.10.2.1, but I was

Re: Why Kafka treat the value of TopicCommand's topic parameter as regex expression?

Sorry, typos. "test.a" and "test-a" On Mon, Jul 17, 2017 at 9:20 PM, Shuai Zhang wrote: > Hi Team, > > In TopicCommand class, the description for "--topic" parameter is "The > topic to be create, alter or describe. Can also accept a regular expression > except for --create option". > > It means

答复: [kafka question] leader of partitions is none

Thank you my friend! You have helped a lot! 发送自 Windows 10 版邮件应用 发件人: Del Barrio, Alberto 发送时间: 2017年7月17日 20:42 收件人: users@kafka.apache.org 抄送: lichenglon...@outlook.com

Kafka issues during Developement

Hi, I am using kafka as the messaging queue for my application which will run on the windows platform. I am facing issues during the log file deletion with following erros. Also i found some disclaimer is available in the kafka documentation to use kafka in windows platform Caused by: java.nio.

Re: Windows OS platform support

Hello M.Manna, Do you mean we should not use Kafka in windows production until the disk I/O issues(For which you have given workaround) are resolved officilally? ThanksHarish On Monday, July 17, 2017, 5:31:11 PM GMT+5:30, M. Manna wrote: The workaround is to download the sources and create a

Re: Windows OS platform support

I have never said that that we shouldn't use Kafka officially for Windows Production Environment. I am running a major PoC which involves developing solution on Windows 10 - but my production docker containers are using Alpine Linux so I won't have any issues for the final integration pipeline. Wh

Re: Kafka issues during Developement

This is a know issue. You cannot delete shared (or opened files) on Windows, but UNIX/Linux is okay. Please read Kafka online documentation on WIndows usage. A relevant jira case is KAFKA-1194. On 17 July 2017 at 11:31, Dharma Raj wrote: > Hi, > > I am using kafka as the messaging queue for m

Re: Windows OS platform support

Hi KR, Thanks for clarifyication and thank you for spending your valuable time to assist on our issues. Our requirement is to have Kafka in Windows environment. The messages in Kafka gets pushed from external entities; we have consumers which consume those records and insert to mySQL database

Re: DAG processing in Kafka Streams

Sameer, Could you elaborate a bit more what do you mean by "DAG processing"? Guozhang On Sun, Jul 16, 2017 at 11:58 PM, Sameer Kumar wrote: > Currently, we don't have DAG processing in Kafka Streams. Having a DAG has > its own share of advantages in that, it can optimize code on its own and

Re: DAG processing in Kafka Streams

I think he means something like Akka Streams: http://doc.akka.io/docs/akka/2.5.2/java/stream/stream-graphs.html Directed Acyclic Graphs are trivial to construct in Akka Streams and use back-pressure to preclude memory issues. -David On 7/17/17, 12:20 PM, "Guozhang Wang" wrote: Sameer,

Re: DAG processing in Kafka Streams

On that note, akka streams has Kafka integration. We use it heavily and it is quite a bit more flexible than K-Streams (which we also use…but for simpler applications) Akka-streams-Kafka is particularly good for asynchronous processing: http://doc.akka.io/docs/akka-stream-kafka/current/home.ht

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Jason, Onur, thank you for reviewing the KIP. Regarding the default `--describe` option, so far there have been a few suggestions that conflict a bit. Here are the suggestions: - Keep the current behavior exactly as is (Edo, Jeff) - Remove members with no assignments from the current result set (

Re: Windows OS platform support

Okay and are you using Kafka Connect (with JDBC connectors) or using plain Producer Consumers (and your own library calls to perform this SQL operations) ? If you have a maintenance window (and some SLA) before which all the messages are capture in SQL db - you can go ahead and clean up the logs y

Avro Serialization & Schema Registry ..

Hi - I am using Avro Serialization in a Kafka Streams application through the following dependency .. "io.confluent" % "kafka-avro-serializer" % "3.2.2" My question is : Is schema registry mandatory for using Avro Serialization ? Because when I run the application I get the following exception

Re: Avro Serialization & Schema Registry ..

For your first question, no you can use the avro API. On Mon, Jul 17, 2017 at 2:29 PM Debasish Ghosh wrote: > Hi - > > I am using Avro Serialization in a Kafka Streams application through the > following dependency .. > > "io.confluent" % "kafka-avro-serializer" % "3.2.2" > > My question is :

Graceful shutdown discrepancy for 0.10.2.1 out-of-the-box

Hello, I recently started using the 0.10.2.1 distro on Ubuntu Linux and trying some basic startup for a 3 node cluster (using the setup mentioned in the documentation). The following is a custom startup that I have written (I have also tried this by calling them individually in the same order):

Re: Windows OS platform support

Thanks KR. We don't use Kafka connect( I have seen this getting used in in ETL CDC capture such as Maxwell CDC, debezium CDC), we have our own consumers and own logic to place the data into SQL. As you suggested, we will do some basic analysis by running Kafka for longer duration continuously a

Re: Windows OS platform support

No problems. Feel free to come back and share your results with us :). On 17 July 2017 at 19:37, harish jadhav wrote: > Thanks KR. We don't use Kafka connect( I have seen this getting used in in > ETL CDC capture such as Maxwell CDC, debezium CDC), we have our own > consumers and own logic to pl

Re: Avro Serialization & Schema Registry ..

I am using the class io.confluent.kafka.serializers.KafkaAvroSerializer as one of the base abstractions for Avro serialization. From the stack trace I see that the instantiation of this class needs set up of KafkaAvroSerializerConfig which needs a value for the schema registry url .. regards. On

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hey Vahid, Hmm... If possible, it would be nice to avoid cluttering the default option too much, especially if it is information which is going to be the same for all members (such as the generation). My preference would be to use the --state option that you've suggested for that info so that we c

Re: DAG processing in Kafka Streams

If that is what it meant for DAG processing (we still need to confirm with Sameer), then programming-wise I do not see what's the difference with Kafka Streams since inside Streams users is also just specifying the topology as a DAG: https://kafka.apache.org/0110/documentation/streams/core-concept

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Jason, Thanks for your quick feedback. Your suggestions seem reasonable. I'll start updating the KIP accordingly and will send out another note when it's ready. Regards. --Vahid From: Jason Gustafson To: d...@kafka.apache.org Cc: Kafka Users Date: 07/17/2017 02:11 PM Subject

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

+Users Thanks for the KIP. I think tracking the rebalance time separately will help resolve some confusion about the performance results given the rebalance delay in KIP-134. And it seems generally useful to know how much overhead is coming from the rebalance in any case. -Jason On Thu, Jul 13,

Re: DAG processing in Kafka Streams

Hi Guozhang, I was comparing it with DAG processing in Spark. Spark Streaming is a close competitor to Kafka Streams, one difference which might accounts for a faster performance was that Spark submits the code to the code and does a bit of code optimization that its end. Lets consider an exampl