Re: Kafka Connect in different nodes than Kafka.

If you are asking if technically does Kafka Connect need the entire Apache Kafka distribution to run then, then the answer is no, it does not because Connectors just remotely connect to Kafka Brokers on separate machines. If you are asking if there is a separate distribution for a “connect

Re: Confluent platform for Kafka 0.10.1.1

Today! Confluent 3.1.2 supports Kafka 0.10.1.1 https://www.confluent.io/blog/confluent-delivers-upgrades-clients-kafka-streams-brokers-apache-kafka-0-10-1-1/ -hans > On Jan 25,

Re: Question about messages in __consumer_offsets topic

The __consumer_offsets topic should also get a tombstone message as soon as a topic is deleted. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Feb 22, 2017 at 5:59 PM, Jun MA <mj.saber1...@gmail.com> wrote: >

Re: Kafka Protocol : about "Offset" in the MessageSet

Producer will not know the offset of the message(s) at the time they send to the broker but they can receive that information back as confirmation of successful publish. -hans > On Jan 18, 2017, at 1:25 AM, Paolo Patierno wrote: > > Hi, > > > reading about the Kafka

Re: Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

but when it’s down, there is not a replica on any of the other brokers. Try creating a new topic with replication-factor 3 and you should get better availability in the event of one or even two broker failures. -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h

Re: creating partitions programmatically

The current Java AdminUtils are older code that talks to zookeeper and does not support a secured Kafka cluster. There will be a new Java Admin API in the future that talks to the Kafka brokers directly using the admin extensions to the Kafka protocol which are already in the 0.10.2 brokers. I

Re: creating partitions programmatically

. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sun, Feb 26, 2017 at 5:44 PM, VIVEK KUMAR MISHRA 13BIT0066 < vivekkumar.mishra2...@vit.ac.in> wrote: > My question is can we create partitions in topic using any pyt

Re: Kafka Connect

Maybe look at this Kafka source connector for salesforce https://github.com/jcustenborder/kafka-connect-salesforce -hans Sent from my iPhone > On Feb 27, 2017, at 4:06 PM, VIVEK KUMAR MISHRA 13BIT0066 > wrote: > > Actually my data sources are salesforce and

Re: kafka-connect-salesforce

Contact the author via github if the readme is not clear. -hans Sent from my iPhone > On Mar 1, 2017, at 7:01 AM, VIVEK KUMAR MISHRA 13BIT0066 > wrote: > > Hi All, > > I want to use kafka-connect-salesforce but i am not able to use it . > can any one provide

Re: kafka new consumer is ready for production setup?

be why the 0.10.1 docs are hard to find. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Oct 4, 2016 at 11:42 PM, Gaurav Shaha <grvsh...@gmail.com> wrote: > Hi, > > I want to use kafka new consumer. But in th

Re: Deleting a message after all consumer have consumed it

Kafka doesn’t work that way. Kafka is “Publish-subscribe messaging rethought as a distributed commit log”. The messages in the log do not get deleted just because “all" clients have consumed the messages. Besides you could always have a late joining consumer come along and if you mistakenly

Re: Regarding Kafka

if its not getting consumed the way I sent, then my analytics will go haywire. Abhi On Sun, Oct 9, 2016 at 12:50 PM, Hans Jespersen <h...@confluent.io> wrote: > You don't even have to do that because the default partitioner will spread > the data you publish to the topic over the av

Re: Regarding Kafka

Why do you have 10 topics? It seems like you are treating topics like partitions and it's unclear why you don't just have 1 topic with 10, 20, or even 30 partitions. Ordering is only guaranteed at a partition level. In general if you want to capacity plan for partitions you benchmark a single

Re: Regarding Kafka

Oct 9, 2016 at 10:13 AM, Hans Jespersen <h...@confluent.io> wrote: > Why do you have 10 topics?  It seems like you are treating topics like > partitions and it's unclear why you don't just have 1 topic with 10, 20, or > even 30 partitions. Ordering is only guaranteed at a partition l

Re: Regarding Kafka

-08 23:37:07.500]AxThreadId 23516 ->ID:4495 offset: 81][ID date: 2016-09-28 20:07:39.000 ] On Sun, Oct 9, 2016 at 1:31 PM, Hans Jespersen <h...@confluent.io> wrote: > Then publish with the user ID as the key and all messages for the same key > will be guaranteed to go to the same

Re: Kafka Multi DataCenter HA/Failover

Are you willing to have a maximum throughput of 6.67 messages per second? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Oct 28, 2016 at 9:07 AM, Mudit Agarwal <mudit...@yahoo.com> wrote: > Hi Hans, > > The

Re: Kafka Multi DataCenter HA/Failover

What is the latency between the two datacenters? I ask because unless they are very close, you probably don’t want to do any form of synchronous replication. The Confluent Replicator (coming very soon in Confluent Enterprise 3.1) will do async replication of both messages and configuration

Re: is there a way to make sure two consumers receive the same message from the broker?

raints. >>> 2) Add a common service on top of two storages and do some sort of 2PC >> but >>> that would degrade the write performance. Morever we dont really have a >>> control over how fast each write/store can happen at each storage layer >>> (

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

I believe that the new topics are picked up at the next metadata refresh which is controlled by the metadata.max.age.ms parameter. The default value is 30 (which is 5 minutes). -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670

Re: is there a way to make sure two consumers receive the same message from the broker?

this is no longer a Kafka question and has become more of a a distributed database design question. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sun, Nov 6, 2016 at 7:08 PM, kant kodali <kanth...@gmail.com> wrote: >

Re: is there a way to make sure two consumers receive the same message from the broker?

ean by "write consumer 1 and consumer 2 to share a common >> external offset storage" ? can you please elaborate a bit more. >> >> Thanks! >> >> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen <h...@confluent.io> wrote: >> >>> There

Re: is there a way to make sure two consumers receive the same message from the broker?

There is no built in mechanism to do this in Apache Kafka but if you can write consumer 1 and consumer 2 to share a common external offset storage then you may be able to build the functionality you seek. -hans > On Nov 5, 2016, at 3:55 PM, kant kodali wrote: > > Sorry

Re: Kafka unable to find topic details after sudden EC2 server restart

You are going to lose everything you store in /tmp. In a production system you never configure Kafka or zookeeper to store critical data in /tmp. This has nothing to do with AWS or EBS it is just standard Linux than everything under /tmp is deleted when Linux reboots. -hans /** * Hans Jespersen

RE: Mirror multi-embedded consumer's configuration

Yes. //h...@confluent.io Original message From: ZHU Hua B Date: 10/24/16 12:09 AM (GMT-08:00) To: users@kafka.apache.org Subject: RE: Mirror multi-embedded consumer's configuration Hi, Many thanks for your confirm! I have another question,

Re: Kafka unable to find topic details after sudden EC2 server restart

-linuxfoundationx-lfs101x-0 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 24, 2016 at 1:50 AM, Gourab Chowdhury <gourab@gmail.com> wrote: > Thanks for the reply, I tried changing the data directory as follows:-

Re: Kafka consumer rate

Yes. See the description of quotas. https://kafka.apache.org/documentation#design_quotas -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Oct 20, 2016 at 3:20 PM, Adrienne Kole <adrienneko...@gmail.com> wrote:

Re: Massive SSL performance degredation

d try to pick something with which java is simpatico. > > >> On Thu, Nov 17, 2016 at 4:04 PM, Hans Jespersen <h...@confluent.io> wrote: >> >> What is the difference using the bin/kafka-console-producer and >> kafka-console-consumer as pub/sub clients? >

Re: Massive SSL performance degredation

What is the difference using the bin/kafka-console-producer and kafka-console-consumer as pub/sub clients? see http://docs.confluent.io/3.1.0/kafka/ssl.html -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Nov 17, 2016 at 11

Re: multiple consumers for a same topic in multiple copies

Just make sure they are not in the same consumer group by creating a unique value for group.id for each independent consumer. /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 31, 2016 at 9:42 AM, Patrick Viet <patric

Re: Deadlock using latest 0.10.1 Kafka release

The 0.10.1 broker will use more file descriptor than previous releases because of the new timestamp indexes. You should expect and plan for ~33% more file descriptors to be open. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670

Re: I found kafka lsot message

Watch this talk. Kafka will not lose messages it configured correctly. http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there -hans > On Oct 13, 2016, at 2:48

Re: Disadvantages of Upgrading Kafka server without upgrading client libraries?

The performance impact of upgrading and some settings you can use to mitigate this impact when the majority of your clients are still 0.8.x are documented on the Apache Kafka website https://kafka.apache.org/documentation#upgrade_10_performance_impact -hans /** * Hans Jespersen, Principal

Re: Kafka Queue

are you setting the group.id property to be the same on both consumers? https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Dec 7, 2016 at 12:36 PM

Re: Storing Kafka Message JSON to deep storage like S3

I know several people that use the qubole Kafka Sink Connector for S3 ( see https://github.com/qubole/streamx ) to store Kafka messages in S3 for long term archiving. You can also do this with the Confluent HDFS Kafka Connector if you have access to a Hadoop

Re: Regarding Connection Problem

I would recommend you use either the Blizzard node-rdkafka module ( see https://github.com/Blizzard/node-rdkafka ) or the Confluent kafka-rest-node module ( see https://github.com/confluentinc/kafka-rest-node

Re: Kafka delay

latency guaranteed messaging -- Jiangjie (Becket) Qin (LinkedIn) <https://www.youtube.com/watch?v=oQe7PpDDdzA> which might give you much better context and understanding of what these parameters mean and how they work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc

Re: can kafka 10 stream API read the topic from a Kafka 9 cluster?

No. All Java clients (including Streams) need to be the same version (or lower) as the brokers they connect to. -hans > On Dec 23, 2016, at 1:03 AM, Sachin Mittal wrote: > > Is Kafka streams 0.10.2.0-SNAPSHOT compatible with 0.10.0.1 broker. > I was facing broker

Re: Heavy traffic between Kafka nodes

How much more data is flowing between the two nodes versus the total ingress rate of the cluster? I would expect inter-broker bandwidth use to be about the same as the ingress rate for the entire cluster since all the data coming into node 1 needs to get replicated to node 2 (for redundancy)

Re: How does Kafka emulate exactly once processing?

orage?). > > Thanks! > > On Wed, Dec 21, 2016 at 5:11 PM, Hans Jespersen <h...@confluent.io> wrote: > >> Exactly once Kafka Sink Connectors typically store the offset externally >> in the same atomic write as they store the messages. That way after a >> crash,

Re: How does Kafka emulate exactly once processing?

Exactly once Kafka Sink Connectors typically store the offset externally in the same atomic write as they store the messages. That way after a crash, they can check the external store (HSFS, JSDC, etc) retrieve the last committed offset and seek the the next message and continue processing with

Re: Out of order message processing with Kafka Streams

ndividual messages in a window > > Are those 2 things possible w/ kafka streams? (java) > > On Tue, Mar 21, 2017 at 7:43 PM, Hans Jespersen <h...@confluent.io> wrote: > >> While it's not exactly the same as the window start/stop time you can >> store (in the state store

Re: kafka is not accepting number of partitions from configuration

The num.partitions parameter is a server/broker config but you are using it as a client/producer parameter so it will not work and will be ignored. http://stackoverflow.com/questions/22152269/how-to-specify-number-of-partitions-on-kafka-2-8 I assume the CLI command you are using is the

Re: kafka not throwing any exception not any response in call back

In your producer have you set acks to be “all” and retries to be something higher than “0”? props.put("acks", "all"); props.put("retries", 2147483647); Also if the order of the messages matter you should also set the max inflight messages to 1 so retries happen and succeed before any other

Re: How to assign client-id to a particular kafka producer or topic?

username comes from authenticated clients client.id can be assigned by any client (no authentication required). It’s hard to enforce a quota on a client.id when the clients can just change the code to use a difference client.id, hence the recent enhancement to add user quotas based on the

Re: kafka-topics[.sh]: fail to support connecting via broker / v0.10 style

. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Mar 17, 2017 at 1:20 PM, Andrew Pennebaker < andrew.penneba...@gmail.com> wrote: > If I understand Kafka correctly, since v0.9 / v0.10, users are often >

Re: Out of order message processing with Kafka Streams

the added benefit of getting intermediary result at much lower latency. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 10:29 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Is it possible to have

Re: Out of order message processing with Kafka Streams

. -hans > On Mar 18, 2017, at 11:36 AM, Sabarish Sasidharan <sabarish@gmail.com> > wrote: > > Hans > > What you state would work for aggregations, but not for state machines and > CEP. > > Regards > Sab > >> On 19 Mar 2017 12:01 a.m., "Hans Jes

Re: Out of order message processing with Kafka Streams

sorry I mixed up Message A and B wrt the to question but the answer is still valid. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 11:07 AM, Hans Jespersen <h...@confluent.io> wrote: > The only wa

Re: validate identity of producer in each record

work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin <apache@msqr.us> wrote: > Hello, > > I am new to Kafka and am looking for a way for consumers to be able to > ide

Re: validate identity of producer in each record

an > extension point in Kafka itself on the receiving of records, before they are > stored/distributed? I was thinking along the lines of > > org.apache.kafka.clients.producer.ProducerInterceptor > > but on the server side? > > — m@ > >> On 21/03/2017, at 12:22 PM

Re: Kafka Retention Policy to Indefinite

the latest value for each key forever, but removes the older messages with the same key in order to reduce the total about of messages stored. How much data do you expect to store in your largest topic over the life of the cluster? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent

Re: Performance and Encryption

You are correct that a Kafka broker is not just writing to one file. Jay Kreps wrote a great blog post with lots of links to even greater detail on the topic of Kafka and disk write performance. Still a good read many years later.

Re: Common Identity between brokers

This might be useful reading as it outlines why Cluster ID was added and lists a few ways that clusters can be identifies prior to that feature enhancement. https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id

Re: How to increase network throughput of Kafka cluster?

t; On Sat, Apr 1, 2017 at 3:51 PM, Hans Jespersen <h...@confluent.io> wrote: >> >> What replication factor are you using? If you have a default replication >> factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 >> Gbps of network traffic. If you are

Re: How to increase network throughput of Kafka cluster?

What replication factor are you using? If you have a default replication factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 Gbps of network traffic. If you are also consuming at the same time then it’s actually 4.2 Gbps + 1.4 Gbps = 5.6 Gbps. You would completely

Re: Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

They are both true. The Apache text is talking about the compatibility of the Producer/Consumer API and the Confluent text is talking about the Streams API. -hans > On Mar 31, 2017, at 11:46 PM, Roger Vandusen > wrote: > > Read below and answer: So which is

Re: Kafka MTLS Support?

Are you asking about Multiplexed Transport Layer Security (MTLS) - https://en.wikipedia.org/wiki/Multiplexed_Transport_Layer_Security or Mutual TLS authentication (mTLS) https://en.wikipedia.org/wiki/Mutual_authentication

Re: Kafka producer and consumer within on sync execution

You posted the same question to Stack Overflow so I answered it there https://stackoverflow.com/questions/43302857/handling-sync-api-call-rest-spring-and-async-message-kafka-in-the-same-execu/43312070#43312070

Re: Re: ZK and Kafka failover testing

e as per your case} > > ssl.keystore.location= {appropriate value as per your case} > > ssl.keystore.password= {appropriate value as per your case} > > ssl.truststore.location= {appropriate value as per your case} > > ssl.truststore.password= {appropriate value as per you

Re: Re: Re: ZK and Kafka failover testing

ers​= {point the F5 VS fronting Kafka cluster} > >> > > key.serializer= {appropriate value as per your cases} > >> > > value.serializer= {appropriate value as per your case} acks= all > >> > > retries=3 > >> > > ssl.key.password=

Re: Question on Metadata

You may also be interested to try out the new Confluent JMS client for Kafka. It implements the JMS 1.1. API along with all the JMS metadata fields and access methods. It does this by putting/getting the JMS metadata into the body of an underlying Kafka message which is defined with a special

Re: Kafka Retention Policy to Indefinite

in 0.10.1 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Mar 14, 2017 at 10:09 AM, Joe San <codeintheo...@gmail.com> wrote: > Dear Kafka Users, > > What are the arguments against setting the retention plociy on

Re: Having 4 Node Kafka Cluster

you think a stretch cluster will work? That seems wrong. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 6, 2017 at 5:37 AM, Le Cyberian <lecyber...@gmail.com> wrote: > Hi Guys, > > Thank yo

Re: Having 4 Node Kafka Cluster

that could fail. So it SHOULD be an odd number of zookeeper nodes (not MUST). -hans > On Mar 6, 2017, at 12:20 AM, Jens Rantil <jens.ran...@tink.se> wrote: > > Hi Hans, > >> On Mon, Mar 6, 2017 at 12:10 AM, Hans Jespersen <h...@confluent.io> wrote: >> >> A 4

Re: Performance and encryption

Its not a single message at a time that is encrypted with TLS its the entire network byte stream so a Kafka broker can’t even see the Kafka Protocol tunneled inside TLS unless it’s terminated at the broker. It is true that losing the zero copy optimization impacts performance somewhat but

Re: Having 4 Node Kafka Cluster

lag. > > If i do a Mirror Maker / Replicator then i will not be able to use them at > the same time for writes./ producers. because the consumers / producers > will request from all of them > > BR, > > Lee > > On Mon, Mar 6, 2017 at 2:50 PM, Hans Jespersen <h...@confluen

Re: Having 4 Node Kafka Cluster

gt; >> Additional Hints: >> * You can run any number of Kafka brokers on a ZK cluster. In your case >> this could be 4 Kafka brokers on 3 ZK nodes. >> * You should set topic replication to 2 (can be done at any time) and some >> other producer/broker settings to ensure your mes

Re: Having 4 Node Kafka Cluster

A 4 node zookeeper ensemble will not even work. It MUST be an odd number of zookeeper nodes to start. For Kafka you can start with any number of nodes (including 4). Remember that it is the partitions that are replicated, not the entire broker so if you have a Kafka node crash, its only the

Re: Kafka connector

nd N3 have different names. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee <tus...@levyx.com> wrote: > Hello Sir/Ma'am, > I was trying to write a simple case of using kaf

Re: Out of order message processing with Kafka Streams

rent/streams/concepts.html#windowing >>>>>> >>>>>> >>>>>> On Mon, Mar 20, 2017 at 5:43 PM, Michael Noll < >> mich...@confluent.io> >>>>>> wrote: >>>>>> >>>>>>> Late-arriving and

Re: ZK and Kafka failover testing

in the kafka cluster. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Apr 18, 2017 at 4:10 PM, Shrikant Patel <spa...@pdxinc.com> wrote: > Hi All, > > I am seeing strange behavior between ZK and Kafka. We ha

Re: How does replication affect kafka quota?

Replication will not effect the users quota as it is done under a different replication quota (which you can control separately). The user should still see a 50 MBps maximum rate enforced into each broker. -hans > On Apr 23, 2017, at 11:39 PM, Archie wrote: > >

Re: How to clear a particular partition?

This is an area that is being worked on. See KIP-107 for details. https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient -hans > On Aug 10,

Re: kafka connect

If you are looking for connectors there is a good curated list of connectors here https://www.confluent.io/product/connectors/ I don't see tcp on the list but the general naming scheme for open source connectors on github is to call them

Re: Requires suggestions for Producer request throttling

Request quotas was just added to 0.11. Does that help in your use case? https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas -hans > On Jun 29, 2017, at 12:55 AM, sukumar.np wrote: > > Hi Team, > > > > We are having a Kafka cluster with

Re: about Exactly-once Semantics

When you write the msg results to MySQL, you include the offset of the message with the results. This can be done in one atomic write transaction. Then if your application crashes, when it starts back up, it should read the offset stored with the last message results in the database, then

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

There is a parameter that controls this behavior called max.in. flight.requests.per.connection If you set max.in. flight.requests.per.connection = 1 then the producer waits until previous produce requests returns a response before sending the next one (or retrying). The retries parameter

Re: Adding partitons | Unaffected producers

See the producer param called metadata.max.age.ms which is "The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions." -hans > On Aug 4, 2017, at 5:17 AM, Sameer

Re: How to clear a particular partition?

kafka-delete-records.sh script to delete data. > > On Sun, Aug 13, 2017 at 11:27 PM, Hans Jespersen <h...@confluent.io> wrote: > >> This is an area that is being worked on. See KIP-107 for details. >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>

Re: Pinning clients to specific brokers

Doing that doesn't really make sense in a Kafka cluster because the topic partitions and their replicas are spread out across many brokers in the cluster. That's what enables the parallel processing and fault tolerance features of Kafka. -hans > On Aug 22, 2017, at 3:14 AM, Mohit Chawla

Re: Why do I need to specify replication factor when creating a topic?

If you enable auto topic creation that that is exactly what will happen. There are pros and cons to creating topics with defaults values but if you fell strongly that is the way that you want Kafka to work it is entire possible to setup the system to work that way. -hans > On May 11,

Re: help!Kafka failover do not work as expected in Kafka quick start tutorial

Do you list all three brokers on your consumers bootstrap-server list? -hans > On Jun 22, 2017, at 5:15 AM, 夏昀 wrote: > > hello: > I am trying the quickstart of kafka documentation,link is, > https://kafka.apache.org/quickstart. when I moved to Step 6: Setting up a >

Re: question about document

Correct. The use of the word "server" in that sentence is meant as broker (or KafkaServer as it shows up in the 'jps' command) not as a physical or virtual machine. -hans > On Jun 27, 2017, at 1:22 AM, James <896066...@qq.com> wrote: > > Hello, >At https://kafka.apache.org/intro, I found

Re: Producer Async Issue

The producer is asynchronous (assuming you mean the Java Producer) https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html -hans > On May 27, 2017, at 5:15 AM, Abhimanyu Nagrath > wrote: > > Hi, > I am using Kafka

Re: Producer Async Issue

figured max.block.ms and after that further code is executed. What I am > looking for is that whether the broker is down or not it should not get > stuck. > > > > Regards, > Abhimanyu > >> On Sat, May 27, 2017 at 10:30 PM, Hans Jespersen <h...@confluent.io> wrot

Re: 0.10.0.0 cluster : segments getting latest ts

e.org/jira/browse/KAFKA/fixforversion/12334962>>. I > even tried that. On QA hosts it retains TS for .log files across restart. > But when tried the new version on one of the prod host, same old story. > > So internal or File system ts, it should get deleted when expired. What > c

Re: Kafka Read Data from All Partition Using Key or Timestamp

The timeindex was added in 0.10 so I think you need to use the new Consumer API to access this functionality. Specifically you should call offsetsForTimes() https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/Consumer.html#offsetsForTimes(java.util.Map) -hans > On May 25,

Re: Java APIs for ZooKeeper related operations

ka+admin+operations > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations> > > -hans > > /** > * Hans Jespersen, Principal Systems Engineer, Confluent Inc. > * h...@confluent.io <mailto:h...@conflue

Re: Java APIs for ZooKeeper related operations

Probably important to read and understand these enhancements coming in 0.11 https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924

Re: [E] Re: Kafka Configuration Question

th zookeeper and kafka brokers to help in debugging further -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ > On May 29, 2017, at 1:33 AM, Bennett, Conrad > <conrad.benn...@verizonwireless.com.INVALID> wrote: > &g

Re: Trouble with querying offsets when using new consumer groups API

It is definitely expected behavior that the new consumer version of kafka-consumer-groups.sh —describe only returns metadata for ‘active’ members. It will print an error message if the consumer group you provide has no active members.

Re: Data in kafka topic in Json format

Check which serializer you have configured in your producer. You are probably using an Avro serializer which will add the schema and modify the payload to avro data. You can use a String serializer or a ByteArray serializer and the data will either be Base64 encoded or not encoded at all.

Re: Data in kafka topic in Json format

My earlier comment still applies but in Kafka Connect the equivalent of a serializer/deserializer (serdes) is called a “converter”. Check which converter you have configured for your source connector and if it is overriding whatever the default converter is configured for the connect worker it

Re: Data in kafka topic in Json format

ormatted/stringified json which has extra "\" and is not json any more? > > Best regards, > Mina > > On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen <h...@confluent.io> wrote: > >> >> My earlier comment still applies but in Kafka Connect the e

Re: Async Non Blocking Kafka Producer

Try adding props.put("max.block.ms", "0"); -hans > On Jun 7, 2017, at 12:24 PM, Ankit Jain wrote: > > Hi, > > We want to use the non blocking Kafka producer. The producer thread should > not block if the Kafka is cluster is down or not reachable. > > Currently, we

Re: Async Non Blocking Kafka Producer

loosing the data even the Cluster is > available. > > Thanks > Ankit Jain > >> On Wed, Jun 7, 2017 at 12:56 PM, Hans Jespersen <h...@confluent.io> wrote: >> >> Try adding props.put("max.block.ms", "0"); >> >> -hans >>

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Offset commit is something that is done in the act of consuming (or reading) Kafka messages. Yes technically it is a write to the Kafka consumer offset topic but it's much easier for administers to think of ACLs in terms of whether the user is allowed to write (Produce) or read (Consume)

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

ght.requests.per.connection > 1, then even if the topic only have > one partition, there’s still no guarantee of the ordering? > > Thanks, > Jun > On Apr 30, 2017, at 7:57 AM, Hans Jespersen <h...@confluent.io> wrote: > > There is a parameter that controls this beh

Re: 0.10.0.0 cluster : segments getting latest ts

even if it is expired, unless all the older segment has been expired." If none of the messages in a segment has a timestamp, last modified time will be used. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, May 25, 2017

Re: 0.10.0.0 cluster : segments getting latest ts

together with log.roll.ms to avoid frequent log segment roll out. During the migration phase, if the first message in a segment does not have a timestamp, the log rolling will still be based on the (current time - create time of the segment)." -hans /** * Hans Jespersen, Principal Systems Engineer,

  1   2   >