Re: Newbie Question

I can tell from the terminology you use that you are familiar with traditional message queue products. Kafka is very different. Thats what makes it so interesting and revolutionary in my opinion. Clients do not connect to topics because kafka is a distributed and clustered system where topics

Re: Kafka with RAID 5 on. busy cluster.

RAID 5 typically is slower because Kafka is very write heavy load and that creates a bottleneck because writes to any disk require parity writes on the other disks. -hans > On Mar 28, 2020, at 2:55 PM, Vishal Santoshi > wrote: > > Ny one ? We doing a series of tests to be confident, but

Re: Scaling Apache Kafka Producers & Consumers

nsumer group. > > If you have 12 partitions and you run 4 consumers with 4 consumer threads > then 4 threads will be idle at any time T1. > > I hope this is clear. > > Thanks, > Sunil. > > On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen wrote: > >>> As

Re: Reg : Slowness in Kafka

Yes it should be going much faster than that. Something is wrong in your setup. -hans > On Mar 26, 2020, at 5:58 PM, Vidhya Sakar wrote: > > Hi Team, > > The Kafka consumer is reading only 8 records per second.We have implemented > apache Kafka and confluent connect S3. The confluent connect

Re: Scaling Apache Kafka Producers & Consumers

> As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Not correct. A single consumer from a consumer group can consume from many partitions. For example if you had a topic with 12 partitions and 4 consumers in a

Re: kafka connection from docker

This is a great blog post that explains how kafka works with advertised listeners and docker https://rmoff.net/2018/08/02/kafka-listeners-explained/ -hans > On Oct 18, 2019, at 5:36 AM, Mich Talebzadeh > wrote: > > I do not understand this. > > You have on a physical host running

Re: Requirements

Gwen Shapira published a great whitepaper with Reference Architectures for all Kafka and Confluent components in big and small environements and for bare metal, VMs, and all 3 major public clouds. https://www.confluent.io/resources/apache-kafka-confluent-enterprise-reference-architecture/ On

Re: Update Replication Factor

Take a look at the Admin Client API here https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html -hans On Mon, Jun 17, 2019 at 4:27 PM shubhmeet kaur wrote: > hi, > > I wish to updater the replciation factor of already created topic through > Java code

Re: Customers are getting same emails for roughly 30-40 times

Its not just the config, you need to change your code. kafka.auto.commit.interval.ms=3000 means that consumers only commit offsets every 3 seconds so if there is any failure or rebalance they will reconsume up to 3 seconds of data per partition. That could be many hundreds or thousands of

Re: Performance Testing Using Consumer-Perf-Test

1) Are all 10 publishers producing to the same topic? What level of ACKs do you have set? How many partitions are in your topic? Are all 10 consumers in the same consumer group or are they supposed to be independent consumers that each get the full set of messages published? 2) Depends what you

Re: Kafka Connect - HDFS or FileStream

Can you just use kafka-console-consumer and just redirect the output into a file? -hans On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote: > Hi > > The data needs to be transferred to some other system in other network, and > due to some security reasons, the other systems cannot be exposed .

Re: Source Connector Task in a distributed env

Your connector sounds a lot like this one https://github.com/jcustenborder/kafka-connect-spooldir I do not think you can run such a connector in distributed mode though. Typically something like this runs in standalone mode to avoid conflicts. -hans On Wed, Apr 24, 2019 at 1:08 AM Venkata S A

Re: Something like a unique key to prevent same record from being inserted twice?

.me...@concept-solutions.com > wrote: > > > >> On 2019/04/02 22:43:31, jim.me...@concept-solutions.com >> wrote: >> >> >>> On 2019/04/02 22:25:16, jim.me...@concept-solutions.com >>> wrote: >>> >>> >>>> On

Re: Something like a unique key to prevent same record from being inserted twice?

yes. Idempotent publish uses a unique messageID to discard potential duplicate messages caused by failure conditions when publishing. -hans > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > wrote: > > Does Kafka have something that behaves like a unique key so a producer

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

use location specific SKU pricing and send specific price lists to the > specific POS terminal. > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen : > >> Doesn’t every one of the 20,000 POS terminals want to get the same price >> list messages? If so then there is no need

Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

Doesn’t every one of the 20,000 POS terminals want to get the same price list messages? If so then there is no need for 20,000 partitions. -hans > On Mar 31, 2019, at 7:24 PM, wrote: > > Hello! > > > > I ask for your help in connection with the my recent task: > > - Price lists are

Re: Question on performance data for Kafka vs NATS

Thats a 4.5 year old benchmark and it was run with a single broker node and only 1 producer and 1 consumer all running on a single MacBookPro. Definitely not the target production environment for Kafka. -hans > On Mar 21, 2019, at 11:43 AM, M. Manna wrote: > > HI All, > >

Re: Proxying the Kafka protocol

You might want to take a look at kafka-proxy ( see https://github.com/grepplabs/kafka-proxy ). It’s a true kafka protocol proxy and modified the metadata like advertized listeners so it works when there is no ip routing between the client and the

Re: Kafka - Connect for logs processing

Take a look at kafka-connect-spooldir and see if it meets your needs. https://www.confluent.io/connector/kafka-connect-spooldir/ This connector can monitor a directory and pick up any new files that are created. Great for picking up batch files, parsing them, and publishing each line as if it

Re: How to balance messages in kafka topics with newly added partitions?

Yes but I find this even easier to do with KSQL. CREATE STREAM OUTPUTTOPIC AS SELECT * FROM INPUTTOPIC; There are similar examples like this that also filter messages while copying, or change the message format while copying on the KSQL Recipe page here

Re: How to acknowledge after consuming the message from Kafka topic?

this one. -hans > On Jan 21, 2019, at 10:02 AM, Rahul Singh > wrote: > > I am using node-kafka, I have used consumer.commit to commit offsets but > don't know why when I restart the consumer it consume the committed offsets. > > Thanks > >> On Mon, Jan 21,

Re: How to acknowledge after consuming the message from Kafka topic?

Are you using kafka-node or node-rdkafka? In either case you should call Consumer.commit(cb) or something similar to manually commit offsets (aka acknowledge messages). Alternatively so can set a config parameter on the consumer to autoCommit.

Re: The asynchronous sending of a message returns no error if the Kafka server is not started

That is expected behavior. Typically there are multiple kafka brokers and so if one is down the client retries to send to a newly elected leader. A send should not be considered successful until an ACK is received in the client from the kafka cluster. By default the ACK is async for

Re: Security for individual partitions

Kafka ACLs are at the topic level, not partition level. Probably better to make 10 topics of 1 partition each and use topic ACLs to control access. -hans > On Jun 25, 2018, at 9:50 PM, Yash Ganthe wrote: > > Hi, > > If I have a topic with 10 partitions, I would like each partition to be >

Re: Is there expiration for committed Offset in the partition

You should just recommit the same offsets sooner than every 24 hours (or whatever your commit topic retention period is set to). The expiry of offsets is based on the timestamp of the commits. -hans > On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian > wrote: > > Hi, > > Facing duplication in

Re: Round-Robin assignment when non-nullable record key

Why don’t to just put the metadata in the header and leave the key null so it defaults to round robin? -hans > On May 31, 2018, at 6:54 AM, M. Manna wrote: > > Hello, > > I can see the this has been set as "KIP required". > > https://issues.apache.org/jira/browse/KAFKA- > > I have a

Re: Facing Duplication Issue in kakfa

Are you seeing 1) duplicate messages stored in a Kafka topic partition or 2) duplicate consumption and processing of a single message stored in a Kafka topic? If it’s #1 then you can turn on the idempotent producer feature to get Exactly Once Semantics (EOS) while publishing. If it’s #2 then

Re: Can anyone help me to send messages in their original order?

wrong > order? Note there are only 9 lines from 1 to 9, but on consumer side their > original order becomes messed up. > > ~~~sent from my cell phone, sorry if there is any typo > > Hans Jespersen <h...@confluent.io> 于 2018年5月26日周六 上午12:16写道： >> If you create a t

Re: Can anyone help me to send messages in their original order?

If you create a topic with one partition they will be in order. Alternatively if you publish with the same key for every message they will be in the same order even if your topic has more than 1 partition. Either way above will work for Kafka. -hans > On May 25, 2018, at 8:56 PM, Raymond Xie

Re: Kafka mirror maker help

Sorry I hit send a bit too soon. I was so focused on the systemd part of the email and not the Mirror Maker part. Confluent packages include Mirror Maker but the systemd scripts are setup to use Confluent Replicator rather than Mirror Maker. My apologies. -hans /** * Hans Jespersen, Director

Re: Kafka mirror maker help

The latest Confluent packages now ship with systemd scripts. That is since Confluent Version 4.1 - which included Apache Kafka 1.1 -hans /** * Hans Jespersen, Director Systems Engineering, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto &l

Re: Is Restart needed after change in trust store for Kafka 1.1 ?

Kafka 1.1.0 https://issues.apache.org/jira/browse/KAFKA-6240 <https://issues.apache.org/jira/browse/KAFKA-6240> which seems in include dynamic reconfiguration of SSL keystores https://issues.apache.org/jira/browse/KAFKA-6241 <https://issues.apache.org/jira/browse/KAFKA-6241> -- /**

Re: Is Kafka Streams right for me ?

"If your system is stateless and the transformations are not interdependent" then I would just look at using Kafka Connect's Single Message Transform (SMT) feature. -hans /** * Hans Jespersen, Director Systems Engineering, Confluent Inc. * h...@confluent.io (650)924-2670 */ On T

Re: replica.fetch.max.bytes split message or not ?

This is the KIP-74 write up if you want to learn more about the motivation and implementation of the fetch.max.bytes feature. For example, the 5th message in your example can now be fetched even if it is larger than fetch.max.bytes which was a great improvement in large message handling.

Re: Kafka Consumer Offsets unavailable during rebalancing

r consumer group 'X' for > topics 'A' and 'B' again and the consumer group is still "Stable". > > I expected the state of consumer group 'X' during step 2b to be > "PreparingRebalance" or "AwaitingSync". > > Best regards, > W

Re: Kafka Consumer Offsets unavailable during rebalancing

I believe this is expected behavior. If there are no subscriptions to a new topic, and therefor no partition assignments, and definitely no committed offsets, then lag is an undefined concept. When the consumers subscribe to this new topic they may chose to start at the beginning or end of the

Re: Capturing and storing these Kafka events for query.

Another approach would be to create the query first (in something like KSQL) and then send the Kafka data through the pre-existing streaming query. In this case the results would be going into various result topics. Tools like KSQL also let you query historical data but you need to be sure

Re: Consumer client not able to receive messages when one of broker is pushed down in the cluster

Check that your __consumer_offsets topic is also setup with replication factor of 3 and has In Sync Replicas. Often it gets setup first as a one node cluster with RF=1 and then when the cluster is expanded to 3 nodes the step to increase the replication factor of this topic gets missed. -hans

Re: Seeking advice on Kafka Streams and Kafka Connect

It might be possible to do all the transformations in #2 inside Kafka Connect. Connect has a simple one message at a time transformation capability called Single Message Transforms (SMT). There are built in Transformation functions that you can declaratively add to any existing connector via

Re: Kafka streams for golang

You can call the REST endpoints in KSQL from any programming language. I wrote some stuff in node.js to call KSQL this way and it works great. The results don't even have to go to a Kafka topic as the results of and POST to /query all stream using HTTP. -hans /** * Hans Jespersen, Principal

Re: Failure to reset consumer offsets for specific topics

I think you are just missing the —execute flag. -hans > On Oct 25, 2017, at 1:24 PM, Ted Yu wrote: > > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin

Re: Debugging invalid_request response from a .10.2 server for list offset api using librdkafka client

tps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI(AKAListOffset) -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Wed, Sep 27, 2017 at 10:20 AM, Vignesh <vignesh.v...@gma

Re: KSQL with Apache Kafka

Those prerequisites are just for the Confluent CLI used in the quickstart. The Apache Kafka and Zookeeper versions included in the Confluent distribution are the latest and the same as the Apache Kafka download so it will work. You will just need to start Zookeeper and Kafka with the shell

Re: Flush Kafka topic

in 0.11 and above see the CLI command bin//kafka-delete-records.sh -hans > On Aug 23, 2017, at 7:28 PM, Rahul Singh wrote: > > Hello all, > > I am unable to purge the topic data from Kafka. Is there any class to flush > all topic data. > > Thank you

Re: Pinning clients to specific brokers

We (Confluent) run Kafka as a SaaS-based cloud offering and we do not see any reason for this feature so I just don’t understand the motivation for it. Please explain. -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670

Re: Pinning clients to specific brokers

Doing that doesn't really make sense in a Kafka cluster because the topic partitions and their replicas are spread out across many brokers in the cluster. That's what enables the parallel processing and fault tolerance features of Kafka. -hans > On Aug 22, 2017, at 3:14 AM, Mohit Chawla

Re: How to clear a particular partition?

kafka-delete-records.sh script to delete data. > > On Sun, Aug 13, 2017 at 11:27 PM, Hans Jespersen <h...@confluent.io> wrote: > >> This is an area that is being worked on. See KIP-107 for details. >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>

Re: How to clear a particular partition?

This is an area that is being worked on. See KIP-107 for details. https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient -hans > On Aug 10,

Re: Adding partitons | Unaffected producers

See the producer param called metadata.max.age.ms which is "The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions." -hans > On Aug 4, 2017, at 5:17 AM, Sameer

Re: kafka connect

If you are looking for connectors there is a good curated list of connectors here https://www.confluent.io/product/connectors/ I don't see tcp on the list but the general naming scheme for open source connectors on github is to call them

Re: about Exactly-once Semantics

When you write the msg results to MySQL, you include the offset of the message with the results. This can be done in one atomic write transaction. Then if your application crashes, when it starts back up, it should read the offset stored with the last message results in the database, then

Re: Requires suggestions for Producer request throttling

Request quotas was just added to 0.11. Does that help in your use case? https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas -hans > On Jun 29, 2017, at 12:55 AM, sukumar.np wrote: > > Hi Team, > > > > We are having a Kafka cluster with

Re: question about document

Correct. The use of the word "server" in that sentence is meant as broker (or KafkaServer as it shows up in the 'jps' command) not as a physical or virtual machine. -hans > On Jun 27, 2017, at 1:22 AM, James <896066...@qq.com> wrote: > > Hello, >At https://kafka.apache.org/intro, I found

Re: help!Kafka failover do not work as expected in Kafka quick start tutorial

Do you list all three brokers on your consumers bootstrap-server list? -hans > On Jun 22, 2017, at 5:15 AM, 夏昀 wrote: > > hello: > I am trying the quickstart of kafka documentation,link is, > https://kafka.apache.org/quickstart. when I moved to Step 6: Setting up a >

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Offset commit is something that is done in the act of consuming (or reading) Kafka messages. Yes technically it is a write to the Kafka consumer offset topic but it's much easier for administers to think of ACLs in terms of whether the user is allowed to write (Produce) or read (Consume)

Re: Async Non Blocking Kafka Producer

loosing the data even the Cluster is > available. > > Thanks > Ankit Jain > >> On Wed, Jun 7, 2017 at 12:56 PM, Hans Jespersen <h...@confluent.io> wrote: >> >> Try adding props.put("max.block.ms", "0"); >> >> -hans >>

Re: Async Non Blocking Kafka Producer

Try adding props.put("max.block.ms", "0"); -hans > On Jun 7, 2017, at 12:24 PM, Ankit Jain wrote: > > Hi, > > We want to use the non blocking Kafka producer. The producer thread should > not block if the Kafka is cluster is down or not reachable. > > Currently, we

Re: Data in kafka topic in Json format

ormatted/stringified json which has extra "\" and is not json any more? > > Best regards, > Mina > > On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen <h...@confluent.io> wrote: > >> >> My earlier comment still applies but in Kafka Connect the e

Re: Data in kafka topic in Json format

My earlier comment still applies but in Kafka Connect the equivalent of a serializer/deserializer (serdes) is called a “converter”. Check which converter you have configured for your source connector and if it is overriding whatever the default converter is configured for the connect worker it

Re: Data in kafka topic in Json format

Check which serializer you have configured in your producer. You are probably using an Avro serializer which will add the schema and modify the payload to avro data. You can use a String serializer or a ByteArray serializer and the data will either be Base64 encoded or not encoded at all.

Re: Java APIs for ZooKeeper related operations

ka+admin+operations > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations> > > -hans > > /** > * Hans Jespersen, Principal Systems Engineer, Confluent Inc. > * h...@confluent.io <mailto:h...@conflue

Re: Java APIs for ZooKeeper related operations

Probably important to read and understand these enhancements coming in 0.11 https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924

Re: Trouble with querying offsets when using new consumer groups API

I can confirm that in 0.10.2.1 I get offset information for disconnected consumers. The note in the output is a bit misleading because it also works with non-Java clients as long as they implement the new consumer. For example below is what I get when using the blizzard/node-rdkafka client

Re: Trouble with querying offsets when using new consumer groups API

It is definitely expected behavior that the new consumer version of kafka-consumer-groups.sh —describe only returns metadata for ‘active’ members. It will print an error message if the consumer group you provide has no active members.

Re: [E] Re: Kafka Configuration Question

th zookeeper and kafka brokers to help in debugging further -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ > On May 29, 2017, at 1:33 AM, Bennett, Conrad > <conrad.benn...@verizonwireless.com.INVALID> wrote: > &g

Re: Producer Async Issue

figured max.block.ms and after that further code is executed. What I am > looking for is that whether the broker is down or not it should not get > stuck. > > > > Regards, > Abhimanyu > >> On Sat, May 27, 2017 at 10:30 PM, Hans Jespersen <h...@confluent.io> wrot

Re: Producer Async Issue

The producer is asynchronous (assuming you mean the Java Producer) https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html -hans > On May 27, 2017, at 5:15 AM, Abhimanyu Nagrath > wrote: > > Hi, > I am using Kafka

Re: 0.10.0.0 cluster : segments getting latest ts

e.org/jira/browse/KAFKA/fixforversion/12334962>>. I > even tried that. On QA hosts it retains TS for .log files across restart. > But when tried the new version on one of the prod host, same old story. > > So internal or File system ts, it should get deleted when expired. What > c

Re: 0.10.0.0 cluster : segments getting latest ts

even if it is expired, unless all the older segment has been expired." If none of the messages in a segment has a timestamp, last modified time will be used. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, May 25, 2017

Re: 0.10.0.0 cluster : segments getting latest ts

together with log.roll.ms to avoid frequent log segment roll out. During the migration phase, if the first message in a segment does not have a timestamp, the log rolling will still be based on the (current time - create time of the segment)." -hans /** * Hans Jespersen, Principal Systems Engineer,

Re: Kafka Read Data from All Partition Using Key or Timestamp

The timeindex was added in 0.10 so I think you need to use the new Consumer API to access this functionality. Specifically you should call offsetsForTimes() https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/Consumer.html#offsetsForTimes(java.util.Map) -hans > On May 25,

Re: Why do I need to specify replication factor when creating a topic?

If you enable auto topic creation that that is exactly what will happen. There are pros and cons to creating topics with defaults values but if you fell strongly that is the way that you want Kafka to work it is entire possible to setup the system to work that way. -hans > On May 11,

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

ght.requests.per.connection > 1, then even if the topic only have > one partition, there’s still no guarantee of the ordering? > > Thanks, > Jun > On Apr 30, 2017, at 7:57 AM, Hans Jespersen <h...@confluent.io> wrote: > > There is a parameter that controls this beh

Re: Does Kafka producer waits till previous batch returns responce before sending next one?

There is a parameter that controls this behavior called max.in. flight.requests.per.connection If you set max.in. flight.requests.per.connection = 1 then the producer waits until previous produce requests returns a response before sending the next one (or retrying). The retries parameter

Re: How does replication affect kafka quota?

Replication will not effect the users quota as it is done under a different replication quota (which you can control separately). The user should still see a 50 MBps maximum rate enforced into each broker. -hans > On Apr 23, 2017, at 11:39 PM, Archie wrote: > >

Re: Re: Re: ZK and Kafka failover testing

ers​= {point the F5 VS fronting Kafka cluster} > >> > > key.serializer= {appropriate value as per your cases} > >> > > value.serializer= {appropriate value as per your case} acks= all > >> > > retries=3 > >> > > ssl.key.password=

Re: Re: ZK and Kafka failover testing

e as per your case} > > ssl.keystore.location= {appropriate value as per your case} > > ssl.keystore.password= {appropriate value as per your case} > > ssl.truststore.location= {appropriate value as per your case} > > ssl.truststore.password= {appropriate value as per you

Re: ZK and Kafka failover testing

in the kafka cluster. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Apr 18, 2017 at 4:10 PM, Shrikant Patel <spa...@pdxinc.com> wrote: > Hi All, > > I am seeing strange behavior between ZK and Kafka. We ha

Re: Kafka MTLS Support?

Are you asking about Multiplexed Transport Layer Security (MTLS) - https://en.wikipedia.org/wiki/Multiplexed_Transport_Layer_Security or Mutual TLS authentication (mTLS) https://en.wikipedia.org/wiki/Mutual_authentication

Re: Kafka producer and consumer within on sync execution

You posted the same question to Stack Overflow so I answered it there https://stackoverflow.com/questions/43302857/handling-sync-api-call-rest-spring-and-async-message-kafka-in-the-same-execu/43312070#43312070

Re: Kafka connector

nd N3 have different names. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee <tus...@levyx.com> wrote: > Hello Sir/Ma'am, > I was trying to write a simple case of using kaf

Re: How to increase network throughput of Kafka cluster?

t; On Sat, Apr 1, 2017 at 3:51 PM, Hans Jespersen <h...@confluent.io> wrote: >> >> What replication factor are you using? If you have a default replication >> factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 >> Gbps of network traffic. If you are

Re: How to increase network throughput of Kafka cluster?

What replication factor are you using? If you have a default replication factor = 3 then a publish rate of 1.4 Gbps is actually 1.4 Gbps *3 = 4.2 Gbps of network traffic. If you are also consuming at the same time then it’s actually 4.2 Gbps + 1.4 Gbps = 5.6 Gbps. You would completely

Re: Which is True? Kafka site vs Confluent 3.2 site upgrade doc details contradiction regarding 0.10.2 clients backward compatible to resp. 0.10.0 vs 0.10.1?

They are both true. The Apache text is talking about the compatibility of the Producer/Consumer API and the Confluent text is talking about the Streams API. -hans > On Mar 31, 2017, at 11:46 PM, Roger Vandusen > wrote: > > Read below and answer: So which is

Re: How to assign client-id to a particular kafka producer or topic?

username comes from authenticated clients client.id can be assigned by any client (no authentication required). It’s hard to enforce a quota on a client.id when the clients can just change the code to use a difference client.id, hence the recent enhancement to add user quotas based on the

Re: kafka not throwing any exception not any response in call back

In your producer have you set acks to be “all” and retries to be something higher than “0”? props.put("acks", "all"); props.put("retries", 2147483647); Also if the order of the messages matter you should also set the max inflight messages to 1 so retries happen and succeed before any other

Re: kafka is not accepting number of partitions from configuration

The num.partitions parameter is a server/broker config but you are using it as a client/producer parameter so it will not work and will be ignored. http://stackoverflow.com/questions/22152269/how-to-specify-number-of-partitions-on-kafka-2-8 I assume the CLI command you are using is the

Re: Out of order message processing with Kafka Streams

ndividual messages in a window > > Are those 2 things possible w/ kafka streams? (java) > > On Tue, Mar 21, 2017 at 7:43 PM, Hans Jespersen <h...@confluent.io> wrote: > >> While it's not exactly the same as the window start/stop time you can >> store (in the state store

Re: Out of order message processing with Kafka Streams

rent/streams/concepts.html#windowing >>>>>> >>>>>> >>>>>> On Mon, Mar 20, 2017 at 5:43 PM, Michael Noll < >> mich...@confluent.io> >>>>>> wrote: >>>>>> >>>>>>> Late-arriving and

Re: validate identity of producer in each record

an > extension point in Kafka itself on the receiving of records, before they are > stored/distributed? I was thinking along the lines of > > org.apache.kafka.clients.producer.ProducerInterceptor > > but on the server side? > > — m@ > >> On 21/03/2017, at 12:22 PM

Re: validate identity of producer in each record

work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin <apache@msqr.us> wrote: > Hello, > > I am new to Kafka and am looking for a way for consumers to be able to > ide

Re: Out of order message processing with Kafka Streams

. -hans > On Mar 18, 2017, at 11:36 AM, Sabarish Sasidharan <sabarish@gmail.com> > wrote: > > Hans > > What you state would work for aggregations, but not for state machines and > CEP. > > Regards > Sab > >> On 19 Mar 2017 12:01 a.m., "Hans Jes

Re: Out of order message processing with Kafka Streams

the added benefit of getting intermediary result at much lower latency. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 10:29 AM, Ali Akhtar <ali.rac...@gmail.com> wrote: > Is it possible to have

Re: Out of order message processing with Kafka Streams

sorry I mixed up Message A and B wrt the to question but the answer is still valid. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Sat, Mar 18, 2017 at 11:07 AM, Hans Jespersen <h...@confluent.io> wrote: > The only wa

Re: kafka-topics[.sh]: fail to support connecting via broker / v0.10 style

. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Fri, Mar 17, 2017 at 1:20 PM, Andrew Pennebaker < andrew.penneba...@gmail.com> wrote: > If I understand Kafka correctly, since v0.9 / v0.10, users are often >

Re: Performance and Encryption

You are correct that a Kafka broker is not just writing to one file. Jay Kreps wrote a great blog post with lots of links to even greater detail on the topic of Kafka and disk write performance. Still a good read many years later.

Re: Common Identity between brokers

This might be useful reading as it outlines why Cluster ID was added and lists a few ways that clusters can be identifies prior to that feature enhancement. https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id

Re: Kafka Retention Policy to Indefinite

the latest value for each key forever, but removes the older messages with the same key in order to reduce the total about of messages stored. How much data do you expect to store in your largest topic over the life of the cluster? -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent

Re: Kafka Retention Policy to Indefinite

in 0.10.1 -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Mar 14, 2017 at 10:09 AM, Joe San <codeintheo...@gmail.com> wrote: > Dear Kafka Users, > > What are the arguments against setting the retention plociy on

Re: Question on Metadata

You may also be interested to try out the new Confluent JMS client for Kafka. It implements the JMS 1.1. API along with all the JMS metadata fields and access methods. It does this by putting/getting the JMS metadata into the body of an underlying Kafka message which is defined with a special

  1   2   >