Re: [ Questions on log4j file & version ]

Hi Ashok, Kafka 2.7.1 was built from the 2.7.1 tag [1] and looking at the dependencies in that version [2], it should have shipped with 1.2.17. You can verify this by looking for the log4j jar in your installation. Because of the security vulnerabilities you mention, Kafka switched to reload4j in

[ Questions on log4j file & version ]

Hello Kafka Team, Request your help... We are using Apache Kafka kafka_2.13-2.7.1 & installed on a server. I understand it uses log4j java for logger purposes. But we don't know, what is the log4j version it is using? Recently we came to know that log4j_1.2.17 has some security issues, how to

RE: questions regarding the tiered storage feature

with Partner Manager. -Original Message- From: Christophe Kalenzaga [mailto:christophe.kalenz...@gmail.com] Sent: 17 October 2023 20:09 To: users@kafka.apache.org Subject: questions regarding the tiered storage feature Hi, Congratulation for the release of kafka 3.6 that contains a feature I

RE: questions regarding the tiered storage feature

Manager. -Original Message- From: Josep Prat [mailto:josep.p...@aiven.io.INVALID] Sent: 17 October 2023 20:38 To: Users Subject: Re: questions regarding the tiered storage feature Hi Christophe, We highly encourage the community to try Tiered Storage in their staging environments

RE: questions regarding the tiered storage feature

Manager. -Original Message- From: Josep Prat [mailto:josep.p...@aiven.io.INVALID] Sent: 17 October 2023 21:18 To: Users Subject: Re: questions regarding the tiered storage feature Hi Christophe, I'll check and let you know when we plan to have Azure support. I'd also encourage you

RE: questions regarding the tiered storage feature

Manager. -Original Message- From: Christophe Kalenzaga [mailto:christophe.kalenz...@gmail.com] Sent: 17 October 2023 21:10 To: users@kafka.apache.org Subject: Re: questions regarding the tiered storage feature Thanks for the answer, Joseph. Aiven's implementation was already in my TODO

RE: questions regarding the tiered storage feature

Manager. -Original Message- From: Christophe Kalenzaga [mailto:christophe.kalenz...@gmail.com] Sent: 17 October 2023 23:24 To: users@kafka.apache.org Subject: Re: questions regarding the tiered storage feature thanks Le mar. 17 oct. 2023 à 17:48, Josep Prat a écrit : > Hi Christo

Re: questions regarding the tiered storage feature

thanks Le mar. 17 oct. 2023 à 17:48, Josep Prat a écrit : > Hi Christophe, > > I'll check and let you know when we plan to have Azure support. I'd also > encourage you to raise a ticket in the repo so we can track interest as > well. > > But I can tell you that Azure is quite up in the priority

Re: questions regarding the tiered storage feature

Hi Christophe, I'll check and let you know when we plan to have Azure support. I'd also encourage you to raise a ticket in the repo so we can track interest as well. But I can tell you that Azure is quite up in the priority list. Best, ——— Josep Prat Aiven Deutschland GmbH Alexanderufer 3-7,

Re: questions regarding the tiered storage feature

Thanks for the answer, Joseph. Aiven's implementation was already in my TODO list . I also saw in the README that you plan to add an Azure implementation. Do you know when you plan to have it (I'm working for a company that uses both AWS & Azure)? For the tests & benchmarks, I guess we'll use

Re: questions regarding the tiered storage feature

Hi Christophe, We highly encourage the community to try Tiered Storage in their staging environments and report any deficiencies they might encounter. That being said, at Aiven to develop our S3 and GCS plugin [1] we did some extensive testing and we will be enabling it to our service offering

questions regarding the tiered storage feature

Hi, Congratulation for the release of kafka 3.6 that contains a feature I was waiting for many years! Speaking of the devil, the tiered storage is currently in early access. When do you expect it to be production ready? I know it's not an easy question since the early access was just released,

Re: Some questions on Kafka on order of messages with mutiple partitions

>>> > >>>>> > >>>>> https://en.everybodywiki.com/Mich_Talebzadeh > >>>>> > >>>>> > >>>>> > >>>>> *Disclaimer:* Use it at your own risk. Any and all re

Re: Some questions on Kafka on order of messages with mutiple partitions

mich.talebza...@gmail.com >>>>> >>>>> wrote: >>>>> >>>>>> This json file seemed to work >>>>>> >>>>>> cat reduce_replication_factor.json >>>>>> { >>>>>> "ver

Re: Some questions on Kafka on order of messages with mutiple partitions

ot;partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}, > > > >> {"topic":"md","partition":3,"replicas":[1,12,9,11,7,3,10,8,2]}, > > > >> {"topic":"md","partition":4,"replicas":[7,9,11,1,3,10,

Re: Some questions on Kafka on order of messages with mutiple partitions

hes76:9092,rhes76:9093,rhes76:9094,rhes76:9095,rhes76:9096, > > >> rhes76:9097 --reassignment-json-file ./reduce_replication_factor.json > > >> --execute > > >> > > >> The output > > >> > > >> Successfully started partition reassig

Re: Some questions on Kafka on order of messages with mutiple partitions

started partition reassignments for > >> md-0,md-1,md-2,md-3,md-4,md-5 > >> > >> > >> I guess it is going to take sometime before it is completed. > >> > >> Thanks > >> > >> > >> > >> > >> On F

Re: Some questions on Kafka on order of messages with mutiple partitions

t;>> >>> with regard to your point below: >>> >>> A replication factor of 9 sounds very high. For production, a replication >>> factor of 3 is recommended. >>> >>> Is it possible to dynamically reduce this number to 3 when the topic is >>

Re: Some questions on Kafka on order of messages with mutiple partitions

way. > > > "Yes" to all other questions. > > > -Matthias > > > > On 5/12/23 9:50 AM, Mich Talebzadeh wrote: > > Hi, > > > > I have used Apache Kafka in conjunction with Spark as a messaging > > source. This rather dated diagram describes

Re: Some questions on Kafka on order of messages with mutiple partitions

topic is >> actively consumed)? >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly

Re: Some questions on Kafka on order of messages with mutiple partitions

ry damages arising from > such loss, damage or destruction. > > > > > On Fri, 12 May 2023 at 19:38, Matthias J. Sax wrote: > >> > Does having 9 partitions with 9 replication factors make sense here? >> >> A replication factor of 9 sounds very high. For production,

Re: Some questions on Kafka on order of messages with mutiple partitions

question, and cannot be > answered in a general way. > > > "Yes" to all other questions. > > > -Matthias > > > > On 5/12/23 9:50 AM, Mich Talebzadeh wrote: > > Hi, > > > > I have used Apache Kafka in conjunction with Spark as a messaging &g

Re: Some questions on Kafka on order of messages with mutiple partitions

all other questions. -Matthias On 5/12/23 9:50 AM, Mich Talebzadeh wrote: Hi, I have used Apache Kafka in conjunction with Spark as a messaging source. This rather dated diagram describes it I have two physical hosts each 64 GB, running RHES 7.6, these are called rhes75 and rhes76 re

Some questions on Kafka on order of messages with mutiple partitions

1,8 Topic: md Partition: 8Leader: 2 Replicas: 2,3,10,8,12,9,11,1,7 Isr: 10,1,9,2,12,7,3,11,8 However, I have a number of questions 1. Does having 9 partitions with 9 replication factors make sense here? 2. As I understand the parallelism is equal to the num

Re: Questions about creating jira account

Hi Jimmy, You can create an account request via https://selfserve.apache.org/jira-account.html Thank you. Luke On Sat, Mar 18, 2023 at 12:03 AM zw wrote: > Hi, > It seems that JIRA access are disabled by default and I can't create an > account by myself. > > I would appreciate that if anyone

Re: Apache Kafka Questions

Hello Team Thank you for answering the questions I reached out earlier. We have a new set, and I would really like to have them verified and confirmed by an expert. Request to answer them with specification in a way they can be replicated in local user environment. f. How are Consumer Groups

Re: Apache Kafka Questions

From: Jatin Chhabriya Date: Wednesday, March 16, 2022 at 9:20 AM To: users@kafka.apache.org Cc: Murali Krishna Subject: [EXTERNAL] Apache Kafka Questions Hello Team Upon careful perusal of documentation and tutorials, our team has a few open questions, and we would appreciate to have

Apache Kafka Questions

Hello Team Upon careful perusal of documentation and tutorials, our team has a few open questions, and we would appreciate to have these clarified by a hands-on practitioner. a. Can there by more than one Producer in a Publish-Subscribe model? b. More details on level of abstraction between

Re: Questions around static membership partition assignment

good, no rebalance will be triggered. > > So, answering your questions below: > > 1. The desired behavior is partitions should be balanced eventually (but > then it conflicts with "no rebalance" nature of static membership with the > healthy backbone). Could you point

Re: Questions around static membership partition assignment

the consumer group to rebalance twice (once when old pods down, and once after new pods up). We hold the rebalance, and after the new pods up, we check everything is good, no rebalance will be triggered. So, answering your questions below: 1. The desired behavior is partitions should be balanced eventually

Questions around static membership partition assignment

there are total 10 partitions and 2 pods. 8 partitions are assigned one of them and 2 partitions go to the other), it will be unbalanced forever because pods are kept healthy by k8s and no rebalancing ever occurs. And I don't think it is the desired behavior. My questions are 1. The desired

Re: Kafka Security Questions

You can read about the security available in Apache Kafka here: https://kafka.apache.org/documentation/#security_ssl -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 16 Feb 2021 at 15:18, Jones, Isaac wrote: > Hello, > > I have a couple important

Kafka Security Questions

Hello, I have a couple important questions regarding kafka and its security. 1. Is data encrypted in transit when streaming in kafka? 2. How does one endpoint get authenticated before data is sent to it? If someone can answer/explain this to me that'd be great. Isaac Jones Full Stack

kafka schema registry - some queries and questions

Hi All, Wanted to understand a bit more on the schema registry 1. With Apache kafka , can we use schema registry 2. Amazon MSK , can we use Schema registry ? Thanks Manoj A This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain

Re: kafka schema registry - some queries and questions

Hi Pushkar, This question is better suited for https://groups.google.com/g/confluent-platform since the Schema Registry is part of the Confluent Platform but not of Apache Kafka. Best, Bruno On 21.09.20 16:58, Pushkar Deole wrote: Hi All, Wanted to understand a bit more on the schema

kafka schema registry - some queries and questions

Hi All, Wanted to understand a bit more on the schema registry provided by confluent. Following are the queries: 1. Is the schema registry provided by confluent over the top of Apache Kafka? 2. If a managed kafka service is used in cloud e.g. say Aiven Kafka, then does the schema registry

Questions around Time based retention policy

Hi All, I have a few questions around time based retention policy. It will be really helpful if you can answer. 1. Does Kafka use CreateTime for building its time based indices? Does CreateTime have any role in deciding time based log rotation ? 2. If CreateTime is provided by the clients

Concept and implementation questions

Hi, I have some questions about kafka concept, Thanks in advance for your answer: 1.I want when kafka are deleting kafka's partition (because it achieved to its time or max volume) it send me a notification and says that "I was deleted offset 22 od topic a and partition 2"

Want to ask three questions about the connector to kafka!

background: I want to use the flink to get the log stream once and according to different demand to add or remove some params in each line from kafka, then distribute to different kafka cluster or different topic. questions: The attachment is the simply example I used to test distribute

Re: Some questions about copyright

Hi Michael, You can find the ASF trademark policies here: https://www.apache.org/foundation/marks/#books -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Sun, 23 Feb 2020 at 19:47, Michi Bertschi wrote: > Hello > I'm Michael > > > I am writing a thesis for the

Some questions about copyright

Hello I'm Michael I am writing a thesis for the evaluation of Kafka the right one for our new project. I wanted to ask if I can use the Kafkalogo in my diploma thesis? If so, which regulations do you have what it should look like? I look forward to your answer Regards michael

Some questions about kafka/kafka-connect

Hello all, We’ve been testing out kafka-connect as a possible way of streaming data and have run into some issues. I am a kafka/kafka connect newbie so hopefully these aren’t too basic. I have listed a couple questions below. Thank you 1. Currently our connected-distributed.properties

Re: [External] Mirror Maker Questions

For 1), MM2 will work with older versions of Kafka. I've gotten it to work with clusters as old as 0.10.2 but with some features disabled iirc. Ryanne On Thu, Feb 20, 2020, 2:49 AM Dean Barlan wrote: > Hi everyone, > > I have a few small questions for you regarding MirrorMaker. >

Re: [External] Mirror Maker Questions

On #2, you can provide an implementation of a MirrorMakerMessageHandler that will called for each record - you ensure it's in the classpath and pass the class name to MM using --message.handler. On Thu, 20 Feb. 2020, 9:49 pm Dean Barlan, wrote: > Hi everyone, > > I have a few small

[External] Mirror Maker Questions

Hi everyone, I have a few small questions for you regarding MirrorMaker. 1. I know that MirrorMaker 2.0 is only available starting with Kafka version 2.4. Does that mean that if I was mirroring from cluster A to cluster B, that both clusters need to be running Kafka 2.4? 2. For MirrorMaker

ops questions MM2

Got some operation questions for MM2. 1. Whats is the best practice way to start MM2 after a reboot of host? Add connect-mirror-maker.sh config/connect-mirror-maker.properties to a systemd script to run after kafka starts? 2. We will use MM2 to mirror primary to a backup(secondary) cluster

Re: Questions about Producer per Task/Partition in Streams EoS Impl.

n in > > `TaskCreator` a `createProducer` method is defined which will create a > new > > `KafkaProducer` per task id when the `threadProducer` is null [3]. This > > behaviour seems to be confirmed with the > > `shouldInjectProducerPerTaskUsingClientSupplierOnCreateIfEosEn

Re: Questions about Producer per Task/Partition in Streams EoS Impl.

cer` is null [3]. This > behaviour seems to be confirmed with the > `shouldInjectProducerPerTaskUsingClientSupplierOnCreateIfEosEnable` test > [4]. > > I assume this just a case of the design doc falling out of sync with the > impl., or maybe I misunderstood its original meaning. Either way, I still > have some q

Questions about Producer per Task/Partition in Streams EoS Impl.

confirmed with the `shouldInjectProducerPerTaskUsingClientSupplierOnCreateIfEosEnable` test [4]. I assume this just a case of the design doc falling out of sync with the impl., or maybe I misunderstood its original meaning. Either way, I still have some questions: 1. It's my understanding that a `TaskId` is 1:1 with a partition, so is it the case that t

Re: Questions for platform to choose

C Hi Eliza, Kafka Streaming, Spark Streaming, Flink and Storm are all good. They also all have their caveats. It's really hard to say that X is the best. For example, Kafka Streaming can't read from one Kafka cluster and write to another, but Spark can. But then Spark offers two flavours of

Questions for platform to choose

Hello, We have all of spark, flink, storm, kafka installed. For realtime streaming calculation, which one is the best above? Like other big players, the logs in our stack are huge. Thanks.

Re: Questions about KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hello Shurong, What you bumped into seems to be the same issue as is reported tracked by Luying Liu here: https://issues.apache.org/jira/browse/KAFKA-8676. Guozhang On Thu, Jun 27, 2019 at 10:48 AM srlin wrote: > Hi, team, > > We are using Kafka Connect at present and have encountered a

Questions about KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hi, team, We are using Kafka Connect at present and have encountered a problem with version 2.1.0 that all connectors kept restarting when one new connector was added into the cluster which then failed to start due to some network problem (firewall not open). And the Connect daemon failed to

Re: Kafka ISR to server count questions

ion factor is 2 and the min.isr is 2 as well, you can not afford to lose one Broker. I guest your producers are using ack = all or -1 Hope that helps. Cheers! -- Jonathan On Wed, Jun 5, 2019 at 7:25 PM Hamer, Scott wrote: > I've have some questions regarding Kafka ISR's and the

Kafka ISR to server count questions

I've have some questions regarding Kafka ISR's and the number of servers one should have. In our production environment we have a 6 node Kafka cluster with min.insync.replicas=2, and all of our topics have a ReplicationFactor: of 2 When we create topics, we do not pin them to a specific

Re: Questions on Exactly Once Semantics

confluent.io > <mailto:matth...@confluent.io>) wrote: > >> Done. Feel free to extend/correct/complete etc. >> >> -Matthias >> >> On 2/20/19 9:56 AM, Guozhang Wang wrote: >> > Since we've seen quite a lot of questions recently about EOS on the >

Re: Questions on Exactly Once Semantics

. Thanks On 20 February 2019 at 23:57:17, Matthias J. Sax (matth...@confluent.io) wrote: Done. Feel free to extend/correct/complete etc. -Matthias On 2/20/19 9:56 AM, Guozhang Wang wrote: > Since we've seen quite a lot of questions recently about EOS on the > mailing list. I think it worth

Re: Questions on Exactly Once Semantics

Since we've seen quite a lot of questions recently about EOS on the mailing list. I think it worth adding an FAQ entry here: https://cwiki.apache.org/confluence/display/KAFKA/FAQ So that we can refer future questions to the page than answering them repeatedly. @Matthias J Sax : would you like

Re: Questions on Exactly Once Semantics

nsactionl.id comes > into picture which is user defined and hence can survive producer restarts. > > I have few questions regarding the same: > > 1. If the above statement is correct, why do we need PID in the first place > and instead use transactionl.id all over? >

Questions regarding Exactly-Once semantics

the same / single producer session for exactly-once guarantees? I presume it is not required. As per my understanding, this is where transactionl.id comes into picture which is user defined and hence can survive producer restarts. I have few questions regarding the same: 1. If the above statement

Questions - Exactly Once Semantics

the same / single producer session for exactly-once guarantees? I presume it is not required. As per my understanding, this is where transactionl.id comes into picture which is user defined and hence can survive producer restarts. I have few questions regarding the same: 1. If the above statement

Questions on Kafka Exactly Once Semantics

the same / single producer session for exactly-once guarantees? I presume it is not required. As per my understanding, this is where transactionl.id comes into picture which is user defined and hence can survive producer restarts. I have few questions regarding the same: 1. If the above statement

Questions on Exactly Once Semantics

the same / single producer session for exactly-once guarantees? I presume it is not required. As per my understanding, this is where transactionl.id comes into picture which is user defined and hence can survive producer restarts. I have few questions regarding the same: 1. If the above statement

Can anyone help me with these questions on kafka?

https://stackoverflow.com/questions/54039216/how-come-kafka-fails-to-commit-offset-for-a-particular-partition https://stackoverflow.com/questions/54020753/why-is-kafka-producer-perf-test-sh-throwing-error -- Thanks, Ankur Rana Software Developer FarEye

Re: Questions about manage offset in external storage and consumer failure detect

is addressed via https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior (available in version 2.0) -Matthias On 9/18/18 7:58 PM, ???? wrote: > Hi, > > I have two questions about Kafka consumer api: > > > 1. I read the section

Questions about manage offset in external storage and consumer failure detect

Hi, I have two questions about Kafka consumer api: 1. I read the section "Storing Offsets Outside Kafka" in jdoc of KafkaConsumer. Is it safer or more efficient to manage offset with external storage compared to manage with Kafka(Zookeeper or topic _consumer_offset)? Or is there an

Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions

Questions The release plan: https://url.emailprotection.link/?aqtDT9nGZlDfQ6CDagckDXG_qA-bYuc0oOhrPxAJMZ8V49Mx-5R9QbkoG-ohfO3xR7MyCRpUWi-rh1Kp61WUZMr-Xw7nXKVaUIA47JtgMw67rm6OX5fAeAX74xZRRyVgE Ismael On Wed, Sep 12, 2018 at 7:51 AM Jeremiah Adams wrote: > Thanks Ismael, > > Is there a rough

Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions

gt; Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Ismael Juma > Sent: Tuesday, September 11, 2018 6:04 PM > To: Kafka Users > Subject: Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions > > H

Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions

: Re: Java 11 OpenJDK/Oracle Java Release Cadence Questions Hi, Information about Java 11 support: https://url.emailprotection.link/?aXbNFujPDP4Cfs-6OQ7jlf8T9J_NulRd7SGvkTYwMFSBSs8xuTBKzTMBG5jwB2lC5SehEZ9nI4Ie-EKOjkgegrw~~. In the short/medium term, we expect to support Java 8, Java 11

Java 11 OpenJDK/Oracle Java Release Cadence Questions

Hello, Are there any documents available concerning Kafka's support for Java 11 and documents regarding models supporting Oracle's new licensing and release cadences? Thanks.?? Jeremiah Adams Software Engineer www.helixeducation.com

Re: Questions about state stores and KSQL

To understand joins better, you might want to check out: https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/ KSQL uses the same join semantics as Kafka Streams. -Matthias On 7/11/18 8:01 AM, Guozhang Wang wrote: > Hello Jonathan, > > At the very high-level, KSQL statements is

Re: Questions about state stores and KSQL

Hello Jonathan, At the very high-level, KSQL statements is compiled into a Kafka Streams topology for execution. And the concept "state stores" are for Kafka Streams, not for KSQL, where inside the topology for those processor nodes that need stateful processing, like Joins, one or more state

Questions about state stores and KSQL

Hi Kafka users, I am very new to Kafka and more globally to stream processing, and am trying to understand some of the concepts used by Kafka. From what I understand, a key-value state store is created on each processor node that performs stateful operations such as aggregations or joins.

Re: message queueing questions?

Hi, multiple answers to this question: 1) it depends of you send messages sync or async to the brokers. Producers do buffer messages in-memory for more efficient writes to the brokers. If messages are successfully sent to the brokers, you can get an acknowledgment back the you can check on the

message queueing questions?

Can someone explain how message queueing in partition(s) work? Specifically: Is there any buffering before message is committed to log (written to disk) ? Does the term "partition" refer to sequence of messages written to disk, or are the messages stored in memory queue and written to disk only

Re: Questions regarding undesired offset reset in Kafka Streams Application

Matthias, Thank you for the quick response. I was able to verify that was the problem. I had seen similar solutions but thought that the expiration would not happen with an active consumer group (but it makes sense considering it is just another topic). I appreciate the help and timely response

Re: Questions regarding undesired offset reset in Kafka Streams Application

Jordon, not sure, if this applies to your situation, but brokers only maintain committed offsets for 24h by default. This offset-retention time is applied for each partitions individually and starts when the commit was done (ie, offset can expired even if the consumer group is active). Thus, if

Questions regarding undesired offset reset in Kafka Streams Application

Hello, I am seeing an issue where I have a single streams app running (so a consumer group of one) that is subscribed to about 10 topics. If the streams app gets killed and restarted, many of the offsets for the consumer group are reset to 0 and a lot of data is unintentionally reprocessed. The

questions about replication factor of __consumer_offsets topic

We noticed that the replication factor is six for *__consumer_offsets* and as we have been moving consumers over from zk to kf offsets latency for FetchFollower on the group coordinator for a busy consumer_group increases significantly (in hundreds of ms), but offset commit latency and cpu

Re: Questions about upgrading to Kafka 1.0 from 0.10.0

I understand it now. I must've done something wrong last time. Thank you. On Wed 22 Nov, 2017, 5:21 PM Ismael Juma, wrote: > Anish, > > That's correct, the broker will down convert messages for older consumers > after log.message.format.version is increased. As I said,

Re: Questions about upgrading to Kafka 1.0 from 0.10.0

Thanks Ismael. Just need a clarification on something because I observed getting errors from the v0.9 and v0.10 consumer for invalid message format. Is it true that after bumping the consumer version post rolling upgrade will not cause message format mismatch errors in the Consumer? On Tue, Nov

Re: Questions about upgrading to Kafka 1.0 from 0.10.0

Hi Anish, The documentation is a bit misleading, see the following JIRA: https://issues.apache.org/jira/browse/KAFKA-6238 All of your clients will still work after the upgrade, but there is an efficiency hit if the message format used for the topic is newer than the message format supported by

Questions about upgrading to Kafka 1.0 from 0.10.0

Hello Kafka users! The first question that I have is related to the documentation. I see that we no longer have to change the message format version when upgrading to 1.0. So, will all clients continue to work after performing the rolling upgrade? We are running Kafka v0.10.0.0. The Kafka

Re: Questions about kafka-consumer-groups output

-started-with-the-new-apache-kafka-0-9-consumer-client/ --Vahid From: Michael Scofield <bayinam...@gmail.com> To: users@kafka.apache.org Date: 11/09/2017 10:43 PM Subject: Questions about kafka-consumer-groups output Hello all: I’m using Kafka version 0.11.0.1, with the ne

Questions about kafka-consumer-groups output

109929174 109929190 16 -- - foo2 109929222 109929240 18 -- - foo1 109929004 109929023 19 -- - I have 2 questions regarding the output above: 1. What does

Questions about Apache Kafka messages type/size and publish/subscribe

Hi, I'm not entirely certain this is the right place to ask, but I have questions about the functioning of Apache Kafka to implement a publish/subscribe messaging system. I am investigating Kafka to see if it fits the needs of the company I work for, and I have quite a few questions I can't find

Re: Few questions about how Kafka Streams manages tasks

I see. For normal maintenance operations, before you kill your container you could shuts down the Streams application by calling `KafkaStreams#close()`. Upon shutting down it would write a local checkpoint file indicating at which point in terms of offsets it has stopped at. So on resuming if the

Re: Few questions about how Kafka Streams manages tasks

Hi Guozhang, Thanks for the reply. By taking a lot of time I meant that I see a log message `Restoring state from changelog topics `, followed by just some kafka consumer logs like `Discovered coordinator`. Looking at this I assumed that the Stream threads are waiting for the states to be

Re: Few questions about how Kafka Streams manages tasks

Hi Anish, 1. The committed offsets are for the messages that have been through the entire pipeline. Plus when we are committing, we make sure all state store caches are flushed so there should be no messages that are "in the middle of the topology". If there is a failure before the commit, then

Few questions about how Kafka Streams manages tasks

First question: We know that Kafka Streams commits offsets on intervals. But what offsets are committed? Are the offsets for messages committed are the ones which have just arrived at the source node? Or the messages that have been through the entire pipeline? If the latter, how do we avoid data

Re: Kafka streams questions

1) > > .. > > KafkaStreams streams = new KafkaStreams(blah, props) > > > Thanks, > > > Neil > > > From: Matthias J. Sax <matth...@confluent.io> > Sent: 28 February 2017 22:26:39 > To: users@kafka.apache.or

Re: Kafka streams questions

reams(blah, props) Thanks, Neil From: Matthias J. Sax <matth...@confluent.io> Sent: 28 February 2017 22:26:39 To: users@kafka.apache.org Subject: Re: Kafka streams questions Adding partitions: You should not add partitions at runtime -- it might break the s

Re: Kafka streams questions

va program that manipulate the offsets before you start your Streams application. However, be aware the this will result in duplicate processing as there is currently no way to reset your state stores. -Matthias On 2/28/17 1:31 PM, Neil Moore wrote: > Hello, > > > I have a few question

Kafka streams questions

Hello, I have a few questions that I couldn't find answers to in the documentation: * Can added partitions be auto-discovered by kafka-streams? In my informal tests I have had to restart the stream nodes. * Is it possible to rewind the consumer for a particular topic-partitions. e.g

Re: Upgrade questions

upgrades and are clearly described in the docs you already linked: http://kafka.apache.org/0100/documentation.html#upgrade -Ewen On Thu, Jan 26, 2017 at 3:22 PM, Fernando Vega <fernando.v...@turn.com> wrote: > I have a few questions regarding an upgrade that Im attempting t

Upgrade questions

I have a few questions regarding an upgrade that Im attempting to perform. - Currently we are running version 0.8.2_beta. - We are trying to upgrade to 10.1.1 - Our setup uses the following path /server/kafka where kafka is a symlink to kafka-{version} - I attempt to perform the upgrade

Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question Does the number of App instances and Zookeeper servers should be the same?

Does the number of App instances and Zookeeper servers should be the same? I understand the requirement of 2F+1 to tolerate F failures but this is to tolerate failures of Zookeeper instances itself. But how about the number of App instances ? For example say I have 3 zookeeper servers and I have 2

Re: Streams: Global state & topic multiplication questions

Thank you both for the directions, I'll dive into these. Peter On Jan 20, 2017 9:55 AM, "Michael Noll" wrote: > As Eno said I'd use the interactive queries API for Q2. > > Demo apps: > - > https://github.com/confluentinc/examples/blob/3. >

Re: Streams: Global state & topic multiplication questions

As Eno said I'd use the interactive queries API for Q2. Demo apps: - https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java -

Re: Streams: Global state & topic multiplication questions

For Q2: one way to export the state on demand would be to use the Interactive Queries API (https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/ ).

  1   2   3   4   >