Re: Architecture recommendations for a tricky use case

Kafka into spark streaming and that will be online or near real time (defined by your window). Then you have a a serving layer to present data from both speed (the one from SS) and batch layer. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Architecture recommendations for a tricky use case

Tableau or Zeppelin to query data You will also need spark streaming to query data online for speed layer. That data could be stored in some transient fabric like ignite or even druid. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Architecture recommendations for a tricky use case

Hi Ali, What is the business use case for this? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Re: Architecture recommendations for a tricky use case

s well and from Flume to Hbase I would have thought that if one wanted to do real time analytics with SS, then that would be a good fit with a real time dashboard. What is not so clear is the business use case for this. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/v

Re: Architecture recommendations for a tricky use case

Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Setting up Kafka with zookeeper

pics. Wouk there be one set of partitions for each topic, say for two topics there will be 6 patitions? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV

3 node Kafka cluster with 3 node ZK ensemble

and on a new version of Kafka say 0.9 or above the offsets are not stored in ZK anymore. In contrast ZK uses random access which tends towards faster disk? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <ht

RAID10 for kafka cluster

is RAID 10 (mirrored and striped) essential for kafka cluster with three nodes in DEV as I have been told that disk failures can shutdown the whole kafka clusters? this is a non-prod environment thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

What is viable hardware for Zookeeper and Kafka brokers?

) and I am wondering if it refers to Kafka < 0.8). However, the calculation's seems to be plausible. Any comments on it please? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com

Re: kafka instalation

Hi Elahe, Did you unzip KAFKA binaries and installed it? Have you set up $KAFKA_HOME like below cd $KAFKA_HOME /usr/lib/kafka> ls LICENSE NOTICE bin config libs logs site-docs Can you see directories? What is missing please. HTH, Dr Mich Talebzadeh LinkedIn * ht

Schema changes in Kafka and impact on topic

etc ? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. An

Re: Accessing Zookepper and Kafka through designated ports in a container

okerid":1,"timestamp":"1531512017622"} cZxid = 0x170 ctime = Fri Jul 13 21:00:17 BST 2018 mZxid = 0x170 mtime = Fri Jul 13 21:00:17 BST 2018 pZxid = 0x170 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x164953a1010 dataLength = 54 numChildren =

Re: Accessing Zookepper and Kafka through designated ports in a container

ete on server rhes75/50.140.197.220:4300, sessionid = 0x164953a101a, negotiated timeout = 3 WATCHER:: WatchedEvent state:SyncConnected type:None path:null Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linked

Accessing Zookepper and Kafka through designated ports in a container

test with key: 55 bytes, value: 11 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for test-0: 1542 ms has passed since batch creation plus linger time I have been working on this for a day or two and

Re: Real time streaming as a microservice

Hi Deepak, I will put it there once all the bits and pieces come together. At the moment I am drawing the diagrams. I will let you know. Definitely everyone's contribution is welcome. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Real time streaming as a microservice

eeper). For Hbase I specified a zookeeper instance running on another host and Hbase works fine. Anyway I will provide further info and diagrams. Cheers, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://

Re: Accessing Zookepper and Kafka through designated ports in a container

to authenticate using SASL (unknown error) 2018-07-14 21:08:42,300 [myid:] - WARN [main-SendThread(rhes75:3400):ClientCnxn$SendThread@1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com

Re: Real time streaming as a microservice

Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any a

Re: Accessing Zookepper and Kafka through designated ports in a container

resolved this now Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it a

Re: Zookeeper and Kafka cluster docker issue

. (org.apache.kafka.clients.NetworkClient) Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is mapped to host port 9094 Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

Re: Zookeeper and Kafka cluster docker issue

that the leader will remain the same until in this case Broker 0 fails? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Re: Zookeeper and Kafka cluster docker issue

Replicas: 2,0,1 Isr: 2,0,1 Topic: finalPartition: 2Leader: 0 Replicas: 0,1,2 Isr: 0 Surely something is not working here although there reports that the problem goes away when brokers are restarted! Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com

Zookeeper and Kafka cluster docker issue

ader for that topic-partition. I believe these Kafka brokers have problem talking to each other and the message is lost! Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linked

Re: Zookeeper and Kafka cluster docker issue

with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it

Re: Real time streaming as a microservice

8dd84a174834ubuntu "bash" 19 hours agoUp 11 hours dockerZooKeeperKafka What would be an option to add a fixed port to the running container.? Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profi

Real time streaming as a microservice

properties)) So in summary some changes need to be made to Flink to be able to interact with the new version of Kafka. And more importantly if one can use an abstract notion of microservice here? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxia

Re: [External] Re: Cannot run the inbuild zookeeper server.

/hduser/zookeeper-3.4.6/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /home/hduser/zookeeper-3.4.6/bin/../conf/zoo.cfg 3) confirm it jps|grep 4226 4226 QuorumPeerMain HTH, Dr Mich Talebzadeh LinkedIn

Re: Real time streaming as a microservice

(each with sub-services) providing three components? Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebza

Re: Need info

Hi James, As a matter of interest this streaming data is fed into some Operational Data Store ODS) like MongoDB? In general using this method will create a near real time snapshot for business users and customers. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id

Re: Need info

on the architecture you have created and your Kafka cluster plus Zookeeper ensemble.), you will be constraint by the rate that you will be allowed to extract data from Mainframe and your shared network. Also for Kafka and Zookeeper you may think about deploying these as microservices. HTH, Dr Mich

Re: Streaming Data

but much simplified [image: image.png] HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer

Issue facing, Aerospike Connect for Kafka - Inbound

I generate a simple pricing as follows: echo "${UUID}:${TICKER}" >> ${IN_FILE} UUID is the key and ticker is the name of security like “IBM” This is what I have in my shell script cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh \ --broker-list

Re: Issue facing, Aerospike Connect for Kafka - Inbound

Hi, I sorted this one out. It was to do with the format of published prices. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Re: Topic migration patterns

Hi Elliot, As you are moving the topic from one cluster to another, I assume it implies a new zookeeper ensemble plus sets of new brokers? Can you describe the current topic? ${KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper , , --topic HTH Dr Mich Talebzadeh LinkedIn * https

Re: Topic migration patterns

it will be involved unless someone else has better ideas. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*

Re: Kafka Topics

Hi Ashock, Check this article of mine Real Time Processing of Trade Data with Kafka, Flume, Spark, Hbase and MongoDB https://www.linkedin.com/pulse/real-time-processing-trade-data-kafka-flume-spark-talebzadeh-ph-d-/ under *Kafka setup* HTH Dr Mich Talebzadeh LinkedIn * https

Re: kafka connection from docker

will use the host port as its own. Indeed other mappings will be discarded. You can either map dockers to host port or use other posts but not both! HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.

Re: Is there any difference between conflunt kafka and apache kafka？

around for 10 years. It is only supported as part of third-party suits of products on Hadoop like Cloudera or Hortonworks. The great Apache community supports it and I believe that is the reason it is so successful and widely used. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com

Re: Is there any difference between conflunt kafka and apache kafka？

Sure we agree to differ. That is indeed a sign of civilised mind to entertain an idea without accepting it. HTH, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view

Re: Client session timed out

on, 19 Oct 2020 at 17:11, Sabina Marx wrote: > Thanks for your answer, but the zookeeper ensemble is running and ports > are ok. > > Am 19.10.20, 17:38 schrieb "Mich Talebzadeh" : > > Start the zookeeper ensemble first before starting Kafka cluster. They > nee

Re: Client session timed out

Start the zookeeper ensemble first before starting Kafka cluster. They need to select a leader and ensure that they all come online OK. Check port 2181, 2888, 3888 using netstat -plten|egrep '2181|2888|3888' tcp0 0 :::2181 :::* LISTEN 1005 9934134

Re: Client session timed out

erything and start it, then it works. But then, if I > restart one server with zookeeper and kafka the kafka gets the timeouts. > > Am 19.10.20, 18:16 schrieb "Mich Talebzadeh" : > > OK so the issue seems to be kafka cluster. Is this a new setup? > > HTH > >

Re: Client session timed out

struction. On Mon, 19 Oct 2020 at 18:17, Sabina Marx wrote: > Yes, you have it > > Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef> > ____ > Von: Mich Talebzadeh > Gesendet: Monday, October 19, 2020 7:09:53 PM > An: users@kafka.apache.org >

Re: Client session timed out

lt;https://aka.ms/o0ukef> > ________ > Von: Mich Talebzadeh > Gesendet: Monday, October 19, 2020 6:46:49 PM > An: users@kafka.apache.org > Betreff: Re: Client session timed out > > Can you pls clarify when you say you start one pair (Zookeeper and K

Re: Apache Kafka project CLA requirement

Hi Radoslava, I believe you can get your answers from here ASF Contributor Agreements (apache.org) specifically I refer you to this sentence and I quote: "The ASF desires that all contributors of ideas, code, or documentation to any

Re: Help !! Apache-Kafka On-Premises Setup.

to Setup a Public Kafka Broker Using a Dynamic DNS? <https://stackoverflow.com/questions/53705573/how-to-setup-a-public-kafka-broker-using-a-dynamic-dns> HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk.

Re: Help !! Apache-Kafka On-Premises Setup.

Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disc

Re: Kafka S3 Connector: Sort by a field within a partition

Ok thanks for the info. One question I forgot to ask is what is the streaming interval that the source is sending messages to Kafka to be processed inside SSS? For example are these market data etc? HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-520

Re: Kafka S3 Connector: Sort by a field within a partition

view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical conten

Re: Kafka S3 Connector: Sort by a field within a partition

e enriched data from SSS? Any other reason except hourly partition of your data? HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or an

Re: Kafka S3 Connector: Sort by a field within a partition

and then go from there. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's tec

Re: Spark Streams vs Kafka Streams

trary operations and custom logic on the output of each micro-batch*. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other propert

Re: Kafka Connectors output to topic.

of choice, Python, Scala, Javas etc and then feed data to sink. It handles json data as well. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction o

Error in Kafka property file contains no connector type

Hi, Trying to stream from Kafka to Google BigQuery. The connect-standalone.properties is as follows key.converter=org.apache.kafka.connect.storage.StringConverter ##value.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.kafka.connect.json.JsonConverter

Re: Error in Kafka property file contains no connector type

: > I feel your CLASSPATH env var might need to include $KAFKA_HOME/lib (it > should already have the common conf lib in there). > > > On Fri, 12 Mar. 2021, 10:33 pm Mich Talebzadeh, > > wrote: > > > Thanks Liam for the suggestion. > > > > This is the red

Re: Kafka Connectors output to topic.

so you want to do ETL on the input data Kafka source (topic 1) -> Processing -> Another Kafka sink (topic2) LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Error in Kafka property file contains no connector type

is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 12 Mar 2021 at 14:44, Mich Talebzadeh wrote: > Thanks again Liam. > > This is the error > > [2021-03-12 14:17:54,670] ERROR Stopping after

Re: Error in Kafka property file contains no connector type

liable for any monetary damages arising from such loss, damage or destruction. On Fri, 12 Mar 2021 at 20:33, Mich Talebzadeh wrote: > Ok this try to stream kafka topic to BigQuery has moved on but I am now > getting this error > > [2021-03-12 20:17:41,870] ERROR Sto

Schema registry for Kafka topic

Hi, We have an in-house cluster of Kafka brokers and ZooKeepers. Kafka version kafka_2.12-2.7.0 ZooKeeper version apache-zookeeper-3.6.2-bin The topic is published to Google BigQuery. We would like to use a schema registry as opposed to sending schema with payload for each message that

Re: Issue writing to Google BigQuery

ABUrV8Pw>* *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary

Issue writing to Google BigQuery

This is what is termed as fun and game. Trying to write a single column (for the sake of test) in this case to BigQuery from Kafka. I am sending the schema and payload as per docs This is message sent that I can get it from console $KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server

Kafka-BigQuery connector issue

;https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will

Re: Error in Kafka property file contains no connector type

: > Hi Mich, > > Your bigquery-sink.properties file is in a JSON format - which won't work. > It needs to follow the usual format of a Java properties file. > > Kind regards, > > Liam Clarke-Hutchinson > > On Fri, Mar 12, 2021 at 12:13 AM Mich Talebzadeh < &

kafka-bigquery connector error - Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException

Hi, Trying to stream into bigQuery table from on-premise cluster. This works with spark structured streaming foreachBatch This is the command I use: $KAFKA_HOME/bin/connect-standalone.sh \ /d4T/hduser/bigquery-kafka-connect-sink/etc/connect-standalone.properties \

Kafka connector for google BigQuery

Hi, To stream data from kafka topic to Google BigQuery, i see some references to wepay-kafka-connect-bigquery-2.1.0 However, this sounds like being depreciated. Currently we can stream directory through Spark Structured Streaming with writeStream. \

Kafka-connect to Google BigQuery benchmarks

. Thanks, Mich view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's tec

Re: GCP Dataproc - getting error in importing KafkaProducer

on. *>>> from kafka import KafkaProducer* *>>>* view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destru

Re: GCP Dataproc - getting error in importing KafkaProducer

isclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 18 Feb 2022 at 08:39, Mich Talebzadeh wrote: > Have you installed the correct package kafka-python? > > *pip install kafka-python* > Collecting kafka-p

Fwd: Error in Kafka property file contains no connector type

@Aakarsh Gopi, check this email - Forwarded message - From: Mich Talebzadeh Date: Sun, 14 Mar 2021 at 19:44 Subject: Re: Error in Kafka property file contains no connector type To: Thanks all. I was using an older Kafka version. I upgraded Kafka in the cluster from

Re: Some questions on Kafka on order of messages with mutiple partitions

question, and cannot be > answered in a general way. > > > "Yes" to all other questions. > > > -Matthias > > > > On 5/12/23 9:50 AM, Mich Talebzadeh wrote: > > Hi, > > > > I have used Apache Kafka in conjunction with Spark as a messaging &g

Re: Some questions on Kafka on order of messages with mutiple partitions

:9097 --reassignment-json-file ./reduce_replication_factor.json --execute The output Successfully started partition reassignments for md-0,md-1,md-2,md-3,md-4,md-5 I guess it is going to take sometime before it is completed. Thanks On Fri, 12 May 2023 at 20:16, Mich Talebzadeh wrote:

Trying to reduce the number of replicates from 9 to 3

Hi, >From the following list kafka-topics.sh --describe --bootstrap-server rhes75:9092 --topic md Topic: md TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9 ReplicationFactor: 9Configs: segment.bytes=1073741824,retention.bytes=1073741824 Topic: md Partition: 0

Re: Some questions on Kafka on order of messages with mutiple partitions

My bad. Only need 3 partitions { "version":1, "partitions":[ {"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]}, {"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},

Re: Trying to reduce the number of replicates from 9 to 3

struction. On Mon, 15 May 2023 at 07:41, Luke Chen wrote: > Hi Mich, > > You might want to take a look at this section: "Increasing replication > factor" in the doc: > > https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor > >

Re: Some questions on Kafka on order of messages with mutiple partitions

uot;md","partition":4,"replicas":[7,9,11]}, > {"topic":"md","partition":5,"replicas":[3,11,1]} > ] > } > > You may want to adjust where the replicas sit to achieve a better balance > across the cluster, but this arrangement

Re: Some questions on Kafka on order of messages with mutiple partitions

roperty which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 17 May 2023 at 00:21, Miguel A. Sotomayor wrote: > Hi Mich, > > You can u

Some questions on Kafka on order of messages with mutiple partitions

Hi, I have used Apache Kafka in conjunction with Spark as a messaging source. This rather dated diagram describes it I have two physical hosts each 64 GB, running RHES 7.6, these are called rhes75 and rhes76 respectively. The Zookeeper version is 3.7.1 and kafka version is 3.4.0 [image:

Re: Some questions on Kafka on order of messages with mutiple partitions

Replicas: 3,11,1 Isr: 1,3,11 Topic: md Partition: 6Leader: 10 Replicas: 10,1,7 Isr: 10,1,7 Topic: md Partition: 7Leader: 8 Replicas: 8,7,3 Isr: 7,3,8 Topic: md Partition: 8Leader: 2 Replicas: 2,3,10 I

Re: Zookeeper not starting

-3.1.1/logs -Dzookeeper.log.file=zookeeper-hduser-server-rhes75.log Go to that directory and look at the log file(example here -> /home/hduser/hadoop-3.1.1/logs/zookeeper-hduser-server-rhes75.log To find out the cause of error HTH Mich Talebzadeh, Lead Solutions Architect/Engineering L

Re: Zookeeper not starting

Hi Lemi, How many ZooKeepers do you have in your topology? Also ZooKeeper and Kafka versions. If it says Kafka is running there must be ZooKeeper somewhere already started HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Palantir Technologies Limited London United Kingdom