KRaft controller listener with SASL SCRAM-SHA-512

2023-07-28 Thread Robin
ng all loggers to TRACE but I wasn't able to figure it out. It's working on my other listeners (INTERNAL, CLIENT, EXTERNAL) but I can't seem to get it working for the CONTROLLER listener for the KRaft communication. Any pointers for how to debug this would be much appreciated! Kind Regards, Robin

Re: Monitoring of consumer group lag

2015-03-17 Thread Robin Yamaguchi
Hi Mathias, We call bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker via NRPE, and alert through Nagios. -Robin On Tue, Mar 17, 2015 at 2:46 AM, Kasper Mackenhauer Jacobsen < kas...@falconsocial.com> wrote: > Hi Mathias, > > We're currently using a custom solutio

Changing replication factor for an existing topic

2015-06-10 Thread Robin Yamaguchi
27;s kafka manager. Thought I give this list another shot. Thank you, Robin

Re: Changing replication factor for an existing topic

2015-06-12 Thread Robin Yamaguchi
10, 2015, at 4:20 PM, Aditya Auradkar > wrote: > > > > The replica list that you specify can be used to increment/decrement the > replication factor. > > > http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor > > > > Aditya > > >

Consumer seek on 0.9.0 API

2016-02-17 Thread Péricé Robin
is will be greatly appreciated ! Regards, Robin

Re: Consumer seek on 0.9.0 API

2016-02-18 Thread Péricé Robin
Hi, Ok I did a poll() before my seek() and poll() again and now my consumer starts at offset. Thanks you a lot ! But I don't really understand why I have to do that, if anyone can explain me. Regards, Robin 2016-02-17 20:39 GMT+01:00 Alex Loddengaard : > Hi Robin, > > I believ

Re: Consumer seek on 0.9.0 API

2016-02-22 Thread Péricé Robin
OK I understand the explanation. Thanks you for sharing your knowledge ! Regards, Robin 2016-02-18 18:56 GMT+01:00 Jason Gustafson : > Woops. Looks like Alex got there first. Glad you were able to figure it > out. > > -Jason > > On Thu, Feb 18, 2016 at 9:55 AM, Jason

KafkaConsumer poll() problems

2016-03-01 Thread Péricé Robin
lBasicTestConsumer with old generation 1 (kafka.coordin ator.GroupCoordinator) [2016-03-01 14:42:55,809] INFO [GroupCoordinator 0]: Group KafkaChannelBasicTestConsumer generation 1 is dead and removed (kafka.coordinator.GroupCoor dinator) I really need help on this ! Regards, Robin

Re: KafkaConsumer poll() problems

2016-03-01 Thread Péricé Robin
or Pastebin, > so > it's formatted and easier to read? > > On Tue, Mar 1, 2016 at 8:49 AM, Péricé Robin > wrote: > > > Hello everybody, > > > > I'm having troubles using KafkaConsumer 0.9.0.0 API. My Cons

Consumers doesn't always poll first messages

2016-03-02 Thread Péricé Robin
ery random. Anybody have an idea on that issue ? Best Regards, Robin

Re: Writing a Producer from Scratch

2016-03-03 Thread Péricé Robin
Hi, Maybe you should look at this : https://github.com/edenhill/librdkafka Regards, Robin 2016-03-03 11:47 GMT+01:00 Hopson, Stephen : > Hi, > > Not sure if this is the right forum for this question, but if it not I’m > sure someone will direct me to the proper one. > >

Re: Writing a Producer from Scratch

2016-03-03 Thread Péricé Robin
Oups didn't see your response. Sorry 2016-03-03 17:23 GMT+01:00 Péricé Robin : > Hi, > > Maybe you should look at this : https://github.com/edenhill/librdkafka > > Regards, > > Robin > > 2016-03-03 11:47 GMT+01:00 Hopson, Stephen : > >> Hi, >>

Re: Kafka streams for golang

2017-12-19 Thread Robin Bartholdson
e Apache Beam will eventually provide some of that functionality as well now that it has a go SDK (https://github.com/apache/beam/tree/master/sdks/go <https://github.com/apache/beam/tree/master/sdks/go>) -Robin > On 19 Dec 2017, at 21:52, Victor Denisov wrote: > > Hi, > > Is

Re: Developing with kafka and other non jvm languages

2018-03-26 Thread Robin Bartholdson
entio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions> - optiopay/kafka: Metadata(): https://godoc.org/github.com/optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/optiopay/kafka#Broker.Metadata> -Robin > On 25 Mar 2018, a

Re: how to use SMT to transform a filed value in kafka connect?

2018-07-10 Thread Robin Moffatt
inc/ksql/issues/620> ( https://github.com/confluentinc/ksql/issues/620) -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Tue, 10 Jul 2018 at 03:48, 刘波 <1091643...@qq.com> wrote: > Hi all, > I'm new to kafak. I'm using kafka to build a ETL prog

Re: Any one using oracle event hub - dedicated ?

2018-07-25 Thread Robin Moffatt
Compared to other managed solutions, or compared to running it yourself? On Wed, 25 Jul 2018 at 12:44, Vaibhav Jain wrote: > Hi There, Any pros/cons ? Regards, Vaibhav -- -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff

Re: Exposing Kafka on WAN

2018-08-31 Thread Robin Moffatt
Shameless plug for an article I wrote on understanding Kafka listeners better, which might help in this case: https://rmoff.net/2018/08/02/kafka-listeners-explained/ On Thu, 30 Aug 2018 at 14:14, Andrew Otto wrote: > The trouble is that the producer and consumer clients need to discover the >

Best way to manage topics using Java API

2018-09-03 Thread Robin Perice
way to manage topics from Java ? Regards, Robin * * *--* Robin PERICE INGENIEUR DEVELOPPEMENT LOGICIEL Tel.: +33 (0) 5 62 88 80 39

Re: Best way to manage topics using Java API

2018-09-06 Thread Robin Perice
Hi, Yes this help a lot ! Thanks ! Robin Le 05/09/2018 à 22:29, John Roesler a écrit : Hi Robin, The AdminClient is what you want. "Evolving" is just a heads-up that the API is relatively new and hasn't stood the test of time, so you shouldn't be *too* surprised to

Re: Need info

2018-09-14 Thread Robin Moffatt
kafka-connect-s3/). There are also options for pulling from the mainframe, depending on specific platform and system. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji < chanchal.chatte...@infosys.com> wrote: > Hi, >

Re: How to config the servers when Kafka cluster is behind a NAT?

2018-09-17 Thread Robin Moffatt
This should help: https://rmoff.net/2018/08/02/kafka-listeners-explained/ -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Mon, 17 Sep 2018 at 12:18, XinYi Long wrote: > Hello, Guys, > > I have a Kafka cluster, which is behind a NAT. > > I have

Re: Add company name on the Powered by page

2018-11-02 Thread Robin Moffatt
Hi Rohan, Please contact a...@confluent.io and she can arrange this. thanks, Robin. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Fri, 26 Oct 2018 at 18:26, Rohan Rasane wrote: > Hi, > What is the email address, if I want to add ServiceNow to the powe

Re: Problem with Kafka Connector

2018-12-06 Thread Robin Moffatt
If the properties are not available per-connector, then you will have to set them on the worker and have independent Kafka Connect clusters delineated by connector requirements. So long as you configure the ports not to clash, there's no reason these can't exist on the same host.

Re: Magic byte error when trying to consume Avro data with Kafka Connect

2018-12-07 Thread Robin Moffatt
xplained -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Thu, 6 Dec 2018 at 18:30, Marcos Juarez wrote: > We're trying to use Kafka Connect to pull down data from Kafka, but we're > having issues with the Avro deserialization. > > When we attempt t

Re: Kafka data log timestamp

2019-01-10 Thread Robin Moffatt
lue (79 bytes): {"uid":2,"name":"Nick","locale":"en_US","address_city":"Palo Alto","elite":"G"} Timestamp: 1520618381093Partition: 0Offset: 1 -- -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @

Re: Kafka servers starting and stopping abnormally

2019-01-14 Thread Robin Moffatt
It would help if you provide a bit more detail. For example, you mention an OS upgrade - but what OS? from what version to what version? Do you see any errors in the broker logs at the time at which they shut down? -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Mon

Re: Kafka Connect Skip Messages

2019-01-25 Thread Robin Moffatt
-transformations-not-transformations-youre-looking/ You might also want to look at Kafka Streams for applying logic to a stream before you ingest it with Kafka Connect. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Fri, 25 Jan 2019 at 13:29, Sönke Liebau wrote: >

Re: Streaming Data

2019-04-10 Thread Robin Moffatt
mples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java Depending on the processing you want to do KSQL could also be useful, writing the results of queries (e.g. spotting exceptions in the data) to a Kafka topic that can be queried by your app. -- Robin Moffatt | Developer Advocate

Re: Kafka Connect - HDFS or FileStream

2019-05-12 Thread Robin Moffatt
ntegration that can be done in a better way. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Mon, 13 May 2019 at 01:35, Vinay Jain wrote: > Hi > > I would like to consume a Topic and save the AVRO messages in local > directory in the AVRO file formats. As pe

Re: quick start does not work...

2019-05-23 Thread Robin Moffatt
Hi Tom, Can you link to which quickstart instructions you're following? thanks. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Thu, 23 May 2019 at 07:35, Tom Kwong wrote: > Hi, > I’m getting the following error (NoNode for /brokers) on a fresh new &

Re: Kafka Connect issues when running in Docker

2019-07-19 Thread Robin Moffatt
Do you get any output from *docker logs*? And does it work if you don't use authentication? How about if you try one of the dockerised Kafka Connect examples here? https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero -- Robin Moffatt | Senior Developer Adv

Re: Ksql with schema registry

2019-10-07 Thread Robin Moffatt
You can specify VALUE_FORMAT='AVRO' to use Avro serialisation and the Schema Registry. See docs for more details https://docs.confluent.io/current/ksql/docs/installation/server-config/avro-schema.html -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On

Re: Monitor Kafka connect jobs

2019-10-25 Thread Robin Moffatt
column -s : -t| sed 's/\"//g'| sort Which returns: source | source-debezium-mysql-02 | RUNNING | RUNNING | io.debezium.connector.mysql.MySqlConnector -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 24 Oct 2019 at 15:52, KhajaAsmath Mo

Re: Install kafka-connect-storage-cloud

2019-11-12 Thread Robin Moffatt
REST API after starting the worker. See https://rmoff.dev/berlin19-kafka-connect for examples and discussion -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 12 Nov 2019 at 16:40, Miguel Silvestre wrote: > Hi, > > I'm new to kafka (really new

Re: help for regarding my question

2019-11-15 Thread Robin Moffatt
We can try, but you'll have to tell us what the problem is :) This is a good checklist for asking a good question: http://catb.org/~esr/faqs/smart-questions.html#beprecise -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 14 Nov 2019 at 05:04, prashant

Re: kafka connect incoming port

2019-11-26 Thread Robin Moffatt
Kafka Connect workers listen on the port defined in *rest.port* in the worker configuration - typically 8083. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 25 Nov 2019 at 18:41, Johnson, Paul wrote: > Hello, > > I am working on setting u

Re: Powered By: add BookingSync and Smily

2019-12-06 Thread Robin Moffatt
Hi Karol, Thanks for submitting these - I'll arrange for a PR to be created to add them to the page. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 6 Dec 2019 at 08:50, Karol Galanciak wrote: > Hi, > > Could you please add Book

Re: Which container should you use when deploying on docker ?

2019-12-16 Thread Robin Moffatt
lse? <https://hub.docker.com/r/confluentinc/cp-kafka/> -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Sat, 14 Dec 2019 at 04:18, Yu Watanabe wrote: > Hello. > > I would like to ask question related to kafka on docker engine. > Which container

Re: Free Kafka Stream Data

2020-01-08 Thread Robin Moffatt
://www.confluent.io/blog/easy-ways-generate-test-data-kafka/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 8 Jan 2020 at 20:42, cool girl wrote: > Hi , > > I am trying to learn Kafka. Is there any free API which I can use like > twitter? I created twitter accou

Re: Public IPs for brokers

2020-01-15 Thread Robin Moffatt
If you use a load balancer bear in mind the importance of the advertised.listeners setting: https://rmoff.net/2018/08/02/kafka-listeners-explained/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 15 Jan 2020 at 08:15, Himanshu Shukla wrote: > is

Re: ingesting web server logs, or log4j log files from a boss server

2020-01-15 Thread Robin Moffatt
f.dev/crunch19-zero-to-hero-kafka-connect -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 15 Jan 2020 at 03:43, George wrote: > Hi all. > > Please advise, a real noob here still, unpacking how the stack still > works... > > if I have a mySQL

Re: Any available cassandra source connector for kafka

2020-01-15 Thread Robin Moffatt
Pretty sure WePay are working on this: https://github.com/debezium/debezium-incubator -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 15 Jan 2020 at 13:55, Sachin Mittal wrote: > Hi, > I was looking for any cassandra source connector to read dat

Re: ingesting web server logs, or log4j log files from a boss server

2020-01-15 Thread Robin Moffatt
If spooldir doesn't suit, there's also https://github.com/streamthoughts/kafka-connect-file-pulse to check out. Also bear in mind tools like filebeat from Elastic support Kafka as a target. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 15 Jan

Re: Stand alone connector SW deployment on a web server

2020-01-16 Thread Robin Moffatt
Not sure what you mean by "SQ", but Kafka Connect workers can indeed be deployed on their own, connecting to a remote Kafka cluster. Standalone would make sense in that case, yes. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 16 Jan 2020 at 03:

Re: User interface for kafka: Management/administration

2020-01-23 Thread Robin Moffatt
#x27;s a useful list of projects to look at: https://dev.to/guyshiloizy/kafka-administration-and-monitoring-ui-tools-hf4 -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 23 Jan 2020 at 04:10, Sunil CHAUDHARI wrote: > Hi all, > Please help me to get so

Re: connect to bitnami kafka cluster from djang app-engine

2020-02-07 Thread Robin Moffatt
The error you get puts you on the right lines: > Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and resolvable? This article explains why and how to fix it: https://rmoff.net/2018/08/02/kafka-listeners-explained/ -- Robin Moffatt | Senior Develo

Re: Kafka compatibility metrics

2020-02-18 Thread Robin Moffatt
Hi, Can you be a bit more specific about what you're looking for? thanks, Robin. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 18 Feb 2020 at 00:28, Ajith Kumar Rajendran wrote: > Hi team, > > Need a kafka compatibility metrics details &

Re: Some questions about copyright

2020-02-24 Thread Robin Moffatt
Hi Michael, You can find the ASF trademark policies here: https://www.apache.org/foundation/marks/#books -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Sun, 23 Feb 2020 at 19:47, Michi Bertschi wrote: > Hello > I'm Michael > > > I am writ

Re: Connect - Membership Protocol

2020-03-18 Thread Robin Moffatt
g-multiple-kafka-connect-workers/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 17 Mar 2020 at 17:58, mandeep gandhi wrote: > Gentle reminder. > > + users. > > On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi > wrote: > > >

Re: I'm trying to connect to a kafka broker running in AWS EKS (from outside the EKS cluster).

2020-03-19 Thread Robin Moffatt
You need to make sure you've configured your listeners & advertised listeners correctly. This should help: https://rmoff.net/2018/08/02/kafka-listeners-explained/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 19 Mar 2020 at 01:49, Dan Hi

Re: How to disable OPTIONS method in confluent-control-center

2020-03-31 Thread Robin Moffatt
You probably want the Confluent Platform mailing list for this: https://groups.google.com/forum/#!forum/confluent-platform (or Confluent Platform slack group: http://cnfl.io/slack with the #control-center channel). Or if you have a Confluent support contract, contact support :) -- Robin

Re: Adding new connector Jars dynamically in Kafka Connect

2020-04-13 Thread Robin Moffatt
No. You have you restart the worker. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 7 Apr 2020 at 18:24, nitin agarwal wrote: > Hi, > > I have a use case where new connectors will keep on adding to existing > running Kafka Connect cluster.

New blog post: A quick and dirty way to monitor data arriving on Kafka

2020-04-17 Thread Robin Moffatt
I thought I'd share here a short blog I wrote up recently: https://rmoff.net/2020/04/16/a-quick-and-dirty-way-to-monitor-data-arriving-on-kafka/ Enjoy :) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff

Confluent Virtual User Group (VUG)

2020-04-17 Thread Robin Moffatt
published here ( https://airtable.com/shrzKAKFMcfIgFVK9/iCal?timeZone=Europe%2FLondon&userLocale=en-gb) or an online view here (https://cnfl.io/online-community-meetups) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff

Re: Sybase Connection

2020-04-20 Thread Robin Moffatt
if Sybase supports JDBC then you can probably use the JDBC Source connector for Kafka Connect Ref: https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 20 Apr 2020 at 17:31, Jonatan

Re: JDBC source connector to Kafka topic

2020-04-29 Thread Robin Moffatt
How are you running Kafka? Do you mean when you shut it down you have to reconfigure the connector? -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 29 Apr 2020 at 20:03, vishnu murali wrote: > Hi guys > I am trying that JDBC source connector to ge

Re: Connector For MirrorMaker

2020-05-01 Thread Robin Moffatt
Are you talking about MirrorMaker 2? That runs as a connector. If not, perhaps you can clarify your question a bit as to what it is you are looking for. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 1 May 2020 at 13:57, vishnu murali wrote: > Hi G

Re: JDBC Sink Connector

2020-05-07 Thread Robin Moffatt
See https://www.youtube.com/watch?v=b-3qN_tlYR4&t=627s -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 7 May 2020 at 06:48, vishnu murali wrote: > Hey Guys, > > i am working on JDBC Sink Conneector to take data from kafka topic to > mysql. &g

Re: JDBC Sink Connector

2020-05-11 Thread Robin Moffatt
Schema Registry and its serde libraries are part of Confluent Platform, licensed under Confluent Community Licence ( https://www.confluent.io/confluent-community-license-faq/) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 8 May 2020 at 13:39, vishnu

Re: JDBC SINK SCHEMA

2020-05-11 Thread Robin Moffatt
You can use Apache Kafka as you are currently using, and just deploy Schema Registry alongside it. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Sat, 9 May 2020 at 02:16, Chris Toomey wrote: > You have to either 1) use one of the Confluent serializ

Re: Write to database directly by referencing schema registry, no jdbc sink connector

2020-05-11 Thread Robin Moffatt
> wirite to target database. I want to use self-written java code instead of kafka jdbc sink connector. Out of interest, why do you want to do this? Why not use the JDBC sink connector (or a fork of it if you need to amend its functionality)? -- Robin Moffatt | Senior Developer Advoc

Re: JDBC SINK SCHEMA

2020-05-11 Thread Robin Moffatt
components as is. When you configure Schema Registry you point it at your existing Apache Kafka cluster ( https://docs.confluent.io/current/schema-registry/installation/config.html#schemaregistry-config ) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 11 May 2020

Re: Jdbc Source Connector Config

2020-05-13 Thread Robin Moffatt
line-world-thereabouts-part-3/ - https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/ - http://rmoff.dev/ksldn19-kafka-connect -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 13 May 2020 at 10:34, vishnu murali wrote: > Hi Guys, >

Re: JDBC source connector

2020-05-14 Thread Robin Moffatt
If you just want it once then delete the connector once it's processed all the data -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 14 May 2020 at 16:14, vishnu murali wrote: > Thanks Liam > > But I am asking like assume I am having 100

Re: Problem in Docker

2020-05-15 Thread Robin Moffatt
Check this out: https://rmoff.dev/fix-jdbc-driver-video (specifically here for Docker instructions https://www.youtube.com/watch?v=vI_L9irU9Pc&t=665s) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 15 May 2020 at 08:16, vishnu murali wrote: >

Re: Cannot access to kafka by server domain and port.

2020-05-19 Thread Robin Moffatt
This should help your understanding: https://rmoff.net/2018/08/02/kafka-listeners-explained/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 19 May 2020 at 11:48, 深大李展发 wrote: > Thank you, Ixy, thanks for your reply, it's working now! > I wil

Re: Persist Kafka Topics and ksqldb

2020-05-19 Thread Robin Moffatt
You need to externalise your container data stores. Here's an example Docker Compose that does that: https://github.com/confluentinc/demo-scene/blob/master/wifi-fun/docker-compose.yml <https://github.com/confluentinc/demo-scene/blob/master/wifi-fun/docker-compose.yml#L10> -- Ro

Re: Exception in SFTP CSV SOURCE

2020-05-19 Thread Robin Moffatt
Hi Vishnu, I think there is a problem with your email client, it's just sent a duplicate of each of your emails from yesterday? thanks, Robin. On Tue, 19 May 2020 at 16:44, vishnu murali wrote: > Hi Guys > > By Trying SFTP CSV SOURCE i am getting this exception by using this &

Re: Kafka Connect Connector Tasks Uneven Division

2020-05-20 Thread Robin Moffatt
nting a problem to you (if so please describe why), or if you're just interested in the theory behind the rebalancing protocol? -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 20 May 2020 at 06:46, Deepak Raghav wrote: > Hi > > Please,

Re: Kafka Connect Connector Tasks Uneven Division

2020-05-20 Thread Robin Moffatt
s any difference where the tasks execute? The point of a cluster is that Kafka Connect manages the workload allocation. If you need workload separation and guaranteed execution locality I would suggest separate Kafka Connect distributed clusters. -- Robin Moffatt | Senior Developer Advocate

Re: Kafka Connect Connector Tasks Uneven Division

2020-05-20 Thread Robin Moffatt
urces available. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 20 May 2020 at 16:39, Deepak Raghav wrote: > Hi Robin > > Replying to your query i.e > > One thing I'd ask at this point is though if it makes any difference where > the ta

Re: Kafka Connect Connector Tasks Uneven Division

2020-05-21 Thread Robin Moffatt
blem or an actual one. If this is an actual problem that you're encountering and need a solution to then since the task allocation is not deterministic it sounds like you need to deploy separate worker clusters based on the workload patterns that you are seeing and machine resources available.

Re: User ID / Password based authentication for Kafka Connect Source connector REST Interface

2020-05-22 Thread Robin Moffatt
See here: https://docs.confluent.io/current/security/basic-auth.html#basic-auth-kconnect -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 22 May 2020 at 15:42, wrote: > Hi Team, > > I have configured a Kafka Connect source connector on Distribut

Re: Kafka Connect Connector Tasks Uneven Division

2020-05-26 Thread Robin Moffatt
The KIP for the current rebalancing protocol is probably a good reference: https://cwiki.apache.org/confluence/display/KAFKA/KIP-415:+Incremental+Cooperative+Rebalancing+in+Kafka+Connect -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 26 May 2020 at 14:25

Re: How to manually start ingesting in kafka source connector ?

2020-05-28 Thread Robin Moffatt
When you create the connector, it will start. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 28 May 2020 at 04:12, Yu Watanabe wrote: > Dear community . > > I would like to ask question related to source connector in kafka > connect (2.4

Re: How to manually start ingesting in kafka source connector ?

2020-05-28 Thread Robin Moffatt
You could look at https://rmoff.net/2019/08/15/reset-kafka-connect-source-connector-offsets/ and experiment with creating the connector elsewhere to see if you can pre-empt the key value that Kafka Connect will use when writing the offsets, and so do your list 2 - 1 - 3 instead -- Robin

Re: handle multiple requests

2020-06-10 Thread Robin Moffatt
You might also find this resource useful in general: https://www.confluent.io/whitepaper/comparing-confluent-platform-with-traditional-messaging-middleware/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff ‪On Wed, 10 Jun 2020 at 16:00, ‫נתי אלמגור‬‎ wrote

Re: Kafka - entity-default Vs entity-name

2020-06-12 Thread Robin Moffatt
If you're going to cross-post, it's useful to include where else you've asked this before, so that people can see what else has been suggested and answered to avoid duplicating their effort :) https://stackoverflow.com/questions/62343035/kafka-entity-default-vs-entity-name --

Re: Problem in reading From JDBC SOURCE

2020-07-02 Thread Robin Moffatt
Check out this article where it covers decimal handling: https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/#bytes-decimals-numerics -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 2 Jul 2020 at 13:54, vishnu murali wrote: >

Re: MONGODB KAFKA CONNECT

2020-07-28 Thread Robin Moffatt
ka-02 For MongoDB connectors see https://www.confluent.io/hub/#mongodb -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 27 Jul 2020 at 18:19, Milka Wafula wrote: > Hello, > > Please how may we connect mongoDB with ka

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

2020-08-19 Thread Robin Moffatt
I echo what Michael says here. Another consideration is that logos are often shrunk (when used on slides) and need to work at lower resolution (think: printing swag, stitching socks, etc) and so whatever logo we come up with needs to not be too fiddly in the level of detail - something that I thin

Re: Amazon MSK Feeback

2020-09-07 Thread Robin Moffatt
, Robin. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 7 Sep 2020 at 10:00, Himanshu Shukla wrote: > Hi, > We are planning to go for amazon MSK instead of having our own self > launched cluster on bare EC2 machines. > > It would be very help

Re: KSQL Enrichment issue with converting decimal to string column

2020-09-08 Thread Robin Moffatt
Why is quantity a STRING? Should it be a numeric? -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 8 Sep 2020 at 08:52, Rainer Schamm wrote: > Hi all > > we are struggling a bit with a fairly simple stream that is supposed to > map columns

Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-09 Thread Robin Moffatt
-timestamp/ Reply here with any feedback, any particular SMT you'd like me to explore, or scenarios to solve :) -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff

Re: Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-10 Thread Robin Moffatt
Here's the second instalment of 🎅🎄 #TwelveDaysOfSMT 🎄 🎅 Day 2: ValueToKey and ExtractField : https://rmoff.net/2020/12/09/twelve-days-of-smt-day-2-valuetokey-and-extractfield/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 9 Dec 2020 at 09:07,

Re: Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-10 Thread Robin Moffatt
Here's the third instalment of 🎅🎄 #TwelveDaysOfSMT 🎄 🎅 Flatten :https://rmoff.net/2020/12/10/twelve-days-of-smt-day-3-flatten/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 10 Dec 2020 at 09:49, Robin Moffatt wrote: > Here's the seco

Re: Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-11 Thread Robin Moffatt
🎅🎄 #TwelveDaysOfSMT 🎄 🎅 Day 4️⃣: RegexRouter https://rmoff.net/2020/12/11/twelve-days-of-smt-day-4-regexrouter/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 10 Dec 2020 at 17:56, Robin Moffatt wrote: > Here's the third instalment

Re: Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-14 Thread Robin Moffatt
🎅🎄 #TwelveDaysOfSMT 🎄 🎅 Day 5️⃣: MaskField https://rmoff.net/2020/12/14/twelve-days-of-smt-day-5-maskfield/ -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 11 Dec 2020 at 17:14, Robin Moffatt wrote: > 🎅🎄 #TwelveDaysOfSMT 🎄 🎅 > >

Re: Announcing 🎅🎄#TwelveDaysOfSMT🎄🎅

2020-12-15 Thread Robin Moffatt
edaysofsmt/ thanks, Robin. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 14 Dec 2020 at 16:30, Ryanne Dolan wrote: > Robin, I don't think this is an appropriate use of the user group. > > Ryanne > > On Mon, Dec 14, 2020 at 10:20 AM R

Re: print.partition=true not working

2021-02-10 Thread Robin Moffatt
%p Offset: %o Headers: %h\n' -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 10 Feb 2021 at 17:59, Rovere Lorenzo wrote: > Hi > > > > We are on Kafka version: 2.2.1-kafka-4.1.0 > > We have some issues when trying to dump some

Re: Kafka Security Questions

2021-02-16 Thread Robin Moffatt
You can read about the security available in Apache Kafka here: https://kafka.apache.org/documentation/#security_ssl -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 16 Feb 2021 at 15:18, Jones, Isaac wrote: > Hello, > > I have a couple important

Loading delimited data into Apache Kafka - quick & dirty (but effective)

2021-03-01 Thread Robin Moffatt
I wrote a blog about loading data into Apache Kafka using kafkacat. It's kinda hacky, but it works and was an interesting journey through some other bash tools too :D https://rmoff.net/2021/02/26/loading-delimited-data-into-kafka-quick-dirty-but-effective/ -- Robin Moffatt | Senior Deve

Re: Filter Kafka Messages

2021-03-10 Thread Robin Moffatt
of it here: https://forum.confluent.io/t/kafka-connect-jmespath-expressive-content-based-record-filtering/1104 -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 10 Mar 2021 at 09:55, raman gugnani wrote: > HI Tom, > > For which connector class conne

Re: Kafka Connectors output to topic.

2021-03-11 Thread Robin Moffatt
For what it's worth, that sounds less like a Kafka Connector and more like a Kafka Streams app. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Thu, 11 Mar 2021 at 18:27, Nick Siviglia wrote: > Hi Everyone, > > I'd like to create a Kafka con

Re: [ANNOUNCE] New committer: Tom Bentley

2021-03-15 Thread Robin Moffatt
Congratulations Tom! -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Mon, 15 Mar 2021 at 18:00, Mickael Maison wrote: > Hi all, > > The PMC for Apache Kafka has invited Tom Bentley as a committer, and > we are excited to announce that he accep

Re: Docker images

2021-03-16 Thread Robin Moffatt
There is a similar discussion about Kafka Connect images here: https://issues.apache.org/jira/browse/KAFKA-9774 -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 16 Mar 2021 at 16:04, Francois Papon wrote: > Hi, > > It would be nice to push an

Re: Kafka Connect Distributed Mode Doubt

2021-03-23 Thread Robin Moffatt
still have a properties file to configure the worker itself. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 23 Mar 2021 at 02:02, Himanshu Shukla wrote: > Hi All, > > I am having these below understanding regarding Kafka connect. > > 1

Re: Confluence Control Center On Windows 10

2021-03-26 Thread Robin Moffatt
You can use WSL2: https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2 -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 26 Mar 2021 at 05:09, Satendra Negi wrote: > Hello Guys, > > Is there any way to run the confluent kafk

Re: Confluence Control Center On Windows 10

2021-03-26 Thread Robin Moffatt
Satendra - there was a new blog published just today about running Confluent Control Center on Windows: https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-and-wsl-2 -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 26 Mar 2021 at 10:07, Robin

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

2021-04-18 Thread Robin Moffatt
Great news, congratulations Randall! -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Sat, 17 Apr 2021 at 00:45, Matthias J. Sax wrote: > Hi, > > It's my pleasure to announce that Randall Hauch in now a member of the > Kafka PMC. > > R

  1   2   >