Thanks Omar. I was looking at ProducerRecord because I was thinking about trying to use Spring Kafka.
I am not sure yet, but part of the issue is that in the camel code, what is being called the paritionKey is not. It is the partition (number). The docs do the same thing. The partitionKey is a value that is converted by kafka to determine the partition. Ok so, yeah. I tested it. Setting just the KafkaConstants.KEY works. There is no Kafka partition key and message key. There is only "Key". It is used for partitioning and compaction. Only use the value in Camel called "KafkaConstants.PARTITION_KEY" if you want to set it to a particular partition by number. The docs are correct in that it is the partition the message will be assigned to. There is also constant called KafkaConstants.PARTITION but it is for consumers. Thanks for helping me think through this Omar. Mark On Mon, Apr 19, 2021 at 12:05 PM Omar Al-Safi <o...@oalsafi.com> wrote: > Weird, from experience, I have seen this timeout happen if the specified > partition doesn't exist. > > However, since you are using Camel 3.9 and you didn't set a message key, > the partition key won't be set in the ProducerRecord, it will be only set > if you set the message key, however this behavior has changed in Camel 3.10 > where the partition key won't require setting the message key. Thus, even > if you set the partition key but you didn't set a message key, I will > expect camel-kafka to ignore the partition (based on Camel 3.9) as you can > see in this code > < > https://github.com/apache/camel/blob/camel-3.9.0/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java#L275 > > > . > > How about if you try using camel-vertx-kafka > <https://camel.apache.org/components/3.9.x/vertx-kafka-component.html> > component > and see if this solves the problem? > > Regards, > Omar > > On Mon, Apr 19, 2021 at 5:26 PM Mark Nuttall <mknutt...@gmail.com> wrote: > > > Setting the message key does not help. :( > > > > On Mon, Apr 19, 2021 at 11:20 AM Mark Nuttall <mknutt...@gmail.com> > wrote: > > > > > I've never set the message key. I might give that a shot. > > > > > > The route is pretty simple. And the only headers being set are the > > > partition key and another string value i need to be passed. The body is > > > JSON. > > > I am running Kafka locally (so whatever partitions is the default) and > > AWS > > > MSK - 2 partitions. > > > > > > Mark > > > > > > > > > > > > On Mon, Apr 19, 2021 at 4:14 AM Omar Al-Safi <o...@oalsafi.com> wrote: > > > > > >> Hi Mark, > > >> > > >> When you send your message to Kafka, do you as well set the message > key > > or > > >> you just set the partition key? Perhaps it would be great if you can > > post > > >> an example for your route with the headers/body that you set, that > could > > >> help us here to troubleshoot the issue. Also, how many partitions do > you > > >> have in your topic? > > >> > > >> Regards, > > >> Omar > > >> > > >> On Fri, Apr 16, 2021 at 5:10 PM Mark Nuttall <mknutt...@gmail.com> > > wrote: > > >> > > >> > Camel version: 3.9.0 > > >> > > > >> > Issue: when consuming from one kafka topic and producing to another, > > >> > setting the partition key causes the write to fail. After a few > > seconds > > >> it > > >> > will fail with this error: > > >> > > > >> > Error during processing. Exchange[2729AA824AE4E97-0000000000000000]. > > >> Caused > > >> > by: [org.apache.kafka.common.errors.TimeoutException - Topic > > some-topic > > >> not > > >> > present in metadata after 60000 ms.] > > >> > > > >> > The topic does exist. If i produce to the topic without the > partition > > >> key > > >> > OR I produce to the topic with a partition key via the > > producerTemplate > > >> it > > >> > works just fine. > > >> > I am pretty sure in the past i have done this and it worked. > > >> > > > >> > Has anyone else seen this? Any ideas? I do have a work around (i.e. > > use > > >> > Spring's kafkaTemplate.). > > >> > > > >> > Mark > > >> > > > >> > > > > > >
