Re: Camel-K / Kafka / Avro: something broke overnight

2021-01-15 Thread mark 
Andrea,
Thank you very much for the pointer. Yes we've hacked something
together using the test code you linked above, and a hand-rolled
deserializer based on
https://github.com/Apicurio/apicurio-registry/blob/1.3.x/app/src/test/java/io/apicurio/registry/RegistrySerdeTest.java
.

Omar,
Thank you for confirming the regression, and for preparing a fix.

Regards,
Mark

On Fri, 15 Jan 2021 at 09:23, Omar Al-Safi  wrote:

> Hi Mark,
>
> indeed it looks like the additionalProperties option is broken in 3.7.0. I
> am pushing a fix for both branches, 3.7.x and master branch.
>
> Regards,
> Omar
>
>
> On Thu, Jan 14, 2021 at 6:41 PM Andrea Cosentino 
> wrote:
>
> > I don't think it is feasible from our side to get to 3.6.0, it's up to
> you.
> >
> > Can you try to set the additionalProperties at component level like this
> > example?
> >
> >
> >
> https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63
> >
> > Il giorno gio 14 gen 2021 alle ore 17:28 mark  ha
> > scritto:
> >
> > > Hello,
> > > Until yesterday we were using Camel-K to send Avro-encoded messages to
> > > Kafka using code of the form,
> > >
> > > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > >   +
> > >
> > >
> >
> "=org.apache.kafka.common.serialization.StringSerializer"
> > >   +
> > > "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > >   + "=" +
> > > schemaRegistry
> > >
> > > Running this today, our Integration had picked up camel-kafka 3.7.0 and
> > > issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed
> > the
> > > first two properties to keySerializer and valueSerializer. However
> > > adjusting the code to,
> > >
> > > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > > +
> > > "=org.apache.kafka.common.serialization.StringSerializer"
> > > +
> > > "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > > + "=" +
> schemaRegistry
> > >
> > > Leaves us no longer able to set additionalProperties. Attempts such as
> > > adding
> > >
> > > -d mvn:org.apache.camel/camel-kafka:[3.6.0]
> > >
> > > have no effect: we always get camel-kafka 3.7 and cannot pin the
> > > Integration to 3.6 or below. The relevant stack trace is below - please
> > can
> > > anyone help us either adjust the apicurio.registry.url for the new
> > library,
> > > or pin things back to 3.6 please?
> > >
> > > [1] Caused by: org.apache.kafka.common.KafkaException: Failed to
> > construct
> > > kafka producer
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:434)
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> > > [1] at
> > >
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> > > [1] at
> > >
> org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> > > [1] at
> > >
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> > > [1] at
> > >
> > >
> >
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> > > [1] ... 37 more
> > > [1] Caused by: java.lang.IllegalArgumentException: Missing registry
> base
> > > url, set apicurio.registry.url
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> > > [1] at
> > >
> > >
> >
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> > > [1] at
> > >
> > >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:369)
> > > [1] ... 49 more
> > >
> > > My dependencies are currently,
> > >
> > > // camel-k:
> > > dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> > > // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> > > // camel-k:
> dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> > > // camel-k:
> > >

Re: Camel-K / Kafka / Avro: something broke overnight

2021-01-15 Thread Omar Al-Safi 
Hi Mark,

indeed it looks like the additionalProperties option is broken in 3.7.0. I
am pushing a fix for both branches, 3.7.x and master branch.

Regards,
Omar


On Thu, Jan 14, 2021 at 6:41 PM Andrea Cosentino  wrote:

> I don't think it is feasible from our side to get to 3.6.0, it's up to you.
>
> Can you try to set the additionalProperties at component level like this
> example?
>
>
> https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63
>
> Il giorno gio 14 gen 2021 alle ore 17:28 mark  ha
> scritto:
>
> > Hello,
> > Until yesterday we were using Camel-K to send Avro-encoded messages to
> > Kafka using code of the form,
> >
> > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> >   +
> >
> >
> "=org.apache.kafka.common.serialization.StringSerializer"
> >   +
> > "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> >   + "=" +
> > schemaRegistry
> >
> > Running this today, our Integration had picked up camel-kafka 3.7.0 and
> > issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed
> the
> > first two properties to keySerializer and valueSerializer. However
> > adjusting the code to,
> >
> > .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> > +
> > "=org.apache.kafka.common.serialization.StringSerializer"
> > +
> > "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> > + "=" + schemaRegistry
> >
> > Leaves us no longer able to set additionalProperties. Attempts such as
> > adding
> >
> > -d mvn:org.apache.camel/camel-kafka:[3.6.0]
> >
> > have no effect: we always get camel-kafka 3.7 and cannot pin the
> > Integration to 3.6 or below. The relevant stack trace is below - please
> can
> > anyone help us either adjust the apicurio.registry.url for the new
> library,
> > or pin things back to 3.6 please?
> >
> > [1] Caused by: org.apache.kafka.common.KafkaException: Failed to
> construct
> > kafka producer
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:434)
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
> > [1] at
> >
> >
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> > [1] at
> > org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > [1] at
> >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> > [1] at
> >
> >
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> > [1] at
> > org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> > [1] at
> > org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> > [1] at
> >
> >
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> > [1] at
> >
> >
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> > [1] at
> >
> >
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> > [1] ... 37 more
> > [1] Caused by: java.lang.IllegalArgumentException: Missing registry base
> > url, set apicurio.registry.url
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> > [1] at
> >
> >
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> > [1] at
> >
> >
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:369)
> > [1] ... 49 more
> >
> > My dependencies are currently,
> >
> > // camel-k:
> > dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> > // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> > // camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> > // camel-k:
> > dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
> >
> > Many thanks in advance,
> > Regards,
> >
> > Mark
> >
>

Re: Camel-K / Kafka / Avro: something broke overnight

2021-01-14 Thread Andrea Cosentino 
I don't think it is feasible from our side to get to 3.6.0, it's up to you.

Can you try to set the additionalProperties at component level like this
example?

https://github.com/apache/camel/blob/master/components/camel-kafka/src/test/java/org/apache/camel/component/kafka/KafkaComponentTest.java#L63

Il giorno gio 14 gen 2021 alle ore 17:28 mark  ha
scritto:

> Hello,
> Until yesterday we were using Camel-K to send Avro-encoded messages to
> Kafka using code of the form,
>
> .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
>   +
>
> "=org.apache.kafka.common.serialization.StringSerializer"
>   +
> "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
>   + "=" +
> schemaRegistry
>
> Running this today, our Integration had picked up camel-kafka 3.7.0 and
> issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed the
> first two properties to keySerializer and valueSerializer. However
> adjusting the code to,
>
> .to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
> +
> "=org.apache.kafka.common.serialization.StringSerializer"
> +
> "=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
> + "=" + schemaRegistry
>
> Leaves us no longer able to set additionalProperties. Attempts such as
> adding
>
> -d mvn:org.apache.camel/camel-kafka:[3.6.0]
>
> have no effect: we always get camel-kafka 3.7 and cannot pin the
> Integration to 3.6 or below. The relevant stack trace is below - please can
> anyone help us either adjust the apicurio.registry.url for the new library,
> or pin things back to 3.6 please?
>
> [1] Caused by: org.apache.kafka.common.KafkaException: Failed to construct
> kafka producer
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:434)
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
> [1] at
>
> org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
> [1] at
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> [1] at
>
> org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
> [1] at
>
> org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
> [1] at
> org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
> [1] at
> org.apache.camel.support.service.BaseService.start(BaseService.java:115)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
> [1] at
>
> org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
> [1] at
>
> org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
> [1] at
>
> org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
> [1] ... 37 more
> [1] Caused by: java.lang.IllegalArgumentException: Missing registry base
> url, set apicurio.registry.url
> [1] at
>
> io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
> [1] at
>
> io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
> [1] at
>
> io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
> [1] at
>
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:369)
> [1] ... 49 more
>
> My dependencies are currently,
>
> // camel-k:
> dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
> // camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
> // camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
> // camel-k:
> dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final
>
> Many thanks in advance,
> Regards,
>
> Mark
>

Camel-K / Kafka / Avro: something broke overnight

2021-01-14 Thread mark 
Hello,
Until yesterday we were using Camel-K to send Avro-encoded messages to
Kafka using code of the form,

.to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
  +
"=org.apache.kafka.common.serialization.StringSerializer"
  +
"=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
  + "=" +
schemaRegistry

Running this today, our Integration had picked up camel-kafka 3.7.0 and
issue https://issues.apache.org/jira/browse/CAMEL-15770 which changed the
first two properties to keySerializer and valueSerializer. However
adjusting the code to,

.to("kafka:{{topic}}?brokers={{kafka-bootstrap}}"
+
"=org.apache.kafka.common.serialization.StringSerializer"
+
"=io.apicurio.registry.utils.serde.AvroKafkaSerializer"
+ "=" + schemaRegistry

Leaves us no longer able to set additionalProperties. Attempts such as
adding

-d mvn:org.apache.camel/camel-kafka:[3.6.0]

have no effect: we always get camel-kafka 3.7 and cannot pin the
Integration to 3.6 or below. The relevant stack trace is below - please can
anyone help us either adjust the apicurio.registry.url for the new library,
or pin things back to 3.6 please?

[1] Caused by: org.apache.kafka.common.KafkaException: Failed to construct
kafka producer
[1] at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:434)
[1] at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:298)
[1] at
org.apache.camel.component.kafka.KafkaProducer.doStart(KafkaProducer.java:115)
[1] at
org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
[1] at
org.apache.camel.impl.engine.AbstractCamelContext.internalAddService(AbstractCamelContext.java:1425)
[1] at
org.apache.camel.impl.engine.AbstractCamelContext.addService(AbstractCamelContext.java:1343)
[1] at
org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:236)
[1] at
org.apache.camel.support.service.BaseService.start(BaseService.java:115)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
[1] at
org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
[1] at
org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
[1] at
org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
[1] ... 37 more
[1] Caused by: java.lang.IllegalArgumentException: Missing registry base
url, set apicurio.registry.url
[1] at
io.apicurio.registry.utils.serde.AbstractKafkaSerDe.configure(AbstractKafkaSerDe.java:120)
[1] at
io.apicurio.registry.utils.serde.AbstractKafkaStrategyAwareSerDe.configure(AbstractKafkaStrategyAwareSerDe.java:75)
[1] at
io.apicurio.registry.utils.serde.AvroKafkaSerializer.configure(AvroKafkaSerializer.java:78)
[1] at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:369)
[1] ... 49 more

My dependencies are currently,

// camel-k:
dependency=mvn:org.apache.camel.quarkus/camel-quarkus-kafka:1.5.0
// camel-k: dependency=mvn:org.apache.avro/avro:1.10.1
// camel-k: dependency=mvn:org.glassfish.jersey.core/jersey-common:2.22.2
// camel-k:
dependency=mvn:io.apicurio/apicurio-registry-utils-serde:1.3.2.Final

Many thanks in advance,
Regards,

Mark