Re: Camel 4 migration blog post - Feedback welcome

[ski n]

 Not being the typical Camel user, I see that as a compliment :)

For the blog it may be also good to highlight not just how to upgrade
(migration path), but also why to upgrade.

There are technical reason like being up to date with Jakarta
EE/Spring/Quarkus, but I think most users are even more interested in the
functional improvements of the framework.

Some technical and functional improvements were highlighted here:

https://camel.apache.org/blog/2023/08/camel4-whatsnew/

But some improvements are already refined in the coming 4.1 release, think
of

- Better support for Yaml DSL
- Better support for XML DSL (xml-io improvements)
- Better Kafka support (camel-kafka)
- Jbang improvements
- Better OpenTelemetry support
- Improvements in dumping of routes

Most Camel developers (at least here on the mailing list) are probably
eager to migrate, but we also need to sell it to our colleagues or
customers. The whole Jakarta thing says nothing to them, but better
observability and better performance (like
https://camel.apache.org/blog/2023/05/camel-4-performance-improvements/) is
much better to communicate.

Raymond

On Fri, Oct 6, 2023 at 4:08 PM Mark Nuttall  wrote:

> one thing that got us was, since we were using kafka with security but not
> using Kafka idempotency, we had to add the following flag. As usual, kafka
> messages were not helpful.
>
> camel.component.kafka.enable-idempotence=false
>
> On Fri, Oct 6, 2023 at 3:49 AM Claus Ibsen  wrote:
>
> > Hi
> >
> > If you have been migrating to Camel 4 already, then any feedback is
> > welcome.
> >
> > You can comment in the PR, or post here in the mailing list
> > https://github.com/apache/camel-website/pull/1071
> >
> >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>

Re: Camel 4 migration blog post - Feedback welcome

[Mark Nuttall]

one thing that got us was, since we were using kafka with security but not
using Kafka idempotency, we had to add the following flag. As usual, kafka
messages were not helpful.

camel.component.kafka.enable-idempotence=false

On Fri, Oct 6, 2023 at 3:49 AM Claus Ibsen  wrote:

> Hi
>
> If you have been migrating to Camel 4 already, then any feedback is
> welcome.
>
> You can comment in the PR, or post here in the mailing list
> https://github.com/apache/camel-website/pull/1071
>
>
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Re: Camel Kafka consumer dynamic topic

[Mark Nuttall]

ya. was just saying you need a plan if you wanted to keep consuming the
route after restart.

anyway, like i said, you have all the pieces to do this. Anything more
would be writing it. But here is some pseudo code.

from(some Trigger)
   to(http) // get the topic
   bean(someServiceWithCamelContextInjected.class, "createRoute");

createRoute(theRestResponseStructure) {
// see stackoverflow link to create the route
}

On Fri, Oct 6, 2023 at 9:56 AM Chirag  wrote:

> Mark,
>
> Scenario for service restart will cause any dynamic route from being
> removed.
>
> It would require ReST to be idempotent and return or more complex -
> persisting routes and reloading them.
>
> ચિરાગ/चिराग/Chirag
> --
> Sent from My Gmail Account
>
>
> On Thu, Oct 5, 2023 at 12:44 AM Mark Nuttall  wrote:
> >
> > The link that I provided shows you how to dynamically create a new route.
> > So all you need to do is after you've called the rest in point using the
> > HTTP component you use what's returned to call a service method in some
> > Class via the bean EIP. The issue is that if the service restarts you
> will
> > lose anything dynamically created. The pieces are all there it's just a
> > matter of you putting them all together
> >
> > On Wed, Oct 4, 2023, 9:38 PM girish vasmatkar 
> wrote:
> >
> > > It's different to what my requirements are. I think dynamic routing in
> case
> > > of a Kafka consumer is different to the examples you have quoted.
> > >
> > > I can't have Kafka topic declared in property file or a yaml file for
> that
> > > matter because I wouldn't be knowing. In that case it would be pretty
> easy
> > > to just use property place holders and build the route
> > >
> > > I need to be able to get a random kafka topic at runtime and create a
> kafka
> > > consumer from it.
> > >
> > > - Girish
> > >
> > > On Wed, Oct 4, 2023, 2:58 AM Mark Nuttall  wrote:
> > >
> > > > Pretty sure I've done this kind of thing before. I think the info is
> here
> > > >
> > > >
> > >
> https://stackoverflow.com/questions/48380456/dynamic-routing-apache-camel
> > > >
> > > > On Tue, Oct 3, 2023, 3:22 AM Claus Ibsen 
> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > If you use Java RouteBuilder, then you can maybe use spring
> dependency
> > > > > injection, to inject the topic name into a String field,
> > > > > and then use that field when building the route model.
> > > > >
> > > > > However I have also thought of adding {{bean:xxx?method=aaa}} as a
> > > > function
> > > > > to property placeholder.
> > > > > Though need to give it a bit more thought as this then have
> > > > > dependency ordering implication (bean must be created beforehand
> etc).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 2, 2023 at 12:40 PM girish vasmatkar <
> girish...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Dear Members,
> > > > > >
> > > > > > I have been searching for this for quite some time but haven't
> been
> > > > able
> > > > > to
> > > > > > figure out a way to do this.
> > > > > >
> > > > > > I need to subscribe to a kafka topic which I will know only at
> > > runtime
> > > > -
> > > > > by
> > > > > > calling a REST endpoint. The REST API I am calling is creating
> topics
> > > > and
> > > > > > returning the topic name.
> > > > > >
> > > > > > I am using spring camel starter and have camel route defined up
> > > until I
> > > > > am
> > > > > > able to fetch the topic name.  How do I then have a route
> created at
> > > > > > runtime that uses the topic name from the rest endpoint?
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Girish Vasmatkar
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Claus Ibsen
> > > > > -
> > > > > @davsclaus
> > > > > Camel in Action 2: https://www.manning.com/ibsen2
> > > > >
> > > >
> > >
>

Re: Camel 4 migration blog post - Feedback welcome

[Claus Ibsen]

On Fri, Oct 6, 2023 at 11:34 AM Mikael Koskinen  wrote:

> Hi,
>
> For me the biggest issues regarding Camel 4 have been when Karavan VS Code
> extension updated itself to 4.0.0. I'm not sure things related to Karavan
> are discussed in the blog post but here's couple things we've encountered.
>
> In short, if you use Karavan 4.x.x to open a route which has been designed
> with the 3.x.x version, in many cases the route breaks as the designer
> removes parts of it. I think the yaml schema has changed a bit between the
> releases and the editor only supports the latest version with no backward
> compatibility. In addition to that, the latest Karavan editor has some
> issues UI issues (scrolling the property panel doesn't work as it should,
> some textboxes are broken). We decided to downgrade the Karavan back to
> 3.x.x even though we updated Camel to 4.x.x.
>
>
You should generally use the Karavan release that is released for a given
Camel version, eg 3.21 for 3.21 and so on.
Having a single continued release of Karavan (and any Camel tooling for the
matter) is hard to have it support many older versions going back.
And also Karavan is in heavy development and Camel v4 is going to be where
it will be ready for mass adoption.




> The Camel 4 itself has been a rock solid release. No issues.
>
pe 6. lokak. 2023 klo 11.52 ski n (raymondmees...@gmail.com) kirjoitti:
>
> > Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.
> >
> > Currently, thus not migrating yet, but planning has started.
> >
> > The things I need to consider:
> >
> > - I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
> > the docs for 2.0 are gone, is the archive to Camel 2.0 documentation
> still
> > available? Like to compare sometimes.
> > - Jakarta stuff in my own code/dependencies.
> > - Jakarta stuff in other libraries (then Camel). For example ActiveMQ
> > Classic (embedded) server doesn't support it yet.
> > - Deprecated components: I use the following components:
> >
> > camel-atlasmap
> > camel-directvm
> > camel-elasticsearch-rest
> > camel-rabbitmq
> > camel-rest-swagger
> > camel-websocket
> > camel-vm
> >
> > So I need to change these dependencies and some code here
> >
> > - I use extendedCamelContext a lot, so need to check the API changes out.
> > For example use the PluginHelper instead of the extendedCamelContext
> > - Update SLFJ to version 2.0
> >
> > Will let you know if I'm running into any issues.
> >
> > I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
> > https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
> > recipe to move to Camel 4 then it would be probably easier to do the
> > migration.
> >
> > Raymond
> >
> >
> > On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen 
> wrote:
> >
> > > Hi
> > >
> > > If you have been migrating to Camel 4 already, then any feedback is
> > > welcome.
> > >
> > > You can comment in the PR, or post here in the mailing list
> > > https://github.com/apache/camel-website/pull/1071
> > >
> > >
> > >
> > >
> > > --
> > > Claus Ibsen
> > > -
> > > @davsclaus
> > > Camel in Action 2: https://www.manning.com/ibsen2
> > >
> >
>


-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Camel Kafka consumer dynamic topic

[Chirag]

Mark,

Scenario for service restart will cause any dynamic route from being removed.

It would require ReST to be idempotent and return or more complex -
persisting routes and reloading them.

ચિરાગ/चिराग/Chirag
--
Sent from My Gmail Account


On Thu, Oct 5, 2023 at 12:44 AM Mark Nuttall  wrote:
>
> The link that I provided shows you how to dynamically create a new route.
> So all you need to do is after you've called the rest in point using the
> HTTP component you use what's returned to call a service method in some
> Class via the bean EIP. The issue is that if the service restarts you will
> lose anything dynamically created. The pieces are all there it's just a
> matter of you putting them all together
>
> On Wed, Oct 4, 2023, 9:38 PM girish vasmatkar  wrote:
>
> > It's different to what my requirements are. I think dynamic routing in case
> > of a Kafka consumer is different to the examples you have quoted.
> >
> > I can't have Kafka topic declared in property file or a yaml file for that
> > matter because I wouldn't be knowing. In that case it would be pretty easy
> > to just use property place holders and build the route
> >
> > I need to be able to get a random kafka topic at runtime and create a kafka
> > consumer from it.
> >
> > - Girish
> >
> > On Wed, Oct 4, 2023, 2:58 AM Mark Nuttall  wrote:
> >
> > > Pretty sure I've done this kind of thing before. I think the info is here
> > >
> > >
> > https://stackoverflow.com/questions/48380456/dynamic-routing-apache-camel
> > >
> > > On Tue, Oct 3, 2023, 3:22 AM Claus Ibsen  wrote:
> > >
> > > > Hi
> > > >
> > > > If you use Java RouteBuilder, then you can maybe use spring dependency
> > > > injection, to inject the topic name into a String field,
> > > > and then use that field when building the route model.
> > > >
> > > > However I have also thought of adding {{bean:xxx?method=aaa}} as a
> > > function
> > > > to property placeholder.
> > > > Though need to give it a bit more thought as this then have
> > > > dependency ordering implication (bean must be created beforehand etc).
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Oct 2, 2023 at 12:40 PM girish vasmatkar 
> > > > wrote:
> > > >
> > > > > Dear Members,
> > > > >
> > > > > I have been searching for this for quite some time but haven't been
> > > able
> > > > to
> > > > > figure out a way to do this.
> > > > >
> > > > > I need to subscribe to a kafka topic which I will know only at
> > runtime
> > > -
> > > > by
> > > > > calling a REST endpoint. The REST API I am calling is creating topics
> > > and
> > > > > returning the topic name.
> > > > >
> > > > > I am using spring camel starter and have camel route defined up
> > until I
> > > > am
> > > > > able to fetch the topic name.  How do I then have a route created at
> > > > > runtime that uses the topic name from the rest endpoint?
> > > > >
> > > > > --
> > > > > Best,
> > > > > Girish Vasmatkar
> > > > >
> > > >
> > > >
> > > > --
> > > > Claus Ibsen
> > > > -
> > > > @davsclaus
> > > > Camel in Action 2: https://www.manning.com/ibsen2
> > > >
> > >
> >

Re: Camel 4 migration blog post - Feedback welcome

[Claus Ibsen]

Hi Raymond

Thanks for the feedback. Yes you are not the typical Camel end user, so
your migration "work" is more than normal.
The note about using PluginHelper etc is something we should highlight in
the blog post, thanks.



On Fri, Oct 6, 2023 at 10:52 AM ski n  wrote:

> Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.
>
> Currently, thus not migrating yet, but planning has started.
>
> The things I need to consider:
>
> - I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
> the docs for 2.0 are gone, is the archive to Camel 2.0 documentation still
> available? Like to compare sometimes.
> - Jakarta stuff in my own code/dependencies.
> - Jakarta stuff in other libraries (then Camel). For example ActiveMQ
> Classic (embedded) server doesn't support it yet.
> - Deprecated components: I use the following components:
>
> camel-atlasmap
> camel-directvm
> camel-elasticsearch-rest
> camel-rabbitmq
> camel-rest-swagger
> camel-websocket
> camel-vm
>
> So I need to change these dependencies and some code here
>
> - I use extendedCamelContext a lot, so need to check the API changes out.
> For example use the PluginHelper instead of the extendedCamelContext
> - Update SLFJ to version 2.0
>
> Will let you know if I'm running into any issues.
>
> I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
> https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
> recipe to move to Camel 4 then it would be probably easier to do the
> migration.
>
> Raymond
>
>
> On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen  wrote:
>
> > Hi
> >
> > If you have been migrating to Camel 4 already, then any feedback is
> > welcome.
> >
> > You can comment in the PR, or post here in the mailing list
> > https://github.com/apache/camel-website/pull/1071
> >
> >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>


-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Camel 4 migration blog post - Feedback welcome

[Andrea Cosentino]

The reason for camel-atlasmap to be removed was the fact Atlasmap was
discontinued.

There were no releases since October.

Il giorno ven 6 ott 2023 alle ore 11:34 Gerald Kallas
 ha scritto:

> Hi folks,
>
> what's the reason that camel-atlasmap has been removed in Camel 4. Is
> there any alternative planned for mapping processors that don't need to be
> coded?
>
> Best
> Gerald
>
> > ski n  hat am 06.10.2023 10:51 CEST
> geschrieben:
> >
> >
> > Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.
> >
> > Currently, thus not migrating yet, but planning has started.
> >
> > The things I need to consider:
> >
> > - I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
> > the docs for 2.0 are gone, is the archive to Camel 2.0 documentation
> still
> > available? Like to compare sometimes.
> > - Jakarta stuff in my own code/dependencies.
> > - Jakarta stuff in other libraries (then Camel). For example ActiveMQ
> > Classic (embedded) server doesn't support it yet.
> > - Deprecated components: I use the following components:
> >
> > camel-atlasmap
> > camel-directvm
> > camel-elasticsearch-rest
> > camel-rabbitmq
> > camel-rest-swagger
> > camel-websocket
> > camel-vm
> >
> > So I need to change these dependencies and some code here
> >
> > - I use extendedCamelContext a lot, so need to check the API changes out.
> > For example use the PluginHelper instead of the extendedCamelContext
> > - Update SLFJ to version 2.0
> >
> > Will let you know if I'm running into any issues.
> >
> > I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
> > https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
> > recipe to move to Camel 4 then it would be probably easier to do the
> > migration.
> >
> > Raymond
> >
> >
> > On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen 
> wrote:
> >
> > > Hi
> > >
> > > If you have been migrating to Camel 4 already, then any feedback is
> > > welcome.
> > >
> > > You can comment in the PR, or post here in the mailing list
> > > https://github.com/apache/camel-website/pull/1071
> > >
> > >
> > >
> > >
> > > --
> > > Claus Ibsen
> > > -
> > > @davsclaus
> > > Camel in Action 2: https://www.manning.com/ibsen2
> > >
>

Re: Camel 4 migration blog post - Feedback welcome

[Mikael Koskinen]

Hi,

For me the biggest issues regarding Camel 4 have been when Karavan VS Code
extension updated itself to 4.0.0. I'm not sure things related to Karavan
are discussed in the blog post but here's couple things we've encountered.

In short, if you use Karavan 4.x.x to open a route which has been designed
with the 3.x.x version, in many cases the route breaks as the designer
removes parts of it. I think the yaml schema has changed a bit between the
releases and the editor only supports the latest version with no backward
compatibility. In addition to that, the latest Karavan editor has some
issues UI issues (scrolling the property panel doesn't work as it should,
some textboxes are broken). We decided to downgrade the Karavan back to
3.x.x even though we updated Camel to 4.x.x.

The Camel 4 itself has been a rock solid release. No issues.

pe 6. lokak. 2023 klo 11.52 ski n (raymondmees...@gmail.com) kirjoitti:

> Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.
>
> Currently, thus not migrating yet, but planning has started.
>
> The things I need to consider:
>
> - I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
> the docs for 2.0 are gone, is the archive to Camel 2.0 documentation still
> available? Like to compare sometimes.
> - Jakarta stuff in my own code/dependencies.
> - Jakarta stuff in other libraries (then Camel). For example ActiveMQ
> Classic (embedded) server doesn't support it yet.
> - Deprecated components: I use the following components:
>
> camel-atlasmap
> camel-directvm
> camel-elasticsearch-rest
> camel-rabbitmq
> camel-rest-swagger
> camel-websocket
> camel-vm
>
> So I need to change these dependencies and some code here
>
> - I use extendedCamelContext a lot, so need to check the API changes out.
> For example use the PluginHelper instead of the extendedCamelContext
> - Update SLFJ to version 2.0
>
> Will let you know if I'm running into any issues.
>
> I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
> https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
> recipe to move to Camel 4 then it would be probably easier to do the
> migration.
>
> Raymond
>
>
> On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen  wrote:
>
> > Hi
> >
> > If you have been migrating to Camel 4 already, then any feedback is
> > welcome.
> >
> > You can comment in the PR, or post here in the mailing list
> > https://github.com/apache/camel-website/pull/1071
> >
> >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>

Re: Camel 4 migration blog post - Feedback welcome

[Gerald Kallas]

Hi folks,

what's the reason that camel-atlasmap has been removed in Camel 4. Is there any 
alternative planned for mapping processors that don't need to be coded?

Best
Gerald

> ski n  hat am 06.10.2023 10:51 CEST geschrieben:
> 
>  
> Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.
> 
> Currently, thus not migrating yet, but planning has started.
> 
> The things I need to consider:
> 
> - I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
> the docs for 2.0 are gone, is the archive to Camel 2.0 documentation still
> available? Like to compare sometimes.
> - Jakarta stuff in my own code/dependencies.
> - Jakarta stuff in other libraries (then Camel). For example ActiveMQ
> Classic (embedded) server doesn't support it yet.
> - Deprecated components: I use the following components:
> 
> camel-atlasmap
> camel-directvm
> camel-elasticsearch-rest
> camel-rabbitmq
> camel-rest-swagger
> camel-websocket
> camel-vm
> 
> So I need to change these dependencies and some code here
> 
> - I use extendedCamelContext a lot, so need to check the API changes out.
> For example use the PluginHelper instead of the extendedCamelContext
> - Update SLFJ to version 2.0
> 
> Will let you know if I'm running into any issues.
> 
> I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
> https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
> recipe to move to Camel 4 then it would be probably easier to do the
> migration.
> 
> Raymond
> 
> 
> On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen  wrote:
> 
> > Hi
> >
> > If you have been migrating to Camel 4 already, then any feedback is
> > welcome.
> >
> > You can comment in the PR, or post here in the mailing list
> > https://github.com/apache/camel-website/pull/1071
> >
> >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >

Re: Camel 4 migration blog post - Feedback welcome

[ski n]

Sure happy to give feedback. I plan to upgrade from 3.20.x to 4.x.

Currently, thus not migrating yet, but planning has started.

The things I need to consider:

- I first need to finish some stuff to upgrade from 2.x to 3.x (Currently
the docs for 2.0 are gone, is the archive to Camel 2.0 documentation still
available? Like to compare sometimes.
- Jakarta stuff in my own code/dependencies.
- Jakarta stuff in other libraries (then Camel). For example ActiveMQ
Classic (embedded) server doesn't support it yet.
- Deprecated components: I use the following components:

camel-atlasmap
camel-directvm
camel-elasticsearch-rest
camel-rabbitmq
camel-rest-swagger
camel-websocket
camel-vm

So I need to change these dependencies and some code here

- I use extendedCamelContext a lot, so need to check the API changes out.
For example use the PluginHelper instead of the extendedCamelContext
- Update SLFJ to version 2.0

Will let you know if I'm running into any issues.

I'm planning to use OpenRewrite to update to Spring Boot 3/Jakarta (
https://docs.openrewrite.org/recipes/java/spring/boot3). If there was a
recipe to move to Camel 4 then it would be probably easier to do the
migration.

Raymond


On Fri, Oct 6, 2023 at 9:49 AM Claus Ibsen  wrote:

> Hi
>
> If you have been migrating to Camel 4 already, then any feedback is
> welcome.
>
> You can comment in the PR, or post here in the mailing list
> https://github.com/apache/camel-website/pull/1071
>
>
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Camel 4 migration blog post - Feedback welcome

[Claus Ibsen]

Hi

If you have been migrating to Camel 4 already, then any feedback is welcome.

You can comment in the PR, or post here in the mailing list
https://github.com/apache/camel-website/pull/1071




-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Behavior of aggregation and file path in Camel Kafka Connector

[Andrea Cosentino]

Hello,

The aggregation works in the following way, if you have the message 1  and
2 and an aggregation size of 2, you'll get a file with 1,2 as content and
an exchange Id as file name if you don't specify the header, otherwise it
should be the last header you sent as file name.

It doesn't distinguish between file name, because it's a general camel
concept and is not related to the specific component.

If you have to do something like that, probably it would be better to use
plain camel, so you should be able to cover your case.

Cheers.

Il giorno ven 6 ott 2023 alle ore 08:57 Satoshi Yamada <
satoshi.yamada@gmail.com> ha scritto:

> Hi community,
>
> I'm using camel-azure-storage-blob-sink-kafka-connector from k8s, and let
> kafkaconnect upload the logs (Kafka messages) to Azure Blob.
> I've used message header to specify the file path, and also use aggregation
> options in Camel as follows
>
> camel.aggregation.size: 2
> camel.aggregation.timeout: 360
>
> I've expected Camel Kafka Connector to aggregate the logs per file
> name, which is specified in the message header. For example, if we
> produce 6 messages with the following messages headers in order
>
> Message 1: CamelHeader.file=A
> Message 2: CamelHeader.file=B
> Message 3: CamelHeader.file=C
> Message 4: CamelHeader.file=A
> Message 5: CamelHeader.file=B
> Message 6: CamelHeader.file=C
>
> Then, I expected files named "A" containing message 1 and 4, "B"
> containing message 2 and 5, and "C" containing message 3 and 6 will be
> uploaded to Azure, but actually it doesn't and it seems a file
> containing message 1 and 2 are there with file name "A" or "B", a file
> with message 3 and 4 with file name "C" or "A", and a file with
> message 5 and 6 with file name "B" or "C" are there.
>
> I want to confirm with the community if this is the expected behavior
> in Camel Kafka Connector.
> I'm using camel-azure-storage-blob-sink-kafka-connector-3.18.2 and use
> `uploadBlockBlob` for `CamelHeader.CamelAzureStorageBlobOperation`.
>
> Thanks in advance,
> Satoshi
>

How to configure camel.sink.marshal ?

[Satoshi Yamada]

Hi community,

Let me repost this cause I don't get replies in github issue here
https://github.com/apache/camel-kafka-connector/issues/1568

I'm testing camel-azure-storage-blob-sink-kafka-connector from k8s.
So far, I've confirmed CamelHeader option works
 and the log
is uploaded to an arbitrary path.
Now I'm testing camel.sink.marshal option to save some usage on Azure Blob
Storage, but haven't been successful yet.
I've followed some configuration like zipfile for camel.sink.marshal option,
but faced the following errors.

2023-09-27 03:14:35,322 ERROR [blob-connector|task-0]
WorkerSinkTask{id=blob-connector-0} Task threw an uncaught and
unrecoverable exception. Task is being killed and will not recover
until manually restarted (org.apache.kafka.connect.runtime.Work
erTask) [task-thread-blob-connector-0]
org.apache.kafka.connect.errors.ConnectException: Failed to create and
start Camel context
at 
org.apache.camel.kafkaconnector.CamelSinkTask.start(CamelSinkTask.java:159)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:315)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:200)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)
at 
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)
at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.camel.RuntimeCamelException:
org.apache.camel.VetoCamelContextStartException: Failure creating
route from template: ckcMarshal
at 
org.apache.camel.RuntimeCamelException.wrapRuntimeException(RuntimeCamelException.java:66)
at 
org.apache.camel.support.service.BaseService.doFail(BaseService.java:413)
at 
org.apache.camel.impl.engine.AbstractCamelContext.doFail(AbstractCamelContext.java:3550)
at 
org.apache.camel.support.service.BaseService.fail(BaseService.java:342)
at 
org.apache.camel.impl.engine.AbstractCamelContext.failOnStartup(AbstractCamelContext.java:5204)
at 
org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:2642)
at 
org.apache.camel.support.service.BaseService.start(BaseService.java:111)
at 
org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2649)
at 
org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:262)
at org.apache.camel.main.SimpleMain.doStart(SimpleMain.java:43)
at 
org.apache.camel.support.service.BaseService.start(BaseService.java:119)
at 
org.apache.camel.kafkaconnector.CamelSinkTask.start(CamelSinkTask.java:152)
... 9 more
Caused by: org.apache.camel.VetoCamelContextStartException: Failure
creating route from template: ckcMarshal
at 
org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.onContextInitialized(KameletComponent.java:433)
at 
org.apache.camel.impl.engine.AbstractCamelContext.doInit(AbstractCamelContext.java:2956)
at 
org.apache.camel.support.service.BaseService.init(BaseService.java:83)
at 
org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:2630)
... 15 more
Caused by: org.apache.camel.component.kamelet.KameletNotFoundException:
Kamelet with id ckcMarshal not found in locations: classpath:/kamelets
at 
org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.createRouteForEndpoint(KameletComponent.java:421)
at 
org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.onContextInitialized(KameletComponent.java:430)
... 18 more
Caused by: org.apache.camel.FailedToCreateRouteException: Failed to
create route ckcMarshal-4 at: >>>
Marshal[CustomDataFormat[{{marshal}}]] <<< in route:
Route(ckcMarshal-4)[From[kamelet://source?routeId=ckcMarshal...
because of Cannot find data format in registry with ref: zipfile
at 
org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:241)
at 
org.apache.camel.reifier.RouteReifier.createRoute(RouteReifier.java:75)
at 
org.apache.camel.impl.DefaultModelReifierFactory.createRoute(DefaultModelReifierFactory.java:49)
at 
org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:862)
at 
org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.createRouteForEndpoint(KameletComponent.java:416)
... 19 more
Caused by: java.lang.IllegalArgumentException: Cannot find data format
in 

Behavior of aggregation and file path in Camel Kafka Connector

[Satoshi Yamada]

Hi community,

I'm using camel-azure-storage-blob-sink-kafka-connector from k8s, and let
kafkaconnect upload the logs (Kafka messages) to Azure Blob.
I've used message header to specify the file path, and also use aggregation
options in Camel as follows

camel.aggregation.size: 2
camel.aggregation.timeout: 360

I've expected Camel Kafka Connector to aggregate the logs per file
name, which is specified in the message header. For example, if we
produce 6 messages with the following messages headers in order

Message 1: CamelHeader.file=A
Message 2: CamelHeader.file=B
Message 3: CamelHeader.file=C
Message 4: CamelHeader.file=A
Message 5: CamelHeader.file=B
Message 6: CamelHeader.file=C

Then, I expected files named "A" containing message 1 and 4, "B"
containing message 2 and 5, and "C" containing message 3 and 6 will be
uploaded to Azure, but actually it doesn't and it seems a file
containing message 1 and 2 are there with file name "A" or "B", a file
with message 3 and 4 with file name "C" or "A", and a file with
message 5 and 6 with file name "B" or "C" are there.

I want to confirm with the community if this is the expected behavior
in Camel Kafka Connector.
I'm using camel-azure-storage-blob-sink-kafka-connector-3.18.2 and use
`uploadBlockBlob` for `CamelHeader.CamelAzureStorageBlobOperation`.

Thanks in advance,
Satoshi