Connection Pooling in (S)FTP Component

[ICS460-BAI]

Hi,

can somebody support us here please?

We tried without pooling the connections and just using disconnect!
However with hundreds of flows these constant reconnects are too much for the 
servers

Any idea what we can do here?

Thanks Christian

[signature_3350928388]

Dipl.-Ing. Christian Müller
Software Engineer
ICS460-BAI



Am 31.07.23, 09:56 schrieb "MÜLLER Christian (ICS460-BAI)" 
mailto:christian.muel...@spar-ics.com.inva>LID>:

Hello,

due to the fact that we have many transports on a single server we need the 
ability to pool/reuse ftp connections.
Otherwise, we have to many reconnects or to many open connections!

Do you have any recommendations how we can realize such a connection pool. We 
are currently trying to pool the entire SftpOperations Object.
This does not work because the SftpOperations Object includes the SftpEndpoint 
which cannot be reused in different routes!

Please support us here. We really would like to achieve this with camel!

Keep up the great work!
Regards Christian

[signature_449552271]

Dipl.-Ing. Christian Müller
Software Engineer
ICS460-BAI



[cid:image510439.PNG@7db0b49f.4ca7cecf]

DI Christian Müller
Software Engineer
Business Automation & Integration

Mobile:
+43 664 88748100
E-Mail:
christian.muel...@spar-ics.com

SPAR Business Services GmbH
Information & Communication Services
Europastrasse 3, 5015 Salzburg, Austria

[cid:image5e16dd.PNG@36ca1112.45abf134]

Sollten Sie diese E-Mail unbeabsichtigt bzw. irrtümlich erhalten haben, so 
weisen wir Sie darauf hin, dass gemäß § 161 Abs 4 TKG 2021 der Inhalt sowie die 
Tatsache des Empfangs dieser E-Mail weder aufgezeichnet noch verwertet oder 
Unbefugten mitgeteilt werden dürfen. Wir ersuchen Sie, die Nachricht von Ihrem 
System zu löschen und sich mit uns in Verbindung zu setzen.
Unter diesem Link finden Sie die Informationen über die Verarbeitung Ihrer 
personenbezogenen Daten sowie zu Ihren Rechten als betroffene Person:
https://www.spar.at/service/datenschutzhinweise

If you have received this email accidentally or in error, we point out that, in 
accordance with § 161 para. 4 TKG 2021 (Telecommunications Act), the contents 
of this email and the fact of its receipt must not be recorded, exploited or 
communicated to unauthorized persons. We ask you to delete the message from 
your system and to contact us.
Under this link you will find information about the processing of your personal 
data and about your rights as a data subject
https://www.spar.at/service/datenschutzhinweise

Re: ZeroMQ component

[Mark Webb]

Ahh that makes sense and is also very unfortunate.

Thanks for the info.

On Tue, Aug 1, 2023, 11:45 AM Andrea Cosentino  wrote:

> Until the license is not compatible with ASF license, we can't take into
> account to move it in the Apache Camel repository.
>
> Il mar 1 ago 2023, 17:40 Nicolas Filotto  ha scritto:
>
> > Hi,
> >
> > AFAIK the components in camel-extra have been moved there because their
> > license is not compatible with an Apache License which is a requirement
> to
> > be in the camel repository.
> >
> > In the case of ZeroMQ, it seems that the license is a Mozilla Public
> > License version 2.0 which is more restrictive than an Apache License
> which
> > is why it is not part of the Camel repository.
> >
> > Regards,
> > Nicolas
> > 
> > From: Mark Webb 
> > Sent: Tuesday, August 1, 2023 17:20
> > To: users@camel.apache.org 
> > Subject: ZeroMQ component
> >
> > I know that the camel-zeromq component exists in the camel-extra repo, is
> > there any chance it could once again find its way into the current
> > "official" trunk?  I'd even be willing to lead and support the effort.
> >
> > I'd prefer to make it based off of JeroMQ and not the Akka version.
> >
> > Would this be possible?
> >
> > - Mark
> >
> > As a recipient of an email from the Talend Group, your personal data will
> > be processed by our systems. Please see our Privacy Notice <
> > https://www.talend.com/privacy-policy/> for more information about our
> > collection and use of your personal information, our security practices,
> > and your data protection rights, including any rights you may have to
> > object to automated-decision making or profiling we use to analyze
> support
> > or marketing related communications. To manage or discontinue promotional
> > communications, use the communication preferences portal<
> > https://info.talend.com/emailpreferencesen.html>. To exercise your data
> > protection rights, use the privacy request form<
> >
> https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl
> >.
> > Contact us here  or by mail to either
> of
> > our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400,
> San
> > Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150
> Suresnes,
> > France
> >
>

Re: ZeroMQ component

[Andrea Cosentino]

Until the license is not compatible with ASF license, we can't take into
account to move it in the Apache Camel repository.

Il mar 1 ago 2023, 17:40 Nicolas Filotto  ha scritto:

> Hi,
>
> AFAIK the components in camel-extra have been moved there because their
> license is not compatible with an Apache License which is a requirement to
> be in the camel repository.
>
> In the case of ZeroMQ, it seems that the license is a Mozilla Public
> License version 2.0 which is more restrictive than an Apache License which
> is why it is not part of the Camel repository.
>
> Regards,
> Nicolas
> 
> From: Mark Webb 
> Sent: Tuesday, August 1, 2023 17:20
> To: users@camel.apache.org 
> Subject: ZeroMQ component
>
> I know that the camel-zeromq component exists in the camel-extra repo, is
> there any chance it could once again find its way into the current
> "official" trunk?  I'd even be willing to lead and support the effort.
>
> I'd prefer to make it based off of JeroMQ and not the Akka version.
>
> Would this be possible?
>
> - Mark
>
> As a recipient of an email from the Talend Group, your personal data will
> be processed by our systems. Please see our Privacy Notice <
> https://www.talend.com/privacy-policy/> for more information about our
> collection and use of your personal information, our security practices,
> and your data protection rights, including any rights you may have to
> object to automated-decision making or profiling we use to analyze support
> or marketing related communications. To manage or discontinue promotional
> communications, use the communication preferences portal<
> https://info.talend.com/emailpreferencesen.html>. To exercise your data
> protection rights, use the privacy request form<
> https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl>.
> Contact us here  or by mail to either of
> our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San
> Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes,
> France
>

Re: ZeroMQ component

[Nicolas Filotto]

Hi,

AFAIK the components in camel-extra have been moved there because their license 
is not compatible with an Apache License which is a requirement to be in the 
camel repository.

In the case of ZeroMQ, it seems that the license is a Mozilla Public License 
version 2.0 which is more restrictive than an Apache License which is why it is 
not part of the Camel repository.

Regards,
Nicolas

From: Mark Webb 
Sent: Tuesday, August 1, 2023 17:20
To: users@camel.apache.org 
Subject: ZeroMQ component

I know that the camel-zeromq component exists in the camel-extra repo, is
there any chance it could once again find its way into the current
"official" trunk?  I'd even be willing to lead and support the effort.

I'd prefer to make it based off of JeroMQ and not the Akka version.

Would this be possible?

- Mark

As a recipient of an email from the Talend Group, your personal data will be 
processed by our systems. Please see our Privacy Notice 
 for more information about our 
collection and use of your personal information, our security practices, and 
your data protection rights, including any rights you may have to object to 
automated-decision making or profiling we use to analyze support or marketing 
related communications. To manage or discontinue promotional communications, 
use the communication preferences 
portal. To exercise your data 
protection rights, use the privacy request 
form.
 Contact us here  or by mail to either of our 
co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San Mateo, 
CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes, France

ZeroMQ component

[Mark Webb]

I know that the camel-zeromq component exists in the camel-extra repo, is
there any chance it could once again find its way into the current
"official" trunk?  I'd even be willing to lead and support the effort.

I'd prefer to make it based off of JeroMQ and not the Akka version.

Would this be possible?

- Mark

Re: loading routes from external xml files

[Ricardo Garcia]

@ski n
Simply:
camelContext.getRoutesLoader() ->
org.apache.camel.support.
PluginHelper.getRoutesLoader(camelContext)

On Tue, Aug 1, 2023 at 8:34 AM ski n  wrote:

> OK, good to know as I use this functionality also heavily. Some
> questions/remarks about this:
>
> 1. What's the reason that it was (re)moved? Is there a new concept behind
> the PluginHelper?
> 2. Maybe good to add this to the migration guide (
> https://camel.apache.org/manual/camel-4-migration-guide.html).
> 3. @chipu. Would be nice if you can post your new solution here?
>
> --
>
> Raymond
>
> On Tue, Aug 1, 2023 at 7:56 AM Claus Ibsen  wrote:
>
> > Hi
> >
> > You can use org.apache.camel.support.PluginHelper
> >
> > On Mon, Jul 31, 2023 at 11:04 PM Ricardo Garcia 
> > wrote:
> >
> > > Hi, I am migrating to camel 4, as part of a broader migration to all
> > > jakarta API instead of javax API
> > > In camel 3 I could load routes from multiple external xml files:
> > >
> > > try (DirectoryStream ds = Files.newDirectoryStream(routesDir,
> > "*.xml"
> > > );
> > > DefaultResourceResolvers.FileResolver resolver = new
> > > DefaultResourceResolvers.FileResolver()) {
> > > resolver.setCamelContext(camelContext);
> > > for(Path file : ds)
> > > try {
> > > log.debug("Loading routes in {}", file);
> > > Resource resource = resolver.resolve(file.toUri().toString());
> > > camelContext.getRoutesLoader().loadRoutes(resource);
> > > } catch(Exception ex) {
> > > log.error("Cannot load routes in " + file, ex);
> > > }
> > > }
> > > But in camel 4 RC2, "getRoutesLoader()" does not exist in
> > > DefaultCamelContext, nor in ExtendedCamelContext... so how should we do
> > it
> > > then?
> > > Thanks
> > >
> > > --
> > > Chipu
> > >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>


-- 
Chipu

Re: EventNotifier and direct/seda endpoints in Camel 4

[Claus Ibsen]

Hi

Yes you can use steps to group X together and have events emitted for
start/complete

On Tue, Aug 1, 2023 at 9:13 AM ski n  wrote:

> Would this possibly work?
>
> 
>  
>
>
> 
>  
>
>  
>   
>
> And then use the StepStartEvent (or StepCompletedEvent). Or is this on
> on lifecycle level and not on a message exchange level?
>
> --
>
> Raymond
>
>
> On Tue, Aug 1, 2023 at 8:58 AM Claus Ibsen  wrote:
>
> > Hi
> >
> > No there is not
> >
> > On Tue, Aug 1, 2023 at 8:49 AM ski n  wrote:
> >
> > > Currently, I use the EventNotifier to collect events in Camel:
> > >
> > >
> > >
> >
> https://www.javadoc.io/doc/org.apache.camel/camel-api/3.21.0/org/apache/camel/spi/CamelEvent.html
> > >
> > > I have integrations that consist of 1 or more routes like this:
> > >
> > > 
> > >  
> > >  
> > >  
> > > 
> > >  
> > >  
> > >  
> > >
> > > To route between two routes I use either:
> > >
> > > - direct-vm
> > > - vm
> > > - activemq
> > >
> > > When I filter the Events on ExchangeCreated I get for each route that
> is
> > > created exactly one message. 2 routes gives 2 events, 10 routes give 10
> > > events.
> > >
> > > In camel 4 the direct-vm and vm are removed and the direct and seda
> > > endpoints can be used as alternatives. This however sees all the routes
> > as
> > > one exchange, so that there is only one ExchangeCreated event. Other
> > events
> > > like ExchangeSent give more events, but they are not 1 on 1 with a
> > specific
> > > route.
> > >
> > > Is there a way to get/keep this functionality in Camel 4 when switching
> > to
> > > the direct endpoint?
> > >
> > > --
> > >
> > > Raymond
> > >
> >
> >
> > --
> > Claus Ibsen
> > -
> > @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>


-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: EventNotifier and direct/seda endpoints in Camel 4

[ski n]

Would this possibly work?


 
   
   

 
   
 
  

And then use the StepStartEvent (or StepCompletedEvent). Or is this on
on lifecycle level and not on a message exchange level?

--

Raymond


On Tue, Aug 1, 2023 at 8:58 AM Claus Ibsen  wrote:

> Hi
>
> No there is not
>
> On Tue, Aug 1, 2023 at 8:49 AM ski n  wrote:
>
> > Currently, I use the EventNotifier to collect events in Camel:
> >
> >
> >
> https://www.javadoc.io/doc/org.apache.camel/camel-api/3.21.0/org/apache/camel/spi/CamelEvent.html
> >
> > I have integrations that consist of 1 or more routes like this:
> >
> > 
> >  
> >  
> >  
> > 
> >  
> >  
> >  
> >
> > To route between two routes I use either:
> >
> > - direct-vm
> > - vm
> > - activemq
> >
> > When I filter the Events on ExchangeCreated I get for each route that is
> > created exactly one message. 2 routes gives 2 events, 10 routes give 10
> > events.
> >
> > In camel 4 the direct-vm and vm are removed and the direct and seda
> > endpoints can be used as alternatives. This however sees all the routes
> as
> > one exchange, so that there is only one ExchangeCreated event. Other
> events
> > like ExchangeSent give more events, but they are not 1 on 1 with a
> specific
> > route.
> >
> > Is there a way to get/keep this functionality in Camel 4 when switching
> to
> > the direct endpoint?
> >
> > --
> >
> > Raymond
> >
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Re: EventNotifier and direct/seda endpoints in Camel 4

[Claus Ibsen]

Hi

No there is not

On Tue, Aug 1, 2023 at 8:49 AM ski n  wrote:

> Currently, I use the EventNotifier to collect events in Camel:
>
>
> https://www.javadoc.io/doc/org.apache.camel/camel-api/3.21.0/org/apache/camel/spi/CamelEvent.html
>
> I have integrations that consist of 1 or more routes like this:
>
> 
>  
>  
>  
> 
>  
>  
>  
>
> To route between two routes I use either:
>
> - direct-vm
> - vm
> - activemq
>
> When I filter the Events on ExchangeCreated I get for each route that is
> created exactly one message. 2 routes gives 2 events, 10 routes give 10
> events.
>
> In camel 4 the direct-vm and vm are removed and the direct and seda
> endpoints can be used as alternatives. This however sees all the routes as
> one exchange, so that there is only one ExchangeCreated event. Other events
> like ExchangeSent give more events, but they are not 1 on 1 with a specific
> route.
>
> Is there a way to get/keep this functionality in Camel 4 when switching to
> the direct endpoint?
>
> --
>
> Raymond
>


-- 
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

EventNotifier and direct/seda endpoints in Camel 4

[ski n]

Currently, I use the EventNotifier to collect events in Camel:

https://www.javadoc.io/doc/org.apache.camel/camel-api/3.21.0/org/apache/camel/spi/CamelEvent.html

I have integrations that consist of 1 or more routes like this:


 
 
 

 
 
 

To route between two routes I use either:

- direct-vm
- vm
- activemq

When I filter the Events on ExchangeCreated I get for each route that is
created exactly one message. 2 routes gives 2 events, 10 routes give 10
events.

In camel 4 the direct-vm and vm are removed and the direct and seda
endpoints can be used as alternatives. This however sees all the routes as
one exchange, so that there is only one ExchangeCreated event. Other events
like ExchangeSent give more events, but they are not 1 on 1 with a specific
route.

Is there a way to get/keep this functionality in Camel 4 when switching to
the direct endpoint?

--

Raymond

Re: loading routes from external xml files

[ski n]

OK, good to know as I use this functionality also heavily. Some
questions/remarks about this:

1. What's the reason that it was (re)moved? Is there a new concept behind
the PluginHelper?
2. Maybe good to add this to the migration guide (
https://camel.apache.org/manual/camel-4-migration-guide.html).
3. @chipu. Would be nice if you can post your new solution here?

--

Raymond

On Tue, Aug 1, 2023 at 7:56 AM Claus Ibsen  wrote:

> Hi
>
> You can use org.apache.camel.support.PluginHelper
>
> On Mon, Jul 31, 2023 at 11:04 PM Ricardo Garcia 
> wrote:
>
> > Hi, I am migrating to camel 4, as part of a broader migration to all
> > jakarta API instead of javax API
> > In camel 3 I could load routes from multiple external xml files:
> >
> > try (DirectoryStream ds = Files.newDirectoryStream(routesDir,
> "*.xml"
> > );
> > DefaultResourceResolvers.FileResolver resolver = new
> > DefaultResourceResolvers.FileResolver()) {
> > resolver.setCamelContext(camelContext);
> > for(Path file : ds)
> > try {
> > log.debug("Loading routes in {}", file);
> > Resource resource = resolver.resolve(file.toUri().toString());
> > camelContext.getRoutesLoader().loadRoutes(resource);
> > } catch(Exception ex) {
> > log.error("Cannot load routes in " + file, ex);
> > }
> > }
> > But in camel 4 RC2, "getRoutesLoader()" does not exist in
> > DefaultCamelContext, nor in ExtendedCamelContext... so how should we do
> it
> > then?
> > Thanks
> >
> > --
> > Chipu
> >
>
>
> --
> Claus Ibsen
> -
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Re: How to read data from a paginated API and upload it as multipart file to S3 page by page

[Claussnitzer, Ralf]

Hi Sahil,

as far as I know the HTTP producer does not support polling based on
headers. You could trigger the individual HTTP requests by using sort
of an event loop:

Event/Message driven consumer route (direct or jms) that makes the
actual request and emits a follow-up event for the next page based on
the returned result (like tail recursion). Now you just need to figure
out who sends the first initial event.

Some folks uses a timer based approach, which in my opinion has to many
drawbacks, because you get additional latency, difficult error handling
and have to stop polling using the control bus.

Regards,
Ralf

On Mon, 2023-07-31 at 13:09 +0530, Sahil Upneja wrote:
> Hi Everyone !
> 
> I am trying to read data from a paginated API , Since the whole data
> is
> huge so I want to read one page from paginated API and upload it as
> multipart file to S3 and then read 2nd page & upload it to S3 and so
> on.
> 
> Could you please let me know if this usecase is achievable through
> Camel.
> 
> If yes, could you please also share a little example on how to
> achieve this.
> 
> Regards
> Sahil


smime.p7s
Description: S/MIME cryptographic signature