Help with Camel Spark component

Hi. Our company has been using Apache Camel for various purposes for some time. Now we want to integrate with a Apache Spark cluster for big data processing, and we would like to use Camel as much as possible. The latest release of Camel came with a new component, for running Spark jobs. The docum

Camel + ActiveMQ shutdown timeout problem

Hello, We are using ServiceMix as a container, with an embedded ActiveMQ broker and some Camel routes deployed as JBI service assemblies. ServiceMix is registered as a Windows service, so when the server machine shuts down, the service is also shut down. The current timeout set on ServiceMix is 10

Re: onCompletion not triggerd in testing

Everything makes more sense now, thank you for the detailed explanation. -- View this message in context: http://camel.465427.n5.nabble.com/onCompletion-not-triggerd-in-testing-tp5743163p5743559.html Sent from the Camel - Users mailing list archive at Nabble.com.

Re: onCompletion not triggerd in testing

Thank you for pointing that out for me, now the test is working. Still, I find it weird that the consumer is not retrieving anything even though JConsole shows that the QueueSize > 0. -- View this message in context: http://camel.465427.n5.nabble.com/onCompletion-not-triggerd-in-testing-tp57431

Re: onCompletion not triggerd in testing

I found out the problem which was making my test not reach the onComplete syncProcessor but I am having another issue now. Inside the processor I use producer.sendBody(buildDurableSubscription(register), holder);, where producer is created in the context.xml with and return consumer.receiveBodyNoW

Re: onCompletion not triggerd in testing

My bad, I was looking over the libs of the project and I was tricked by a activemq-camel library. The version is 2.7.3-fuse-00-89. -- View this message in context: http://camel.465427.n5.nabble.com/onCompletion-not-triggerd-in-testing-tp5743163p5743269.html Sent from the Camel - Users mailing l

Re: onCompletion not triggerd in testing

Hi Claus, I'm using Apache Camel 5.5.1.fuse-70-097 and I have to stick with this version. But as I mentioned above, I have the same route in production and is working just fine, the onCompletion is always triggered and it does its job. The only problem is that I would have liked to have a unit te

Re: Loop in Camel - Calling a service till I receive required status back

Maybe I don't see the whole picture, but why wouldn't you do this in the processor? For example: public void process(Exchange exchange) throws Exception { processMessage(); callWebService(JSON); do { // add some kind of sleep or timer task or anything else response

Re: onCompletion not triggerd in testing

Hello, I'm back with some code. The mock route builder: @Component public class MockedTransactionToDBPublisherRouteBuilder extends TransactionToDBPublisherRouteBuilder { /** * @throws java.sql.SQLException */ public MockedTransactionToDBPublisherRouteBuilder(DataSource datasourc

Re: count of processed messages when using aggregation

That's the beauty of the stream resequencer, it allows the messages to go one by one as they get ordered and the difference in speed is very little for the overall process. The batch aggregator waits for all the exchanges to completed and then it send them to the aggregator, which will bring additi

Re: count of processed messages when using aggregation

This problem can be solved by a stream resequencer. (http://camel.apache.org/resequencer.html - stream resequencing) It will reorder the exchanges after being processed and send them to the aggregator. You have to reorder them so that the exchange with the split complete is always the last one, and

Re: onCompletion not triggerd in testing

I will post something tomorrow from work, but the basic idea is that I send 2 kinds of messages and Message Type 2 should be integrated only when all the Message Type 1 messages are in the database. If this hasn't happened when MT2 arrives on the "when" route and is processed by the syncProcessor t

Re: count of processed messages when using aggregation

You could set a count header on each exchange which leaves ".bean("myProcessor", "doWork")" and in the end log the number of the last exchange or have the ".bean("myProcessor", "doWork")" increase an internal counter and when you receive CamelSplitComplete you go into the myProcessor bean again and

onCompletion not triggerd in testing

Hello, I am using the following route: protected void createRoute(Processor processor) { from(getInputEndpoint()) .onCompletion() .onCompleteOnly() .onWhen(customPredicate) .process(syncProcessor) .end()

Re: Exceptions when aggregating messages

Thanks for making things more clear. I read the documentation and I understand your point. -- View this message in context: http://camel.465427.n5.nabble.com/Exceptions-when-aggregating-messages-tp5735523p5735830.html Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Exceptions when aggregating messages

I'm afraid that I don't understand which option you are talking about. And how can I "care about the the property value which could not be serialized", please explain in more details if possible? Thanks. -- View this message in context: http://camel.465427.n5.nabble.com/Exceptions-when-aggrega

Re: Exceptions when aggregating messages

It seems that I was trying to serialize the exchange properties, which is nonsense because the exchange that arrives on the new route will contain it's own properties, and everything was failing probably when trying to serialize the following property: *CamelFileExchangeFile=GenericFile[path\filena

Re: Exceptions when aggregating messages

Hello and thank you for your reply. This is the code for aggregating the whole camel message: class ObjectsAggregationStrategy implements AggregationStrategy { final Logger logger = Logger.getLogger(ObjectsAggregationStrategy.class); @SuppressWarnings("unchecked") @O

Exceptions when aggregating messages

Hello, I'm trying to aggregate a bunch of messages, each containing a String as body, the In message from each exchange that comes to the aggregator, so that I can send only one exchange containing the list of strings to the next route in the end. If I set the body as a List the send operation fa

Synchronizing camel processors

Hello, I'm using camel routes for processing and I thought that the process is synchronous, unless parallel processing is specified. I have a processor that unmarshalls from xml strings and I noticed the following in the logging: DefaultMessageListenerContainer-2 | UnmarshallerProcessor| roces

Re: Web server in ServiceMix with camel-jetty or camel-servlet

We have decided to go with an embedded Apache MINA FtpServer for increased performance and scalability. I think that the FtpServer is lighter than an embedded Jetty server and it servers our needs perfectly. Thank you for your support. -- View this message in context: http://camel.465427.n5.n

Re: Mixing ChoiceDefinition with DelayDefinition

Hello Claus, I have one more question. This is the working route, as per your advice: from(startEndpoint) .choice() .when(fileIsNotComplete()) .to(tempEndpoint) .otherwise() .delay(5 * COMPLETION_TIMEOUT)

Re: Mixing ChoiceDefinition with DelayDefinition

Thanks for your quick answer. Now I only have to adjust the delay. -- View this message in context: http://camel.465427.n5.nabble.com/Mixing-ChoiceDefinition-with-DelayDefinition-tp5734524p5734536.html Sent from the Camel - Users mailing list archive at Nabble.com.

Mixing ChoiceDefinition with DelayDefinition

Hello, I have the following route: from(startEndpoint) .choice() .when(fileIsComplete()).delay(COMPLETION_TIMEOUT).process(processor).to(outputEndpoint) .otherwise().aggregate(new ObjectsAggregator()).constant(true) .completionSize(BATCH_SIZE).completionTimeout(COMPLETION_TIMEOUT) .process(process

Web server in ServiceMix with camel-jetty or camel-servlet

Hello, We are trying to send blob messages (large files) through an ActiveMQ broker integrated in ServiceMix. It seems that this operation requires a ftp or a http server for upload and download and we are wondering if this can be achieved with a servicemix camel component implementation. I am loo

CXFRS producer deployed in Fuse ESB

Hello everybody, I'm using Apache Fuse ESB 5.4.0-fuse-00-00 with Apache Camel 2.4.0-fuse-00-00. I need to expose a REST service that can be called by a consumer located on another machine. I read about Camel CXFRS but I'm not sure if it is what I need. After reading here

Re: Aggregate exchanges until before EoF

Yes, I already used aggregators to build batches of data that I will later use for db operations and they work great. But I don't know how to implement a custom splitter that, for example, sets the file complet flag only on the last split line, is this possible? -- View this message in context:

Re: Aggregate exchanges until before EoF

No, I want to send that last line without aggregating it, or aggregating it as the first and only exchange of that aggregation strategy. What I actually need is to have a final exchange, containing a line from the file and the FILE_COMPLETE header, that gets sent through a few service units, each p

Aggregate exchanges until before EoF

Hello, Hi guys, I'm processing a file line by line and I want to aggregate all the exchanges (lines) until the one that has "Exchange.SPLIT_COMPLETE", basically I want to send the aggregated exchanges just before aggregating the final one. When the last exchange arrives it should be in the case w

Re: handling large files

I know that this is an old post but I'm in the exact situation and I can't find a way out. Some of you already know what I process with Apache Camel, but I will add the link again: http://camel.465427.n5.nabble.com/Large-file-processing-with-Apache-Camel-td5727977.html And here is a detailed descr

Re: Large file processing with Apache Camel

After digging more into my problem I found that the slow db access was the main issue, maybe you heard before of setting sendStringParametersAsUnicode=false in the jdbc driver to dramatically increase the performance. Since the last time I posted here I learned a lot about apache camel and I imple

Re: Large file processing with Apache Camel

Thank you everybody for your help. After reading and trying, I found how to implement splitter and aggregators and I managed to achieve my scope. Here is the new status: 1. read 500 lines from the file and send them in an exchange to the next service unit, to a certain endpoint according to the fi

Re: Large file processing with Apache Camel

I will try to provide the steps that are in the current version: 1. read one line from the file, set it as the outbound message's body of an exchange, and, according to the file type, send the exchange to an activemq queue 2. the exchange will arrive on another service unit that has a processor whi

Re: Large file processing with Apache Camel

Many thanks for your reply. When I read from the file I read simple lines, not XML. It takes more than one hour to read, process and insert into the db 20.000 lines so I took out the service unit that does the db operations and I was left with 26 minutes for reading from the file line by line, con

Large file processing with Apache Camel

Hello everybody, I'm using Apache Fuse ESB with Apache Camel 2.4.0 (I think) to process some large files. Until now a service unit deployed in servicemix would read the file line by line, create and send an exchange containing that line to another service unit that would analyze the line and trans