Hi, I am new to Camel and in Spring as well. I am facing a problem. We have a
big CSV file with around 50,000 record. In the database, we have two tables
IN_MSG and its child table IN_SPLIT_REC (One-to-many relationship)
Our task is to read the file (from a local directory), insert a record in
IN_MSG table corresponding to the whole file, Splitting the file based on
new line (So, we will have 50,000 split records) and insert a record in
IN_SPLIT_REC table corresponding to each split record (linking that with
parent record) and also writing the split record in output queue.
Also, we need to persist the data in IN_SPLIT_REC table and split record in
output queue one-by-one (Not at-a-time through a single transaction)
For this, we have defined a camel route as follows:
...
final Policy required = new
SpringTransactionPolicy(lookup("PROPAGATION_REQUIRED",
TransactionTemplate.class));
final Policy requires_new = new
SpringTransactionPolicy(lookup("PROPAGATION_REQUIRES_NEW",
TransactionTemplate.class));
from(fromURI)
.routeId(getRoute().getRouteId())
.policy(required)
.bean(auditMessagePersister, "storeAuditMessage") // persists data in IN_MSG
table.
.bean(auditMessagePersister, "invokeResolver") // do some unrelated task.
.end()
.choice() // start of outer choice
.when(header(ElementKey.IS_SPLIT_REQUIRED.getId()).isEqualTo(Boolean.TRUE))
// start of outer when
.split().method(SplitHelper.class,
"invokeSplitter").streaming().parallelProcessing()
.policy(requires_new)
.bean(SplitHelper.class, "invokeSplitEnricher") // persists
data in
IN_SPLIT_REC table and links to parent record.
.bean(recipientList, "determineDestRuleEndPoints")
.recipientList().header(Constants.ENDPOINT_LIST) // persists
split record
in output queue.
.end()
.end()
.endChoice() // end of outer when
.end() // end of outer choice
.end();
...
By this, we are able to achive our goal, that is the split records are
written in DB one-by-one and also the split messages are written in out
queue one-by-one. But after processing around 150-200 split records, the
Camel Route just hangs without throwing any error. Nothing is written in log
also.
Can anybody have any idea about what could go wrong. Thanks in advance.
NOTE: Our Max DB Connection Pool size is 20. Also, while we are trying to
shut down the Route Manually, it gives following message:
....
[2016-02-24 17:43:54,495]-[INFO ]-[ -
ShutdownTask]-[org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask.run():623]-Waiting
as there are still 2 inflight and pending exchanges to complete, timeout in
299 seconds. Inflights per route: [RP_CONSUMER = 2]
[2016-02-24 17:43:54,495]-[DEBUG]-[ -
ShutdownTask]-[org.apache.camel.impl.DefaultShutdownStrategy.logInflightExchanges():752]-There
are 22 inflight exchanges:
....
--
View this message in context:
http://camel.465427.n5.nabble.com/Camel-Route-hangs-while-processing-splitted-records-parallelly-tp5778194.html
Sent from the Camel Development mailing list archive at Nabble.com.
