Hi Petr,
Missed this email last week.
1. The thread dump should be captured when the issue happens. From the thread
dump you sent looks like everything is fine. Also we need to get the state of
all threads, not just one. Thread dumps are long. So redirect your dump to a
file and search for:" java.lang.Thread.State ". You should not see all your
threads waiting or worse: dead lock !
This thread dump looks ok: Camel is waiting (TIMED_WAIT) for a file(*.csv) to
come into the folder: {//appl/cdr/rated-cdr }
If all you threads are in TIMED_WAIT, then we have a problem in the read
part[the from(...) part)
2. Looked at your old log messages again: looks like the problem is with writing
{thread #23 org.apache.camel.component.file.remote.RemoteFileProducer -
handleFailedWrite...}
Do you have enough space allocated on the target disk to save the rated CDRs ?
Telco cdr files are normally huge.
3. To confirm if the threads are stuck in read(file) or write(sftp), put a lot
of log messages on the route
Regards,
Prajod
-----Original Message-----
From: Petr Sponer [mailto:[email protected]]
Sent: Wednesday, August 13, 2014 3:10 AM
To: [email protected]
Subject: RE: sftp hang-ups
Hi Prajod,
I am sending attached java thread dump (dataexporter.jstack). Most likely the
problematic threads are these, right?
PSp
"Camel (dataExporter) thread #29 - file:////appl/cdr/rated-cdr" daemon prio=10
tid=0x00007f0be0962800 nid=0x3e51 in Object.wait() [0x00007f0bb1ad8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.io.PipedInputStream.read(PipedInputStream.java:310)
- locked <0x00000004162e9e68> (a
com.jcraft.jsch.Channel$MyPipedInputStream)
at java.io.PipedInputStream.read(PipedInputStream.java:361)
- locked <0x00000004162e9e68> (a
com.jcraft.jsch.Channel$MyPipedInputStream)
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2665)
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2691)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1556)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1466)
at
org.apache.camel.component.file.remote.SftpOperations.existsFile(SftpOperations.java:929)
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:134)
at
org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:49)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:132)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:307)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:127)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:352)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:199)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:165)
at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Thursday, August 7, 2014 10:51 AM
To: [email protected]
Subject: RE: sftp hang-ups
Hi Petr,
This could be :
1. Thread pool exhausted(camel uses thread pools, the size of which is
configurable) 2. Thread blocked due to sftp not completing. If one thread hangs
in this state, a new thread will handle the next file. This second thread could
also get blocked; soon all threads in your thread pool will be exhausted, and
all file transfer will stop 3. Some programming logic error which is blocking
your file read listeners(this is a low probability cause)
Easiest was is to capture a "java thread dump" of the Camel JVM and send it to
us. it will show the state of each thread used by Camel.
Prajod
________________________________________
From: Petr Šponer [[email protected]]
Sent: Thursday, August 07, 2014 1:31 PM
To: [email protected]
Subject: sftp hang-ups
Hi,
I have a camel (2.12.0) in a production environment of a telco virtual
operator. We are having a strange behavior in the route defined like this:
if (ftpUploadEnabled){
from("file:///{{rated_cdr.output_folder}}?include=VPOB_X_FUOM_CONSUM.D.*\\.c
sv&delete=true").routeId("exportCdrCounterConsumptionDwhFtp")
.onException(Exception.class).handled(true)
.log(LoggingLevel.ERROR, "Sending
${file:name} file to FTP has failed.")
.log(LoggingLevel.ERROR, "Exception
occurred due - ${exception.message}")
.log(LoggingLevel.ERROR, "Stacktrace
- ${exception.stacktrace}")
.end()
.to("bean:fileUtility?method=copyToError")
.log(LoggingLevel.DEBUG, "Sending ${file:name} file to
FTP.")
.to("{{rated_cdr.ftp.dwh.proto}}://{{rated_cdr.ftp.dwh.username}}@{{rated_cd
r.ftp.dwh.url}}{{rated_cdr.ftp.dwh.output_folder}}" +
"?password=RAW({{rated_cdr.ftp.dwh.password}})" +
"&disconnect=true" +
"&tempFileName={{rated_cdr.ftp.dwh.work_folder_relative_to_output}}/${file:n
ame}")
.to("bean:fileUtility?method=moveFromErrorToArchive")
.log(LoggingLevel.INFO, "${file:name} file has been
sent to DWH.");
rated_cdr.ftp.dwh.enabled=true
rated_cdr.ftp.dwh.url=some.server
rated_cdr.ftp.dwh.proto=sftp
rated_cdr.ftp.dwh.work_folder_relative_to_output=../tmp
rated_cdr.ftp.dwh.output_folder=/daily_extracts_storage/rated_cdr
rated_cdr.ftp.dwh.username= some.user
rated_cdr.ftp.dwh.password= some.pass
The route starts and is working well in the beginning. It works fine for some
time but then it stops delivering files to SFTP (I don't see any deterministic
pattern). The last file remains in a locked state
(*.camelLock) and when I turn off the application I have warnings in a log:
10:36:55.348 [Thread-4] WARN org.apache.camel.impl.DefaultShutdownStrategy
- doShutdown - Timeout occurred. Now forcing the routes to be shutdown now.
10:36:55.349 [Camel (dataExporter) thread #31 - ShutdownTask] WARN
org.apache.camel.impl.DefaultShutdownStrategy - run - Interrupted while waiting
during graceful shutdown, will force shutdown now.
10:36:55.442 [Camel (dataExporter) thread #23 - file:////appl/cdr/rated-cdr]
WARN org.apache.camel.component.file.remote.RemoteFileProducer -
handleFailedWrite - Writing file failed with:
10:36:55.443 [Camel (dataExporter) thread #23 - file:////appl/cdr/rated-cdr]
WARN org.apache.camel.component.file.GenericFileOnCompletion -
processStrategyRollback - Rollback file strategy:
org.apache.camel.component.file.strategy.GenericFileDeleteProcessStrategy@35
3158cf for file:
GenericFile[/appl/cdr/rated-cdr/VPOB_X_FUOM_CONSUM.D.201407311334.D.004.csv]
(END)
Anyone have an idea what could be the problem? I'd even appreciate a one-time
commercial support because the problem is going to be a bit too critical :)
Thanks Petr Sponer
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should
check this email and any attachments for the presence of viruses. The company
accepts no liability for any damage caused by any virus transmitted by this
email.
www.wipro.com
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should
check this email and any attachments for the presence of viruses. The company
accepts no liability for any damage caused by any virus transmitted by this
email.
www.wipro.com
