from("
sftp://X.X.X.X:2222/ts/sdf?username=sftpuser&password=*****&delay=100&initialDelay=100&streamDownload=false&recursive=false&readLock=changed&disconnect=true&KnownHostsFile=~/.ssh/known_hosts&localWorkDirectory=/tmp&moveFailed=/ts/error&include=.*\.xml&move=inprogress";)
.routePolicy(new ZooKeeperRoutePolicy("{{zookeeper." +
feedName + ".forwarder.endpoint}}", 1))
.to("sftp://Y.Y.Y.T:2222/sftp/sdf?username=sftpuser&password=*****&fileExist=move&moveExisting=20170112221528-***.xml&KnownHostsFile=~/.ssh/known_hosts";);
I narrow done the issue to above code snippet, its FTPing the file from 1
FTP server to another with move=inprogress folder using the
ZooKeeperRoutePolicy.
- it works as expected on the single instance.
- on having multiple instance(of camel route) , FTPing is successful but
move file to inprogress is failing due to below error. However this error
happen only on the first file(205.xml). As you can see on the log, the
subsequence file(206.xml) are transfer sucessfully as expected
according to error message it is "Caused by: com.jcraft.jsch.SftpException:
No such file."
But if i remove the "move=inprogress" from the route, i can see the file in
/sftp/sd folder.
is there any other way to FTP the file and move the file to folder(using
zookeeper for leader election)? or how do i eliminate this error?
2017-01-12T00:11:31.768517747Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[o.a.c.c.z.policy.ZooKeeperElection :138] Initializing ZookeeperElection
with uri '{{zookeeper.sdf.forwarder.endpoint}}'
2017-01-12T00:11:31.788697843Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
2017-01-12T00:11:31.788763764Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:host.name=a810a44c767c
2017-01-12T00:11:31.788774252Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.version=1.8.0_60
2017-01-12T00:11:31.788783141Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.vendor=Oracle Corporation
2017-01-12T00:11:31.788791331Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.home=/opt/jre1.8.0_60
2017-01-12T00:11:31.789350564Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.class.path=/app.jar
2017-01-12T00:11:31.789388119Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-12T00:11:31.790108758Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.io.tmpdir=/tmp
2017-01-12T00:11:31.791108364Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:java.compiler=<NA>
2017-01-12T00:11:31.791583396Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client environment:os.name=Linux
2017-01-12T00:11:31.792170295Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client environment:os.arch=amd64
2017-01-12T00:11:31.792194966Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:os.version=4.4.39-moby
2017-01-12T00:11:31.792379046Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client environment:user.name=root
2017-01-12T00:11:31.793107176Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:user.home=/root
2017-01-12T00:11:31.793138639Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :100] Client
environment:user.dir=/etc/service/dataf_gateway
2017-01-12T00:11:31.794761800Z 2017-01-12 00:11:31 INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper :438] Initiating client connection,
connectString=68.128.182.149:2181 sessionTimeout=5000
watcher=org.apache.camel.component.zookeeper.ConnectionHolder@4ec951ba
2017-01-12T00:11:31.813834319Z 2017-01-12 00:11:31 INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn :032] Opening socket connection to
server 68.128.182.149/68.128.182.149:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-01-12T00:11:31.904467961Z 2017-01-12 00:11:31 INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn :876] Socket connection established to
68.128.182.149/68.128.182.149:2181, initiating session
2017-01-12T00:11:32.006541339Z 2017-01-12 00:11:32 INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn :299] Session establishment complete on
server 68.128.182.149/68.128.182.149:2181, sessionid = 0x159855fc87400c2,
negotiated timeout = 5000
2017-01-12T00:11:32.109418356Z 2017-01-12 00:11:32 WARN [vzw/sdf]
[o.a.c.c.zookeeper.ZookeeperProducer:237] Node
'/zk/file_forwarder_node/a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a'
did not exist, creating it.
2017-01-12T00:11:32.398124509Z 2017-01-12 00:11:32 INFO [vzw/sdf]
[o.a.c.c.z.policy.ZooKeeperElection :151] Candidate node
'/zk/file_forwarder_node/a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a'
has been created
2017-01-12T00:11:32.431922439Z 2017-01-12 00:11:32 INFO [vzw/sdf]
[o.a.camel.impl.DefaultCamelContext :570] Route:
election-route-a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a started and
consuming from:
Endpoint[zookeeper://68.128.182.149:2181/zk/file_forwarder_node?createMode=EPHEMERAL]
2017-01-12T00:11:32.620071603Z 2017-01-12 00:11:32 INFO [vzw/sdf]
[Datafeed.Sdf.dc3Forwarder :159] Moving Starts fileName=205.xml
from /ts/vzw/sdf
2017-01-12T00:11:39.674491004Z 2017-01-12 00:11:39 INFO [vzw/sdf]
[Datafeed.Sdf.dc3Forwarder :159] Moving Completed fileName=205.xml
transfer to sftp://68.128.182.149:2222
2017-01-12T00:11:40.692277702Z 2017-01-12 00:11:40 WARN [vzw/sdf]
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[o.a.c.c.f.GenericFileOnCompletion :213] Error during commit.
Exchange[ID-a810a44c767c-44403-1484179857573-0-2]. Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file from: ts/sdf/205.xml to: ts/sdf/inprogress/205.xml]
2017-01-12T00:11:40.692309664Z
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
rename file from: ts//sdf/205.xml to: ts/sdf/inprogress/205.xml
2017-01-12T00:11:40.692319752Z at
org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:431)
2017-01-12T00:11:40.692331238Z at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
2017-01-12T00:11:40.692336731Z at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
2017-01-12T00:11:40.692341626Z at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
2017-01-12T00:11:40.692349716Z at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
2017-01-12T00:11:40.692354510Z at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
2017-01-12T00:11:40.692359205Z at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
2017-01-12T00:11:40.692364099Z at
org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
2017-01-12T00:11:40.692375485Z at
org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
2017-01-12T00:11:40.692381378Z at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:674)
2017-01-12T00:11:40.692386372Z at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:629)
2017-01-12T00:11:40.692390967Z at
org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:246)
2017-01-12T00:11:40.692395761Z at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:179)
2017-01-12T00:11:40.692409245Z at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454)
2017-01-12T00:11:40.692414039Z at
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:137)
2017-01-12T00:11:40.692418534Z at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:226)
2017-01-12T00:11:40.692422929Z at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:190)
2017-01-12T00:11:40.692427323Z at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
2017-01-12T00:11:40.692432817Z at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
2017-01-12T00:11:40.692437211Z at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-01-12T00:11:40.692441606Z at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2017-01-12T00:11:40.692449497Z at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2017-01-12T00:11:40.692457887Z at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2017-01-12T00:11:40.692464779Z at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-01-12T00:11:40.692469273Z at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-01-12T00:11:40.692474567Z at java.lang.Thread.run(Thread.java:745)
2017-01-12T00:11:40.692479761Z Caused by: com.jcraft.jsch.SftpException: No
such file
2017-01-12T00:11:40.692484055Z at
com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
2017-01-12T00:11:40.692488550Z at
com.jcraft.jsch.ChannelSftp.rename(ChannelSftp.java:1923)
2017-01-12T00:11:40.692493245Z at
org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:428)
2017-01-12T00:11:40.692503133Z ... 25 common frames omitted
2017-01-12T00:12:08.133521922Z 2017-01-12 00:12:08 INFO [/sdf]
[Dataf.Sdf.dc3Forwarder :159] Moving Starts fileName=206.xml from
/ts/sdf
2017-01-12T00:12:11.826120872Z 2017-01-12 00:12:11 INFO [/sdf]
[Dataf.Sdf.dc3Forwarder :159] Moving Completed fileName=206.xml
transfer to sftp://X.X.X.X:2222
--
View this message in context:
http://camel.465427.n5.nabble.com/SFTP-leader-election-using-ZooKeeper-RoutePolicy-Error-on-moving-file-tp5792480.html
Sent from the Camel - Users mailing list archive at Nabble.com.
