Hi all, I'm submitting jobs to gw77 server to test functionalities of the GridChem client. I have input files for a gaussian experiment in the same server. So I passed the value for input file as
file://airavata@localhost :/home/airavata/oodt/archive/hpn40deftk7r1tvgo2545lfoqc/c4b4nhtwbs3.inp When I launch the experiment, I get an error saying 2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata What could be the reason for that? As far as I understood, it seems like Airavata is unable to scp a file from localhost. Full log is as follows. 2015-08-17 02:45:10,995 [pool-3-thread-5] INFO org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so creating all the experiment docs from the scratch 2015-08-17 02:45:11,017 [pool-3-thread-5] INFO org.apache.airavata.gfac.core.utils.GFacUtils - Created the node: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/state successfully ! 2015-08-17 02:45:11,028 [pool-3-thread-5] INFO org.apache.airavata.gfac.server.GfacServerHandler - -----------------------------------------------------90----------------------------------------------------- 2015-08-17 02:45:11,028 [pool-3-thread-5] INFO org.apache.airavata.gfac.server.GfacServerHandler - Id:dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 : GFac Received submit job request for the Experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 TaskId: tempNode_73058234-5ac6-4278-83f2-c27e23474182 2015-08-17 02:45:11,735 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-17 02:45:11,739 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: ACCEPTED 2015-08-17 02:45:11,882 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-17 02:45:11,882 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: INHANDLERSINVOKING 2015-08-17 02:45:11,969 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Invoking ApplicationDescriptorCheckHandler ... 2015-08-17 02:45:11,985 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Recoverable data is saving to zk: /oasis/scratch/comet/scigap/temp_project/airavata-workdirs,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stdout,/oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262/Gaussian.stderr 2015-08-17 02:45:12,206 [pool-8-thread-54] INFO org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Listing directory: comet.sdsc.edu:~/ 2015-08-17 02:45:12,412 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.util.GFACSSHUtils - Re-using the same connection used with the connection string:scigapcomet.sdsc.edu22 2015-08-17 02:45:12,413 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Setup SSH job directorties 2015-08-17 02:45:12,463 [pool-8-thread-54] INFO org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Successfully retrieved the Security Context 2015-08-17 02:45:12,463 [pool-8-thread-54] INFO org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster - Creating directory: comet.sdsc.edu: /oasis/scratch/comet/scigap/temp_project/airavata-workdirs/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 2015-08-17 02:45:12,973 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Previously stored data is wrong so we continue the operations 2015-08-17 02:45:13,162 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.util.GFACSSHUtils - An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error adding security Context org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,164 [pool-8-thread-54] ERROR org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler - Error adding security Context 2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error while input File Staging org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,247 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while input File Staging org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,248 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,253 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 2015-08-17 02:45:13,253 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 2015-08-17 02:45:13,254 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during updating the statuses of Experiments,tasks or Job statuses to failed, NullPointerException occurred because at this point there might not have Job Created org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:13,254 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262: Updating workflow node status for tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED 2015-08-17 02:45:13,617 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-17 02:45:14,721 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED 2015-08-17 02:45:14,771 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-17 02:45:14,777 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Clean up task - Acknowledging delivery Tag - 91 2015-08-17 02:45:15,017 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:15,099 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:15,100 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job with experiment ID: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262:Error launching the Job 2015-08-17 02:45:15,218 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 2015-08-17 02:45:15,218 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 2015-08-17 02:45:15,718 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-17 02:45:15,769 [pool-8-thread-54] INFO org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - Id:tempNode_73058234-5ac6-4278-83f2-c27e23474182 : Updated task tempNode_73058234-5ac6-4278-83f2-c27e23474182 status to TaskStatus(executionState:FAILED, timeOfStateChange:1439793915661). 2015-08-17 02:45:15,787 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-17 02:45:15,797 [pool-8-thread-54] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262: Updating workflow node status for tempNode_41ca8687-37ab-492c-bce0-09058ccc1c02:FAILED 2015-08-17 02:45:16,225 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-17 02:45:17,316 [pool-8-thread-54] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262 to FAILED 2015-08-17 02:45:17,376 [pool-8-thread-54] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-17 02:45:17,385 [pool-8-thread-54] ERROR org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery Tag in path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau_10_5f0ea67d-f2c7-42a1-820c-d1c6ed712262-deliveryTag for this experiment 2015-08-17 02:45:17,397 [pool-8-thread-54] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) ... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:17,401 [pool-8-thread-54] ERROR org.apache.airavata.gfac.core.utils.InputHandlerWorker - org.apache.airavata.gfac.GFacException: Error launching the Job org.apache.airavata.gfac.GFacException: org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:200) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:505) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:183) ... 4 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:743) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:492) ... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:910) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:686) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:899) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error adding security Context at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:287) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.prepareSecurityContext(GFACSSHUtils.java:547) at org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler.invoke(AdvancedSCPInputHandler.java:148) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:895) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - localhost:22 connecting user name - airavata at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:271) ... 10 more Caused by: com.jcraft.jsch.JSchException: Auth cancel at com.jcraft.jsch.Session.connect(Session.java:511) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 14 more 2015-08-17 02:45:25,721 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-17 02:45:35,723 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-17 02:45:45,724 [Thread-9] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails Thanks Dimuthu -- Regards W.Dimuthu Upeksha Undergraduate Department of Computer Science And Engineering University of Moratuwa, Sri Lanka
