Hi, I'm getting following error when I tried to launch a Gaussian experiment on gw111.iu.xsede.org <[email protected]> server. What could be the reason for that?
Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90 Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537 2015-08-01 04:54:58,754 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:08,756 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:18,756 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:28,756 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:38,757 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:48,758 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:55:54,691 [pool-3-thread-23] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:55:54,700 [pool-3-thread-23] INFO org.apache.airavata.api.server.handler.AiravataServerHandler - Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created new experiment with experiment name dimuthu_experiment_gau4 2015-08-01 04:55:58,758 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:56:08,759 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:56:18,760 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:56:28,760 [Thread-18] INFO org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job Monitor Map is empty, no need to retrieve emails 2015-08-01 04:56:31,929 [pool-10-thread-17] INFO org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl - Validation of org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is SUCCESSFUL 2015-08-01 04:56:31,931 [pool-10-thread-17] INFO org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl - Validation of org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator is SUCCESSFUL 2015-08-01 04:56:32,843 [pool-18-thread-14] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:33,025 [pool-18-thread-14] INFO org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000). 2015-08-01 04:56:33,592 [pool-18-thread-14] INFO org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000). 2015-08-01 04:56:34,060 [pool-18-thread-14] INFO org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher - Publishing to launch queue ... 2015-08-01 04:56:34,061 [pool-18-thread-14] INFO org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher - Successfully published to launch queue ... 2015-08-01 04:56:34,130 [pool-13-thread-9] INFO org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so creating all the experiment docs from the scratch 2015-08-01 04:56:34,147 [pool-13-thread-9] INFO org.apache.airavata.gfac.core.utils.GFacUtils - Created the node: /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state successfully ! 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO org.apache.airavata.gfac.server.GfacServerHandler - -----------------------------------------------------14----------------------------------------------------- 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO org.apache.airavata.gfac.server.GfacServerHandler - Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac Received submit job request for the Experiment: dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId: tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-01 04:56:34,750 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: ACCEPTED 2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-01 04:56:34,831 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: INHANDLERSINVOKING 2015-08-01 04:56:34,880 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Invoking ApplicationDescriptorCheckHandler ... 2015-08-01 04:56:34,886 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - Recoverable data is saving to zk: /oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr 2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error occurred... org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Error occurred... 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error while creating SSHSecurityContext org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while creating SSHSecurityContext org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b 2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during updating the statuses of Experiments,tasks or Job statuses to failed, NullPointerException occurred because at this point there might not have Job Created org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:35,832 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b: Updating workflow node status for tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED 2015-08-01 04:56:36,070 [pool-19-thread-28] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:36,656 [pool-19-thread-28] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED 2015-08-01 04:56:36,715 [pool-19-thread-28] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:36,718 [pool-19-thread-28] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Clean up task - Acknowledging delivery Tag - 14 2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error Executing a InFlow Handler org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job with experiment ID: dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error launching the Job 2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created at this point 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac internal state changed to: FAILED 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Experiment Failed, So removing the ZK entry for the experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Zookeeper experiment Path: /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978). 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b: Updating workflow node status for tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED 2015-08-01 04:56:37,267 [pool-19-thread-28] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:37,889 [pool-19-thread-28] INFO org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - Updating the experiment status of experiment: dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED 2015-08-01 04:56:37,947 [pool-19-thread-28] INFO org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - Publishing status to rabbitmq... 2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery Tag in path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag for this experiment 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR org.apache.airavata.gfac.GFacException - Error launching the Job org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) ... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR org.apache.airavata.gfac.core.utils.InputHandlerWorker - org.apache.airavata.gfac.GFacException: Error launching the Job org.apache.airavata.gfac.GFacException: org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196) at org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) ... 4 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) ... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) ... 6 more Caused by: org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) ... 7 more Caused by: org.apache.airavata.gfac.GFacException: Error occurred... at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) at org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) at org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) ... 7 more Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred while connecting to server.Connecting server - comet.sdsc.edu:22 connecting user name - ogce at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) at org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) ... 9 more Caused by: com.jcraft.jsch.JSchException: Auth fail at com.jcraft.jsch.Session.connect(Session.java:512) at org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) ... 13 more -- Regards W.Dimuthu Upeksha Undergraduate Department of Computer Science And Engineering University of Moratuwa, Sri Lanka
