Hi Dimuthu, do you still have this issue?I just checked and gw111 has access to comet resource.
Thanks, Shameera. On Sat, Aug 1, 2015 at 5:05 AM DImuthu Upeksha <[email protected]> wrote: > Hi, > > I'm getting following error when I tried to launch a Gaussian experiment > on gw111.iu.xsede.org <[email protected]> server. What could be > the reason for that? > > Application interface: Gaussian_cc471eee-35e0-4d2b-8cdc-0023481dda90 > Resource host id: comet-ln1.sdsc.edu_0bb9bd78-b5e7-40cf-a5dd-fd6f8bd6b537 > > 2015-08-01 04:54:58,754 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:08,756 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:18,756 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:28,756 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:38,757 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:48,758 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:55:54,691 [pool-3-thread-23] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:55:54,700 [pool-3-thread-23] INFO > org.apache.airavata.api.server.handler.AiravataServerHandler - > Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : Created > new experiment with experiment name dimuthu_experiment_gau4 > 2015-08-01 04:55:58,758 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:56:08,759 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:56:18,760 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:56:28,760 [Thread-18] INFO > org.apache.airavata.gfac.monitor.email.EmailBasedMonitor - [EJM]: Job > Monitor Map is empty, no need to retrieve emails > 2015-08-01 04:56:31,929 [pool-10-thread-17] INFO > org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl - > Validation of > org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is > SUCCESSFUL > 2015-08-01 04:56:31,931 [pool-10-thread-17] INFO > org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl - > Validation of > org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator > is SUCCESSFUL > 2015-08-01 04:56:32,843 [pool-18-thread-14] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:33,025 [pool-18-thread-14] INFO > org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - > Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task > tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to > TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000). > 2015-08-01 04:56:33,592 [pool-18-thread-14] INFO > org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - > Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task > tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to > TaskStatus(executionState:UNKNOWN, timeOfStateChange:1438419390000). > 2015-08-01 04:56:34,060 [pool-18-thread-14] INFO > org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher - > Publishing to launch queue ... > 2015-08-01 04:56:34,061 [pool-18-thread-14] INFO > org.apache.airavata.messaging.core.impl.RabbitMQTaskLaunchPublisher - > Successfully published to launch queue ... > 2015-08-01 04:56:34,130 [pool-13-thread-9] INFO > org.apache.airavata.gfac.core.utils.GFacUtils - This is a new Job, so > creating all the experiment docs from the scratch > 2015-08-01 04:56:34,147 [pool-13-thread-9] INFO > org.apache.airavata.gfac.core.utils.GFacUtils - Created the node: > /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/state > successfully ! > 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO > org.apache.airavata.gfac.server.GfacServerHandler - > -----------------------------------------------------14----------------------------------------------------- > 2015-08-01 04:56:34,154 [pool-13-thread-9] INFO > org.apache.airavata.gfac.server.GfacServerHandler - > Id:dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b : GFac > Received submit job request for the Experiment: > dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b TaskId: > tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 > 2015-08-01 04:56:34,749 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created > at this point > 2015-08-01 04:56:34,750 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac > internal state changed to: ACCEPTED > 2015-08-01 04:56:34,831 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created > at this point > 2015-08-01 04:56:34,831 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac > internal state changed to: INHANDLERSINVOKING > 2015-08-01 04:56:34,880 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - > Invoking ApplicationDescriptorCheckHandler ... > 2015-08-01 04:56:34,886 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.handler.AppDescriptorCheckHandler - > Recoverable data is saving to zk: > /oasis/scratch/comet/ogce/temp_project/gta-work-dirs,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stdout,/oasis/scratch/comet/ogce/temp_project/gta-work-dirs/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b/Gaussian.stderr > 2015-08-01 04:56:35,765 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error occurred... > org.apache.airavata.gsi.ssh.api.SSHApiException: An exception occurred > while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,766 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler - Error > occurred... > 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error while creating > SSHSecurityContext > org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,829 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.handler.GFacHandlerException - Error while > creating SSHSecurityContext > org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error Executing a InFlow Handler > org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error > Executing a InFlow Handler > org.apache.airavata.gfac.GFacException: Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,830 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created > at this point > 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac > internal state changed to: FAILED > 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - > Experiment Failed, So removing the ZK entry for the > experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b > 2015-08-01 04:56:35,830 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - > Zookeeper experiment Path: > /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b > 2015-08-01 04:56:35,831 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error occured during > updating the statuses of Experiments,tasks or Job statuses to failed, > NullPointerException occurred because at this point there might not have > Job Created > org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error > Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error Executing a > InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > ... 7 more > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:35,832 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - > expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b: > Updating workflow node status for > tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED > 2015-08-01 04:56:36,070 [pool-19-thread-28] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:36,656 [pool-19-thread-28] INFO > org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - > Updating the experiment status of experiment: > dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED > 2015-08-01 04:56:36,715 [pool-19-thread-28] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:36,718 [pool-19-thread-28] INFO > org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - > Clean up task - Acknowledging delivery Tag - 14 > 2015-08-01 04:56:36,770 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error Invoking Handlers:Error > Executing a InFlow Handler > org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error > Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error Executing a > InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > ... 7 more > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error launching the Job > org.apache.airavata.gfac.GFacException: Error Invoking Handlers:Error > Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error Invoking > Handlers:Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > ... 6 more > Caused by: org.apache.airavata.gfac.GFacException: Error Executing a > InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > ... 7 more > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:36,820 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.cpi.BetterGfacImpl - Error inovoking the job > with experiment ID: > dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b:Error > launching the Job > 2015-08-01 04:56:36,868 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.monitor.MonitorID - There is not job created > at this point > 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - Gfac > internal state changed to: FAILED > 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - > Experiment Failed, So removing the ZK entry for the > experimentdimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b > 2015-08-01 04:56:36,868 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.GfacInternalStatusUpdator - > Zookeeper experiment Path: > /gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b > 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO > org.apache.airavata.persistance.registry.jpa.impl.ExperimentRegistry - > Id:tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 : Updated task > tempNode_dc6ca8f5-5093-4efc-a983-fc4ae387b1a6 status to > TaskStatus(executionState:FAILED, timeOfStateChange:1438419396978). > 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:37,059 [pool-19-thread-28] INFO > org.apache.airavata.gfac.core.monitor.AiravataWorkflowNodeStatusUpdator - > expId - dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b: > Updating workflow node status for > tempNode_50386c5c-2d49-4f04-b061-024c134be4dc:FAILED > 2015-08-01 04:56:37,267 [pool-19-thread-28] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:37,889 [pool-19-thread-28] INFO > org.apache.airavata.api.server.listener.AiravataExperimentStatusUpdator - > Updating the experiment status of experiment: > dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b to FAILED > 2015-08-01 04:56:37,947 [pool-19-thread-28] INFO > org.apache.airavata.messaging.core.impl.RabbitMQStatusPublisher - > Publishing status to rabbitmq... > 2015-08-01 04:56:37,949 [pool-19-thread-28] ERROR > org.apache.airavata.common.utils.AiravataZKUtils - Cannot find delivery > Tag in > path:/gfac-experiments/gfac-node0/dimuthu_experiment_gau4_e9539503-c171-4d6e-a491-44a97ea3968b-deliveryTag > for this experiment > 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.GFacException - Error launching the Job > org.apache.airavata.gfac.GFacException: Error launching the Job > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error Invoking > Handlers:Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > ... 5 more > Caused by: org.apache.airavata.gfac.GFacException: Error Invoking > Handlers:Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > ... 6 more > Caused by: org.apache.airavata.gfac.GFacException: Error Executing a > InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > ... 7 more > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > 2015-08-01 04:56:37,951 [pool-19-thread-28] ERROR > org.apache.airavata.gfac.core.utils.InputHandlerWorker - > org.apache.airavata.gfac.GFacException: Error launching the Job > org.apache.airavata.gfac.GFacException: > org.apache.airavata.gfac.GFacException: Error launching the Job > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:196) > at > org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.airavata.gfac.GFacException: Error launching the Job > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:498) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179) > ... 4 more > Caused by: org.apache.airavata.gfac.GFacException: Error Invoking > Handlers:Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:736) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:485) > ... 5 more > Caused by: org.apache.airavata.gfac.GFacException: Error Invoking > Handlers:Error Executing a InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:903) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:679) > ... 6 more > Caused by: org.apache.airavata.gfac.GFacException: Error Executing a > InFlow Handler > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:892) > ... 7 more > Caused by: org.apache.airavata.gfac.GFacException: Error occurred... > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:184) > at > org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler.invoke(SSHDirectorySetupHandler.java:47) > at > org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:888) > ... 7 more > Caused by: org.apache.airavata.gsi.ssh.api.SSHApiException: An exception > occurred while connecting to server.Connecting server - comet.sdsc.edu:22 > connecting user name - ogce > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:235) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:100) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.<init>(GSISSHAbstractCluster.java:94) > at org.apache.airavata.gsi.ssh.impl.PBSCluster.<init>(PBSCluster.java:44) > at > org.apache.airavata.gfac.ssh.util.GFACSSHUtils.addSecurityContext(GFACSSHUtils.java:172) > ... 9 more > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:512) > at > org.apache.airavata.gsi.ssh.impl.GSISSHAbstractCluster.reconnect(GSISSHAbstractCluster.java:230) > ... 13 more > > -- > Regards > > W.Dimuthu Upeksha > Undergraduate > Department of Computer Science And Engineering > > University of Moratuwa, Sri Lanka > -- Shameera Rathnayaka
