Hi Jeff, Storage resource is not compute resource(e.g.: comet) specific.
It is a generic storage location on gateway (PGA) side, common for all the gateway jobs. In storage per user/project/experiment, a storage directory will be created for each experiment. The storage resource registration is simply identifying the actual storage by registering and giving a storage resource ID, and this needs to to be a location in the same machine/ server as PGA. Since we use SCP for file transfer and use SSH keys, we need to assign an ssh key in the admin dashboard --> gateway management --> storage preferences and then actually add the public key in the authorized_keys file. Hope this helps. Thanks, Eroma On Mon, Jun 12, 2017 at 1:44 PM, Sale, Jeff <[email protected]> wrote: > Supun, > > I believe I configured my storage resource settings for Comet correctly > but I am still getting errors. However, the job is at least able to open a > directory on Comet in the path that I specified as my scratch. > Unfortunately the directory is empty. The directory name is: > > PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 > > > The same thing happened back on June 5 the last time I tried, but it was > also empty. > > The errors I am getting appear to be related to ssh authentication. My > Data Movement Protocol is set to SCP using SSH Keys. I assume this is > correct, even though the instructions are not specific about these details. > I used the testdrive.airavata.org for some guidance. I believe I have > added my Comet Storage resource to the pga_config.php file correctly, i.e. > as follows: > > > /** > > * Gateway data store resource id > > */ > > 'gateway-data-store-resource-id' => 'CometStorage_efa7abdf-ade7- > 4adc-b97c-49f81d4dbcf0', > > > > Here are the errors I am still getting. > > > [INFO] Started Gfac Server on Port 8950 ... > > DriverClassName=com.mysql.jdbc.Driver,Url=jdbc:mysql:// > localhost:3306/replica_catalog?autoReconnect=true, > Username=airavata,Password=airavata,validationQuery=SELECT 1 from > CONFIGURATION > > 0 replicacatalog_data INFO [pool-1-thread-1] openjpa.Runtime - > Starting OpenJPA 2.2.0 > > 16 replicacatalog_data INFO [pool-1-thread-1] openjpa.jdbc.JDBC - > Using dictionary class "org.apache.openjpa.jdbc.sql.MySQLDictionary" > (MySQL 5.5.5-10.1.21-MariaDB ,MySQL Connector Java > mysql-connector-java-5.1.41 ( Revision: > 83c6dc41b96809df81444362933043b20a1d49d5 > )). > > [INFO] Message Received with message id > 'LAUNCH.EXP-6956f6ca-3237-4f27-b0fd-5f085f24f4e5' > and with message type 'EXPERIMENT' for experimentId: > Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb > > [INFO] Launching experiment with experimentId: Clone_of_Gaussian_Job_5_ > 849cde15-5a1b-4b14-adec-f3b5af2c48cb gateway Id: php_reference_gateway > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid user login > name, using gateway computer resource preference login name jsale > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid user login > name, using gateway computer resource preference login name jsale > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid user login > name, using gateway computer resource preference login name jsale > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid user login > name, using gateway computer resource preference login name jsale > > [WARN] Process compute resource scheduling doesn't have valid scratch > location, using gateway computer resource preference scratch > location/oasis/scratch/comet/jsale/temp_project/pga/ > > [WARN] Process compute resource scheduling doesn't have valid user login > name, using gateway computer resource preference login name jsale > > [INFO] Validation of > org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator > is SUCCESSFUL > > [INFO] Validation of > org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator > is SUCCESSFUL > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > Launched experiment > > [INFO] Message Received with message id > 'LAUNCH.PROCESS-7a545822-c32b-43a1-87b2-2496b59da9a8 > and with message type:LAUNCHPROCESS, for > processId:PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, > expId:Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb > > [INFO] Message Received with message id LAUNCHPROCESS and with message > type: {}LAUNCH.PROCESS-7a545822-c32b-43a1-87b2-2496b59da9a8 > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed event received for status STARTED > > [INFO] expId : Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb > :- Experiment status updated to EXECUTING > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, get process > cancel data from zookeeper node /experiments/Clone_of_ > Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb/ > PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7/cancelListener > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed STARTED -> CONFIGURING_WORKSPACE > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed event received for status CONFIGURING_WORKSPACE > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_fb013659-b4c4-427a-967f-e5e5b4be0c8a, type: ENV_SETUP : Task status > changed CREATED -> EXECUTING > > [INFO] Creating directory: comet.sdsc.edu:/oasis/scratch/ > comet/jsale/temp_project/pga/PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 > > [INFO] Session validation failed, key :jsale_comet.sdsc.edu_22_ > c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] Initialize a new SSH session for :jsale_comet.sdsc.edu_22_ > c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_fb013659-b4c4-427a-967f-e5e5b4be0c8a, type: ENV_SETUP : Task status > changed EXECUTING -> COMPLETED > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed CONFIGURING_WORKSPACE -> INPUT_DATA_STAGING > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING : Task > status changed CREATED -> EXECUTING > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed event received for status INPUT_DATA_STAGING > > [INFO] SSH Session validation succeeded, key :jsale_comet.sdsc.edu_22_ > c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] Channel creation test succeeded, key :jsale_comet.sdsc.edu_22_ > c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] Reuse SSH session for :jsale_comet.sdsc.edu_22_ > c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] Transferring from:/var/www/experimentData/ > jsale37/GaussianonComet/Clone_of_Gaussian_Job_51497288328/water_opt_8c.dat > To: /oasis/scratch/comet/jsale/temp_project/pga//PROCESS_ > 7a041582-fd6a-4d68-9237-3ff1f5d059e7/ > > [INFO] Session validation failed, key :jsale_Comet > Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [INFO] Initialize a new SSH session for :jsale_Comet > Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115 > > [ERROR] JSch initialization error > > com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet Storage > > at com.jcraft.jsch.Util.createSocket(Util.java:349) > > at com.jcraft.jsch.Session.connect(Session.java:215) > > at com.jcraft.jsch.Session.connect(Session.java:183) > > at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession( > HPCRemoteCluster.java:137) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession( > HPCRemoteCluster.java:274) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty( > HPCRemoteCluster.java:173) > > at org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging( > SCPDataStageTask.java:247) > > at org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute( > SCPDataStageTask.java:158) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask( > GFacEngineImpl.java:814) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging( > GFacEngineImpl.java:604) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom( > GFacEngineImpl.java:347) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( > GFacEngineImpl.java:286) > > at org.apache.airavata.gfac.impl.GFacWorker.executeProcess( > GFacWorker.java:227) > > at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) > > at org.apache.airavata.common.logging.MDCUtil.lambda$ > wrapWithMDC$0(MDCUtil.java:40) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.net.UnknownHostException: Comet Storage > > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java: > 184) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > at java.net.Socket.connect(Socket.java:589) > > at java.net.Socket.connect(Socket.java:538) > > at java.net.Socket.<init>(Socket.java:434) > > at java.net.Socket.<init>(Socket.java:211) > > at com.jcraft.jsch.Util.createSocket(Util.java:343) > > ... 18 common frames omitted > > [ERROR] Data staging failed > > org.apache.airavata.gfac.core.GFacException: JSch initialization error > > at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession( > HPCRemoteCluster.java:137) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession( > HPCRemoteCluster.java:274) > > at org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty( > HPCRemoteCluster.java:173) > > at org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging( > SCPDataStageTask.java:247) > > at org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute( > SCPDataStageTask.java:158) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask( > GFacEngineImpl.java:814) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging( > GFacEngineImpl.java:604) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom( > GFacEngineImpl.java:347) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( > GFacEngineImpl.java:286) > > at org.apache.airavata.gfac.impl.GFacWorker.executeProcess( > GFacWorker.java:227) > > at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) > > at org.apache.airavata.common.logging.MDCUtil.lambda$ > wrapWithMDC$0(MDCUtil.java:40) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: > Comet Storage > > at com.jcraft.jsch.Util.createSocket(Util.java:349) > > at com.jcraft.jsch.Session.connect(Session.java:215) > > at com.jcraft.jsch.Session.connect(Session.java:183) > > at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542) > > ... 15 common frames omitted > > Caused by: java.net.UnknownHostException: Comet Storage > > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java: > 184) > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > at java.net.Socket.connect(Socket.java:589) > > at java.net.Socket.connect(Socket.java:538) > > at java.net.Socket.<init>(Socket.java:434) > > at java.net.Socket.<init>(Socket.java:211) > > at com.jcraft.jsch.Util.createSocket(Util.java:343) > > ... 18 common frames omitted > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING : Task > status changed EXECUTING -> FAILED > > [ERROR] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_f3129550-e851-4699-8b09-e8909d17168b type: DATA_STAGING,:- > DATA_STAGING failed, reason: Data staging failed > > [ERROR] GFac Worker throws an exception > > org.apache.airavata.gfac.core.GFacException: Error: userFriendly msg > :Error while executing DATA_STAGING task, actual msg :expId: > Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, processId: > PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId: > TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING :- > DATA_STAGING failed. Reason: Data staging failed > > at org.apache.airavata.gfac.impl.GFacEngineImpl.checkFailures( > GFacEngineImpl.java:628) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging( > GFacEngineImpl.java:609) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom( > GFacEngineImpl.java:347) > > at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( > GFacEngineImpl.java:286) > > at org.apache.airavata.gfac.impl.GFacWorker.executeProcess( > GFacWorker.java:227) > > at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) > > at org.apache.airavata.common.logging.MDCUtil.lambda$ > wrapWithMDC$0(MDCUtil.java:40) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed INPUT_DATA_STAGING -> FAILED > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status > changed event received for status FAILED > > [INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, > processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Sent ack for > deliveryTag 1 > > [INFO] expId : Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb > :- Experiment status updated to FAILED > > > > ------------------------------ > *From:* Supun Nakandala [[email protected]] > *Sent:* Monday, June 05, 2017 12:12 PM > *To:* [email protected] > *Subject:* Re: Running Gaussian on Comet > > Jeff, > > Basically you need to some setup work in your gateway as the gateway admin > which includes creating and configuring a storage resource. From the error > that I see it seems that the storage resource is not properly configured. > More details on setting up storage resources and other configurations can > be found here https://cwiki.apache.org/confluence/display/AIRAVATA/ > Tutorial+05+-+PHP+Reference+Gateway+for+Airavata+-+Gateway+Admin+Guide# > Tutorial05-PHPReferenceGatewayforAiravata-GatewayAdminGuide-TutorialIV- > RegisterandManageStorageResources > > Can you try to setup it following this tutorial? If it still has issues we > can set up a video call. > > On Mon, Jun 5, 2017 at 3:05 PM, Sale, Jeff <[email protected]> wrote: > >> OK, I made that change and tried again and it looks like I am making >> progress. Still some errors with data staging. Here are the errors: >> >> DriverClassName=com.mysql.jdbc.Driver,Url=jdbc:mysql://local >> host:3306/replica_catalog?autoReconnect=true,Username= >> airavata,Password=airavata,validationQuery=SELECT 1 from CONFIGURATION >> >> 1 replicacatalog_data INFO [pool-1-thread-1] openjpa.Runtime - >> Starting OpenJPA 2.2.0 >> >> 10 replicacatalog_data INFO [pool-1-thread-1] openjpa.jdbc.JDBC - >> Using dictionary class "org.apache.openjpa.jdbc.sql.MySQLDictionary" >> (MySQL 5.5.5-10.1.21-MariaDB ,MySQL Connector Java >> mysql-connector-java-5.1.41 ( Revision: >> 83c6dc41b96809df81444362933043b20a1d49d5 >> )). >> >> [INFO] Message Received with message id >> 'LAUNCH.EXP-3399a423-f598-4ef2-b33f-6c07b983ef2c' >> and with message type 'EXPERIMENT' for experimentId: >> Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 >> >> [INFO] Launching experiment with experimentId: >> Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 gateway Id: >> php_reference_gateway >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid user login >> name, using gateway computer resource preference login name jsale >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid user login >> name, using gateway computer resource preference login name jsale >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid user login >> name, using gateway computer resource preference login name jsale >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid user login >> name, using gateway computer resource preference login name jsale >> >> [WARN] Process compute resource scheduling doesn't have valid scratch >> location, using gateway computer resource preference scratch >> location/oasis/scratch/comet/jsale/temp_project/pga/ >> >> [WARN] Process compute resource scheduling doesn't have valid user login >> name, using gateway computer resource preference login name jsale >> >> [INFO] Validation of org.apache.airavata.orchestrat >> or.core.validator.impl.BatchQueueValidator is SUCCESSFUL >> >> [INFO] Validation of org.apache.airavata.orchestrat >> or.core.validator.impl.ExperimentStatusValidator is SUCCESSFUL >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, Launched experiment >> >> [INFO] Message Received with message id >> 'LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529 >> and with message type:LAUNCHPROCESS, for >> processId:PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> expId:Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 >> >> [INFO] Message Received with message id LAUNCHPROCESS and with message >> type: {}LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529 >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed event received for status STARTED >> >> [INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 >> :- Experiment status updated to EXECUTING >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> get process cancel data from zookeeper node /experiments/Clone_of_Gaussian >> _Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8/PROCESS_ >> e2a359f3-5caa-4ed7-b203-2fde6ce7962c/cancelListener >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed STARTED -> CONFIGURING_WORKSPACE >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed event received for status CONFIGURING_WORKSPACE >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> taskId: TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : >> Task status changed CREATED -> EXECUTING >> >> [INFO] Creating directory: comet.sdsc.edu:/oasis/scratch/ >> comet/jsale/temp_project/pga/PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> >> [INFO] Session validation failed, key :jsale_comet.sdsc.edu_22_c7eb6 >> c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] Initialize a new SSH session for :jsale_comet.sdsc.edu_22_c7eb6 >> c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> taskId: TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : >> Task status changed EXECUTING -> COMPLETED >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed CONFIGURING_WORKSPACE -> INPUT_DATA_STAGING >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> taskId: TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : >> Task status changed CREATED -> EXECUTING >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed event received for status INPUT_DATA_STAGING >> >> [INFO] SSH Session validation succeeded, key >> :jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] Channel creation test succeeded, key :jsale_comet.sdsc.edu_22_c7eb6 >> c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] Reuse SSH session for :jsale_comet.sdsc.edu_22_c7eb6 >> c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] Transferring from:/var/www/experimentData/j >> sale37/GaussianonComet/Clone_of_Gaussian_Job_51496689416/water_opt_8c.dat >> To: /oasis/scratch/comet/jsale/temp_project/pga//PROCESS_e2a359f >> 3-5caa-4ed7-b203-2fde6ce7962c/ >> >> [INFO] Session validation failed, key :jsale_Comet >> Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115 >> >> [INFO] Initialize a new SSH session for :jsale_Comet >> Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115 >> >> [ERROR] JSch initialization error >> >> com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet >> Storage >> >> at com.jcraft.jsch.Util.createSocket(Util.java:349) >> >> at com.jcraft.jsch.Session.connect(Session.java:215) >> >> at com.jcraft.jsch.Session.connect(Session.java:183) >> >> at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession >> (HPCRemoteCluster.java:137) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HP >> CRemoteCluster.java:274) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty >> (HPCRemoteCluster.java:173) >> >> at org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDat >> aStaging(SCPDataStageTask.java:247) >> >> at org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute( >> SCPDataStageTask.java:158) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFa >> cEngineImpl.java:814) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStagin >> g(GFacEngineImpl.java:604) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskList >> From(GFacEngineImpl.java:347) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( >> GFacEngineImpl.java:286) >> >> at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFac >> Worker.java:227) >> >> at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) >> >> at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMD >> C$0(MDCUtil.java:40) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.net.UnknownHostException: Comet Storage >> >> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket >> Impl.java:184) >> >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> >> at java.net.Socket.connect(Socket.java:589) >> >> at java.net.Socket.connect(Socket.java:538) >> >> at java.net.Socket.<init>(Socket.java:434) >> >> at java.net.Socket.<init>(Socket.java:211) >> >> at com.jcraft.jsch.Util.createSocket(Util.java:343) >> >> ... 18 common frames omitted >> >> [ERROR] Data staging failed >> >> org.apache.airavata.gfac.core.GFacException: JSch initialization error >> >> at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession >> (HPCRemoteCluster.java:137) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HP >> CRemoteCluster.java:274) >> >> at org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty >> (HPCRemoteCluster.java:173) >> >> at org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDat >> aStaging(SCPDataStageTask.java:247) >> >> at org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute( >> SCPDataStageTask.java:158) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFa >> cEngineImpl.java:814) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStagin >> g(GFacEngineImpl.java:604) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskList >> From(GFacEngineImpl.java:347) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( >> GFacEngineImpl.java:286) >> >> at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFac >> Worker.java:227) >> >> at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) >> >> at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMD >> C$0(MDCUtil.java:40) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: >> Comet Storage >> >> at com.jcraft.jsch.Util.createSocket(Util.java:349) >> >> at com.jcraft.jsch.Session.connect(Session.java:215) >> >> at com.jcraft.jsch.Session.connect(Session.java:183) >> >> at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542) >> >> ... 15 common frames omitted >> >> Caused by: java.net.UnknownHostException: Comet Storage >> >> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket >> Impl.java:184) >> >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> >> at java.net.Socket.connect(Socket.java:589) >> >> at java.net.Socket.connect(Socket.java:538) >> >> at java.net.Socket.<init>(Socket.java:434) >> >> at java.net.Socket.<init>(Socket.java:211) >> >> at com.jcraft.jsch.Util.createSocket(Util.java:343) >> >> ... 18 common frames omitted >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> taskId: TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : >> Task status changed EXECUTING -> FAILED >> >> [ERROR] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, >> taskId: TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26 type: DATA_STAGING,:- >> DATA_STAGING failed, reason: Data staging failed >> >> [ERROR] GFac Worker throws an exception >> >> org.apache.airavata.gfac.core.GFacException: Error: userFriendly msg >> :Error while executing DATA_STAGING task, actual msg :expId: >> Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId: >> TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING :- >> DATA_STAGING failed. Reason: Data staging failed >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.checkFailures(G >> FacEngineImpl.java:628) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStagin >> g(GFacEngineImpl.java:609) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskList >> From(GFacEngineImpl.java:347) >> >> at org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess( >> GFacEngineImpl.java:286) >> >> at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFac >> Worker.java:227) >> >> at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86) >> >> at org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMD >> C$0(MDCUtil.java:40) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed INPUT_DATA_STAGING -> FAILED >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Process status changed event received for status FAILED >> >> [INFO] expId: Clone_of_Gaussian_Job_5_f7a276 >> af-dec5-460e-a002-9c5de4d883e8, processId: >> PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c >> :- Sent ack for deliveryTag 1 >> >> [INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 >> :- Experiment status updated to FAILED >> >> >> >> > > > -- > Thank you > Supun Nakandala > Dept. Computer Science and Engineering > University of Moratuwa > -- Thank You, Best Regards, Eroma
