Supun,
I believe I configured my storage resource settings for Comet correctly but I
am still getting errors. However, the job is at least able to open a directory
on Comet in the path that I specified as my scratch. Unfortunately the
directory is empty. The directory name is:
PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7
The same thing happened back on June 5 the last time I tried, but it was also
empty.
The errors I am getting appear to be related to ssh authentication. My Data
Movement Protocol is set to SCP using SSH Keys. I assume this is correct, even
though the instructions are not specific about these details. I used the
testdrive.airavata.org for some guidance. I believe I have added my Comet
Storage resource to the pga_config.php file correctly, i.e. as follows:
/**
* Gateway data store resource id
*/
'gateway-data-store-resource-id' =>
'CometStorage_efa7abdf-ade7-4adc-b97c-49f81d4dbcf0',
Here are the errors I am still getting.
[INFO] Started Gfac Server on Port 8950 ...
DriverClassName=com.mysql.jdbc.Driver,Url=jdbc:mysql://localhost:3306/replica_catalog?autoReconnect=true,Username=airavata,Password=airavata,validationQuery=SELECT
1 from CONFIGURATION
0 replicacatalog_data INFO [pool-1-thread-1] openjpa.Runtime - Starting
OpenJPA 2.2.0
16 replicacatalog_data INFO [pool-1-thread-1] openjpa.jdbc.JDBC - Using
dictionary class "org.apache.openjpa.jdbc.sql.MySQLDictionary" (MySQL
5.5.5-10.1.21-MariaDB ,MySQL Connector Java mysql-connector-java-5.1.41 (
Revision: 83c6dc41b96809df81444362933043b20a1d49d5 )).
[INFO] Message Received with message id
'LAUNCH.EXP-6956f6ca-3237-4f27-b0fd-5f085f24f4e5' and with message type
'EXPERIMENT' for experimentId:
Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb
[INFO] Launching experiment with experimentId:
Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb gateway Id:
php_reference_gateway
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
SUCCESSFUL
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
is SUCCESSFUL
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
Launched experiment
[INFO] Message Received with message id
'LAUNCH.PROCESS-7a545822-c32b-43a1-87b2-2496b59da9a8 and with message
type:LAUNCHPROCESS, for processId:PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7,
expId:Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb
[INFO] Message Received with message id LAUNCHPROCESS and with message type:
{}LAUNCH.PROCESS-7a545822-c32b-43a1-87b2-2496b59da9a8
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed event received for status STARTED
[INFO] expId : Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb :-
Experiment status updated to EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, get process cancel
data from zookeeper node
/experiments/Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb/PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7/cancelListener
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed STARTED -> CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed event received for status CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_fb013659-b4c4-427a-967f-e5e5b4be0c8a, type: ENV_SETUP : Task status
changed CREATED -> EXECUTING
[INFO] Creating directory:
comet.sdsc.edu:/oasis/scratch/comet/jsale/temp_project/pga/PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7
[INFO] Session validation failed, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_fb013659-b4c4-427a-967f-e5e5b4be0c8a, type: ENV_SETUP : Task status
changed EXECUTING -> COMPLETED
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed CONFIGURING_WORKSPACE -> INPUT_DATA_STAGING
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING : Task status
changed CREATED -> EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed event received for status INPUT_DATA_STAGING
[INFO] SSH Session validation succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Channel creation test succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Reuse SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Transferring
from:/var/www/experimentData/jsale37/GaussianonComet/Clone_of_Gaussian_Job_51497288328/water_opt_8c.dat
To:
/oasis/scratch/comet/jsale/temp_project/pga//PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7/
[INFO] Session validation failed, key :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[ERROR] JSch initialization error
com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: Comet Storage
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[ERROR] Data staging failed
org.apache.airavata.gfac.core.GFacException: JSch initialization error
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet
Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
... 15 common frames omitted
Caused by: java.net.UnknownHostException: Comet Storage
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING : Task status
changed EXECUTING -> FAILED
[ERROR] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_f3129550-e851-4699-8b09-e8909d17168b type: DATA_STAGING,:- DATA_STAGING
failed, reason: Data staging failed
[ERROR] GFac Worker throws an exception
org.apache.airavata.gfac.core.GFacException: Error: userFriendly msg :Error
while executing DATA_STAGING task, actual msg :expId:
Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb, processId:
PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7, taskId:
TASK_f3129550-e851-4699-8b09-e8909d17168b, type: DATA_STAGING :- DATA_STAGING
failed. Reason: Data staging failed
at
org.apache.airavata.gfac.impl.GFacEngineImpl.checkFailures(GFacEngineImpl.java:628)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:609)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed INPUT_DATA_STAGING -> FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Process status
changed event received for status FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb,
processId: PROCESS_7a041582-fd6a-4d68-9237-3ff1f5d059e7 :- Sent ack for
deliveryTag 1
[INFO] expId : Clone_of_Gaussian_Job_5_849cde15-5a1b-4b14-adec-f3b5af2c48cb :-
Experiment status updated to FAILED
________________________________
From: Supun Nakandala [[email protected]]
Sent: Monday, June 05, 2017 12:12 PM
To: [email protected]
Subject: Re: Running Gaussian on Comet
Jeff,
Basically you need to some setup work in your gateway as the gateway admin
which includes creating and configuring a storage resource. From the error that
I see it seems that the storage resource is not properly configured. More
details on setting up storage resources and other configurations can be found
here
https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+05+-+PHP+Reference+Gateway+for+Airavata+-+Gateway+Admin+Guide#Tutorial05-PHPReferenceGatewayforAiravata-GatewayAdminGuide-TutorialIV-RegisterandManageStorageResources
Can you try to setup it following this tutorial? If it still has issues we can
set up a video call.
On Mon, Jun 5, 2017 at 3:05 PM, Sale, Jeff
<[email protected]<mailto:[email protected]>> wrote:
OK, I made that change and tried again and it looks like I am making progress.
Still some errors with data staging. Here are the errors:
DriverClassName=com.mysql.jdbc.Driver,Url=jdbc:mysql://localhost:3306/replica_catalog?autoReconnect=true,Username=airavata,Password=airavata,validationQuery=SELECT
1 from CONFIGURATION
1 replicacatalog_data INFO [pool-1-thread-1] openjpa.Runtime - Starting
OpenJPA 2.2.0
10 replicacatalog_data INFO [pool-1-thread-1] openjpa.jdbc.JDBC - Using
dictionary class "org.apache.openjpa.jdbc.sql.MySQLDictionary" (MySQL
5.5.5-10.1.21-MariaDB ,MySQL Connector Java mysql-connector-java-5.1.41 (
Revision: 83c6dc41b96809df81444362933043b20a1d49d5 )).
[INFO] Message Received with message id
'LAUNCH.EXP-3399a423-f598-4ef2-b33f-6c07b983ef2c' and with message type
'EXPERIMENT' for experimentId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8
[INFO] Launching experiment with experimentId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 gateway Id:
php_reference_gateway
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
SUCCESSFUL
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
is SUCCESSFUL
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
Launched experiment
[INFO] Message Received with message id
'LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529 and with message
type:LAUNCHPROCESS, for processId:PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c,
expId:Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8
[INFO] Message Received with message id LAUNCHPROCESS and with message type:
{}LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status STARTED
[INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 :-
Experiment status updated to EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, get process cancel
data from zookeeper node
/experiments/Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8/PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c/cancelListener
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed STARTED -> CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : Task status
changed CREATED -> EXECUTING
[INFO] Creating directory:
comet.sdsc.edu:/oasis/scratch/comet/jsale/temp_project/pga/PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c
[INFO] Session validation failed, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : Task status
changed EXECUTING -> COMPLETED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed CONFIGURING_WORKSPACE -> INPUT_DATA_STAGING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : Task status
changed CREATED -> EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status INPUT_DATA_STAGING
[INFO] SSH Session validation succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Channel creation test succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Reuse SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Transferring
from:/var/www/experimentData/jsale37/GaussianonComet/Clone_of_Gaussian_Job_51496689416/water_opt_8c.dat
To:
/oasis/scratch/comet/jsale/temp_project/pga//PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c/
[INFO] Session validation failed, key :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[ERROR] JSch initialization error
com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: Comet Storage
at
java.net<http://java.net>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[ERROR] Data staging failed
org.apache.airavata.gfac.core.GFacException: JSch initialization error
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet
Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
... 15 common frames omitted
Caused by: java.net.UnknownHostException: Comet Storage
at
java.net<http://java.net>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : Task status
changed EXECUTING -> FAILED
[ERROR] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26 type: DATA_STAGING,:- DATA_STAGING
failed, reason: Data staging failed
[ERROR] GFac Worker throws an exception
org.apache.airavata.gfac.core.GFacException: Error: userFriendly msg :Error
while executing DATA_STAGING task, actual msg :expId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8, processId:
PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING :- DATA_STAGING
failed. Reason: Data staging failed
at
org.apache.airavata.gfac.impl.GFacEngineImpl.checkFailures(GFacEngineImpl.java:628)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:609)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed INPUT_DATA_STAGING -> FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Sent ack for
deliveryTag 1
[INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 :-
Experiment status updated to FAILED
--
Thank you
Supun Nakandala
Dept. Computer Science and Engineering
University of Moratuwa
