OK, I made that change and tried again and it looks like I am making progress.
Still some errors with data staging. Here are the errors:
DriverClassName=com.mysql.jdbc.Driver,Url=jdbc:mysql://localhost:3306/replica_catalog?autoReconnect=true,Username=airavata,Password=airavata,validationQuery=SELECT
1 from CONFIGURATION
1 replicacatalog_data INFO [pool-1-thread-1] openjpa.Runtime - Starting
OpenJPA 2.2.0
10 replicacatalog_data INFO [pool-1-thread-1] openjpa.jdbc.JDBC - Using
dictionary class "org.apache.openjpa.jdbc.sql.MySQLDictionary" (MySQL
5.5.5-10.1.21-MariaDB ,MySQL Connector Java mysql-connector-java-5.1.41 (
Revision: 83c6dc41b96809df81444362933043b20a1d49d5 )).
[INFO] Message Received with message id
'LAUNCH.EXP-3399a423-f598-4ef2-b33f-6c07b983ef2c' and with message type
'EXPERIMENT' for experimentId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8
[INFO] Launching experiment with experimentId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 gateway Id:
php_reference_gateway
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[WARN] Process compute resource scheduling doesn't have valid scratch location,
using gateway computer resource preference scratch
location/oasis/scratch/comet/jsale/temp_project/pga/
[WARN] Process compute resource scheduling doesn't have valid user login name,
using gateway computer resource preference login name jsale
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.BatchQueueValidator is
SUCCESSFUL
[INFO] Validation of
org.apache.airavata.orchestrator.core.validator.impl.ExperimentStatusValidator
is SUCCESSFUL
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
Launched experiment
[INFO] Message Received with message id
'LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529 and with message
type:LAUNCHPROCESS, for processId:PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c,
expId:Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8
[INFO] Message Received with message id LAUNCHPROCESS and with message type:
{}LAUNCH.PROCESS-aeeddab5-cd7b-466e-bbef-2bff4cc6a529
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status STARTED
[INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 :-
Experiment status updated to EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, get process cancel
data from zookeeper node
/experiments/Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8/PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c/cancelListener
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed STARTED -> CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status CONFIGURING_WORKSPACE
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : Task status
changed CREATED -> EXECUTING
[INFO] Creating directory:
comet.sdsc.edu:/oasis/scratch/comet/jsale/temp_project/pga/PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c
[INFO] Session validation failed, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_4490424a-12a7-4d4f-95d2-1850b8cc5ce6, type: ENV_SETUP : Task status
changed EXECUTING -> COMPLETED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed CONFIGURING_WORKSPACE -> INPUT_DATA_STAGING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : Task status
changed CREATED -> EXECUTING
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status INPUT_DATA_STAGING
[INFO] SSH Session validation succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Channel creation test succeeded, key
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Reuse SSH session for
:jsale_comet.sdsc.edu_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Transferring
from:/var/www/experimentData/jsale37/GaussianonComet/Clone_of_Gaussian_Job_51496689416/water_opt_8c.dat
To:
/oasis/scratch/comet/jsale/temp_project/pga//PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c/
[INFO] Session validation failed, key :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[INFO] Initialize a new SSH session for :jsale_Comet
Storage_22_c7eb6c4d-d2b1-430b-9082-3f94a53d6115
[ERROR] JSch initialization error
com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: Comet Storage
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[ERROR] Data staging failed
org.apache.airavata.gfac.core.GFacException: JSch initialization error
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:545)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSshSession(HPCRemoteCluster.java:137)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.getSession(HPCRemoteCluster.java:274)
at
org.apache.airavata.gfac.impl.HPCRemoteCluster.scpThirdParty(HPCRemoteCluster.java:173)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.inputDataStaging(SCPDataStageTask.java:247)
at
org.apache.airavata.gfac.impl.task.SCPDataStageTask.execute(SCPDataStageTask.java:158)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTask(GFacEngineImpl.java:814)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:604)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: java.net.UnknownHostException: Comet
Storage
at com.jcraft.jsch.Util.createSocket(Util.java:349)
at com.jcraft.jsch.Session.connect(Session.java:215)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.airavata.gfac.impl.Factory.getSSHSession(Factory.java:542)
... 15 common frames omitted
Caused by: java.net.UnknownHostException: Comet Storage
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at com.jcraft.jsch.Util.createSocket(Util.java:343)
... 18 common frames omitted
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING : Task status
changed EXECUTING -> FAILED
[ERROR] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26 type: DATA_STAGING,:- DATA_STAGING
failed, reason: Data staging failed
[ERROR] GFac Worker throws an exception
org.apache.airavata.gfac.core.GFacException: Error: userFriendly msg :Error
while executing DATA_STAGING task, actual msg :expId:
Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8, processId:
PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c, taskId:
TASK_bd1d0984-ef2c-4f84-b327-ea5f09e10c26, type: DATA_STAGING :- DATA_STAGING
failed. Reason: Data staging failed
at
org.apache.airavata.gfac.impl.GFacEngineImpl.checkFailures(GFacEngineImpl.java:628)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.inputDataStaging(GFacEngineImpl.java:609)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeTaskListFrom(GFacEngineImpl.java:347)
at
org.apache.airavata.gfac.impl.GFacEngineImpl.executeProcess(GFacEngineImpl.java:286)
at org.apache.airavata.gfac.impl.GFacWorker.executeProcess(GFacWorker.java:227)
at org.apache.airavata.gfac.impl.GFacWorker.run(GFacWorker.java:86)
at
org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed INPUT_DATA_STAGING -> FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Process status
changed event received for status FAILED
[INFO] expId: Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8,
processId: PROCESS_e2a359f3-5caa-4ed7-b203-2fde6ce7962c :- Sent ack for
deliveryTag 1
[INFO] expId : Clone_of_Gaussian_Job_5_f7a276af-dec5-460e-a002-9c5de4d883e8 :-
Experiment status updated to FAILED
