[
https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380833#comment-16380833
]
Eroma edited comment on AIRAVATA-2620 at 2/28/18 6:58 PM:
----------------------------------------------------------
h4. Tested in:
[https://dev.ultrascan.scigap.org|https://dev.ultrascan.scigap.org/]
h4. Test Cases
h5. NOTE:
# The implementation is mainly tested for Ultrascan gateway.
# This service will be added as a new method to the bridge code.
# Single experiment to execute at a time.
# User should be notified when the file transfer is completed. Currently when
the force processing is run if there are no files to transfer it will not throw
any errors or messages.
h5. Test Cases
# User save and launch an experiment. Post processing is run while experiment
is in EXECUTING, Job in QUEUED. Results: No files will be transferred as the
job is still QUEUED. No errors in the FORCE_PROCESSING. Experiment status will
be changed to FORCE_PROCESSING_COMPLETED. - PASS
# User launch an experiment. Post processing is run while experiment is in
EXECUTING, Job in QUEUED (Email monitoring has stopped and emails have received
for both job start and completion). Results: Files are transferred as the job
is actually done. No errors in the FORCE_PROCESSING. Experiment status will be
changed to FORCE_PROCESSINF_EXECUTING and to FORCE_PROCESSING_COMPLETED. - PASS
# User creates an experiment and when its in CREATED runs FORCE_PROCESSING.
Results: No files will be transferred as the experiment is in CREATED. Results:
No files transferred. No experiment status changed. In api-orch log warning
message [1] exists. – NEED IMPROVEMENT
# User runs the post processing with incorrect non-existing gateway ID – If
experiment in created log has [2] If experiment is failed log has [3] - FAIL
# Use runs FORCE_PROCESSING with NULL as experiment ID. Error returned [7] and
error in Log [8] = PASS
# User runs FORCE_RPOCESSING giving NULL as gateway ID –From PGA point of view
nothing happened but exception thrown in the backend api-orch level. Log has
the exception [6]. Exception does not indicate the gateway ID is non existing.
User should have been informed that the gateway ID is missing. - FAIL
# When experiment in COMPLETED runs FORCE_PROCESSING. Results: No files will
be transferred as the experiment is in COMPLETED as files should be already
transferred. –
# User runs POST_PROCESSING for an experiment which is FAILED. Log has [4] –
No exceptions should be thrown. – This situation should provide user with a
descriptive message. - NEED IMPROVEMENT
# User runs post processing when the job is failed due to incorrect input file.
# Test post processing for experiments with ARCHIVING enabled before job
completion. – Ran FORCE_PROCESSING before the job completion. Existing files
along with ARCHIVE got transferred but ARCHIVE folder is empty. When the job
and experiment actually got completed after FORCE_PROCESSING archive.tar was
not removed. GFAC log has the command to remove it but its not removed. Check
exp ID Ultrascan-T2_96439421-467d-4b15-884c-5a8fb42c6d36 on 27^th^ February, 18
– FAIL
# Test post processing for experiments with ARCHIVING enabled. Run
FORCE_PROCESSING after job completion. ARCHIVE folder is empty. – FAIL
# Test an experiment which only have ARCHIVE, no other output files defined to
stage in app interface. ARCHIVE folder exists but empty. - FAIL
# Run FORCE_PROCESSING when GFAC server is down, but job is all completed.
# Run FORCE_PROCESSING when GFAC server is down, but job is all completed.
Once the FORCE_PROCESSING is completed start the GFAC server.
# Run FORCE_PROCESSING when GFAC server is down, job is also not launched. No
errors or exceptions. FORCE_PROCESSING didn’t run - PASS
# Run FORCE_PROCESSING when GFAC server is down, and job is not launched.
Start FORCE_PROCESSING and then start the GFAC server. – When GFAC server was
started the FORCE_PROCESSING completed and created the empty ARCHIVE directory.
(After the actual experiment completion was run using email monitoring the
archive.tar exists in the folder, still ARCHIVE folder is empty) - FAIL
h5. Improvements
# When the force processing is done, user should be messaged.
# If there were no files to transfer or less number of files than expected
should be notified.
# List the files that got transferred.
# When user runs the force processing for experiments in CRETED, COMPLETED or
FAILED where files would not be transferred or when gateway ID is wrong should
throw exceptions with meaningful messages to the user.
h5. Exceptions (Log messages)
[1]
2018-02-21 16:53:02,882 [pool-30-thread-11] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post
processing of the CREATED experiment with id
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,125 [pool-30-thread-12] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-ffe76d33-62e4-4d91-977a-05bba1d14f98' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,126 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Starting the post
processing of the experiment with experimentId:
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527 gateway Id: ultrascan
2018-02-21 16:53:24,126 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan -
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,129 [pool-30-thread-12] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post
processing of the CREATED experiment with id
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
[2]
2018-02-27 18:59:48,549 [pool-30-thread-9] INFO o.a.a.m.c.i.ExperimentConsumer
- Message Received with message id
'POSTPROCESS.EXP-97bff249-1b22-434d-8988-35a3fe5f5100' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 18:59:48,553 [pool-30-thread-9] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post
processing of the experiment with experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd
2018-02-27 18:59:48,554 [pool-30-thread-9] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd -
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 18:59:48,557 [pool-30-thread-9] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Can't run post
processing of the CREATED experiment with id
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
[3]
2018-02-27 19:08:02,567 [pool-30-thread-13] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-b185b4e7-150f-4b41-9a91-6bdcc55e491d' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 19:08:02,568 [pool-30-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post
processing of the experiment with experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd
2018-02-27 19:08:02,568 [pool-30-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd -
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 19:08:03,641 [pool-30-thread-13] ERROR
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Error while launching post
processing
java.lang.NullPointerException: null
at
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createOutputDataSatagingTasks(SimpleOrchestratorImpl.java:486)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:438)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)
at
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)
at
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[4]
2018-02-27 19:04:11,298 [pool-28-thread-6] INFO
o.a.a.o.s.OrchestratorServerHandler - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated
to FORCE_POST_PROCESSING_EXECUTING
2018-02-27 19:04:11,708 [pool-28-thread-8] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,109 [pool-28-thread-11] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,455 [pool-28-thread-10] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,768 [pool-28-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status COMPLETED
2018-02-27 19:04:13,008 [pool-28-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated
to FORCE_POST_PROCESSING_COMPLETED
[6]
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-e0fe9590-c609-467f-9bb6-9c27dbbbf955' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id= - Starting the post processing
of the experiment with experimentId:
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 gateway Id:
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id= -
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756
2018-02-28 17:03:54,075 [pool-30-thread-12] ERROR
o.a.a.o.s.OrchestratorServerHandler gateway_id= - expId :
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 :- Error while launching post
processing
java.lang.NullPointerException: null
at
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createArchiveDataStatgingTask(SimpleOrchestratorImpl.java:470)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:450)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)
at
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)
at
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[7]
objc[39326]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin/java
(0x1064464c0) and
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/libinstrument.dylib
(0x10650e4e0). One of the two will be used. Which one is undefined.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/Eroma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/Eroma/.m2/repository/ch/qos/logback/logback-classic/1.1.6/logback-classic-1.1.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main"
AiravataSystemException(airavataErrorType:INTERNAL_ERROR, message:Error while
instantiate the registry instance. More info : Requested experiment id does
not exist in the system..)
at
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)
at
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)
at
org.apache.airavata.api.Airavata$executePostProcessing_result.read(Airavata.java)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.airavata.api.Airavata$Client.recv_executePostProcessing(Airavata.java:5015)
at
org.apache.airavata.api.Airavata$Client.executePostProcessing(Airavata.java:5000)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.forcePostProcess(ForcePostProcessingExperiment.java:60)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.register(ForcePostProcessingExperiment.java:55)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.main(ForcePostProcessingExperiment.java:50)
Process finished with exit code 1
[8]
2018-02-28 17:31:15,855 [pool-23-thread-47] ERROR
o.a.a.a.s.h.AiravataServerHandler -
org.apache.airavata.model.error.ExperimentNotFoundException: Requested
experiment id does not exist in the system..
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50536)
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50504)
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result.read(RegistryService.java:50435)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.airavata.registry.api.RegistryService$Client.recv_getExperiment(RegistryService.java:3518)
at
org.apache.airavata.registry.api.RegistryService$Client.getExperiment(RegistryService.java:3505)
at
org.apache.airavata.api.server.handler.AiravataServerHandler.executePostProcessing(AiravataServerHandler.java:1769)
at
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17564)
at
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17548)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
was (Author: eroma_a):
h1. Tested in:
[https://dev.ultrascan.scigap.org|https://dev.ultrascan.scigap.org/]
h1. Test Cases
h2. NOTE:
# The implementation is mainly tested for Ultrascan gateway.
# This service will be added as a new method to the bridge code.
# Single experiment to execute at a time.
# User should be notified when the file transfer is completed. Currently when
the force processing is run if there are no files to transfer it will not throw
any errors or messages.
h2. Test Cases
# User save and launch an experiment. Post processing is run while experiment
is in EXECUTING, Job in QUEUED. Results: No files will be transferred as the
job is still QUEUED. No errors in the FORCE_PROCESSING. Experiment status will
be changed to FORCE_PROCESSING_COMPLETED. - PASS
# User launch an experiment. Post processing is run while experiment is in
EXECUTING, Job in QUEUED (Email monitoring has stopped and emails have received
for both job start and completion). Results: Files are transferred as the job
is actually done. No errors in the FORCE_PROCESSING. Experiment status will be
changed to FORCE_PROCESSINF_EXECUTING and to FORCE_PROCESSING_COMPLETED. - PASS
# User creates an experiment and when its in CREATED runs FORCE_PROCESSING.
Results: No files will be transferred as the experiment is in CREATED. Results:
No files transferred. No experiment status changed. In api-orch log warning
message [1] exists. – NEED IMPROVEMENT
# User runs the post processing with incorrect non-existing gateway ID – If
experiment in created log has [2] If experiment is failed log has [3] - FAIL
# Use runs FORCE_PROCESSING with NULL as experiment ID. Error returned [7] and
error in Log [8] = PASS
# User runs FORCE_RPOCESSING giving NULL as gateway ID –From PGA point of view
nothing happened but exception thrown in the backend api-orch level. Log has
the exception [6]. Exception does not indicate the gateway ID is non existing.
User should have been informed that the gateway ID is missing. - FAIL
# When experiment in COMPLETED runs FORCE_PROCESSING. Results: No files will
be transferred as the experiment is in COMPLETED as files should be already
transferred. –
# User runs POST_PROCESSING for an experiment which is FAILED. Log has [4] –
No exceptions should be thrown. – This situation should provide user with a
descriptive message. - NEED IMPROVEMENT
# User runs post processing when the job is failed due to incorrect input file.
# Test post processing for experiments with ARCHIVING enabled before job
completion. – Ran FORCE_PROCESSING before the job completion. Existing files
along with ARCHIVE got transferred but ARCHIVE folder is empty. When the job
and experiment actually got completed after FORCE_PROCESSING archive.tar was
not removed. GFAC log has the command to remove it but its not removed. Check
exp ID Ultrascan-T2_96439421-467d-4b15-884c-5a8fb42c6d36 on 27^th^ February, 18
– FAIL
# Test post processing for experiments with ARCHIVING enabled. Run
FORCE_PROCESSING after job completion. ARCHIVE folder is empty. – FAIL
# Test an experiment which only have ARCHIVE, no other output files defined to
stage in app interface. ARCHIVE folder exists but empty. - FAIL
# Run FORCE_PROCESSING when GFAC server is down, but job is all completed.
# Run FORCE_PROCESSING when GFAC server is down, but job is all completed.
Once the FORCE_PROCESSING is completed start the GFAC server.
# Run FORCE_PROCESSING when GFAC server is down, job is also not launched. No
errors or exceptions. FORCE_PROCESSING didn’t run - PASS
# Run FORCE_PROCESSING when GFAC server is down, and job is not launched.
Start FORCE_PROCESSING and then start the GFAC server. – When GFAC server was
started the FORCE_PROCESSING completed and created the empty ARCHIVE directory.
(After the actual experiment completion was run using email monitoring the
archive.tar exists in the folder, still ARCHIVE folder is empty) - FAIL
h2. Improvements
# When the force processing is done, user should be messaged.
# If there were no files to transfer or less number of files than expected
should be notified.
# List the files that got transferred.
# When user runs the force processing for experiments in CRETED, COMPLETED or
FAILED where files would not be transferred or when gateway ID is wrong should
throw exceptions with meaningful messages to the user.
h2. Exceptions (Log messages)
[1]
2018-02-21 16:53:02,882 [pool-30-thread-11] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post
processing of the CREATED experiment with id
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,125 [pool-30-thread-12] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-ffe76d33-62e4-4d91-977a-05bba1d14f98' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,126 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Starting the post
processing of the experiment with experimentId:
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527 gateway Id: ultrascan
2018-02-21 16:53:24,126 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan -
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
2018-02-21 16:53:24,129 [pool-30-thread-12] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post
processing of the CREATED experiment with id
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527
[2]
2018-02-27 18:59:48,549 [pool-30-thread-9] INFO o.a.a.m.c.i.ExperimentConsumer
- Message Received with message id
'POSTPROCESS.EXP-97bff249-1b22-434d-8988-35a3fe5f5100' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 18:59:48,553 [pool-30-thread-9] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post
processing of the experiment with experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd
2018-02-27 18:59:48,554 [pool-30-thread-9] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd -
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 18:59:48,557 [pool-30-thread-9] WARN
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Can't run post
processing of the CREATED experiment with id
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
[3]
2018-02-27 19:08:02,567 [pool-30-thread-13] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-b185b4e7-150f-4b41-9a91-6bdcc55e491d' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 19:08:02,568 [pool-30-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post
processing of the experiment with experimentId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd
2018-02-27 19:08:02,568 [pool-30-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd -
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb
2018-02-27 19:08:03,641 [pool-30-thread-13] ERROR
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Error while launching post
processing
java.lang.NullPointerException: null
at
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createOutputDataSatagingTasks(SimpleOrchestratorImpl.java:486)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:438)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)
at
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)
at
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[4]
2018-02-27 19:04:11,298 [pool-28-thread-6] INFO
o.a.a.o.s.OrchestratorServerHandler - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated
to FORCE_POST_PROCESSING_EXECUTING
2018-02-27 19:04:11,708 [pool-28-thread-8] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,109 [pool-28-thread-11] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,455 [pool-28-thread-10] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status OUTPUT_DATA_STAGING
2018-02-27 19:04:12,768 [pool-28-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler - expId:
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId:
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event
received for status COMPLETED
2018-02-27 19:04:13,008 [pool-28-thread-13] INFO
o.a.a.o.s.OrchestratorServerHandler - expId :
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated
to FORCE_POST_PROCESSING_COMPLETED
[6]
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
'POSTPROCESS.EXP-e0fe9590-c609-467f-9bb6-9c27dbbbf955' and with message type
'POSTPROCESSING_START' for experimentId:
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id= - Starting the post processing
of the experiment with experimentId:
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 gateway Id:
2018-02-28 17:03:53,882 [pool-30-thread-12] INFO
o.a.a.o.s.OrchestratorServerHandler gateway_id= -
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756
2018-02-28 17:03:54,075 [pool-30-thread-12] ERROR
o.a.a.o.s.OrchestratorServerHandler gateway_id= - expId :
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 :- Error while launching post
processing
java.lang.NullPointerException: null
at
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createArchiveDataStatgingTask(SimpleOrchestratorImpl.java:470)
at
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:450)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)
at
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)
at
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)
at
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)
at
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[7]
objc[39326]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin/java
(0x1064464c0) and
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/libinstrument.dylib
(0x10650e4e0). One of the two will be used. Which one is undefined.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/Eroma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/Eroma/.m2/repository/ch/qos/logback/logback-classic/1.1.6/logback-classic-1.1.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main"
AiravataSystemException(airavataErrorType:INTERNAL_ERROR, message:Error while
instantiate the registry instance. More info : Requested experiment id does
not exist in the system..)
at
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)
at
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)
at
org.apache.airavata.api.Airavata$executePostProcessing_result.read(Airavata.java)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.airavata.api.Airavata$Client.recv_executePostProcessing(Airavata.java:5015)
at
org.apache.airavata.api.Airavata$Client.executePostProcessing(Airavata.java:5000)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.forcePostProcess(ForcePostProcessingExperiment.java:60)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.register(ForcePostProcessingExperiment.java:55)
at
org.apache.airavata.client.samples.ForcePostProcessingExperiment.main(ForcePostProcessingExperiment.java:50)
Process finished with exit code 1
[8]
2018-02-28 17:31:15,855 [pool-23-thread-47] ERROR
o.a.a.a.s.h.AiravataServerHandler -
org.apache.airavata.model.error.ExperimentNotFoundException: Requested
experiment id does not exist in the system..
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50536)
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50504)
at
org.apache.airavata.registry.api.RegistryService$getExperiment_result.read(RegistryService.java:50435)
at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.airavata.registry.api.RegistryService$Client.recv_getExperiment(RegistryService.java:3518)
at
org.apache.airavata.registry.api.RegistryService$Client.getExperiment(RegistryService.java:3505)
at
org.apache.airavata.api.server.handler.AiravataServerHandler.executePostProcessing(AiravataServerHandler.java:1769)
at
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17564)
at
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17548)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> Force post processing functionality
> ------------------------------------
>
> Key: AIRAVATA-2620
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2620
> Project: Airavata
> Issue Type: Improvement
> Affects Versions: 0.16
> Reporter: Suresh Marru
> Assignee: Dimuthu Upeksha
> Priority: Major
> Fix For: 0.17
>
>
> Due to current limitations of only relying on email for job monitoring, the
> post-processing sometimes has inherent delays. Ultrascan science gateway
> would like to have a capability in airavata to request forcing of post
> processing. This will be used when clients have out of band knowledge about
> job completion (for example through code instrumented UDP messages) and would
> like Airavata to force staging of output files.
> This improvement has to be carefully added so existing life cycle of an
> experiment is not hampred.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
