[ 
https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380833#comment-16380833
 ] 

Eroma commented on AIRAVATA-2620:
---------------------------------

h1. Tested in: 
[https://dev.ultrascan.scigap.org|https://dev.ultrascan.scigap.org/]
h1. Test Cases
h2. NOTE:
 # The implementation is mainly tested for Ultrascan gateway.
 # This service will be added as a new method to the bridge code.
 # Single experiment to execute at a time.
 # User should be notified when the file transfer is completed. Currently when 
the force processing is run if there are no files to transfer it will not throw 
any errors or messages.

h2. Test Cases
 # User save and launch an experiment. Post processing is run while experiment 
is in EXECUTING, Job in QUEUED. Results: No files will be transferred as the 
job is still QUEUED. No errors in the FORCE_PROCESSING. Experiment status will 
be changed to FORCE_PROCESSING_COMPLETED. - PASS
 # User launch an experiment. Post processing is run while experiment is in 
EXECUTING, Job in QUEUED (Email monitoring has stopped and emails have received 
for both job start and completion). Results: Files are transferred as the job 
is actually done. No errors in the FORCE_PROCESSING. Experiment status will be 
changed to FORCE_PROCESSINF_EXECUTING and to FORCE_PROCESSING_COMPLETED. - PASS
 # User creates an experiment and when its in CREATED runs FORCE_PROCESSING. 
Results: No files will be transferred as the experiment is in CREATED. Results: 
No files transferred. No experiment status changed. In api-orch log warning 
message [1] exists. – NEED IMPROVEMENT
 # User runs the post processing with incorrect non-existing gateway ID – If 
experiment in created log has [2] If experiment is failed log has [3] - FAIL
 # Use runs FORCE_PROCESSING with NULL as experiment ID. Error returned [7] and 
error in Log [8] = PASS
 # User runs FORCE_RPOCESSING giving NULL as gateway ID –From PGA point of view 
nothing happened but exception thrown in the backend api-orch level. Log has 
the exception [6]. Exception does not indicate the gateway ID is non existing. 
User should have been informed that the gateway ID is missing. - FAIL
 # When experiment in COMPLETED runs FORCE_PROCESSING. Results: No files will 
be transferred as the experiment is in COMPLETED as files should be already 
transferred. –
 # User runs POST_PROCESSING for an experiment which is FAILED. Log has [4] – 
No exceptions should be thrown. – This situation should provide user with a 
descriptive message. - NEED IMPROVEMENT
 # User runs post processing when the job is failed due to incorrect input file.
 # Test post processing for experiments with ARCHIVING enabled before job 
completion. – Ran FORCE_PROCESSING before the job completion. Existing files 
along with ARCHIVE got transferred but ARCHIVE folder is empty. When the job 
and experiment actually got completed after FORCE_PROCESSING archive.tar was 
not removed. GFAC log has the command to remove it but its not removed. Check 
exp ID Ultrascan-T2_96439421-467d-4b15-884c-5a8fb42c6d36 on 27^th^ February, 18 
– FAIL
 # Test post processing for experiments with ARCHIVING enabled. Run 
FORCE_PROCESSING after job completion. ARCHIVE folder is empty. – FAIL
 # Test an experiment which only have ARCHIVE, no other output files defined to 
stage in app interface. ARCHIVE folder exists but empty. - FAIL
 # Run FORCE_PROCESSING when GFAC server is down, but job is all completed.
 # Run FORCE_PROCESSING when GFAC server is down, but job is all completed. 
Once the FORCE_PROCESSING is completed start the GFAC server.
 # Run FORCE_PROCESSING when GFAC server is down, job is also not launched. No 
errors or exceptions. FORCE_PROCESSING didn’t run - PASS
 # Run FORCE_PROCESSING when GFAC server is down, and job is not launched. 
Start FORCE_PROCESSING and then start the GFAC server. – When GFAC server was 
started the FORCE_PROCESSING completed and created the empty ARCHIVE directory. 
(After the actual experiment completion was run using email monitoring the 
archive.tar exists in the folder, still ARCHIVE folder is empty) - FAIL

h2. Improvements
 # When the force processing is done, user should be messaged.
 # If there were no files to transfer or less number of files than expected 
should be notified.
 # List the files that got transferred.
 # When user runs the force processing for experiments in CRETED, COMPLETED or 
FAILED where files would not be transferred or when gateway ID is wrong should 
throw exceptions with meaningful messages to the user.

h2. Exceptions (Log messages)

[1]

2018-02-21 16:53:02,882 [pool-30-thread-11] WARN  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post 
processing of the CREATED experiment with id 
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527

2018-02-21 16:53:24,125 [pool-30-thread-12] INFO  
o.a.a.m.c.i.ExperimentConsumer  -  Message Received with message id 
'POSTPROCESS.EXP-ffe76d33-62e4-4d91-977a-05bba1d14f98' and with message type 
'POSTPROCESSING_START'  for experimentId: 
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527

2018-02-21 16:53:24,126 [pool-30-thread-12] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Starting the post 
processing of the experiment with experimentId: 
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527 gateway Id: ultrascan

2018-02-21 16:53:24,126 [pool-30-thread-12] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - 
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527

2018-02-21 16:53:24,129 [pool-30-thread-12] WARN  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post 
processing of the CREATED experiment with id 
Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527

 

[2]

2018-02-27 18:59:48,549 [pool-30-thread-9] INFO  o.a.a.m.c.i.ExperimentConsumer 
 -  Message Received with message id 
'POSTPROCESS.EXP-97bff249-1b22-434d-8988-35a3fe5f5100' and with message type 
'POSTPROCESSING_START'  for experimentId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb

2018-02-27 18:59:48,553 [pool-30-thread-9] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post 
processing of the experiment with experimentId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd

2018-02-27 18:59:48,554 [pool-30-thread-9] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb

2018-02-27 18:59:48,557 [pool-30-thread-9] WARN  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Can't run post 
processing of the CREATED experiment with id 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb

 

 

[3]

2018-02-27 19:08:02,567 [pool-30-thread-13] INFO  
o.a.a.m.c.i.ExperimentConsumer  -  Message Received with message id 
'POSTPROCESS.EXP-b185b4e7-150f-4b41-9a91-6bdcc55e491d' and with message type 
'POSTPROCESSING_START'  for experimentId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb

2018-02-27 19:08:02,568 [pool-30-thread-13] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - Starting the post 
processing of the experiment with experimentId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb gateway Id: ultrascanmd

2018-02-27 19:08:02,568 [pool-30-thread-13] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb

2018-02-27 19:08:03,641 [pool-30-thread-13] ERROR 
o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascanmd - expId : 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Error while launching post 
processing

java.lang.NullPointerException: null

                  at 
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createOutputDataSatagingTasks(SimpleOrchestratorImpl.java:486)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:438)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)

                  at 
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)

                  at 
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)

                  at 
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)

                  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                  at java.lang.Thread.run(Thread.java:748)

 

[4]

2018-02-27 19:04:11,298 [pool-28-thread-6] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId : 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated 
to FORCE_POST_PROCESSING_EXECUTING

2018-02-27 19:04:11,708 [pool-28-thread-8] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId: 
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event 
received for status OUTPUT_DATA_STAGING

2018-02-27 19:04:12,109 [pool-28-thread-11] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId: 
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event 
received for status OUTPUT_DATA_STAGING

2018-02-27 19:04:12,455 [pool-28-thread-10] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId: 
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event 
received for status OUTPUT_DATA_STAGING

2018-02-27 19:04:12,768 [pool-28-thread-13] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId: 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb, processId: 
PROCESS_8743eef1-a3c8-497f-b49c-cbb7990cbac5 :- Process status changed event 
received for status COMPLETED

2018-02-27 19:04:13,008 [pool-28-thread-13] INFO  
o.a.a.o.s.OrchestratorServerHandler  - expId : 
Ultrascan-T1_e0ff1b3f-6613-48c0-9b29-d6fe8bec5afb :- Experiment status updated 
to FORCE_POST_PROCESSING_COMPLETED

 

[6]

2018-02-28 17:03:53,882 [pool-30-thread-12] INFO  
o.a.a.m.c.i.ExperimentConsumer  -  Message Received with message id 
'POSTPROCESS.EXP-e0fe9590-c609-467f-9bb6-9c27dbbbf955' and with message type 
'POSTPROCESSING_START'  for experimentId: 
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756

2018-02-28 17:03:53,882 [pool-30-thread-12] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id= - Starting the post processing 
of the experiment with experimentId: 
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 gateway Id:

2018-02-28 17:03:53,882 [pool-30-thread-12] INFO  
o.a.a.o.s.OrchestratorServerHandler gateway_id= - 
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756

2018-02-28 17:03:54,075 [pool-30-thread-12] ERROR 
o.a.a.o.s.OrchestratorServerHandler gateway_id= - expId : 
Ultrascan-T6_effdf142-9da8-4e94-afd0-45072a068756 :- Error while launching post 
processing

java.lang.NullPointerException: null

                  at 
org.apache.airavata.orchestrator.core.utils.OrchestratorUtils.getScratchLocation(OrchestratorUtils.java:229)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.getOutputDataStagingTask(SimpleOrchestratorImpl.java:622)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createArchiveDataStatgingTask(SimpleOrchestratorImpl.java:470)

                  at 
org.apache.airavata.orchestrator.cpi.impl.SimpleOrchestratorImpl.createAndSaveOutputDataStagingTasks(SimpleOrchestratorImpl.java:450)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.createAndSavePostProcessingProcess(OrchestratorServerHandler.java:307)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.validateStatesAndStartPostProcessing(OrchestratorServerHandler.java:518)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler.launchPostProcessingOfExperiment(OrchestratorServerHandler.java:402)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.startPostProcessingOfExperiment(OrchestratorServerHandler.java:839)

                  at 
org.apache.airavata.orchestrator.server.OrchestratorServerHandler$ExperimentHandler.onMessage(OrchestratorServerHandler.java:823)

                  at 
org.apache.airavata.messaging.core.impl.ExperimentConsumer.handleDelivery(ExperimentConsumer.java:85)

                  at 
com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:144)

                  at 
com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:99)

                  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                  at java.lang.Thread.run(Thread.java:748)

 

[7]

objc[39326]: Class JavaLaunchHelper is implemented in both 
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/bin/java 
(0x1064464c0) and 
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/libinstrument.dylib
 (0x10650e4e0). One of the two will be used. Which one is undefined.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in 
[jar:file:/Users/Eroma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in 
[jar:file:/Users/Eroma/.m2/repository/ch/qos/logback/logback-classic/1.1.6/logback-classic-1.1.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Exception in thread "main" 
AiravataSystemException(airavataErrorType:INTERNAL_ERROR, message:Error while 
instantiate the registry instance. More info : Requested experiment id  does 
not exist in the system..)

                  at 
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)

                  at 
org.apache.airavata.api.Airavata$executePostProcessing_result$executePostProcessing_resultStandardScheme.read(Airavata.java)

                  at 
org.apache.airavata.api.Airavata$executePostProcessing_result.read(Airavata.java)

                  at 
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)

                  at 
org.apache.airavata.api.Airavata$Client.recv_executePostProcessing(Airavata.java:5015)

                  at 
org.apache.airavata.api.Airavata$Client.executePostProcessing(Airavata.java:5000)

                  at 
org.apache.airavata.client.samples.ForcePostProcessingExperiment.forcePostProcess(ForcePostProcessingExperiment.java:60)

                  at 
org.apache.airavata.client.samples.ForcePostProcessingExperiment.register(ForcePostProcessingExperiment.java:55)

                  at 
org.apache.airavata.client.samples.ForcePostProcessingExperiment.main(ForcePostProcessingExperiment.java:50)

 

Process finished with exit code 1

 

[8]

2018-02-28 17:31:15,855 [pool-23-thread-47] ERROR 
o.a.a.a.s.h.AiravataServerHandler  -

org.apache.airavata.model.error.ExperimentNotFoundException: Requested 
experiment id  does not exist in the system..

                  at 
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50536)

                  at 
org.apache.airavata.registry.api.RegistryService$getExperiment_result$getExperiment_resultStandardScheme.read(RegistryService.java:50504)

                  at 
org.apache.airavata.registry.api.RegistryService$getExperiment_result.read(RegistryService.java:50435)

                  at 
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)

                  at 
org.apache.airavata.registry.api.RegistryService$Client.recv_getExperiment(RegistryService.java:3518)

                  at 
org.apache.airavata.registry.api.RegistryService$Client.getExperiment(RegistryService.java:3505)

                  at 
org.apache.airavata.api.server.handler.AiravataServerHandler.executePostProcessing(AiravataServerHandler.java:1769)

                  at 
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17564)

                  at 
org.apache.airavata.api.Airavata$Processor$executePostProcessing.getResult(Airavata.java:17548)

                  at 
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

                  at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

                  at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)

                  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                  at java.lang.Thread.run(Thread.java:748)

> Force post processing functionality 
> ------------------------------------
>
>                 Key: AIRAVATA-2620
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-2620
>             Project: Airavata
>          Issue Type: Improvement
>    Affects Versions: 0.16
>            Reporter: Suresh Marru
>            Assignee: Dimuthu Upeksha
>            Priority: Major
>             Fix For: 0.17
>
>
> Due to current limitations of only relying on email for job monitoring, the 
> post-processing sometimes has inherent delays. Ultrascan science gateway 
> would like to have a capability in airavata to request forcing of post 
> processing. This will be used when clients have out of band knowledge about 
> job completion (for example through code instrumented UDP messages) and would 
> like Airavata to force staging of output files.
> This improvement has to be carefully added so existing life cycle of an 
> experiment is not hampred. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to