[
https://issues.apache.org/jira/browse/AIRAVATA-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348801#comment-16348801
]
Marcus Christie commented on AIRAVATA-2664:
-------------------------------------------
I think the problem here is that the registryClient in the
OrchestratorServerHandler is shared amongst different threads but it isn't
thread safe. See [this
thrift-user|http://mail-archives.apache.org/mod_mbox/thrift-user/201207.mbox/%3cCALdd-zhraB5CZJfFmFEYp74x93n=1epi9qdksntrajvfz+o...@mail.gmail.com%3e]
discussion about the thrift client not being thread safe.
Using the ThriftClientPool should fix this issue, but we're still waiting on
AIRAVATA-2607 to have it in a common place.
I think for now a short term fix will be to create and destroy a registry
client for reach request.
> Error while updating task status, hence updated experiment status to FAILED
> org.apache.thrift.TApplicationException: getProcessIds failed: out of
> sequence response: expected 88 but got 87
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AIRAVATA-2664
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2664
> Project: Airavata
> Issue Type: Bug
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> {noformat}
> 2018-02-01 15:02:47,912 [pool-37-thread-1] ERROR
> o.a.a.o.s.OrchestratorServerHandler experiment_name=Clone of Test22,
> experiment_id=Clone_of_Test22_d8e660d1-1093-44d3-9d3e-1b965cabc89b,
> gateway_id=seagrid - expId:
> Clone_of_Test22_d8e660d1-1093-44d3-9d3e-1b965cabc89b, Error while updating
> task status, hence updated experiment status to FAILED
> org.apache.thrift.TApplicationException: getProcessIds failed: out of
> sequence response: expected 88 but got 87
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.airavata.registry.api.RegistryService$Client.recv_getProcessIds(RegistryService.java:4257)
> at
> org.apache.airavata.registry.api.RegistryService$Client.getProcessIds(RegistryService.java:4244)
> at
> org.apache.airavata.orchestrator.server.OrchestratorServerHandler$SingleAppExperimentRunner.launchSingleAppExperiment(OrchestratorServerHandler.java:464)
> at
> org.apache.airavata.orchestrator.server.OrchestratorServerHandler$SingleAppExperimentRunner.run(OrchestratorServerHandler.java:453)
> at
> org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01 15:02:47,913 [pool-37-thread-1] ERROR
> o.a.a.o.s.OrchestratorServerHandler experiment_name=Clone of Test22,
> experiment_id=Clone_of_Test22_d8e660d1-1093-44d3-9d3e-1b965cabc89b,
> gateway_id=seagrid - Unable to launch experiment..
> org.apache.thrift.TException: org.apache.thrift.TApplicationException:
> getProcessIds failed: out of sequence response: expected 88 but got 87
> at
> org.apache.airavata.orchestrator.server.OrchestratorServerHandler$SingleAppExperimentRunner.launchSingleAppExperiment(OrchestratorServerHandler.java:486)
> at
> org.apache.airavata.orchestrator.server.OrchestratorServerHandler$SingleAppExperimentRunner.run(OrchestratorServerHandler.java:453)
> at
> org.apache.airavata.common.logging.MDCUtil.lambda$wrapWithMDC$0(MDCUtil.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: getProcessIds failed: out
> of sequence response: expected 88 but got 87
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.airavata.registry.api.RegistryService$Client.recv_getProcessIds(RegistryService.java:4257)
> at
> org.apache.airavata.registry.api.RegistryService$Client.getProcessIds(RegistryService.java:4244)
> at
> org.apache.airavata.orchestrator.server.OrchestratorServerHandler$SingleAppExperimentRunner.launchSingleAppExperiment(OrchestratorServerHandler.java:464)
> ... 5 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
