[
https://issues.apache.org/jira/browse/BEAM-6512?focusedWorklogId=198983&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198983
]
ASF GitHub Bot logged work on BEAM-6512:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Feb/19 22:06
Start Date: 14/Feb/19 22:06
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on issue #7794: [BEAM-6512] Run
GrpcDataServiceTest clients on the main thread
URL: https://github.com/apache/beam/pull/7794#issuecomment-463819584
Ah sorry #7775 is the PR where I added the logging. Hopefully that helps you
make sense of the detailed log.
You're right there are two things that get deferred:
1. The creation of the clients (threads are managed in `clientFutures`)
2. The calls to onNext (managed by gRPC)
I removed both of them in this PR, but #2 is really the root of the problem,
and to your point, #1 is necessary to actually exercise the GrpcDataService's
client creation in the face of concurrency.
What if I just remove #2, i.e. use the direct executor so the onNext calls
are not deferred, but continue to create each client in it's own thread? I
pushed up some commits that revert my previous changes and do that instead. It
seems to fix the problem (tested by adding a Thread.sleep to the onNext
function which makes the previous test fail consistently), and still ensures
GrpcDataService correctly handles concurrency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 198983)
Time Spent: 2h 20m (was: 2h 10m)
> [beam_PreCommit_Java_Cron] [GrpcDataServiceTest] Flake, Multiplexer hanging up
> ------------------------------------------------------------------------------
>
> Key: BEAM-6512
> URL: https://issues.apache.org/jira/browse/BEAM-6512
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Daniel Oliveira
> Assignee: Brian Hulette
> Priority: Minor
> Labels: currently-failing, flake, triaged
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
> * [Jenkins Job|https://builds.apache.org/job/beam_PreCommit_Java_Cron/869/]
> * [Gradle Build Scan|https://scans.gradle.com/s/wodzocvegyy5a]
> * [Test source
> code|https://github.com/apache/beam/blob/f560edc5a4e38cb13d41718540271ae79d7d00ee/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java#L105]
> Initial investigation:
> I see this message:
> {noformat}
> org.apache.beam.runners.fnexecution.data.GrpcDataServiceTest >
> testMessageReceivedBySingleClientWhenThereAreMultipleClients FAILED
> java.lang.AssertionError at GrpcDataServiceTest.java:105
> {noformat}
> And this message:
> {noformat}
> [grpc-default-executor-2] WARN
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Hanged up for unknown
> endpoint.
> Jan 25, 2019 6:16:02 PM
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed@41415b8e
> java.lang.RuntimeException:
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException:
> CANCELLED: Multiplexer hanging up
> at
> org.apache.beam.sdk.fn.test.TestStreams.lambda$throwingErrorHandler$0(TestStreams.java:95)
> at
> org.apache.beam.sdk.fn.test.TestStreams$ForwardingCallStreamObserver.onError(TestStreams.java:144)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException:
> CANCELLED: Multiplexer hanging up
> at
> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:526)
> ... 19 more
> {noformat}
> Note that the second message is something I've seen when successfully ending
> a job without in portable runners (specifically the Java Reference Runner),
> so it might be a red herring.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
