[
https://issues.apache.org/jira/browse/BEAM-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519225#comment-17519225
]
Chamikara Madhusanka Jayalath commented on BEAM-14263:
------------------------------------------------------
Container startup failed due to following:
Info
2022-04-07 09:40:43.746 PDT
"Caused by: org.apache.beam.vendor.grpc.v1p43p2.io.grpc.StatusRuntimeException:
UNAVAILABLE: Network closed for unknown reason "
Info
2022-04-07 09:40:43.746 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.Status.asRuntimeException(Status.java:535)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:479)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:102)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:95)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:751)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:687)
"
Info
2022-04-07 09:40:43.747 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.DelayedStream$DelayedStreamListener$4.run(DelayedStream.java:510)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.DelayedStream$DelayedStreamListener.delayOrExecute(DelayedStream.java:462)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.DelayedStream$DelayedStreamListener.closed(DelayedStream.java:507)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ForwardingClientStreamListener.closed(ForwardingClientStreamListener.java:34)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.InternalSubchannel$CallTracingTransport$1$1.closed(InternalSubchannel.java:693)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState.closeListener(AbstractClientStream.java:459)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState.access$400(AbstractClientStream.java:221)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState$1.run(AbstractClientStream.java:442)
"
Info
2022-04-07 09:40:43.748 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState.deframerClosed(AbstractClientStream.java:278)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.Http2ClientStreamTransportState.deframerClosed(Http2ClientStreamTransportState.java:31)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.MessageDeframer.close(MessageDeframer.java:233)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.MessageDeframer.closeWhenComplete(MessageDeframer.java:191)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:200)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:445)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:401)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.netty.NettyClientHandler$3.visit(NettyClientHandler.java:446)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.forEachActiveStream(DefaultHttp2Connection.java:974)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.handler.codec.http2.DefaultHttp2Connection.forEachActiveStream(DefaultHttp2Connection.java:208)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.grpc.netty.NettyClientHandler.channelInactive(NettyClientHandler.java:441)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
"
Info
2022-04-07 09:40:43.749 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
"
Info
2022-04-07 09:40:43.750 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
"
Info
2022-04-07 09:40:43.750 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
"
Info
2022-04-07 09:40:43.750 PDT
" at
org.apache.beam.vendor.grpc.v1p43p2.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
"
Info
2022-04-07 09:40:43.750 PDT
" at java.lang.Thread.run(Thread.java:750) "
> beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing
> consistently
> --------------------------------------------------------------------------------------
>
> Key: BEAM-14263
> URL: https://issues.apache.org/jira/browse/BEAM-14263
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Daniel Oliveira
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P1
> Labels: currently-failing
>
> testBigQueryStorageWrite30MProto seems to have been failing since being
> originally introduced. First failure I found:
> https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2/1391/, which
> includes when this PR was merged: https://github.com/apache/beam/pull/17038
> I don't see any explicit reason this might be failing either in the console
> logs or Dataflow logs. Here's the console logs:
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes:
> S06:WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)/Reshard/Read+WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)/Reshard/UnreifyWindow+WriteToBQ/StorageApiLoads/GroupIntoBatches/ParDo(GroupIntoBatches)/ParMultiDo(GroupIntoBatches)+WriteToBQ/StorageApiLoads/StorageApiWriteSharded/Write
>
> Records/ParMultiDo(WriteRecords)/Reshard/Reify+WriteToBQ/StorageApiLoads/StorageApiWriteSharded/Write
> Records/ParMultiDo(WriteRecords)/Reshard/Write failed., The job failed
> because a work item has failed 4 times. Look in previous log entries for the
> cause of each one of the 4 failures. For more information, see
> https://cloud.google.com/dataflow/docs/guides/common-errors. The work item
> was attempted on these workers:
> testpipeline-jenkins-0319-03190548-sfjp-harness-263n
> Root cause: The worker lost contact with the service.,
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
