lukecwik opened a new issue, #23588: URL: https://github.com/apache/beam/issues/23588
### What happened? The `use_grpc_for_gcs` experiment was broken by the upgrade of the big dataoss library from 2.2.6 to 2.2.8 (https://github.com/apache/beam/pull/23300) and was caught by internal testing within Google (b/251766179). I used the following flags when creating the pipeline: ``` --experiments=use_grpc_for_gcs --gcsUploadBufferSizeBytes=16777216 ``` The pipeline fails to upload artifacts during pipeline creation. An example dump of the failure is below: ``` WARNING: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/. Oct 11, 2022 3:39:58 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.jobName did not match the service requirements. Using df2-source-bench-text-100g-staging-10111534327073 instead of df2-source-bench-text-100G-staging-10111534327073. Oct 11, 2022 3:39:59 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Oct 11, 2022 3:40:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 2 files from PipelineOptions.filesToStage to staging location to prepare for execution. Oct 11, 2022 3:40:00 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/local/google/_blaze_lcwik/1fc426c8a75c8141859a9e8595f2fd0a/execroot/google3/blaze-out/k8-opt/bin/cloud/dataflow/testing/integration/perf/source_bench_text_100G_gcs_with_grpc.runfiles/google3/cloud/dataflow/testing/framework/options/conscrypt.jar to gs://deft-guitar-sandbox-e2e-test/df2-source-bench-text-100G-staging/10111534327072/staging/lcwik/conscrypt-AfnHQstZKhUeLmK9U5eomAYoqWcAH82s1KpHRGeGhfM.jar Oct 11, 2022 3:40:00 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /usr/local/google/_blaze_lcwik/1fc426c8a75c8141859a9e8595f2fd0a/execroot/google3/blaze-out/k8-opt/bin/cloud/dataflow/testing/integration/perf/source_bench_text_100G_gcs_with_grpc.runfiles/google3/javatests/com/google/cloud/dataflow/v2/beam-testing-bundle.jar to gs://deft-guitar-sandbox-e2e-test/df2-source-bench-text-100G-staging/10111534327072/staging/lcwik/beam-testing-bundle-VMTQVJGM2Ldw1dAThX5bn1C7UwZoBdhVhq6XD-GXCHU.jar Oct 11, 2022 3:43:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Still staging 2 files Oct 11, 2022 3:46:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Still staging 2 files Oct 11, 2022 3:46:11 PM com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 710, then retrying. java.io.IOException: Got transient error for UploadID 'ADPycdtUrXxAYd3LoEhcLfF-VNMfY2kvwpyIzERGjfcVAaDWbGF7Ep1We5NSt0n9gJiLc7MxBGcZgSv5WwfjyKRgK87Ekw' at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.doResumableUpload(GoogleCloudStorageGrpcWriteChannel.java:325) at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66) at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:105) at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.call(GoogleCloudStorageGrpcWriteChannel.java:206) at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.call(GoogleCloudStorageGrpcWriteChannel.java:174) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:830) Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: 408:Request Timeout at io.grpc.Status.asRuntimeException(Status.java:535) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:802) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:436) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ... 3 more Oct 11, 2022 3:49:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Still staging 2 files Oct 11, 2022 3:52:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Still staging 2 files Oct 11, 2022 3:52:20 PM com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 580, then retrying. java.io.IOException: Got transient error for UploadID 'ADPycdtUrXxAYd3LoEhcLfF-VNMfY2kvwpyIzERGjfcVAaDWbGF7Ep1We5NSt0n9gJiLc7MxBGcZgSv5WwfjyKRgK87Ekw' at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.doResumableUpload(GoogleCloudStorageGrpcWriteChannel.java:325) at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66) at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:105) at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.call(GoogleCloudStorageGrpcWriteChannel.java:206) at com.google.cloud.hadoop.gcsio.GoogleCloudStorageGrpcWriteChannel$UploadOperation.call(GoogleCloudStorageGrpcWriteChannel.java:174) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:830) Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: 408:Request Timeout at io.grpc.Status.asRuntimeException(Status.java:535) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:802) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:436) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ... 3 more ``` ### Issue Priority Priority: 0 ### Issue Component Component: io-java-gcp -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
