damccorm opened a new issue, #20771: URL: https://github.com/apache/beam/issues/20771
Hi ALL. It seems there is some regression error in the last Beam version – 2.27.0. There is a streaming dataflow that is runned in Google Cloud. This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0. But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below: ``` Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175) java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197) org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) ``` The error is rare. 99% of a time the dataflow works without this (actually any) error. This issue blocks us to make upgrade of the Beam library for the dataflow. Imported from Jira [BEAM-11825](https://issues.apache.org/jira/browse/BEAM-11825). Original Jira may contain additional context. Reported by: trunikov. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
