Nikolai Romanov created BEAM-12449:
--------------------------------------
Summary: OutOfMemoryError in GoogleDataflow pipeline in 2.29.0
Key: BEAM-12449
URL: https://issues.apache.org/jira/browse/BEAM-12449
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Affects Versions: 2.29.0
Reporter: Nikolai Romanov
Our pipeline reads data from PubSub and writes it to BigQuery.
When we updated our beam version to 2.29.0 we started having lots of errors
like
{quote}"*~*~*~ Channel ManagedChannelImpl\{logId=59,
target=bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
{quote}
which is already mentioned in tickets:
https://issues.apache.org/jira/browse/BEAM-12365
https://issues.apache.org/jira/browse/BEAM-12356
But for us the worst thing was the fact we started having lots of
OutOfMemoryError errors:
{code:JSON}
{
"insertId": "3218987470810364545:19344:0:33525266",
"jsonPayload": {
"thread": "36781",
"job": "2021-06-02_05_56_45-14040068175423437671",
"stage": "P2",
"exception": "java.lang.OutOfMemoryError: unable to create native thread:
possibly out of memory or process/resource limits reached\n\tat
java.base/java.lang.Thread.start0(Native Method)\n\tat
java.base/java.lang.Thread.start(Thread.java:803)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1005)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat
java.base/java.lang.Thread.run(Thread.java:834)\n",
"work": "ac872063622e9bea-1fc5c3cfe1c5e604",
"logger": "org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker",
"worker": "text-events-processor-06020556-7d78-harness-b3z1",
"message": "Uncaught exception in main thread. Exiting with status code 1."
},
"resource": {
"type": "dataflow_step",
"labels": {
"project_id": "bolcom-stg-trex-c7d",
"job_name": "text-events-processor",
"region": "europe-west1",
"job_id": "2021-06-02_05_56_45-14040068175423437671",
"step_id": ""
}
},
"timestamp": "2021-06-02T22:28:07.363Z",
"severity": "ERROR",
"labels": {
"dataflow.googleapis.com/job_id":
"2021-06-02_05_56_45-14040068175423437671",
"compute.googleapis.com/resource_id": "3218987470810364545",
"compute.googleapis.com/resource_name":
"text-events-processor-06020556-7d78-harness-b3z1",
"dataflow.googleapis.com/region": "europe-west1",
"dataflow.googleapis.com/log_type": "supportability",
"compute.googleapis.com/resource_type": "instance",
"dataflow.googleapis.com/job_name": "text-events-processor"
},
"logName":
"projects/bolcom-stg-trex-c7d/logs/dataflow.googleapis.com%2Fworker",
"receiveTimestamp": "2021-06-02T22:28:09.656481597Z"
}
{code}
I suspect that these two errors are related, but I think OutOfMemory shouldn't
be ignored at all.
We had to rollback to 2.28.0. The pipeline works fine now, without any errors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
