Hanghang Liu created GOBBLIN-1432:
-------------------------------------
Summary: JVM hangs on flushing events after OOM
Key: GOBBLIN-1432
URL: https://issues.apache.org/jira/browse/GOBBLIN-1432
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-kafka, gobblin-metrics
Reporter: Hanghang Liu
Assignee: Shirshanka Das
A hang was observed for ingestion flows after an OOM.
This appears to be due to the kafka pusher flush() call not returning.
Hanging stack.
{code:java}
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
at
org.apache.gobblin.metrics.kafka.KafkaProducerPusher.flush(KafkaProducerPusher.java:125)
at
org.apache.gobblin.metrics.kafka.KafkaProducerPusher.close(KafkaProducerPusher.java:137)
at com.google.common.io.Closer.close(Closer.java:214) at
org.apache.gobblin.metrics.reporter.ConfiguredScheduledReporter.close(ConfiguredScheduledReporter.java:245)
at com.google.common.io.Closer.close(Closer.java:214) at
org.apache.gobblin.metrics.MetricContext.close(MetricContext.java:553)
at
org.apache.gobblin.metrics.RootMetricContext$1.run(RootMetricContext.java:194){code}
Shutdown hook call
{code:java}
"DestroyJavaVM" #308 prio=5 os_prio=0 tid=0x00007fb55c01a800 nid=0x5d51 in
Object.wait() [0x00007fb563f5b000] java.lang.Thread.State: WAITING (on
object monitor) at java.lang.Object.wait(Native Method) -
waiting on <0x00000000e036cf30> (a
org.apache.gobblin.metrics.RootMetricContext$1) at
java.lang.Thread.join(Thread.java:1252) - locked <0x00000000e036cf30>
(a org.apache.gobblin.metrics.RootMetricContext$1) at
java.lang.Thread.join(Thread.java:1326) at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
