rangareddy commented on issue #8504: URL: https://github.com/apache/hudi/issues/8504#issuecomment-5102985122
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: This is fixed. The cause was that the jetty server inside `TimelineService` created **non-daemon** threads, so once your Spark job failed and the `SparkContext` stopped, those threads kept the JVM alive and the pod never terminated -- precisely the behaviour in your thread dump. The fix is #8335 (`[HUDI-6009] Let the jetty server in TimelineService create daemon threads`), merged 2023-04-19 -- the PR @stream2000 pointed you at the following day. It did land. Validated against the release tags by checking for daemon-thread handling in `hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java`: absent in `release-0.13.0`, present from **`release-0.13.1`** onward. You reported this on 0.12, which is before the fix, so upgrading to 0.13.1 or later resolves it. On the workaround discussed at the time: prefer upgrading over setting `hoodie.embed.timeline.server=false`. As noted on the thread, the embedded timeline server caches filesystem file handles and improves write performance, so disabling it trades away real throughput to work around a bug that is already fixed. Closing as fixed. If pods still hang after a failed job on 0.13.1+, please reopen with a fresh thread dump -- that would be a different non-daemon thread and worth tracking separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
