GabrielBrascher opened a new pull request #4291: URL: https://github.com/apache/cloudstack/pull/4291
## Description <!--- Describe your changes in detail --> After a few hours running cloudstack 4.13.1.0 storing stats data on InfluxDB CloudSTack hangs with due to OutOfMemoryException raised due to [com.cloud.server.StatsCollector.writeBatches(StatsCollector.java:1510)](https://github.com/apache/cloudstack/blob/1da76d27f13e045ac88e6c494d604d6133486c9c/server/src/main/java/com/cloud/server/StatsCollector.java#L1510): ``` 2020-08-12 21:19:00,972 ERROR [c.c.s.StatsCollector] (StatsCollector-6:ctx-0a4cfe6a) (logid:03a7ba48) Error trying to retrieve host stats java.lang.OutOfMemoryError: unable to create new native thread ... at org.influxdb.impl.BatchProcessor.<init>(BatchProcessor.java:294) at org.influxdb.impl.BatchProcessor$Builder.build(BatchProcessor.java:201) at org.influxdb.impl.InfluxDBImpl.enableBatch(InfluxDBImpl.java:311) at com.cloud.server.StatsCollector.writeBatches(StatsCollector.java:1510) at com.cloud.server.StatsCollector$AbstractStatsCollector.sendMetricsToInfluxdb(StatsCollector.java:1351) at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:522) ``` **Context on InfluxDB Batch:** Enabling batch on InfluxDB is great and speeds writing but it requires caution to avoid Zombie threads. **Solution:** This happens because the batching feature creates an internal thread pool that needs to be shutdown explicitly; therefore, it is important to add: `influxDB.close()`. ## Types of changes <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) ## How Has This Been Tested? <!-- Please describe in detail how you tested your changes. --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> - The `OutOfMemoryException` happens in less than 10 hours on the staging environment. - After patching the proposed fix on the same test environment with the same workload I spend a week waiting and the exception was not thrown. All looking googd with memory consumption. <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document --> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
