jnioche opened a new issue, #2289: URL: https://github.com/apache/stormcrawler/issues/2289
The default value of `status.updater.cache.spec` in `crawler-default.yaml` is `expireAfterAccess=1h,softValues`. It was changed from `maximumSize=10000,expireAfterAccess=1h` in #1854, with the soft values meant to protect against running out of memory now that the size is no longer bounded. The soft values have no effect. `AbstractStatusUpdaterBolt.ack()` does `cache.put(url, "")`, so every value is the `""` literal. That string is interned and referenced from the constant pool, so it is always strongly reachable and its soft references are never cleared. Caffeine only removes an entry with soft values once the value has been collected, which never happens here. The keys (the URLs) are held strongly. The only limit left on the cache is the expiry after one hour, so it holds every DISCOVERED URL acked over the last hour. With a high enough throughput this exhausts the heap. I got an `OutOfMemoryError: Java heap space` in the status updater thread when running `StatusUpdaterBenchmark` through `execute()` on a large input with the default configuration. Using the URL as value would not help either, since the key references the same string. A value that nothing else references, e.g. `new Object()` per entry, would make the soft values work, but it adds to the memory used by each entry and the JVM only clears soft references when the heap is nearly full, after long GC pauses. Which entries go is up to the GC, not based on recency. I suggest going back to a bounded cache by default, e.g. `maximumSize=N,expireAfterAccess=1h`, with N to be agreed. The 10,000 we had before is probably too small for most crawls. If a limit based on memory is preferred, `maximumWeight` with a weigher based on the length of the URL would do it, but the weigher would have to be set in the code as it can't be expressed in a spec string. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
