Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/22138
@koeninger
Yeah I see what you're saying, then IMHO isolating consumers with query
sounds better than others. Adding next offset to the cache key would make
consumer moving bucket in cache every time it is processed, which is not
expected behavior for general pool solution and we have to reinvent the wheel
(and it is not ideal situation for caching, too).
There's an evict thread in Apache Commons Pool running on background, and
we could close consumers being idle for a long time (say 5 mins or higher).
That's another benefit of adopting Apache Commons Pool (maybe available for
most of general pool solutions): we could also evict cached consumers
eventually which topic or partition is removed while query is running. It is
not only evicted because of exceeding cache, but also time of inactivity.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
