Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/22138
I just addressed eviction to consumer pool as well as added relevant test.
This will help closing invalid idle consumers which topic or partition are no
longer be assigned to any tasks. I guess current cache is not capable of
closing invalid consumers.
I haven't find how to add "query id" to the cache key, but IMHO the patch
itself is already providing some values to be merged. It would be even better
if someone could guide how to add "query id" to the cache key.
@koeninger @tdas @zsxwing Please take a look and comment. Thanks in advance!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
