zachjsh opened a new pull request #11863:
URL: https://github.com/apache/druid/pull/11863
### Description
Enhanced the `ExtractionNamespace` interface in `lookups-cached-global` core
extension with the ability to set a `maxSize` in bytes for the cache of the
respective namespace. The reason for adding this functionality, is make it
easier to detect when a lookup table grows to a size that the underlying
service cannot handle, because it does not have enough memory.. The default
value of maxSize for the interface is -1, which indicates that no maxSize has
been set. For the `JdbcExtractionNamespace` and `UriExtractionNamespace`
implementations, the default value is null, which will cause the respective
service that the lookup is loaded in, to warn when its cache is beyond the
maximum of either 5MB, or 10% of the service's configured max heap size. If a
non-null value is set for the namespace's `maxSize` config, this value will be
honored for all services that the respective lookup is loaded onto, and
consequently log warning messages when the cache of the respective lookup grows
beyond
this size. Warnings are logged everytime that either Uri based on Jdbc based
lookups are regenerated, no other implementations will log warnings at this
time. No error is thrown when the size exceeds the maxSize at this time, as
doing so could break functionality for existing users. Previously the
`JdbcCacheGenerator` generated its cache by materializing all rows of the
underling table in memory at once; this made it difficult to log warning
messages in the case that the results from the jdbc query exceeded the max
size. To help with this, I've made it so that the jdbc query results are
instead streamed through an iterator.
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
