jasonk000 commented on issue #8801: URL: https://github.com/apache/druid/issues/8801#issuecomment-793255773
Ideally the `K8sTaskRunner` becomes an extension point where we can plug in any sort of `TaskRunner` extension. Is this the intended approach? I think there would be value in looking at this for a different job scheduler implementations (non-k8s, similar in some ideas). Having this as a standardised extension point will allow us to run this on any scheduler. With regards to comments on HPA / scaling requirements, currently we set task properties in Druid ingestion spec incl `context.druid.indexer.runner.javaOpts` to set thread counts and offheap memory limits, etc. I think we can use a similar mechanism in the ingestion spec to pass specific configuration to the task runner such as the expected resources required, with some defaults provided in configuration. This keeps the spec simple and can be modified for each use case. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
