churromorales commented on PR #13156: URL: https://github.com/apache/druid/pull/13156#issuecomment-1292671526
> @abhishekagarwal87 > > > may not be an issue but is there a limit in k8s on how large the env variable value can be? The task JSON could be large. > > I just checked the defaults for k3s, microk8s, GKE, and EKS: > > * A configMap value is limited to 1MiB > > * Similarly, an inline `env` value is limited to 1MiB > > * The entire final command passed to the container (command + arguments + ENVs) is limited to the `ARG_MAX` configuration of the host node, which defaults to 2MiB on Linux. > > > None of our tasks reach close to a mebibyte, but if that could be a problem, then the best would be retrieving the task from the coordinator. > > Mounting a config map instead of passing as an env var is cleaner IMHO, but would only make a difference regarding limits if there were multiple envs smaller than 1MiB that combined go over 2MiB. I didn't want to waste quotas for no reason. In k8s you can have a quota for configmaps. I was attempting to not eat up quota items for this work. Config map is cleaner, but env doesn't eat up quotas. Also we compress and base64 encode the task.json, I doubt it would ever get large enough to cause a problem. We have some pretty large task.json files running with this patch and they have not come close to hitting the limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
