andygrove opened a new pull request, #2851: URL: https://github.com/apache/datafusion-ballista/pull/2851
# Which issue does this PR close? N/A # Rationale for this change The scripts in `benchmarks/cluster` never set `imagePullPolicy`, so Kubernetes applies its default: `IfNotPresent` for any tag but `:latest`. After pushing a new build to a tag the cluster has already used (`ballista:main`, say), nodes keep the copy they cached. Different nodes can then run different builds, and a run's scheduler and runner can disagree. I hit this after rebuilding the image: the scheduler rejected the runner's session config because it was an older build. # What changes are included in this PR? - A new `image_pull_policy` setting in `cluster.toml` (`Always`, `IfNotPresent` or `Never`), set on every container the scripts create. The default is empty, which leaves the policy to Kubernetes, so existing configs behave as before. - Spark's executor pods take the same value (`spark.kubernetes.container.image.pullPolicy`), still `IfNotPresent` when unset. - An invalid value is rejected when the config is loaded. - Tests, and the setting documented in `cluster.example.toml`. # Are there any user-facing changes? One optional `cluster.toml` setting; nothing changes unless it is set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
