nchammas commented on code in PR #57083: URL: https://github.com/apache/spark/pull/57083#discussion_r3552509207
########## .github/workflows/build_infra_images_cache.yml: ########## @@ -20,6 +20,13 @@ name: Build / Cache base image on: + # Rebuild cache on a schedule to keep it warm despite base image drift. + # The `ubuntu:noble` tag is re-pointed to a new image roughly monthly, + # which invalidates all cached layers built on top of it. Without a + # scheduled rebuild, the cache goes stale between Dockerfile changes + # and every commit on every PR spends ~50 minutes building base images! + schedule: + - cron: '0 4 * * *' Review Comment: I'm fine with any frequency (the PR actually started with a weekly schedule). But I think it's nice to have the predictability of a daily refresh vs. depending on updates to the Dockerfiles to trigger the workflow. With a weekly refresh schedule, if we somehow happen to have a long period of time where no updates are made to the images, that means potentially accepting the ~50 minute image builds on every change to every PR for up to a week. Another, more sophisticated, approach we could take is to persist the version of `ubuntu:noble` we last built against, run the job hourly, but only rebuild images if we see the version has changed. But that requires persisting the last seen version somewhere, perhaps in a [repo variable](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-variables?versionId=free-pro-team%40latest&productId=actions&restPage=how-tos%2Cwrite-workflows%2Cchoose-what-workflows-do%2Cuse-variables#creating-configuration-variables-for-a-repository). And to reiterate an earlier suggestion, we could eliminate the need for regular refreshes _and_ solve the problem of surprise rebuilds being needed on every PR by simply pinning all of our uses of `ubuntu:noble` to a specific version. Is it that important that we always be running the latest version of `noble`? If not, pinning the version would be the best solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
