All,
Over the last several days, I have been updating the build scripts used
by YuniKorn to fix several inconsistencies and annoyances. This work is
now complete, and there are several notable changes that may be
impactful to you:
- Changes common to yunikorn-k8shim, yunikorn-core, and yunikorn-web:
- All build tools are now installed to the "tools" directory within
each repository. This prevents overwriting user or system-level
versions, and ensures the proper versions are used in build
scripts. This includes:
- shellcheck (used for make check_scripts)
- golangci-lint (used for make lint)
- kubectl (used for e2e tests)
- kind (used for e2e tests)
- helm (used for e2e tests)
- spark (used for e2e tests)
The downloaded versions will be cached between builds. To force
re-downloading (such as for a new version), use the new
"make distclean" target.
- All generated build artifacts are now created under a standard
"build" directory (instead of "_output" or random source tree
locations).
- The go compiler can now be overridden during a build by setting
the GO env var to a different version:
$ make GO=go1.20.5 image
This can be useful when testing newer compilers, as you no
longer have to change your system default.
- The Makefile targets have now been updated to do proper
dependency checking. In other words, "make build" if run
twice in a row will run much faster the second time, as
build targets will only be made if the dependencies are out
of date.
- Additional changes for yunikorn-k8shim
- Source-level Dockerfiles are now located under "docker/{component}"
rather than "deployments/image/{component}". With this change,
the existing "deployments" folder now contains only non-essential
examples and documentation. Additionally, the contents of
the "docker/{component}" directories are copied into
"build/docker/{component}" before creating Docker images,
ensuring that the Docker build environments remain pristine and
generated binaries do not end up in the source codd directories.
- The generated binaries now have consistent naming:
- yunikorn-scheduler
- yunikorn-scheduler-plugin
- yunikorn-admission-controller
Additionally, binaries for the local (host-specific) architecture
are written to "build/dev" while target binaries are written to
"build/bin".
There should be no impact from this change for releases as the
Dockerfiles have been updated to reflect the new binary names.
- Docker image tag names can now be overridden during the build:
$ make \
SCHEDULER_TAG=local/yunikorn-scheduler:latest \
PLUGIN_TAG=local/yunikorn-scheduler-plugin:latest \
ADMISSION_TAG=local/yunikorn-admission:latest image
This is useful when deploying to local repositories as it
avoids the necessity of renaming the images after the build.
- Additional changes for yunikorn-web
- The Docker image tag used for the web UI can now be overridden
during the build:
$ make WEB_TAG=local/yunikorn-web:latest image
This is useful when deploying to local repositories as it
avoids the necessity of renaming the images after the build.
Regards,
Craig Condit
PS - See the following JIRAs for more information:
- https://issues.apache.org/jira/browse/YUNIKORN-1862
Shim: Use privately installed build tools
- https://issues.apache.org/jira/browse/YUNIKORN-1865
Shim: Update build process to follow standard conventions
- https://issues.apache.org/jira/browse/YUNIKORN-1868
Core: Update build process to follow standard conventions
- https://issues.apache.org/jira/browse/YUNIKORN-1873
Web: Use standard build conventions in Makefile
