potiuk commented on issue #15635:
URL: https://github.com/apache/airflow/issues/15635#issuecomment-1000138886
Thanks @andormarkus - I understand your CI provides ARM (and I am pretty
sure those are weeks rather than months that GitHub Actions will provide one -
ARM is becoming a "highly reguested" thing. So nice try ;) but GitHub Actions
are not a problem here. This is by far the least (and easy to bypass) of our
blockers
> As I know this project uses Github actions to build images. Github actions
only support amd64 runners does not offer arm64 runner like other CI provider.
You can build arm64 images on amd64 platform with the help on Docker BuildX
however it will have a 3-4X time penalty compared to a native arm64 runner.
Actually we build our images for CI tests with our self-hosted runners of
Amazon. So we could set the pipelines on GitHub actions using those. We will
just need to build a bit of infrastructure for that.
The actual blocker here is native support for ARM for a number of libraries
and tools we are using. Airflow has 550+ pyhon dependencies, where some of
those have native c/C++ libraries and need precompiled versions. We have
complex cross-dependencies which simply make it complex to get the right set
that will work.
For example one of the things that "blocks us" to hav full ARM support is
lack of ARM images for MySQL https://bugs.mysql.com/bug.php?id=103462.
However, while we won't be able to get "full" support" for ARM yet for all
the dependencies I am pretty confident we can get "selective" support. Python
3.9 + Postgres + subset of providers should actually work.
I am going to take a stab on it after Xmas, however. I even have a new
MacBook PRO to play a bit with that. This will require a few upgrades to our
build infrastructure:
* adapt our Dockerfiles to support multi-platform builds
* changing build infrastructure to use BUILDX and produce multiplatform
images (including proper caching)
* adding posibility to select test scope depending on the Architecture
* adding possibility to select and build constraints that will depend on the
architecture (I am pretty sure we will have different set of constraints for
our dependencies for different architectures)
* converting our self-hosted AMIs to be ARM-based
All that is needed to make it works. The final "result" that you see as an
image seems like "simple" thing to do. But in order do that a set of
prerequisites need to be fulfilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
