Just for your information: Thanks to that change - i will soon be adding ARM support for Apache Airflow - including building and publishing the images and running our tests (using self-hosted runners). As soon as I get it I will be able to share the code/experiences with you.
J On Tue, Feb 8, 2022 at 2:50 PM Ismaël Mejía <[email protected]> wrote: > For awareness with the just released Beam 2.36.0 Beam works out of the > box to develop on a Mac M1. > > I tried Java and Python pipelines with success running locally on both > Flink/Spark runner. > I found one issue using zstd and created [1] that was merged today, > with this the sdks:core tests and Spark runner tests fully pass. > > I would see 2.36.0 is the first good enough release for someone > working on a Mac M1 or ARM64 processor. > > There are still some missing steps to have full ARM64 [apart of testing it > :)] > > 1. In theory we could run docker x86 images on ARM but those would be > emulated so way slower so it is probably better to support 'native' > CPUs) via multiarchitecture docker images [2]. > BEAM-11704 Support Beam docker images on ARM64 > > I could create the runners images from master, for the SDK containers > there are some issues with hardcoded paths [2] and virtualenv that > probably will be solved once we move to venv, and we will need to > upgrade our release process to include multiarch images (for user > friendliness). > > Also golang only supports officially ARM64 starting with version > 1.18.0 so we need to move up to that version. > > Anyway Beam is in a waaay better shape for ARM64 now than 1y ago when > I created the initial JIRAs. > > Ismaël > > [1] https://github.com/apache/beam/pull/16755 > [2] https://issues.apache.org/jira/browse/BEAM-11704 > [3] > https://github.com/apache/beam/blob/d1b8e569fd651975f08823a3db49dbee56d491b5/sdks/python/container/Dockerfile#L79 > > > > > Could not find protoc-3.14.0-osx-aarch_64.exe > (com.google.protobuf:protoc:3.14.0). > Searched in the following locations: > > https://jcenter.bintray.com/com/google/protobuf/protoc/3.14.0/protoc-3.14.0-osx-aarch_64.exe > > > > > > On Wed, Jan 12, 2022 at 9:53 PM Luke Cwik <[email protected]> wrote: > > > > The docker container running in an x86 based cloud machine should work > pretty well. This is what Apache Beam's Jenkins setup effectively does. > > > > No experience with developing on an ARM based CPU. > > > > On Wed, Jan 12, 2022 at 9:28 AM Jarek Potiuk <[email protected]> wrote: > >> > >> Comment from the side - If you use Docker - experience from Airflow - > >> until we will get ARM images, docker experience is next to unusable > >> (docker filesystem slowness + emulation). > >> > >> J. > >> > >> On Wed, Jan 12, 2022 at 6:21 PM Daniel Collins <[email protected]> > wrote: > >> > > >> > I regularly develop on a non-m1 mac using intellij, which mostly > works out of the box. Are you running into any particular issues building > or just looking for advice? > >> > > >> > -Daniel > >> > > >> > On Wed, Jan 12, 2022 at 12:16 PM Matt Rudary < > [email protected]> wrote: > >> >> > >> >> Does anyone do Beam development on an M1 Mac? Any tips to getting > things up and running? > >> >> > >> >> > >> >> > >> >> Alternatively, does anyone have a good “workstation in the cloud” > setup? > >> >> > >> >> > >> >> > >> >> Thanks > >> >> > >> >> Matt >
