Cool. Are DoFn (et al) references compatible across cross-compiled binaries?
On Tue, Jan 26, 2021 at 11:23 AM Robert Burke <rob...@frantil.com> wrote: > Go cross compilation is as simple as setting the right flag env variables > [1], but can be as complicated as requiring a cross compiling GCC instance > installed if CGO[2] is necessary. I think we're probably clear on just > needing the flag though for the various Boot executables. > > For go pipelines we'd need to update the shared runner code to support > selecting the cross compiled worker binary environment. I believe it's hard > set to amd64 linux at present, but that's a separate issue. > > [1] https://golangcookbook.com/chapters/running/cross-compiling/ > [2] https://golang.org/cmd/cgo/ > > On Tue, Jan 26, 2021, 10:25 AM Robert Bradshaw <rober...@google.com> > wrote: > >> +1 >> >> I don't think it would be that hard to build and release arm-based docker >> images. (Perhaps just a matter of changing the docker file to depend on a >> different base, and doing some cross-compile. That would suss out whether >> we're inadvertently taking on any incompatible dependencies.) >> >> Theoretically, if one does that and manually specifies the container, it >> could just work for Python (assuming no wheel files are specified as manual >> dependencies). For Java, if one builds/deploys an uberjar (on a different >> architecture), there may be issues in any transitive dependency that has >> JNI code (us or users). I'd imagine this issue is common to and being >> explored by many of the other Java big data systems in use; it'd be >> interesting to know what solutions are out there. >> >> For go, the executable is uploaded directly into the container. We'd >> probably have to do something fancier like cross-compiling the executable >> (and making sure the UserFn references, which I think are just pointers >> into the binary, still work if the launcher is one architecture and the >> workers another). >> >> Definitely worth exploring. >> >> >> >> >> On Tue, Jan 26, 2021 at 10:09 AM Ismaël Mejía <ieme...@gmail.com> wrote: >> >>> I stumbled today on this user request: >>> BEAM-10982 Wheel support for linux aarch64 >>> >>> It made me wonder if with the advent of ARM64 processors not only in >>> the client but server side (Graviton and others) if it is worth that >>> we start to think about having support for this architecture on the >>> python installers and in the docker images. It seems that for the >>> latter it should not be that difficult given that our parent images >>> are already multi-arch. >>> >>> Are there some possible issues or binary/platform specific >>> dependencies that impede us from doing this? >>> >>
