Kerry, I think the next steps would be discussing if pivoting to using docker containers would be better than the current strategy of leaning on java jars, and not filing JIRAs with no clear tasks.
As Daniel points out, the doc is the results of an investigation that may be useful for a future person with those aims, not dictating a specific goal for the community. It's very important we also report our dead ends, and not just our fully completed plans, so others can pick up where we left off of the situation changes. Now, to that end: is it worth moving to docker? I say "Not at this time." I agree that containers would be pretty egalitarian for any cross language transform implementation. However, IIRC Java was chosen because of it's broad operating system support (Linux, OSX, Windows) as well as that's presently where most transforms implementations exist at present. But while Go has a pretty simple path for actuating docker (the CLI and daemon are written in Apache Licensed go packages), i don't think that's true for Python and Java. My understanding is that Windows isn't well supported by docker outside of the WSL layer (effectively linux). Finally: Docker on Docker. It's my understanding that containers that start containers within themselves have significant issues, performance and other wise. Granted, it's been a while since I looked into that. If it's still an issue this would mean implementing a solution that doesn't work for our work on a local development environment container. Ultimately, my opinion (which is just that), is that it's currently a niche problem that has existing work arounds, and that the effort of developing the solution and maintaining the infrastructure isn't currently worth the time necessary. As such, we shouldn't file jiras and start doing this work at present, since the trade off isn't yet compelling. So, as much as this gopher hates to say it, we're best off sticking with Java for now. Maybe once we have all of Java, Python, *and* Go expansion services with compelling transform implementations. Doing so earlier is premature. Robert Burke Beam Go Busybody On Tue, Feb 22, 2022, 6:12 AM Kerry Donny-Clark <[email protected]> wrote: > This addresses a real barrier to development, thanks Daniel! > Can you link to Jira tickets for the work? This should be achievable in > several steps. > Kerry > > On Fri, Feb 18, 2022, 5:29 PM Daniel Oliveira <[email protected]> > wrote: > >> Hey everyone, >> >> This isn't a big deal or a full design doc, but I recently looked into >> what it would take to automatically start up expansion services for xlang >> transforms in Beam development environments (mostly focused on the Go SDK) >> and I put my findings into this one-pager >> <https://docs.google.com/document/d/16Yj3oZYAkw7Xc5xDQ88bdTt3BE94vY6833cAtrjbnRg/edit?usp=sharing> >> (I >> also added it to the cwiki page for design docs >> <https://cwiki.apache.org/confluence/display/BEAM/Design+Documents>). >> >> This is a pretty niche topic so I don't expect that it's useful to people >> immediately, I'm mostly sending this email so this can be easily searchable >> if this topic gets revisited in the future. But hey, if you're particularly >> invested in cross-language pipelines and you're a Beam dev feel free to >> take a look. >> >> Thanks, >> Daniel Oliveira >> >
