Well said Robert! I think that nuance: auto-searching for the dev jars, vs just printing out the command, was definitely lost on me! That seems like a pretty happy medium TBH. As long as one isn't copy pasting things from commands and directories, I'll be a happy camper.
On Tue, Feb 22, 2022, 4:29 PM Robert Bradshaw <[email protected]> wrote: > +1 to not making sense to move to docker (at this time). Note that > this isn't exclusively for expansion services--in Python we use the > automated service startup for launching the Flink and Spark runners as > well (and here docker in docker would be even more of an issue). > > I would like to call out that deciding to use docker vs. the jre > directly is somewhat orthogonal to making auto service startup work > well in dev environments (which is a worthwhile goal). Python solves > this by noting that when you're in a dev environment, rather than > trying to download the image from maven, it looks for the required > jars in the gradle build directories, and if not present, prompts the > user how to build them. No reason something similar couldn't be done > for Go. (Even prompting the user with a one-line command to build and > start up the expansion service when in a dev environment would be a > step forward in usability.) > > On Tue, Feb 22, 2022 at 9:16 AM Jack McCluskey <[email protected]> > wrote: > > > > To echo/confirm this point, running Docker containers within WSL2 is > inconsistent at best. Getting Beam Go working in that context was > unsuccessful for me beyond using the Go direct runner, which is less than > ideal for obvious reasons. Leaning on the JRE for now seems like the best > solution. > > > > On Tue, Feb 22, 2022 at 12:12 PM Robert Burke <[email protected]> > wrote: > >> > >> Kerry, I think the next steps would be discussing if pivoting to using > docker containers would be better than the current strategy of leaning on > java jars, and not filing JIRAs with no clear tasks. > >> > >> As Daniel points out, the doc is the results of an investigation that > may be useful for a future person with those aims, not dictating a specific > goal for the community. > >> > >> It's very important we also report our dead ends, and not just our > fully completed plans, so others can pick up where we left off of the > situation changes. > >> > >> Now, to that end: is it worth moving to docker? I say "Not at this > time." > >> > >> I agree that containers would be pretty egalitarian for any cross > language transform implementation. However, IIRC Java was chosen because of > it's broad operating system support (Linux, OSX, Windows) as well as that's > presently where most transforms implementations exist at present. > >> > >> But while Go has a pretty simple path for actuating docker (the CLI and > daemon are written in Apache Licensed go packages), i don't think that's > true for Python and Java. > >> > >> My understanding is that Windows isn't well supported by docker outside > of the WSL layer (effectively linux). > >> > >> Finally: Docker on Docker. It's my understanding that containers that > start containers within themselves have significant issues, performance and > other wise. Granted, it's been a while since I looked into that. If it's > still an issue this would mean implementing a solution that doesn't work > for our work on a local development environment container. > >> > >> Ultimately, my opinion (which is just that), is that it's currently a > niche problem that has existing work arounds, and that the effort of > developing the solution and maintaining the infrastructure isn't currently > worth the time necessary. As such, we shouldn't file jiras and start doing > this work at present, since the trade off isn't yet compelling. > >> > >> So, as much as this gopher hates to say it, we're best off sticking > with Java for now. Maybe once we have all of Java, Python, *and* Go > expansion services with compelling transform implementations. Doing so > earlier is premature. > >> > >> Robert Burke > >> Beam Go Busybody > >> > >> > >> On Tue, Feb 22, 2022, 6:12 AM Kerry Donny-Clark <[email protected]> > wrote: > >>> > >>> This addresses a real barrier to development, thanks Daniel! > >>> Can you link to Jira tickets for the work? This should be achievable > in several steps. > >>> Kerry > >>> > >>> On Fri, Feb 18, 2022, 5:29 PM Daniel Oliveira <[email protected]> > wrote: > >>>> > >>>> Hey everyone, > >>>> > >>>> This isn't a big deal or a full design doc, but I recently looked > into what it would take to automatically start up expansion services for > xlang transforms in Beam development environments (mostly focused on the Go > SDK) and I put my findings into this one-pager (I also added it to the > cwiki page for design docs). > >>>> > >>>> This is a pretty niche topic so I don't expect that it's useful to > people immediately, I'm mostly sending this email so this can be easily > searchable if this topic gets revisited in the future. But hey, if you're > particularly invested in cross-language pipelines and you're a Beam dev > feel free to take a look. > >>>> > >>>> Thanks, > >>>> Daniel Oliveira >
