I noticed that Environments.getDeferredArtifacts will add a function to DefaultArtifactResolver.INSTANCE which is never removed, so those will accumulate forever. That's the only one I noticed so far from code review, but it's possible there's more.
On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <[email protected]> wrote: > IIRC, the folks at Lyft offered this as a service (internally, not > publicly). We've toyed with the idea of offering it more generally. > You're right that there would be security implications (though some > generic sandboxing could go a long way). What were the memory leaks > that you were noticing? > > On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <[email protected]> wrote: > > > > cool, thanks for the info. I might be the first to try then :) > > > > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <[email protected]> wrote: > >> > >> I believe that was one of the ideas behind an expansion service but to > my knowledge I don't know of anyone who has a long running expansion > service. I was thinking that the Apache Beam community at some point could > host one so that users don't need to spin up Java/Python/Go expansion > services to run a subset of XLang pipelines. > >> > >> The XLang effort has been focusing about getting more transforms > working and solving ease of use issues for users so I don't believe what > you describe has been done. > >> > >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <[email protected]> wrote: > >>> > >>> /cc @Chamikara Jayalath @Robert Bradshaw > >>> > >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <[email protected]> > wrote: > >>>> > >>>> Has anyone ever tried hosting a long-running expansion service as a > real "service", the intent being that users don't need to run it locally, > and can instead connect to the shared one when expanding pipelines? > >>>> > >>>> Looking around the code I already see a few assumptions that it will > only live for a short while (memory leaks, etc), as well as some pretty > large security concerns (eg, the artifact retrieval service can be used to > download any file on the host) curious if anyone has put any work into this > already? >
