> I was imagining a long running expansion service would just serve up a fixed number of transforms, and changing this set would be done via a new release of the service.
This is how I was thinking about it as well, I don't think we'd have any requirement to dynamically add/remove transforms at runtime. On Wed, Oct 6, 2021 at 1:27 PM Robert Bradshaw <[email protected]> wrote: > On Wed, Oct 6, 2021 at 10:24 AM Chamikara Jayalath <[email protected]> > wrote: > > > > Regarding the memory leak, I'm not sure if a function registration is > needed here since it seems like that function is doing something pretty > generic. +Heejong Lee should know more. > > +1 > > > Regarding a long running expansion service, in addition to what others > mentioned, I think we also need protocols for dynamically > registering/unregistering transforms and a protocol for discovering > transforms already registered in an expansion service. > > I was imagining a long running expansion service would just serve up a > fixed number of transforms, and changing this set would be done via a > new release of the service. > > Having a dynamic set of transforms served (e.g. expansion services as > a service) is another level that could be interesting to explore > (though more complicated, e.g. now you're hosting third party jars and > running arbitrary code). > > > On Wed, Oct 6, 2021 at 10:02 AM Steve Niemitz <[email protected]> > wrote: > >> > >> I noticed that Environments.getDeferredArtifacts will add a function to > DefaultArtifactResolver.INSTANCE which is never removed, so those will > accumulate forever. That's the only one I noticed so far from code review, > but it's possible there's more. > >> > >> On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <[email protected]> > wrote: > >>> > >>> IIRC, the folks at Lyft offered this as a service (internally, not > >>> publicly). We've toyed with the idea of offering it more generally. > >>> You're right that there would be security implications (though some > >>> generic sandboxing could go a long way). What were the memory leaks > >>> that you were noticing? > >>> > >>> On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <[email protected]> > wrote: > >>> > > >>> > cool, thanks for the info. I might be the first to try then :) > >>> > > >>> > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <[email protected]> wrote: > >>> >> > >>> >> I believe that was one of the ideas behind an expansion service but > to my knowledge I don't know of anyone who has a long running expansion > service. I was thinking that the Apache Beam community at some point could > host one so that users don't need to spin up Java/Python/Go expansion > services to run a subset of XLang pipelines. > >>> >> > >>> >> The XLang effort has been focusing about getting more transforms > working and solving ease of use issues for users so I don't believe what > you describe has been done. > >>> >> > >>> >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <[email protected]> > wrote: > >>> >>> > >>> >>> /cc @Chamikara Jayalath @Robert Bradshaw > >>> >>> > >>> >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <[email protected]> > wrote: > >>> >>>> > >>> >>>> Has anyone ever tried hosting a long-running expansion service as > a real "service", the intent being that users don't need to run it locally, > and can instead connect to the shared one when expanding pipelines? > >>> >>>> > >>> >>>> Looking around the code I already see a few assumptions that it > will only live for a short while (memory leaks, etc), as well as some > pretty large security concerns (eg, the artifact retrieval service can be > used to download any file on the host) curious if anyone has put any work > into this already? >
