Le 18 févr. 2018 15:39, "Jean-Baptiste Onofré" <j...@nanthrax.net> a écrit :
Hi, I think, as you said, it depends of the protocol and the IO. For instance, in first version of JdbcIO, I created the connections in @Setup and released in @Teardown. But, in case of streaming system, it's not so good (especially for pooling) as the connection stays open for a very long time. Hmm can be discussed in practise (both pooling and connection holding for jdbc in beam context) but lets assume it. So, I updated to deal with connection in @StartBundle and release in @FinishBundle. Which leads to an unpredictable bundle size and therefore very very bad perfs on write size - read size is faked but in mem buffer i guess which breaks the bundle definition but let s ignore it too for now. So, I think it depends of the kind of connections: the kind of connection actually holding resources should be manage in bundle (at least for now), the other kind of connection (just wrapping configuration but not holding resources like Apache HTTP Component Client for instance) could be dealt in DoFn lifecycle. Once again, I would be ok with bundles for now - but it doesnt solve the real issue - if bundles are up to the user. Since it is not, it doesnt help and can just degrade the overall behavior in both batch and streaming. I fully understand beam doesnt handle properly that today. What does block to do it? Nothing technical so why not doing it? Technically: 1. Teardown can be guaranteed 2. Bundle size can be highly influenced / configured by user Both are needed to be able to propose a strong api compared to competitors and aims to not only have disavantages going portable for users. Let just do it, no? Regards JB On 02/18/2018 11:05 AM, Romain Manni-Bucau wrote: > > > Le 18 févr. 2018 00:23, "Kenneth Knowles" <k...@google.com > <mailto:k...@google.com>> a écrit : > > On Sat, Feb 17, 2018 at 3:09 PM, Romain Manni-Bucau < rmannibu...@gmail.com > <mailto:rmannibu...@gmail.com>> wrote: > > If you give an example of a high-level need (e.g. "I'm trying to > write an IO for system $x and it requires the following > initialization and the following cleanup logic and the following > processing in between") I'll be better able to help you. > > > Take a simple example of a transform requiring a connection. Using > bundles is a perf killer since size is not controlled. Using teardown > doesnt allow you to release the connection since it is a best effort > thing. Not releasing the connection makes you pay a lot - aws ;) - or > prevents you to launch other processings - concurrent limit. > > > For this example @Teardown is an exact fit. If things die so badly that > @Teardown is not called then nothing else can be called to close the > connection either. What AWS service are you thinking of that stays open for > a long time when everything at the other end has died? > > > You assume connections are kind of stateless but some (proprietary) protocols > requires some closing exchanges which are not only "im leaving". > > For aws i was thinking about starting some services - machines - on the fly in a > pipeline startup and closing them at the end. If teardown is not called you leak > machines and money. You can say it can be done another way...as the full > pipeline ;). > > I dont want to be picky but if beam cant handle its components lifecycle it can > be used at scale for generic pipelines and if bound to some particular IO. > > What does prevent to enforce teardown - ignoring the interstellar crash case > which cant be handled by any human system? Nothing technically. Why do you push > to not handle it? Is it due to some legacy code on dataflow or something else? > > Also what does it mean for the users? Direct runner does it so if a user udes > the RI in test, he will get a different behavior in prod? Also dont forget the > user doesnt know what the IOs he composes use so this is so impacting for the > whole product than he must be handled IMHO. > > I understand the portability culture is new in big data world but it is not a > reason to ignore what people did for years and do it wrong before doing right ;). > > My proposal is to list what can prevent to guarantee - in the normal IT > conditions - the execution of teardown. Then we see if we can handle it and only > if there is a technical reason we cant we make it experimental/unsupported in > the api. I know spark and flink can, any unknown blocker for other runners? > > Technical note: even a kill should go through java shutdown hooks otherwise your > environment (beam enclosing software) is fully unhandled and your overall system > is uncontrolled. Only case where it is not true is when the software is always > owned by a vendor and never installed on customer environment. In this case it > belongd to the vendor to handle beam API and not to beam to adjust its API for a > vendor - otherwise all unsupported features by one runner should be made > optional right? > > All state is not about network, even in distributed systems so this is key to > have an explicit and defined lifecycle. > > > Kenn > > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
