Re: @TearDown guarantees

Yes 1M. Lets try to explain you simplifying the overall execution. Each instance - one fn so likely in a thread of a worker - has its lifecycle. Caricaturally: "new" and garbage collection. In practise, new is often an unsafe allocate (deserialization) but it doesnt matter here. What i want is

Re: @TearDown guarantees

The kind of whole-transform lifecycle you're mentioning can be accomplished using the Wait transform as I suggested in the thread above, and I believe it should become the canonical way to do that. (Would like to reiterate one more time, as the main author of most design documents related to SDF

Re: Euphoria Java 8 DSL - proposal

Hi Davor, We still have some discussion/paperwork on Euphoria side (SGA, ...). So, it's on track but it takes a little more time than expected. Regards JB On 02/19/2018 05:40 AM, Davor Bonaci wrote: > I may have missed things, but any update on the progress of this donation? > > On Tue, Jan

Re: @TearDown guarantees

Are you sure that focusing on the cleanup of specific DoFn's is appropriate? Many cases where cleanup is necessary, it is around an entire composite PTransform. I think there have been discussions/proposals around a more methodical "cleanup" option, but those haven't been implemented, to the best

Re: @TearDown guarantees

I kind of agree except transforms lack a lifecycle too. My understanding is that sdf could be a way to unify it and clean the api. Otherwise how to normalize - single api - lifecycle of transforms? Le 18 févr. 2018 21:32, "Ben Chambers" a écrit : > Are you sure that

Re: Euphoria Java 8 DSL - proposal

I may have missed things, but any update on the progress of this donation? On Tue, Jan 2, 2018 at 10:52 PM, Jean-Baptiste Onofré wrote: > Great ! > > Thanks ! > Regards > JB > > On 01/03/2018 07:29 AM, David Morávek wrote: > >> Hello JB, >> >> Perfect! I'm already on the Beam

Re: @TearDown guarantees

On Sun, Feb 18, 2018 at 10:50 AM, Romain Manni-Bucau wrote: > > > Le 18 févr. 2018 19:28, "Ben Chambers" a écrit : > > It feels like his thread may be a bit off-track. Rather than focusing on > the semantics of the existing methods -- which have been

Re: @TearDown guarantees

On Sun, Feb 18, 2018 at 11:07 AM, Reuven Lax wrote: > > > On Sun, Feb 18, 2018 at 10:50 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> wrote: > >> >> >> Le 18 févr. 2018 19:28, "Ben Chambers" a écrit : >> >> It feels like his thread may be a bit

Re: @TearDown guarantees

On Sun, Feb 18, 2018 at 10:25 AM Romain Manni-Bucau wrote: > 2018-02-18 19:19 GMT+01:00 Eugene Kirpichov : > >> FinishBundle has a stronger guarantee: if the pipeline succeeded, then it >> has been called for every succeeded bundle, and succeeded

Re: @TearDown guarantees

Le 18 févr. 2018 19:28, "Ben Chambers" a écrit : It feels like his thread may be a bit off-track. Rather than focusing on the semantics of the existing methods -- which have been noted to be meet many existing use cases -- it would be helpful to focus on more on the reason

Re: @TearDown guarantees

Le 18 févr. 2018 00:23, "Kenneth Knowles" a écrit : On Sat, Feb 17, 2018 at 3:09 PM, Romain Manni-Bucau wrote: > > If you give an example of a high-level need (e.g. "I'm trying to write an > IO for system $x and it requires the following initialization

Re: @TearDown guarantees

2018-02-18 18:36 GMT+01:00 Eugene Kirpichov : > "Machine state" is overly low-level because many of the possible reasons > can happen on a perfectly fine machine. > If you'd like to rephrase it to "it will be called except in various > situations where it's logically

Re: @TearDown guarantees

FinishBundle has a stronger guarantee: if the pipeline succeeded, then it has been called for every succeeded bundle, and succeeded bundles together cover the entire input PCollection. Of course, it may not have been called for failed bundles. To anticipate a possible objection "why not also keep

Re: @TearDown guarantees

2018-02-18 19:19 GMT+01:00 Eugene Kirpichov : > FinishBundle has a stronger guarantee: if the pipeline succeeded, then it > has been called for every succeeded bundle, and succeeded bundles together > cover the entire input PCollection. Of course, it may not have been called

Re: @TearDown guarantees

It feels like his thread may be a bit off-track. Rather than focusing on the semantics of the existing methods -- which have been noted to be meet many existing use cases -- it would be helpful to focus on more on the reason you are looking for something with different semantics. Some

Re: @TearDown guarantees

Le 18 févr. 2018 15:39, "Jean-Baptiste Onofré" a écrit : Hi, I think, as you said, it depends of the protocol and the IO. For instance, in first version of JdbcIO, I created the connections in @Setup and released in @Teardown. But, in case of streaming system, it's not so

Re: @TearDown guarantees

2018-02-18 18:00 GMT+01:00 Eugene Kirpichov : > > > On Sun, Feb 18, 2018, 2:06 AM Romain Manni-Bucau > wrote: > >> >> >> Le 18 févr. 2018 00:23, "Kenneth Knowles" a écrit : >> >> On Sat, Feb 17, 2018 at 3:09 PM, Romain Manni-Bucau <

Re: @TearDown guarantees

"Machine state" is overly low-level because many of the possible reasons can happen on a perfectly fine machine. If you'd like to rephrase it to "it will be called except in various situations where it's logically impossible or impractical to guarantee that it's called", that's fine. Or you can

Re: @TearDown guarantees

My bad, I thought you talked about guarantee in the Runner API. If it's semantic point in the SDK (enforcement instead of best effort), and then if the runner doesn't respect that, it's a limitation/bug in the runner, I would agree with that. Regards JB On 18/02/2018 16:58, Romain

Re: @TearDown guarantees

Yes exactly JB, I just want to ensure the sdk/core API is clear and well defined and that any not respect of that falls into a runner bug. What I don't want is that a buggy impl leaks in the SDK/core definition. Romain Manni-Bucau @rmannibucau | Blog

Re: @TearDown guarantees

It will not be called if it's impossible to call it: in the example situation you have (intergalactic crash), and in a number of more common cases: eg in case the worker container has crashed (eg user code in a different thread called a C library over JNI and it segfaulted), JVM bug, crash due to

Re: @TearDown guarantees

Agree Eugene except that "best effort" means that. It is also often used to say "at will" and this is what triggered this thread. I'm fine using "except if the machine state prevents it" but "best effort" is too open and can be very badly and wrongly perceived by users (like I did). Romain

Re: @TearDown guarantees

Hi, I think, as you said, it depends of the protocol and the IO. For instance, in first version of JdbcIO, I created the connections in @Setup and released in @Teardown. But, in case of streaming system, it's not so good (especially for pooling) as the connection stays open for a very long