Re: Limit the number of DoFn instances per worker?

2017-10-17 Thread Lukasz Cwik
The `numberOfWorkerHarnessThreads` is worker wide and not per DoFn. Setting this value to constrain how many threads are executing will impact all parts of your pipeline. One idea is to use a Semaphore as a static object within your DoFn with a fixed number of allowed actors that can enter and

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Yihua Fang
+1 On Tue, Oct 17, 2017 at 9:38 AM Steve Anderson wrote: > +1 > > Sent from my iPhone > > On Oct 17, 2017, at 09:31, Aleksandr wrote: > > +1 > > 17. okt 2017 7:17 PM kirjutas kuupäeval "Ismaël Mejía" >: > > We have discussed

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Aleksandr
+1 17. okt 2017 7:17 PM kirjutas kuupäeval "Ismaël Mejía" : We have discussed recently in the developer mailing list about the idea of removing support for Java 7 on Beam. There are multiple reasons for this: - Java 7 has not received public updates for almost two years and

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Rune Fevang
+1 On Tue, Oct 17, 2017 at 6:52 PM, Henning Rohde wrote: > +1 > > On Tue, Oct 17, 2017 at 9:47 AM, Jean-Baptiste Onofré > wrote: > >> However, it's good to target this for Beam 3.0.0 as it can have an impact >> especially for runners. >> >> Regards >> JB

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Vilhelm von Ehrenheim
+1 On Tue, Oct 17, 2017 at 8:22 PM, Thomas Groh wrote: > I'm pretty strongly in favor of phasing out Java7 support, especially > given that it was EoL'd more than two years ago. However, I'm not sure how > this interacts with the repository's backwards-compatibility guarantees

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Wayne Collins
+1 (snip): So, please vote: +1 Yes, go ahead and move Beam support to Java 8.   0 Do whatever you want. I don’t have a preference. -1 Please keep Java 7 compatibility (if possible add your argument to keep supporting for Java 7).

How to window by quantity of data?

2017-10-17 Thread Jacob Marble
My first streaming pipeline is pretty simple, it just pipes a queue into files: - read JSON objects from PubsubIO - event time = processing time - 5 minute windows ( - write n files to GCS, (TextIO.withNumShards() not dynamic) When the pipeline gets behind (for example, when the pipeline is

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Raghu Angadi
+1. On Tue, Oct 17, 2017 at 2:11 PM, David McNeill wrote: > The final version of Beam that supports Java 7 should be clearly stated in > the docs, so those stuck on old production infrastructure for other java > app dependencies know where to stop upgrading. > > David

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Randal Moore
+1 On Tue, Oct 17, 2017 at 5:21 PM Raghu Angadi wrote: > +1. > > On Tue, Oct 17, 2017 at 2:11 PM, David McNeill > wrote: > >> The final version of Beam that supports Java 7 should be clearly stated >> in the docs, so those stuck on old production

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Eugene Kirpichov
+1 to removing Java 7 support. In terms of release 3.0, we can handle this two ways: - Wait until enough other potentially incompatible changes accumulate, do all of them, and call it a "3.0" release, so that 3.0 will truly differ in a lot of incompatible and hopefully nice ways from 2.x. This

Re: October Apache Beam Newsletter

2017-10-17 Thread Matthias Baetens
Great overview of what happened in Beam-land. Thanks a lot! On Tue, Oct 10, 2017 at 4:17 AM, James wrote: > Cool, very informational, thanks! > > On Tue, Oct 10, 2017 at 2:39 AM Griselda Cuevas wrote: > >> Hi Apache Beam Community, >> >> Our first Apache

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread David McNeill
The final version of Beam that supports Java 7 should be clearly stated in the docs, so those stuck on old production infrastructure for other java app dependencies know where to stop upgrading. David McNeill 021 721 015 On 18 October 2017 at 05:16, Ismaël Mejía wrote: >

How does Beam set up the bundle size in streaming mode (like Pub/Sub)?

2017-10-17 Thread Derek Hao Hu
Hi, Is there any more detailed explanation on how Beam chooses the window size (bundle size) in streaming mode? It seems there is no clear answer in the [Beam Programming Guide]( https://beam.apache.org/documentation/programming-guide/) and I can't find how PubsubIO implements this windowing

Re: [VOTE] [DISCUSSION] Remove support for Java 7

2017-10-17 Thread Griselda Cuevas
+1 On 17 October 2017 at 16:36, Robert Bradshaw wrote: > +1 to removing Java 7 support, pending no major user outcry to the > contrary. > > In terms of versioning, I fall into the camp that this isn't > sufficiently incompatible to warrant a major version increase. >

[Events] Apache Beam Meetup in San Francisco on 11/1, Join us!

2017-10-17 Thread Griselda Cuevas
Hi Everyone, A new Apache Beam Meetup is happening on 11/1! Ahmet Altay & Anand Iyer from Google are speaking at the Next San Francisco Cloud Mafia gathering. You can find details here [1], join us if you're in the area :) I'll

Re: How to window by quantity of data?

2017-10-17 Thread Jacob Marble
Lukasz- That worked. I created a stateful DoFn with a stale timer, an initial timestamp state, and a counter state, along with a buffer of elements to bundle. When the counter or timer exceeds max values, outputWithTimestamp(). I'm happy to post the entire implementation somewhere, not sure