Jenkins build is back to normal : beam_Release_NightlySnapshot #627

2017-12-20 Thread Apache Jenkins Server
See

Re: [DISCUSS] Guidelines for merging new runners and SDKs into master

2017-12-20 Thread Robert Bradshaw
On Wed, Dec 20, 2017 at 6:45 PM, Henning Rohde wrote: >> > (3) Similarly to new runners, new SDKs should handle at least a useful >> > subset of the model, but not necessarily the whole model (at the time of >> > merge). A global-window-batch-only SDK targeting the portability

Re: remove avro from core?

2017-12-20 Thread Lukasz Cwik
I believe that the progress towards creating a Row type within Apache Beam will subsume the responsibilities that Avro provides today. Once the Row type exists, I agree with Romain that Avro should move out to be an extension but would wait for the next Apache Beam major version release before

Re: FlinkRunner restore from save point, CoGroupByKey holds onto state

2017-12-20 Thread Lukasz Cwik
+dev@beam.apache.org The location of the GroupByKey/CoGroupByKey is where all windowing information is being buffered before being fired by a trigger. The windowing strategy is: Window> window = Window.

Re: [DISCUSS] Guidelines for merging new runners and SDKs into master

2017-12-20 Thread Robert Bradshaw
On Tue, Dec 19, 2017 at 4:18 PM, Henning Rohde wrote: > Hi everyone, > > As part of the Go SDK development, I was looking at the guidelines for > merging new runners and SDKs into master [1] and I think they would benefit > from being updated to reflect the emerging

Re: "Maven JVM terminated unexpectedly with exit code 137"

2017-12-20 Thread Valentyn Tymofieiev
I believe committers only have access to Jenkins workspaces, so you can try to open a ticket with INFRA and ask them to ssh and run particular commands on Jenkins machine. Related: https://issues.apache.org/jira/browse/BEAM-3057 On Wed, Dec 20, 2017 at 2:50 PM, Udi Meiri

Re: [DISCUSS] Guidelines for merging new runners and SDKs into master

2017-12-20 Thread Henning Rohde
Thanks for the comments! > > (2) What are the minimal set of IO connectors a new SDK must support? > Given > > the upcoming cross-language feature in the portability framework, can we > > rely on that to meet the requirement > > without implementing any native IO connectors? > > It could be

Re: Pushing daily/test containers for python

2017-12-20 Thread Henning Rohde
+1 It would be great to be able to test this aspect of portability. For testing purposes, I think whatever container registry is convenient to use for distribution is fine. Regarding frequency, I think we should consider something closer to (a). The container images -- although usually quite