Re: Let's make Beam transforms comply with PTransform Style Guide

Hi Eugene, thanks for the update. I'm volunteer to tackle some those IOs (and make them conform with PTransform style guide). I'm pretty sure other people will jump on ;) Regards JB On 04/08/2017 12:20 AM, Eugene Kirpichov wrote: Hey all, More progress has been made and we're nearing

Re: IO ITs: Hosting Docker images

Hi Stephen, I think we should go to 1 and 4: 1. Try to use existing images providing what we need. If we don't find existing image, we can always ask and help other community to provide so. 4. If we don't find a suitable image, and waiting for this image, we can store the image in our own "IT

Re: Proposed Splittable DoFn API changes

Hi Eugene, thanks for the update and nice example. I plan to start to refactor/experiment on some IOs. Regards JB On 04/08/2017 02:44 AM, Eugene Kirpichov wrote: The changes are in. Also included is a handy change that allows one to skip implementing the NewTracker method if the restriction

Re: Combine.Global

I wasn't able to reproduce the issue you're experiencing. I've created a gist with an example that works and is similar to what you have described. Please help us make tweaks to the gist reproduce your problem: https://gist.github.com/aviemzur/ba213d98b4484492099b3cf709ddded0 On Fri, Apr 7, 2017

Re: Proposed Splittable DoFn API changes

The changes are in. Also included is a handy change that allows one to skip implementing the NewTracker method if the restriction type implements HasDefaultTracker, leaving the only two required methods be ProcessElement and GetInitialRestriction. E.g. here's what a minimal SDF example looks

[DISCUSSION] PAssert success/failure count validation for all runners

Currently, PAssert assertions may not happen and tests will pass while silently hiding issues. Up until now, several runners have implemented an assertion that the number of expected successful assertions have actually happened, and that no failed assertions have happened. (runners which check

Re: Combine.Global

Have you set the coder for your input PCollection? The one on which you perform the Combine? On Fri, Apr 7, 2017 at 4:24 PM Paul Gerver wrote: > Hello All, > > I'm trying to test out a Combine.Globally transform which takes in a small > custom class (CustomA) and outputs a

Combine.Global

Hello All, I'm trying to test out a Combine.Globally transform which takes in a small custom class (CustomA) and outputs a secondary custom class (CustomB). I have set the coder for the resulting PCollection, but Beam is arguing that a coder for a KV type is missing (see output at bottom). Since

Re: Update of Pei in Alibaba

Hello Basti, Thanks a lot for answering, I imagined that, that all the improvements of both JStorm and Heron wouldn’t translate perfectly but still a worth goal to try to have the ‘common’ storm parts isolated so they can be shared with the other runners. Really interesting, I wish you guys the