Re: Custom shardingFn for FileIO

So you were able to use this in Flink? Did you see performance gains? On Sun, May 5, 2019 at 5:25 AM Jozef Vilcek wrote: > Sorry, it took a while. I wanted to actually use this extension for > WriteFiles in Flink and see it works and that proved too be a bit bumpy. > PR is at

Re: [Discuss] Publishing pre-release artifacts to repositories

Thank you all for feedback. I filed BEAM-7242 for myself to update the release process with sufficient tooling to add support for this. I am unlikely to be able to do it for the current release but hopefully I will work on it soon. *From: *Michael Luckey *Date: *Mon, May 6, 2019 at 4:48 PM *To:

Question about unbounded in-memory PCollection

Hi Beam Team, I am new to here and recently study the programming guide, I have a question about the in-memory data, https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection Is there a way to create unbounded PCollection from the in-memory collection? I want to test the

Re: Contributing to Beam

Thanks all! On Mon, May 6, 2019 at 2:34 PM Lukasz Cwik wrote: > Welcome. > > On Mon, May 6, 2019 at 2:23 PM Reuven Lax wrote: > >> Welcome! >> >> On Mon, May 6, 2019 at 2:15 PM Kenneth Knowles wrote: >> >>> Welcome! >>> >>> On Mon, May 6, 2019 at 9:20 AM Ahmet Altay wrote: >>> Welcome

Re: Artifact staging in cross-language pipelines

Here's the first draft: https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit?usp=sharing It's rather high-level. We may want to add more details once we have finalized the design. Feel free to make comments and edits. All of this goes back to the idea that I

Re: Artifact staging in cross-language pipelines

Looking forward to your writeup, Max. In the meantime, some comments below. From: Lukasz Cwik Date: Thu, May 2, 2019 at 6:45 PM To: dev > > > On Thu, May 2, 2019 at 7:20 AM Robert Bradshaw wrote: >> >> On Sat, Apr 27, 2019 at 1:14 AM Lukasz Cwik wrote: >> > >> > We should stick with URN +

Re: Better naming for runner specific options

I think using RunnerOptions was an idea at some point, but in Python, we ended up parsing options from the runner api without populating RunnerOptions, and RunnerOptions was eventually removed [1]. If we decide to rename options, a path forward may be to have runners recognize both old and new

Re: PardoLifeCycle: Teardown after failed call to setup

Thanks Kenn and Reuven. Based on your feedback, I amended to the PR [1] implementing the missing calls to teardown. Best, michel [1] https://github.com/apache/beam/pull/8495 On Tue, May 7, 2019 at 6:09 AM Kenneth Knowles wrote: > > > On Mon, May 6, 2019 at 2:19 PM Reuven Lax wrote: > >> >>

Fwd: Beam at Google Summer of Code 2019

Thank You! I'm really excited to work on Beam! I'd like to thank Pablo, Chamikara Jayalath and Tim Robertson for helping out with my proposal[1]. Looking forward to working with everyone and learning a great deal. Regards Tanay Tummalapalli LinkedIn |