Jenkins build is unstable: beam_Release_NightlySnapshot #331

See

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

Raghu, Amit -- +1 to your expertise :) On Thu, Feb 16, 2017 at 3:39 PM, Amit Sela wrote: > I agree with Dan on everything regarding HdfsFileSystem - it's super > convenient for users to use TextIO with HdfsFileSystem rather then > replacing the IO and also specifying the InputFormat type. > > I

Re: Pipeline Surgery and an interception-free future

Awesome! First thing I'm gonna do: 1. traverse the pipeline to determine if streaming. 2. If streaming, replace Read.Bounded with an adapted Read.Unbounded. Current implementation forces translating bounded reads by the unbounded translator and it feels awkward, this makes it right again.

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

I agree with Dan on everything regarding HdfsFileSystem - it's super convenient for users to use TextIO with HdfsFileSystem rather then replacing the IO and also specifying the InputFormat type. I disagree on "HadoopIO" - I think that people who work with Hadoop would find this name intuitive, and

Re: Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2315

This was a PR build. Ignore. On Thu, Feb 16, 2017 at 2:39 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See RunnableOnService_Dataflow/2315/> > >

Re: We've hit 2000 PRs!

It's not just 1000 more PRs, it's also new contributors. We're growing and that's awesome! Congrats to everyone. On Thu, Feb 16, 2017 at 9:43 PM Thomas Groh wrote: > Impressive work everyone. Very cool. > > On Thu, Feb 16, 2017 at 8:05 AM, Dan Halperin > > wrote: > > > Checking my previous clai

Jenkins build is back to normal : beam_SeedJob #116

See

Re: Build failed in Jenkins: beam_SeedJob #115

Another manual build; fixing broken tests in Jenkins now. On Thu, Feb 16, 2017 at 12:48 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > > -- > Started by user davor > [EnvInject] - Load

Build failed in Jenkins: beam_SeedJob #115

See -- Started by user davor [EnvInject] - Loading node environment variables. Building remotely on beam1 (beam) in workspace > git rev-parse --is-inside-wor

Re: Build failed in Jenkins: beam_SeedJob #113

Manual build, safe to ignore. On Thu, Feb 16, 2017 at 12:10 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See > > -- > GitHub pull request #2020 of commit 509b4d61b5b7505d71b5f7d19a541e217d

Jenkins build is back to normal : beam_SeedJob #114

See

Build failed in Jenkins: beam_SeedJob #113

See -- GitHub pull request #2020 of commit 509b4d61b5b7505d71b5f7d19a541e217d04d407, no merge conflicts. Setting status of 509b4d61b5b7505d71b5f7d19a541e217d04d407 to PENDING with url https://builds.apache.

Re: Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2313

This was a PR build. Ignore. On Thu, Feb 16, 2017 at 11:33 AM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See RunnableOnService_Dataflow/2313/> > >

Re: We've hit 2000 PRs!

Impressive work everyone. Very cool. On Thu, Feb 16, 2017 at 8:05 AM, Dan Halperin wrote: > Checking my previous claims: > > PR #1: Feb 26, 2016 > PR #1000: Sep 24, 2016 (211 days later) > PR #2000: Feb 13, 2016 (142 days later) Yep -- much quicker! > > I'm excited to see this community growing

Re: We've hit 2000 PRs!

Awesome news! Congratulations to the community! On Feb 16, 2017 11:20 AM, "Jason Kuster" wrote: > All trends are linear -- PR #3000 in 70 days?? :P > > Huge milestone; congrats to the community. :) > > On Thu, Feb 16, 2017 at 11:16 AM, Raghu Angadi > > wrote: > > > Amazing to see this Dan. > >

Re: We've hit 2000 PRs!

All trends are linear -- PR #3000 in 70 days?? :P Huge milestone; congrats to the community. :) On Thu, Feb 16, 2017 at 11:16 AM, Raghu Angadi wrote: > Amazing to see this Dan. > > Congrats and thanks to the committers. You have been so welcoming and > promptly reviewed so may patches. > > I kn

Re: We've hit 2000 PRs!

Amazing to see this Dan. Congrats and thanks to the committers. You have been so welcoming and promptly reviewed so may patches. I know from my own personal experience how much dedication and time it demands. thank you! Raghu. On Thu, Feb 16, 2017 at 8:05 AM, Dan Halperin wrote: > Checking my

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

FileInputFormat is extremely widely used, pretty much all the file based input formats extend it. All of them call into to list the input files, split (with some tweaks on top of that). The special API ( *FileInputFormat.setMinInputSplitSize(job, desiredBundleSizeBytes)* ) is how the split size is

Re: Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Apex #516

Looks like this was a flake. Filed https://issues.apache.org/jira/browse/BEAM-1501 On Wed, Feb 15, 2017 at 10:37 PM, Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See RunnableOnService_Apex/516/> > >

Re: We've hit 2000 PRs!

An example what community development means :-) Thanks for sharing, Dan! On Feb 16, 2017 5:05 PM, "Dan Halperin" wrote: > Checking my previous claims: > > PR #1: Feb 26, 2016 > PR #1000: Sep 24, 2016 (211 days later) > PR #2000: Feb 13, 2016 (142 days later) Yep -- much quicker! > > I'm excited

Re: We've hit 2000 PRs!

That is really impressive :) Congratulations everyone. Ahmet On Thu, Feb 16, 2017 at 8:57 AM, Jean-Baptiste Onofré wrote: > Thanks for sharing Dan, impressive number indeed ! > > Congrats to the community ! > > Regards > JB > > > On 02/16/2017 05:05 PM, Dan Halperin wrote: > >> Checking my prev

Re: We've hit 2000 PRs!

Thanks for sharing Dan, impressive number indeed ! Congrats to the community ! Regards JB On 02/16/2017 05:05 PM, Dan Halperin wrote: Checking my previous claims: PR #1: Feb 26, 2016 PR #1000: Sep 24, 2016 (211 days later) PR #2000: Feb 13, 2016 (142 days later) Yep -- much quicker! I'm exci

Re: Merge HadoopInputFormatIO and HDFSIO in a single module

Chiming in a bit late, but here's my 2 cents. HdfsFileSystem vs Hadoop*InputFormatIO is a red herring: * HdfsFileSystem is for file-format-specific, Beam-native, parsers of files. It will make TextIO, AvroIO, etc., work for files that happen to be located at hdfs:// URIs. * This is complementa

We've hit 2000 PRs!

Checking my previous claims: PR #1: Feb 26, 2016 PR #1000: Sep 24, 2016 (211 days later) PR #2000: Feb 13, 2016 (142 days later) Yep -- much quicker! I'm excited to see this community growing and innovating as we march towards the true Beam Technical Vision, a first major release, and really empo

Re: Pipeline Surgery and an interception-free future

I might just try and do that. ;-) On Thu, 16 Feb 2017 at 03:55 Thomas Groh wrote: > As of Github PR #1998 (https://github.com/apache/beam/pull/1998), the new > Pipeline Surgery API is ready and available. There are a couple of > refinements coming in PR #2006, but in general pipelines can now, p

Re: Better developer instructions for using Maven?

JB, Maybe I was not clear, when I talked about the tests I was thinking more about execute them in parallel in the same machine, this is not the case today for some test suites, and for these the tests need to be refined to support this, and configured via the pom to execute the tests in parallel p