Re: [DISCUSS] Python SDK status and next steps

Great -- congratulations to everyone who has contributed to the Python SDK! On Mon, Jan 30, 2017 at 11:10 PM, Ahmet Altay wrote: > Hi all, > > This merge is completed. Python SDK is now officially part of the master > branch! Thank you all for the support. Please open an issue, if you notice > a

Re: [DISCUSS] Python SDK status and next steps

Hi all, This merge is completed. Python SDK is now officially part of the master branch! Thank you all for the support. Please open an issue, if you notice a reference to the now obsolete python-sdk branch in the documentation. There will not be any more merges to the python-sdk branch. Going for

Re: Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2473

Thank you Dan. The build is back to green again. Ahmet On Mon, Jan 30, 2017 at 4:27 PM, Dan Halperin wrote: > Hey folks, > > It looks like the python-sdk -> master merge went bad and, unfortunately, > we have it configured to email anyone who ever contributed a commit to the > merge, which I th

Re: Let's make Beam transforms comply with PTransform Style Guide

On Mon, Jan 30, 2017 at 7:56 PM, Dan Halperin wrote: > On Mon, Jan 30, 2017 at 5:42 PM, Eugene Kirpichov < > kirpic...@google.com.invalid> wrote: > >> Hello, >> >> The PTransform Style Guide is live >> https://beam.apache.org/contribute/ptransform-style-guide/ - a natural >> next >> step is to au

Re: Let's make Beam transforms comply with PTransform Style Guide

On Mon, Jan 30, 2017 at 5:42 PM, Eugene Kirpichov < kirpic...@google.com.invalid> wrote: > Hello, > > The PTransform Style Guide is live > https://beam.apache.org/contribute/ptransform-style-guide/ - a natural > next > step is to audit Beam libraries for compliance and file JIRAs for places > that

Re: PTransform style guide PR

I don't think I'll have capacity to review every PR that brings particular Beam transforms in accordance with the style guide - but I'm happy to review some of them and participate in discussions of potentially more controversial changes. In my ideal world, this task would be crowdsourced (I sent o

Re: PTransform style guide PR

Thanks for putting that together. Does this mean you've volunteered to referee bikeshedding? On Mon, Jan 30, 2017 at 5:21 PM Eugene Kirpichov wrote: > The initial PR has been merged and the style guide is live > https://beam.apache.org/contribute/ptransform-style-guide/ - let us > continue discu

Let's make Beam transforms comply with PTransform Style Guide

Hello, The PTransform Style Guide is live https://beam.apache.org/contribute/ptransform-style-guide/ - a natural next step is to audit Beam libraries for compliance and file JIRAs for places that need to be fixed. It'd be great to finish these cleanups before declaring Beam stable API. Please tak

Re: PTransform style guide PR

The initial PR has been merged and the style guide is live https://beam.apache.org/contribute/ptransform-style-guide/ - let us continue discussing and tweaking on this thread and via smaller PRs modifying the document. On Mon, Jan 30, 2017 at 7:50 AM Aljoscha Krettek wrote: > Wow, that's a long

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

It looks good to me, but let's hear Aljoscha's opinion on BEAM-1346. A passing suite of Jenkins jobs: * https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/6870/ * https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/2474/ * https://builds.apache.org/job/beam_PostCommit_Java_

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

I am worried about https://issues.apache.org/jira/browse/BEAM-1346 for RC1 and would at least wait for resolution there before proceeding. On Mon, Jan 30, 2017 at 3:48 AM, Jean-Baptiste Onofré wrote: > Good catch for the PPMC, I'm upgrading the email template in the release > guide (it was a cop

Re: TextIO binary file

P.S. Note that this point (about coders) is also mentioned in the now-being-reviewed PTransform Style Guide https://github.com/apache/beam-site/pull/134 currently staged at http://apache-beam-website-pull-requests.storage.googleapis.com/134/contribute/ptransform-style-guide/index.html#coders On M

Re: Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2473

Hey folks, It looks like the python-sdk -> master merge went bad and, unfortunately, we have it configured to email anyone who ever contributed a commit to the merge, which I think devolves to "anyone who ever committed to that branch". I've disabled further emails in this job's configuration for

Re: TextIO binary file

+1 to what Eugene said. I've seen a number of Python SDK users incorrectly assuming that coder.decode() is needed when developing their own file-based sources (since many users usually refer to text source first). Probably coder parameter should not be configurable for text source/sink and they sh

Re: TextIO binary file

The use of Coder in TextIO is a long standing design issue because coders are not intended to be used for general purpose converting things from and to bytes, their only proper use is letting the runner materialize and restore objects if the runner thinks it's necessary. IMO it should have been cal

Re: TextIO binary file

Stas' comment is the right one. The "canonical" use of TextIO is using something like a TextualIntegerCoder , but that should almost certainly be replaced with TextIO.Read |

Re: PTransform style guide PR

Wow, that's a long read. But quite informative +1 On Sat, 28 Jan 2017 at 06:54 Jean-Baptiste Onofré wrote: > Hi Eugene, > > As said in the PR: great work and thanks a lot ! > > I will take a complete look during the week end. I'm pretty sure it's a > great guide as it's basically the result of o

Re: [BEAM-135] Utilities for "batching" elements in a DoFn

Hi, Le 27/01/2017 à 19:44, Robert Bradshaw a écrit : On Fri, Jan 27, 2017 at 6:55 AM, Etienne Chauchot wrote: Hi Robert, Le 26/01/2017 à 18:17, Robert Bradshaw a écrit : First off, let me say that a *correctly* batching DoFn is a lot of value, especially because it's (too) easy to (often unk

回复：How to implement Timer in runner

@小多 Thank you! I'll taka a look.--发件人：小多 发送时间：2017年1月27日(星期五) 23:22收件人：dev ; JingsongLee 抄　送：Kenneth Knowles ; Aljoscha Krettek 主　题：Re: How to implement Timer in runner Hi Jingsong, Take a look at Blink (a fork of Flink at Alibaba

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

Good catch for the PPMC, I'm upgrading the email template in the release guide (it was a copy/paste). Regards JB On 01/30/2017 11:50 AM, Sergio Fernández wrote: +1 (non-binding) So far I've successfully checked: * signatures and digests * source releases file layouts * matched git tags and co

Re: TextIO binary file

The Javadoc of TextIO states: * By default, {@link TextIO.Read} returns a {@link PCollection} of {@link String Strings}, * each corresponding to one line of an input UTF-8 text file. To convert directly from the raw * bytes (split into lines delimited by '\n', '\r', or '\r\n') to another object

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

+1 (non-binding) So far I've successfully checked: * signatures and digests * source releases file layouts * matched git tags and commit ids * incubator suffix and disclaimer * NOTICE and LICENSE files * license headers * clean build (Java 1.8.0_91, Maven 3.3.9, Debian amd64) Two minor comments t

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

It's normal, and the same in the previous release: it will be fixed when the release will be complete. Regards JB On 01/30/2017 11:19 AM, Alexey Demin wrote: Hi all good but now branch release-0.5.0 can't be build because last commit not reversed files sdks/java/maven-archetypes/examples-jav

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

Hi all good but now branch release-0.5.0 can't be build because last commit not reversed files sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml sdks/java/maven-archetypes/sta

Re: TextIO binary file

Hi Aviem, TextIO is not designed to write/read binary file: it's pure Text, so String. Regards JB On 01/30/2017 09:24 AM, Aviem Zur wrote: Hi, While trying to use TextIO to write/read a binary file rather than String lines from a textual file I ran into an issue - the delimiter TextIO uses se

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

+1 (binding) Regards JB On 01/27/2017 09:55 PM, Jean-Baptiste Onofré wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 0.5.0 as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging

Re: TextIO binary file

Dear Aviem, That's a good point. TextIO seems to make a few "string oriented" assumptions, see TextIO.Write's header and footer support , and IO design pattern: Decouple Parsers and Coders (BEAM-73) that came

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #1

+1 (non-binding) - verified signatures + checksums - run mvn clean verify -Prelease, all artifacts build and the tests run smoothly Great to see a shorter release cycle, the improvements and the new IOs. On Fri, Jan 27, 2017 at 9:55 PM, Jean-Baptiste Onofré wrote: > Hi everyone, > > Please re

TextIO binary file

Hi, While trying to use TextIO to write/read a binary file rather than String lines from a textual file I ran into an issue - the delimiter TextIO uses seems to be hardcoded '\n'. See `findSeparatorBounds` - https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sd