Re: Projects for Google Summer of Code 2017

2017-02-01 Thread Pablo Estrada
ng this to Shakespeare's works, and there are thousands of > times as many public articles so there's non-toy scale issues. And yet, it > does seem like it can be scoped appropriately. > > The deadline for a mentoring organization is Feb 9 so let's put together a > proposal! > > Kenn >

Proposal on porting PAssert away from aggregators

2017-03-30 Thread Pablo Estrada
Hello all, BEAM-775 aims to remove Aggregators from the Java SDK. One of the blocking issues is BEAM-1148, porting PAssert away from using them. Let me summarize the issues that we should consider; and then propose a solution that seems reasonable: 1. We want the test Pipeline run to fail-fast

Re: What is the stepName and how to assign stepNames?

2017-03-31 Thread Pablo Estrada
Hi Shen, That's correct. The Metrics API fills in the step name automatically from the PTransfor, and it is used to group and aggregate Metrics per step. As a user, you're not supposed to set/modify it yourself. Hope that helps solve your question. Best -P. On Fri, Mar 31, 2017 at 11:39 AM Shen

Adding logging for RunnableOnService/ValidatesRunner tests

2017-04-03 Thread Pablo Estrada
Hello there, I'm running RunnableOnService tests on the DirectRunner, with 'mvn clean verify' in runners/direct-java; and I'd like to add some logging to figure out what's going on in some failures. My question is: 1. Is there a way to run only a specific test with maven? 2. Is there extra

How I hit a roadblock with AutoValue and AvroCoder

2017-04-05 Thread Pablo Estrada
Hi all, I was encouraged to write about my troubles to use PCollections of AutoValue classes with AvroCoder; because it seems like currently, this is not possible. As part of the changes to PAssert, I meant to create a SuccessOrFailure class that could be passed in a PCollection to a

Re: [DISCUSSION] PAssert success/failure count validation for all runners

2017-04-17 Thread Pablo Estrada
Hi all, sorry about the long silence. I was on vacation all of last week. Also, the implementation of my proposal is in this PR: https://github.com/apache/beam/pull/2417. I don't currently have plans on the work for https://issues.apache.org/jira/browse/BEAM-1763. My main focus as of right now is

Re: How I hit a roadblock with AutoValue and AvroCoder

2017-04-05 Thread Pablo Estrada
As a note, it seems that SerializableCoder does the trick in this case, as it does not require a no-arg constructor for the class that is being deserialized - so perhaps we should encourage people to use that in the future. Best -P. On Wed, Apr 5, 2017 at 1:48 PM Pablo Estrada <pabl...@google.

Re: [ANNOUNCEMENT] New PMC members, August 2017 edition!

2017-08-11 Thread Pablo Estrada
Good job guys. Well deserved! -P. On Fri, Aug 11, 2017 at 11:34 AM Ismaël Mejía wrote: > Congratulations Ahmet and Aviem, keep up the great work ! > > On Fri, Aug 11, 2017 at 8:30 PM, Thomas Groh > wrote: > > Congratulations to both of you! Looking

contrib package for beam?

2017-08-16 Thread Pablo Estrada
Hi all, What would be an appropriate medium for contributions such as utility Pipelines or PTransforms? Perhaps it's different for each kind of contribution (sources/sinks, PTransforms, or utility pipelines). The question comes from an active user on Stack Overflow[1], and it seems pertinent.

Re: A personal update

2017-12-13 Thread Pablo Estrada
Welcome back Davor. On Wed, Dec 13, 2017 at 4:25 PM Alan Myrvold wrote: > Welcome back! Look forward to your contributions. > > On Wed, Dec 13, 2017 at 11:04 AM, Amit Sela wrote: > >> Congrats and welcome back! can't wait.. >> >> On Wed, Dec 13, 2017

Re: Python SDK build is failing

2017-12-13 Thread Pablo Estrada
I saw this as well, but somehow tests for my PR are now passing again.. On Wed, Dec 13, 2017 at 11:01 AM Robert Bradshaw wrote: > Strange, it passed all precommits (granted, that was a week ago), and > should have been basically an no-op (mostly just added logging). I'll >

Re: The trouble with DisplayData tests

2017-11-01 Thread Pablo Estrada
> > As of today, the Java and Python Dataflow runners actually transmit the > entire original pipeline, which has slots for display data on composites. > So the future in which this is actually used - at least for Dataflow - > might be very near. > I am not 100% sure, but I believe this is not

Re: Jenkins Post Commit Status to Github

2018-05-09 Thread Pablo Estrada
Does this require a Seed Job rerun? I'm seeing Python postcommits breaking as well On Wed, May 9, 2018 at 10:49 AM Alan Myrvold wrote: > I saw ~1-2 hour latency for the trigger happening yesterday. Not sure what > would cause it to stop or be slow. > > On Wed, May 9, 2018

Re: Jenkins Post Commit Status to Github

2018-05-09 Thread Pablo Estrada
8 at 11:08 AM Lukasz Cwik <lc...@google.com> wrote: >>> >>>> I am also seeing the delayed launch of the tests. >>>> >>>> On Wed, May 9, 2018 at 10:56 AM Pablo Estrada <pabl...@google.com> >>>> wrote: >>>> >>&g

Re: Jenkins Post Commit Status to Github

2018-05-09 Thread Pablo Estrada
the seed job, I'll figure out how to test my changes in the future. Sorry > for all the trouble! > > Andrew > > On Wed, May 9, 2018 at 11:36 AM Pablo Estrada <pabl...@google.com> wrote: > >> I was able to trigger a build again just now. >> >> On Wed, May 9

Documenting Metrics API?

2018-05-11 Thread Pablo Estrada
Hello all, I could not find a place were the Beam Metrics API is well detailed in the Beam website. Is there a JIRA tracking this? Perhaps our use-case driven docs + java/pydoc cover it well enough, but I'm not sure that that's the case. Thanks -P -- Got feedback? go/pabloem-feedback

Re: Documenting Metrics API?

2018-05-11 Thread Pablo Estrada
a while back: https://s.apache.org/beam-fn-api-metrics > > Otherwise, I'm not aware of any metrics related documentation for Apache > Beam on the website. > > > > On Fri, May 11, 2018 at 12:02 PM Pablo Estrada <pabl...@google.com> wrote: > >> Hello all, >> I co

Re: [VOTE] Go SDK

2018-05-22 Thread Pablo Estrada
+1 (binding) Very excited to see this! On Tue, May 22, 2018 at 9:09 AM Thomas Weise wrote: > +1 and congrats! > > > On Tue, May 22, 2018 at 8:48 AM, Rafael Fernandez > wrote: > >> +1 ! >> >> On Tue, May 22, 2018 at 7:54 AM Lukasz Cwik

Re: Missing copyright notices for shaded packages

2018-05-24 Thread Pablo Estrada
Thanks Boyuan for the prompt resolution! : D -P/ On Thu, May 24, 2018 at 4:33 PM Boyuan Zhang wrote: > Hey all, > > Release blocker https://issues.apache.org/jira/browse/BEAM-4393 marked as > Resolved. > > > Boyuan > > On Wed, May 23, 2018 at 1:52 PM Scott Wegner

Java PreCommit seems broken

2018-05-17 Thread Pablo Estrada
I'm seeing failures on Maven Archetype-related tests. Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk And the failure is here specifically: https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116

Re: Missing dep in python sdk? "ImportError: No module named builtins"

2018-05-15 Thread Pablo Estrada
That makes sense. Thanks a lot Valentyn! On Tue, May 15, 2018 at 5:25 PM Valentyn Tymofieiev <valen...@google.com> wrote: > Adding dependency on future in the setup.py fixes this for me, sending > https://github.com/apache/beam/pull/5379. > > On Tue, May 15, 2018 at 3:58 PM, P

Fwd: Missing dep in python sdk? "ImportError: No module named builtins"

2018-05-15 Thread Pablo Estrada
I believe this is a consequence of the recent work to futurize Python code[1]. I believe Robbe and Valentyn are working on this effort. We might just need to add the dependency to setup.py? [1]

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-11 Thread Pablo Estrada
> On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré wrote: >> > >> > No problem Pablo. >> > >> > The vote period is a minimum, it can be extended as requested or if we >> > don't have the minimum of 3 binding votes. >> > >> > Regards

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-08 Thread Pablo Estrada
Hello all, I'd like to request an extension of the voting period until Monday evening (US time, so later in other geographical regions). This is because we were only now able to publish Dataflow Workers, and have not had the chance to run release validation tests on them. The extension will allow

Re: [FYI] New Apache Beam Swag Store!

2018-06-08 Thread Pablo Estrada
Nice : D On Fri, Jun 8, 2018, 3:43 PM Raghu Angadi wrote: > Woo-hoo! This is terrific. > > If we are increasing color choices I would like black or charcoal... Beam > logo would really pop on a dark background. > > On Fri, Jun 8, 2018 at 3:32 PM Griselda Cuevas wrote: > >> Hi Beam Community,

Re: Announcement & Proposal: HDFS tests on large cluster.

2018-06-07 Thread Pablo Estrada
made decisions regarding that as >> this will increase monthly billing. >> >> Kamil Szewczyk >> >> 2018-06-07 6:27 GMT+02:00 Kenneth Knowles : >> >>> This is rad. Another +1 from me for a bigger cluster. What do you need >>> to make that happen?

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Pablo Estrada
It's very exciting to see a new runner making it into master. : ) Best -P. On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez wrote: > I've just read this and wanted to share my excitement :D > > > > On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles wrote: > >> One thing that will be necessary is

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-18 Thread Pablo Estrada
; > > > Thanks. > > Boyuan > > > > On Sun, Jun 17, 2018 at 9:22 PM Jean-Baptiste Onofré > <mailto:j...@nanthrax.net>> wrote: > > > > s/add/had/g > > > > On 17/06/2018 19:32, Pablo Estrada wrote: > > > Going to h

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Pablo Estrada
Hi all, I'll humbly declare that after wrestling with he build to stop depending on the wrong findbugs_annotations, I feel somewhat lost. The issue is actually quite small: - The JAR from the non-LGPL findbugs does not contain the SuppressFBWarnings annotation. This means that when building,

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-13 Thread Pablo Estrada
ations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java > [2] https://github.com/stephenc/findbugs-annotations/releases > [3] > https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab > > > > > > > > O

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-12 Thread Pablo Estrada
tests are failing for Windows. JIRA >>> > is https://issues.apache.org/jira/browse/BEAM-4535. >>> > >>> > I don't think this is a release blocker but this should probably go >>> > in release notes (for any user that tries to run tests on Pytho

Re: Beam Dependency Check Report (2018-06-13)

2018-06-13 Thread Pablo Estrada
Ahh very nice... thanks Yifan & Cham! Lots of old dependencies eh... very interesting. Best -P. On Wed, Jun 13, 2018 at 7:45 AM Yifan Zou wrote: > Hi, > > > I want to follow up and explain this email. > > > This is a sample email that reports the results of Beam SDK dependency > check, which

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-13 Thread Pablo Estrada
//github.com/stephenc/findbugs-annotations/issues/4 >> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html >> >> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada >> wrote: >> >>> Hi Tim, >>> you're right. Thanks for pointing that out

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-13 Thread Pablo Estrada
est > in the release branch. My reasoning is: > - ReferenceRunner is not yet the official alternative to existing direct > runners. > - It is bad to have flaky tests on the release branch, and we would not > get good signal during validation. > > On Wed, Jun 13, 2018 at 3:14

Re: Apache Beam June Newsletter

2018-06-13 Thread Pablo Estrada
Thanks Gris! Lots of interesting things. Best -P. On Wed, Jun 13, 2018 at 4:40 PM Griselda Cuevas wrote: > Hi Beam Community! > > Here > > [1] > is the June Edition of our Apache Beam

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

2018-06-13 Thread Pablo Estrada
at 9:54 AM Pablo Estrada wrote: > Precommits for PR https://github.com/apache/beam/pull/5609 are now > passing. For now I've simply set failOnWarning to false to cherrypick into > the release, and fix in master later on. > Best > -P. > > On Wed, Jun 13, 2018 at 9:08 A

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-17 Thread Pablo Estrada
Going to https://github.com/apache/beam/tree/v2.5.0-RC2 gives me a 404 error. May need to push the label? Best -P. On Sun, Jun 17, 2018 at 1:15 AM Romain Manni-Bucau wrote: > +1 (same tests than for take #1) > > Le dim. 17 juin 2018 07:18, Jean-Baptiste Onofré a > écrit : > >> Hi everyone, >>

Re: [DISCUSS] Remove findbugs from sdks/java

2018-05-30 Thread Pablo Estrada
Thank you guys : D On Wed, May 30, 2018 at 9:20 AM Scott Wegner wrote: > Sorry to revive an old thread, but I wanted to give a shout-out and say > thank you to Ismaël and Tim who have been quickly chipping away at the > ErrorProne backlog. We started with 47 ErrorProne JIRA's [1], and in two >

Re: Reducing Committer Load for Code Reviews

2018-05-31 Thread Pablo Estrada
In that case, does it make sense to say: - A code review by a committer is enough to merge. - Committers can have their PRs reviewed by non-committers that are familiar with the code - Non-committers may have their code reviewed by non-committers, but should have a committer do a lightweight

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

2018-05-31 Thread Pablo Estrada
>> On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré >>>>>> wrote: >>>>>> >>>>>>> Congrats ! >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>&g

Re: [VOTE] Code Review Process

2018-06-01 Thread Pablo Estrada
+1 :) glad that we had this discussion On Fri, Jun 1, 2018, 3:38 PM Udi Meiri wrote: > +1 > > On Fri, Jun 1, 2018 at 1:46 PM Andrew Pilloud wrote: > >> +1 - I hope this doesn't reduce the urgency to fix the root cause: not >> having enough committers. >> >> On Fri, Jun 1, 2018 at 1:18 PM

Re: Initial contributor experience

2018-06-05 Thread Pablo Estrada
Thanks Austin for taking the time to go through this! We came out with a few JIRAs to improve the documentation (see doc), and hopefully we'll keep iterating on this. Hopefully we can get more experiences from other people that start to approach Beam. Best -P. On Tue, Jun 5, 2018 at 1:49 PM

Re: Announcement & Proposal: HDFS tests on large cluster.

2018-06-06 Thread Pablo Estrada
This is really cool! +1 for having a cluster with more than one machine run the test. -P. On Wed, Jun 6, 2018 at 9:57 AM Chamikara Jayalath wrote: > On Wed, Jun 6, 2018 at 5:19 AM Łukasz Gajowy > wrote: > >> Hi all, >> >> I'd like to announce that thanks to Kamil Szewczyk, since this PR >>

Re: [PROPOSAL] Preparing 2.5.0 release next week

2018-06-06 Thread Pablo Estrada
This is because the release plugin that we went with[1] only produces release (and release candidate) tags, but it does not create a new branch, so the branch itself had to be created manually. There were other plugins with the extra branching functionality[2], but we decided to use the more

Re: [DISCUSS] [BEAM-4126] Deleting Maven build files (pom.xml) grace period?

2018-06-06 Thread Pablo Estrada
I agree that we should delete the pom.xml files soon, as they create a burden for maintainers. I'd like to be able to extend the grace period by a bit, to allow the internal build systems at Google to move away from using the Beam poms. We use these pom files to build Dataflow workers, and thus

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-20 Thread Pablo Estrada
0, 2018 at 9:37 AM Davor Bonaci wrote: >>>> >>>>> Please take a peek at LEGAL-288 [1], which I learned about recently in >>>>> the context of another project. Looks like an issue, requiring a new RC, >>>>> but I didn't have a chance to look closel

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-20 Thread Pablo Estrada
+1 (binding) On Wed, Jun 20, 2018 at 9:08 AM Lukasz Cwik wrote: > +1 (binding) > > On Tue, Jun 19, 2018 at 10:39 PM Jean-Baptiste Onofré > wrote: > >> +1 (binding) >> >> Regards >> JB >> >> On 17/06/2018 07:18, Jean-Baptiste Onofré wrote: >> > Hi everyone, >> > >> > Please review and vote on

Re: Filtered Pre-commit triggering is BACK!

2018-06-26 Thread Pablo Estrada
This is great. Reducing load on infrastructure should help Beam scale into a larger project : ) - Thanks Scott! On Tue, Jun 26, 2018 at 2:21 PM Scott Wegner wrote: > By popular demand [1], filtered pre-commit triggering is now re-enabled. > Now when submitting pull request, only pre-commit

Re: [Proposal] Apache Beam Swag Store

2017-10-26 Thread Pablo Estrada
I like the designs, and I like the idea of the store. How hard is it to set up and keep the store? : ) On Thu, Oct 26, 2017 at 10:54 AM Griselda Cuevas wrote: > Hi everyone - pinging the thread to see what do you folks think about the > store idea and the initial

Re: A 15x speed-up in local Python DirectRunner execution

2018-02-07 Thread Pablo Estrada
Very cool Charles! Have you considered whether you'll want to remove the direct runner code afterwards? Best -P. On Wed, Feb 7, 2018, 6:25 PM Lukasz Cwik wrote: > That is pretty awesome. > > On Wed, Feb 7, 2018 at 5:55 PM, Charles Chen wrote: > >> Local

Re: Python 3 reviewers

2018-02-22 Thread Pablo Estrada
I'm not a committer, but willing to help review and On Thu, Feb 22, 2018, 8:54 PM Holden Karau wrote: > So I've been hesistant to do large chunks of Python 3 reviews for the same > reason I've been hesistant to do large chunks of Python 3 work (on the > commit side seems

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

2018-02-21 Thread Pablo Estrada
Hello all! As outlined in BEAM-3720, this is due to my adding Gauge metrics without adding support for the portability framework around the same time that the direct runner was swapped. Please don't despair. PR https://github.com/apache/beam/pull/4719 fixes this, and hopefully will be merged soon.

Re: Jenkins job beam_PreCommit_Python_MavenInstall” still fails

2018-02-21 Thread Pablo Estrada
ix the build. > > On Wed, Feb 21, 2018 at 9:03 AM, Pablo Estrada <pabl...@google.com> wrote: > >> Hello all! >> As outlined in BEAM-3720, this is due to my adding Gauge metrics without >> adding support for the portability framework around the same time that the >>

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

2018-06-22 Thread Pablo Estrada
Yes, sorry about the confusion! -P. On Fri, Jun 22, 2018, 3:06 AM Jean-Baptiste Onofré wrote: > +1 (binding) > > Regards > JB > > On 17/06/2018 07:18, Jean-Baptiste Onofré wrote: > > Hi everyone, > > > > Please review and vote on the release candidate #2 for the version > > 2.5.0, as follows: >

Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-27 Thread Pablo Estrada
PM Pablo Estrada wrote: > Hello everyone, > I wanted to do an update on the state of the release, as there haven't > been news on this for a while. > We have found a few issues that broke postcommits a few weeks back, but we > hadn't noticed. Some people are tacking these to t

Re: Parallelizing test runs

2018-08-01 Thread Pablo Estrada
It feels to me like a peak of 60 jobs per minute is pretty high. If I understand correctly, we run up to 20 dataflow jobs in parallel per test suite? Or what's the number here? It is also true that most our tests are simple NeedsRunner tests, that test a couple elements, so the whole pipeline

Re: Removing documentation for old Beam versions

2018-08-02 Thread Pablo Estrada
Is it worth adding a tag / branch to the repositories every time we make a release, so that people are able to dive in and find the docs? Best -P. On Thu, Aug 2, 2018 at 12:09 PM Ahmet Altay wrote: > I would guess that users are still using some of these old releases. It is > unclear from Beam

Re: Removing documentation for old Beam versions

2018-08-02 Thread Pablo Estrada
o a branch occurs), such as done in >>>> the Apex and Flink projects. >>>> >>>> I was told that Scott who was working in the beam-site changes is on >>>> leave now and the migration is still pending (see note at >>>> https://github.com/apache/beam

Re: [Call for items] Beam August Newsletter

2018-08-02 Thread Pablo Estrada
Thanks for doing this Rose! I'll add a couple of things. -P. On Thu, Aug 2, 2018, 4:18 PM Rose Nguyen wrote: > Hi all: > > Here's > > [1] > the template for the August Beam Newsletter! > > *Add the

Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-26 Thread Pablo Estrada
Pablo Estrada wrote: > Hello all! > I've cut the release branch (release-2.6.0), with some help from Ahmet and > Boyuan. From now on, please cherry-pick 2.6.0 blockers into the branch. > Now we start stabilizing it. > > Thanks! > > -P. > > On Tue, Jul 17, 2018 a

[VOTE] Apache Beam, version 2.6.0, release candidate #1

2018-07-31 Thread Pablo Estrada
Hello everyone! I have been able to prepare a release candidate for Beam 2.6.0. : D Please review and vote on the release candidate #1 for the version 2.6.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staged set of

Re: Community Examples Repository

2018-08-03 Thread Pablo Estrada
Hello all, I see a number of mixed responses. I think it would be helpful to push for a decision by calling for a vote. Also, the proposal has a number of parts, so perhaps we could ask David and other contributors of the proposal to outline a couple alternatives the we can all vote on. (e.g. #1

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

2018-08-03 Thread Pablo Estrada
Aug 2, 2018 at 10:07 PM Suneel Marthi >>> wrote: >>> >>>> +1 non-binding >>>> >>>> 1. tested with beam samples >>>> 2. verified sigs and hashes of artifacts >>>> >>>> >>>> On Fri, Aug 3, 2018 at 12:43 AM, Je

[VOTE] Apache Beam, version 2.6.0, release candidate #2

2018-08-03 Thread Pablo Estrada
Hello everyone! Extra, extra! The Apache Beam 2.6.0 release candidate #2 is out. Please review and vote on the release candidate #2 for the version 2.6.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staged set of

Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-30 Thread Pablo Estrada
Hello, is there a change this bug make it into the release? >> > >> > https://issues.apache.org/jira/browse/BEAM-5028 >> > >> > On Sat, Jul 28, 2018 at 1:38 AM Pablo Estrada > > <mailto:pabl...@google.com>> wrote: >> >

[HELP] Blog post for 2.6.0 Release

2018-08-08 Thread Pablo Estrada
Hello all, During my work on the release, I missed that we have started doing blog posts for every release. I decided to announce the release, and start the blog post afterwards - the blog post will only be late for a few days. Please add all your release notes and comments in this doc:

Re: Renewing the Slack channel join link?

2018-08-08 Thread Pablo Estrada
t;Get short-URL" > > On Wed, Aug 8, 2018 at 2:17 PM Pablo Estrada wrote: > >> Hello all, >> Can we renew the link please? Also, if y{all teach me how, I'll gladly >> set reminders to do it every month : ) >> Best >> -P. >> -- >> Got feed

Renewing the Slack channel join link?

2018-08-08 Thread Pablo Estrada
Hello all, Can we renew the link please? Also, if y{all teach me how, I'll gladly set reminders to do it every month : ) Best -P. -- Got feedback? go/pabloem-feedback

[ANNOUNCE] Apache Beam 2.6.0 released!

2018-08-08 Thread Pablo Estrada
Beam 2.6.0. -- Pablo Estrada, on behalf of The Apache Beam team -- Got feedback? go/pabloem-feedback

Re: [Discussion] Clarify the support story for released Beam versions

2018-08-10 Thread Pablo Estrada
I think this all sounds reasonable, and I think it would be a good story for our users. We don't have much experience with patching releases, but I guess it's a matter of learning and improving over time. -P. On Wed, Aug 8, 2018 at 9:04 PM Ahmet Altay wrote: > Hi all, > > I would like us to

Re: [Vote] Create apache-beam-wheels repo like beam-site

2018-08-10 Thread Pablo Estrada
+1 : ) On Thu, Aug 9, 2018 at 1:39 PM Ahmet Altay wrote: > +1 > > On Thu, Aug 9, 2018 at 1:10 PM, Boyuan Zhang wrote: > >> Hey all, >> >> With some ongoing discussions happened in "Apache Beam Python Wheels >> Repository" thread, we are decided to move forward to start this vote >> process. We

Re: Apache Beam Newsletter - August 2018

2018-08-10 Thread Pablo Estrada
;- > >Suneel Marthi and Jose Luis Contreras gave a talk on using streaming >pipelines built on Apache Flink for model training and inference. They >leveraged convolutional Neural Networks (CNNs) built with Apache MXNet to >train Deep Learning models for land

Re: [HELP] Blog post for 2.6.0 Release

2018-08-10 Thread Pablo Estrada
Hello everyone, here's a PR for the blog post: https://github.com/apache/beam-site/pull/533 Last call for opinions : ) Best -P. On Fri, Aug 10, 2018 at 9:47 AM Pablo Estrada wrote: > Hello everyone, > a bump here on asking for your contributions to the 2.6.0 release blog > post. &g

Re: [HELP] Blog post for 2.6.0 Release

2018-08-10 Thread Pablo Estrada
Hello everyone, a bump here on asking for your contributions to the 2.6.0 release blog post. Best -P. On Wed, Aug 8, 2018 at 4:20 PM Pablo Estrada wrote: > Hello all, > During my work on the release, I missed that we have started doing blog > posts for every release. I decided to

Re: Contribute

2018-08-11 Thread Pablo Estrada
An extra tip: ./gradlew tasks --all should show you a bunch of existing tasks that you can run, and ./gradlew projects --all does the same for projects. -P. On Fri, Aug 10, 2018, 10:33 PM Trần Thành Đạt wrote: > Thank you, Rui, > > I have the same question. > I will start running ./gradlew

Re: Apache Beam Python Wheels Repository

2018-08-07 Thread Pablo Estrada
n INFRA ticket to create a new repo to host wheel setup. >>> Does anybody have experience with setting up a new repo similar to this? >>> >>> Ahmet >>> >>> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang wrote: >>> >>>> Hey all, >>>&g

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

2018-08-07 Thread Pablo Estrada
There are no disapproving votes. I will go ahead and finalize the release. Thanks everyone! -P. Thanks everyone! On Tue, Aug 7, 2018, 10:27 AM Reuven Lax wrote: > +1 (binding) > > On Tue, Aug 7, 2018 at 9:57 AM Pablo Estrada wrote: > >> Hello all, >> the voting seems to be going wel

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

2018-08-07 Thread Pablo Estrada
I'll vote +1 as well. On Tue, Aug 7, 2018, 10:27 AM Reuven Lax wrote: > +1 (binding) > > On Tue, Aug 7, 2018 at 9:57 AM Pablo Estrada wrote: > >> Hello all, >> the voting seems to be going well for RC2. As a heads up for anyone to >> express concerns or quest

Re: Rename Nexmark jobs to Perf instead of PostCommit

2018-08-13 Thread Pablo Estrada
I think it makes sense to rename. Also, although we should hold perf tests to a high reliability standard, we should also prioritize fixing and triaging PostCommit tests earlier. Best -P. On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud wrote: > I have no objections with renaming these to Perf

Re: PR# 6181 Needs Review (Spark Runner BoundedSource)

2018-08-15 Thread Pablo Estrada
Hi Kyle! You did the right thing. We don't have guarantees on when someone will get back to you on a PR. It all depends on the availability of the reviewer. You've done the right thing by mentioning here to get more eyes looking. We'll see if JB is able to take a look, or maybe we'll get someone

Re: [Discussion] Clarify the support story for released Beam versions

2018-08-15 Thread Pablo Estrada
page. Do you have any > ideas? > > > On Wed, Aug 15, 2018 at 11:28 AM, Pablo Estrada > wrote: > >> Thanks Ahmet for looking into this. I have a follow-up question. Have you >> thought about the next few releases, and which one will be the first LTS >> r

Re: [Discussion] Clarify the support story for released Beam versions

2018-08-15 Thread Pablo Estrada
gt;>>>>>>> This comes from my anecdotal evidence and I may be wrong. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I believe it would be valuable to ask our users

Re: gradlew :rat broken

2018-08-15 Thread Pablo Estrada
The files that are causing the failure seem to have been generated. They are not part of the checked-in source code: https://github.com/apache/beam/tree/master/sdks/python/container Your files are container/vendor/, and sdks/python/nosetests.xml (which is generated when running nosetests).

Re: Policy for Python ValidatesRunner vs IT tests?

2018-08-14 Thread Pablo Estrada
am/blob/master/sdks/java/io/google-cloud-platform/build.gradle#L85 > > On Tue, Aug 14, 2018 at 3:27 PM Pablo Estrada wrote: > >> Hello, >> In Python, we tag some test methods with @attr('ValidatesRunner') and >> @attr('IT'), which marks them to be run as pipeline tests. &g

Policy for Python ValidatesRunner vs IT tests?

2018-08-14 Thread Pablo Estrada
Hello, In Python, we tag some test methods with @attr('ValidatesRunner') and @attr('IT'), which marks them to be run as pipeline tests. If I understand correctly: - ValidatesRunner tests are more like a component tests[1] as explained in Beam docs - IT tests are more like a E2E test[2] as

Re: [Build Python Wheels] New Repository for Building Python Wheels

2018-08-14 Thread Pablo Estrada
Thanks Boyuan : D - this is very nice to have. On Tue, Aug 14, 2018 at 4:30 PM Boyuan Zhang wrote: > Hey all, > > Now we have beam-wheels as a > wrapper repository to build and stage python wheels for each release. In > order to stage all artifacts

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

2018-08-07 Thread Pablo Estrada
/beam-release-validation >>>> >>>> On Sun, Aug 5, 2018 at 6:37 AM Suneel Marthi >>>> wrote: >>>> >>>>> +1 non-binding >>>>> >>>>> 1. verified Sigs and Hashes of artifacts >>>>>

Re: [ANNOUNCE] Apache Beam 2.6.0 released!

2018-08-09 Thread Pablo Estrada
eat news! > Thanks to Pablo for driving this and to all contributors of this release! > > > On 9 Aug 2018, at 00:48, Pablo Estrada wrote: > > The Apache Beam team is pleased to announce the release of 2.6.0 version! > > Apache Beam is an open source unified programming model t

Re: [ANNOUNCE] Apache Beam 2.6.0 released!

2018-08-09 Thread Pablo Estrada
Of course, I messed that link up again. The release notes should be here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12343392 On Thu, Aug 9, 2018 at 10:11 AM Pablo Estrada wrote: > As was pointed out to me by Ryan Williams, the link I posted with the > r

Re: Travis apache credentials

2018-08-20 Thread Pablo Estrada
This would mean that released artifacts are signed with two different keys (wheels with travis / jars and others with release manager's). Is this consistent with Apache policy? Just checking : ) -P. On Mon, Aug 20, 2018 at 9:47 AM Boyuan Zhang wrote: > Hey Robert, > > I think your idea would be

Should we mention TF Transform in Beam site?

2018-08-20 Thread Pablo Estrada
Other projects mention their ML / Graph / misc tooling libraries in their websites. It may be good for Beam to direct people to use Tensorflow Transform[1] if they want to use beam for ML? What do people think? Best -P. [1] https://www.tensorflow.org/tfx/transform/

Re: [Event] Open Source Challenge Mexico

2018-08-17 Thread Pablo Estrada
Hello everyone, this has finished today. The two teams that worked in Beam prepared 2 PRs (6241, 6242), made one code change in a branch without a PR, and proposed solutions for a couple more issues in JIRA. These PRs and changes will have to be iterated on, but we'll be able to incorporate their

Should Beam Python throw an error if DoFn returns a string?

2018-08-17 Thread Pablo Estrada
Beam Python expects DoFns to return an iterable that contains the actual output elements. This is documented, and visible in examples, but it is also a bit counter-intuitive. We should definitely add a check in _OutputProcessor[1] to throw a more expressive error if it receives a non-iterable.

Re: Should Beam Python throw an error if DoFn returns a string?

2018-08-17 Thread Pablo Estrada
Thanks Robert. You raise a good point that this code is performance-critical. If the check can be fast, then its worth having. > Should we also let Beam error out if users return a string? > > e.g. consider the following pipeline: > > p | Create(['abc']) | ParDo(lambda x: x) |

Re: [Proposal] Beam Mascot

2018-08-23 Thread Pablo Estrada
This sounds great to me. Let's get us some wonderful Beam mascot : D On Thu, Aug 23, 2018 at 2:14 PM Griselda Cuevas wrote: > Hi Folks, > > I'm interested in launching a community contest to design a mascot for > Apache Beam. I'd like to get input from the PMC on how this should be > pursued

Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-17 Thread Pablo Estrada
gt; Thanks, >> Thomas >> >> >> On Fri, Jul 13, 2018 at 11:02 AM Ahmet Altay wrote: >> >>> >>> >>> On Fri, Jul 13, 2018 at 10:48 AM, Pablo Estrada >>> wrote: >>> >>>> Hi all, >>>> I've triaged most i

Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-18 Thread Pablo Estrada
stigating this issue, but it's a little long process. > > So, I propose you start with the release process, cutting the branch, > and then, I will create a cherry-pick PR for this one. > > Regards > JB > > On 17/07/2018 20:19, Pablo Estrada wrote: > > Checking once mo

Any Beamers at OSCON?

2018-07-15 Thread Pablo Estrada
Hello everyone, I am reaching out for people that will be attending this year's OSCon in Portland[1], starting tomorrow 10/16. I am aware of a few Beam people that will be attending along with me, and I'm sure we'll be glad to see others. I am also aware of one talk involving Beam, by Holden

Re: Updating Many Dependency Versions

2018-07-19 Thread Pablo Estrada
This is long overdue. Thanks a lot for working on this Garrett and Luke! -P. On Thu, Jul 19, 2018 at 10:16 AM Lukasz Cwik wrote: > I wanted to raise awareness of https://github.com/apache/beam/pull/5988 > since it updates a lot of the Google dependency client versions and many of > their

Re: Accessing attempted metrics from within a DoFn

2018-08-29 Thread Pablo Estrada
Adding Andrea as she may have thoughts on this. On Wed, Aug 29, 2018 at 10:43 AM Robin Qiu wrote: > Hi everyone, > > I am writing a test [1] for the support of @RequiresStableInput annotation > in Java SDK [2]. In the test I need to have a ParDo make some side effects > and then fail.

  1   2   3   4   >