Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Pablo Estrada]

Hello all,
I will start daily updates of progress on the 2.6.0 release.
As of today, the main release blockers are issues in Dataflow that are
preventing us from cutting the Dataflow workers.

   - One issue in Java SDK related to FnAPI. Specifically PR 5709 requires
   Dataflow worker changes[1].
   - One issue in the Python SDK related to context management. PR 5356
   also requires Dataflow worker changes [2].

Please reach out to me if you have any questions.
Best
-P.

[1] https://github.com/apache/beam/pull/5709
[2] https://github.com/apache/beam/pull/5356


On Thu, Jul 26, 2018 at 2:16 PM Pablo Estrada  wrote:

> Hello everyone,
> I wanted to do an update on the state of the release, as there haven't
> been news on this for a while.
> We have found a few issues that broke postcommits a few weeks back, but we
> hadn't noticed. Some people are tacking these to try to stabilize the
> release branch[1].
>
> In the meantime, the release has been blocked, but Boyuan Zhang has taken
> advantage of this to code up a few scripts to try and automate release
> steps. (Thanks Boyuan!). We will try these as soon as the release is
> unblocked.
>
> Best
> -P.
>
> [1] https://github.com/apache/beam/pull/6072
>
> On Wed, Jul 18, 2018 at 11:03 AM Pablo Estrada  wrote:
>
>> Hello all!
>> I've cut the release branch (release-2.6.0), with some help from Ahmet
>> and Boyuan. From now on, please cherry-pick 2.6.0 blockers into the branch.
>> Now we start stabilizing it.
>>
>> Thanks!
>>
>> -P.
>>
>> On Tue, Jul 17, 2018 at 9:34 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi Pablo,
>>>
>>> I'm investigating this issue, but it's a little long process.
>>>
>>> So, I propose you start with the release process,  cutting the branch,
>>> and then, I will create a cherry-pick PR for this one.
>>>
>>> Regards
>>> JB
>>>
>>> On 17/07/2018 20:19, Pablo Estrada wrote:
>>> > Checking once more:
>>> > What does the communitythink we should do
>>> > about https://issues.apache.org/jira/browse/BEAM-4750 ? Should I bump
>>> it
>>> > to 2.7.0?
>>> > Best
>>> > -P.
>>> >
>>> > On Fri, Jul 13, 2018 at 5:15 PM Ahmet Altay >> > > wrote:
>>> >
>>> > Update:  https://issues.apache.org/jira/browse/BEAM-4784 is not a
>>> > release blocker, details in the JIRA issue.
>>> >
>>> > On Fri, Jul 13, 2018 at 11:12 AM, Thomas Weise >> > > wrote:
>>> >
>>> > Can one of our Python experts please take a look
>>> > at https://issues.apache.org/jira/browse/BEAM-4784 and advise
>>> if
>>> > this should be addressed for the release?
>>> >
>>> > Thanks,
>>> > Thomas
>>> >
>>> >
>>> > On Fri, Jul 13, 2018 at 11:02 AM Ahmet Altay >> > > wrote:
>>> >
>>> >
>>> >
>>> > On Fri, Jul 13, 2018 at 10:48 AM, Pablo Estrada
>>> > mailto:pabl...@google.com>> wrote:
>>> >
>>> > Hi all,
>>> > I've triaged most issues marked for 2.6.0 release. I've
>>> > localized two that need a decision / attention:
>>> >
>>> > - https://issues.apache.org/jira/browse/BEAM-4417 -
>>> > Bigquery IO Numeric Datatype Support. Cham is not
>>> > available to fix this at the moment, but this is a
>>> > critical issue. Is anyone able to tackle this / should
>>> > we bump this to next release?
>>> >
>>> >
>>> > I bumped this to the next release. I think Cham will be the
>>> > best person to address it when he is back. And with the
>>> > regular release cadence, it would not be delayed by much.
>>> >
>>> >
>>> >
>>> > - https://issues.apache.org/jira/browse/BEAM-4750 -
>>> > Performance degradation due to some safeguards in
>>> > beam-sdks-java-core. JB, are you looking to fix this?
>>> > Should we bump? I had the impression that it was an
>>> easy
>>> > fix, but I'm not sure.
>>> >
>>> > If you're aware of any other issue that needs to be
>>> > included as a release blocker, please report it to me.
>>> > Best
>>> > -P.
>>> >
>>> > On Thu, Jul 12, 2018 at 2:15 AM Etienne Chauchot
>>> > mailto:echauc...@apache.org>>
>>> wrote:
>>> >
>>> > +1,
>>> >
>>> > Thanks for volunteering Pablo, thanks also to have
>>> > caught tickets that I forgot to close :)
>>> >
>>> > Etienne
>>> >
>>> > Le mercredi 11 juillet 2018 à 12:55 -0700, Alan
>>> > Myrvold a écrit :
>>> >> +1 Thanks for volunteering, Pablo
>>> >>
>>> >> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster
>>> >> >> >> > wrote:
>>> 

Re: Beam Docs Contributor

[Ahmet Altay]

Welcome Rose! Looking forward to your contributions.

On Fri, Jul 27, 2018 at 4:08 PM, Rose Nguyen  wrote:

> Hi all:
>
> I'm Rose! I've worked on Cloud Dataflow documentation and now I'm starting
> a project to refresh the Beam docs and improve the onboarding experience.
> We're planning on splitting up the programming guide into multiple pages,
> making the docs more accessible for new users. I've got lots of ideas for
> doc improvements, some of which are motivated by the UX research, and am
> excited to share them with you all and work on them.
>
> I look forward to interacting with everybody in the community. I welcome
> comments, thoughts, feedback, etc.
> --
>
>
> Rose Thi Nguyen
>
>   Technical Writer
>
> (281) 683-6900
>

Beam Docs Contributor

[Rose Nguyen]

Hi all:

I'm Rose! I've worked on Cloud Dataflow documentation and now I'm starting
a project to refresh the Beam docs and improve the onboarding experience.
We're planning on splitting up the programming guide into multiple pages,
making the docs more accessible for new users. I've got lots of ideas for
doc improvements, some of which are motivated by the UX research, and am
excited to share them with you all and work on them.

I look forward to interacting with everybody in the community. I welcome
comments, thoughts, feedback, etc.
-- 


Rose Thi Nguyen

  Technical Writer

(281) 683-6900

Re: Cannot create subscription because pipeline option 'project' not specified

[Rui Wang]

The complete story is, TestPubsub invokes TestPipeline so it requires
configuration in build.gradle. BeamSQL JDBC driver does not invoke
TestPipeline so it requires "SET project".

Thanks Andrew.

-Rui

On Thu, Jul 26, 2018 at 6:01 PM Andrew Pilloud  wrote:

> Your second stacktrace isn't going through SQL. It looks like you are
> using the normal test path there. Have you tried setting in both places?
>
> On Thu, Jul 26, 2018, 5:48 PM Rui Wang  wrote:
>
>> Ah,  SET project = apache-beam-testing; gives the following exception:
>>
>> io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to 
>> perform this action.
>>  at 
>> io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
>>  at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
>>  at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
>>  at 
>> com.google.pubsub.v1.PublisherGrpc$PublisherBlockingStub.createTopic(PublisherGrpc.java:666)
>>  at 
>> org.apache.beam.sdk.io.gcp.pubsub.PubsubGrpcClient.createTopic(PubsubGrpcClient.java:302)
>>  at 
>> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.initializePubsub(TestPubsub.java:102)
>>  at 
>> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.access$200(TestPubsub.java:42)
>>  at 
>> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:85)
>>
>>
>> which should not happen because jenkins should already have credentials
>> to access GCP.
>>
>> -Rui
>>
>> On Thu, Jul 26, 2018 at 3:30 PM Andrew Pilloud 
>> wrote:
>>
>>> Beam SQL CLI does not accept beamTestPipelineOptions. Also, gradle is
>>> invoking your test not the Beam SQL CLI. You'll need to set the options in
>>> your integration test by executing 'SET project = ...' in the Beam SQL
>>> connection you've launched for test.
>>>
>>> Andrew
>>>
>>> On Thu, Jul 26, 2018 at 3:06 PM Rui Wang  wrote:
>>>
 The code path of reading pubsub through BeamSQL goes through this line
 of code:

 PubsubIO.Read read = 
 PubsubIO.readMessagesWithAttributes().fromTopic(getTopic());

 -Rui

 On Thu, Jul 26, 2018 at 2:58 PM Rui Wang  wrote:

> Hi Community,
>
> I am facing a runtime exception when I try to read from pubsub by Beam
> SQL in JUnit tests (PR: https://github.com/apache/beam/pull/6006).
> The exception is "Cannot create subscription because pipeline option
> 'project' not specified". Based on existing JUnit tests which also read
> from pubsub by Beam SQL, I added the following code to my
> build.gradle and it seems didn't work. Is there someone who could know
> what's wrong in my .gradle file?
>
>
> task endToEndTest(type: Test) {
>   group = "Verification"
>   def gcpProject = project.findProperty('gcpProject') ?: 
> 'apache-beam-testing'
>   def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 
> 'gs://temp-storage-for-end-to-end-tests/'
>
>   // Disable Gradle cache (it should not be used because the IT's won't 
> run).
>   outputs.upToDateWhen { false }
>
>   def pipelineOptions = [
>   "--project=${gcpProject}",
>   "--tempLocation=${gcsTempRoot}",
>   "--blockOnRun=false"]
>
>   systemProperty "beamTestPipelineOptions", 
> JsonOutput.toJson(pipelineOptions)
>
>   include '**/BeamSqlLineIT.class'
>   classpath = 
> project(":beam-sdks-java-extensions-sql-jdbc").sourceSets.test.runtimeClasspath
>   testClassesDirs = 
> files(project(":beam-sdks-java-extensions-sql-jdbc").sourceSets.test.output.classesDirs)
>   useJUnit { }
> }
>
> task postCommit {
>   group = "Verification"
>   description = "Various integration tests"
>   dependsOn endToEndTest
> }
>
>
> -Rui
>

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

Summary doc for CODEOWNERS, Mention-bot, Prow:
https://docs.google.com/document/d/1S8spggJsxDNYZ7aNwZN6VhLhNW372SVRezjblt-7lNQ/edit?usp=sharing
This doc will get updated as we gain experience with Mention-bot and Prow.

On Wed, Jul 25, 2018 at 5:15 PM Udi Meiri  wrote:

> So I configured Prow using their getting started guide (and found a bug in
> it) on a test repo.
>
> TLDR: Prow can work for us as a review assignment tool if all potential
> reviewers are also added to the https://github.com/apache org.
>
> Some findings:
> 1. Github doesn't allow non-collaborators to be listed as reviewers. :(
> But! anyone added to the Apache org on Github may be added as a reviewer.
> (no write access needed)
> Is this something the ASF is willing to consider?
>
> 2. Prow works pretty well. I've configured it to just assign code
> reviewers.
> Here's an example of it in action:
> https://github.com/udim-org/prow-test/pull/6
> Essentially, the command we would use are:
> "/cc @user" - to explicitly add a reviewer (/uncc to remove)
>
> Other command in the example above are not necessary.
> We can still use our current PR approval and merge process.
>
> 3. Prow currently tries to assign 2 code reviewers, and hopefully that's
> configurable.
>
> Still unsure:
> 1. How does Prow select reviewers? Does it load balance?
>
> On Mon, Jul 23, 2018 at 9:51 PM Jean-Baptiste Onofré 
> wrote:
>
>> It looks interesting but I would like to see the complete video and
>> explanation about prow. Especially what we concretely need.
>>
>> Regards
>> JB
>> Le 24 juil. 2018, à 04:17, Udi Meiri  a écrit:
>>>
>>> I was recently told about Prow
>>> , which
>>> automates testing and merging for Kubernetes and other projects.
>>> It also automates assigning reviewers and suggesting approvers. Example
>>>  PR, video
>>> explanation 
>>> I propose trying out Prow, since is it's a maintained and it uses OWNERS
>>> files to explicitly define both who should be reviewing and who should
>>> approve a PR.
>>>
>>> I'm not suggesting we use it to replace Jenkins or do our merges for us.
>>>
>>>
>>> On Tue, Jul 17, 2018 at 11:04 AM Udi Meiri  wrote:
>>>
 +1 to generating the file.
 I'll go ahead and file a PR to remove CODEOWNERS

 On Tue, Jul 17, 2018 at 9:28 AM Holden Karau 
 wrote:

> So it doesn’t support doing that right now, although if we find it’s a
> problem we can specify an exclude file with folks who haven’t contributed
> in the past year. Would people want me to generate that first?
>
> On Tue, Jul 17, 2018 at 10:22 AM Ismaël Mejía 
> wrote:
>
>> Is there a way to put inactive people as not reviewers for the blame
>> case? I think it can be useful considering that a good amount of our
>> committers are not active at the moment and auto-assigning reviews to
>> them seem like a waste of energy/time.
>> On Tue, Jul 17, 2018 at 1:58 AM Eugene Kirpichov <
>> kirpic...@google.com> wrote:
>> >
>> > We did not, but I think we should. So far, in 100% of the PRs I've
>> authored, the default functionality of CODEOWNERS did the wrong thing 
>> and I
>> had to fix something up manually.
>> >
>> > On Mon, Jul 16, 2018 at 3:42 PM Andrew Pilloud 
>> wrote:
>> >>
>> >> This sounds like a good plan. Did we want to rename the CODEOWNERS
>> file to disable github's mass adding of reviewers while we figure this 
>> out?
>> >>
>> >> Andrew
>> >>
>> >> On Mon, Jul 16, 2018 at 10:20 AM Jean-Baptiste Onofré <
>> j...@nanthrax.net> wrote:
>> >>>
>> >>> +1
>> >>>
>> >>> Le 16 juil. 2018, à 19:17, Holden Karau 
>> a écrit:
>> 
>>  Ok if no one objects I'll create the INFRA ticket after OSCON
>> and we can test it for a week and decide if it helps or hinders.
>> 
>>  On Mon, Jul 16, 2018, 7:12 PM Jean-Baptiste Onofré <
>> j...@nanthrax.net> wrote:
>> >
>> > Agree to test it for a week.
>> >
>> > Regards
>> > JB
>> > Le 16 juil. 2018, à 18:59, Holden Karau <
>> holden.ka...@gmail.com> a écrit:
>> >>
>> >> Would folks be OK with me asking infra to turn on blame based
>> suggestions for Beam and trying it out for a week?
>> >>
>> >> On Mon, Jul 16, 2018, 6:53 PM Rafael Fernandez <
>> rfern...@google.com> wrote:
>> >>>
>> >>> +1 using blame -- nifty :)
>> >>>
>> >>> On Mon, Jul 16, 2018 at 2:31 AM Huygaa Batsaikhan <
>> bat...@google.com> wrote:
>> 
>>  +1. This is great.
>> 
>>  On Sat, Jul 14, 2018 at 7:44 AM Udi Meiri < eh...@google.com>
>> wrote:
>> >
>> > Mention bot looks cool, as it tries to guess the reviewer
>>