Re: [DISCUSS] Use Confluence wiki for non-user-facing stuff

[Mikhail Gryzykhin]

Hello everyone it's Mikhail and I'm here to revive this long-sleeping
thread.

I have summarized discussion above into a design/proposal document

.

The initial proposal is what I consider the best approach, so it is open
for change.

Please comment on following topics:
1. Another engines you have in mind.
2. If you have access to configure corresponding engine
3. General ideas.

Since this is a long-desired change, please, be active.

--Mikhail

Have feedback ?


On Tue, Jun 12, 2018 at 5:24 PM Griselda Cuevas  wrote:

>
> Hi Everyone,
>
>
> (a) should we do it? -- I like the idea of having a wiki, yes. Mainly to
> differentiate the documentation we cater to users and the one we cater to
> contributors. For things like user examples, and more demo-y content, I'd
> suggest we still host it in the Website.
>
> (b) what should go there? -- The ultimate purpose of the wiki should be
> to host everything needed to a) get started (with official documentation)
> and b) how to get the most out of Beam (here is where I see things like
> what Robert suggested could fit, tips, tricks and other cool things created
> by and for our contributors.)
>
> (c) what should not go there? -- Any demos, examples or showcases. I think
> that material should be either embedded or linked (listed) in the website.
>
> Tu summarize, I'd like to see the wiki to be a knowledge collection for*
> people who contribute* to the project and the website the collection of
> information that allows *someone to make the decision to use Beam* (or
> join the community).
>
> When we are ready to vote on the creation of a wiki, I'd like to propose
> that the first thing we document there is the Beam Improvement plan along
> side with a concrete "Get Started Contributing to Beam" cheatsheet.
>
> WDYT?
>
>
> On Tue, 12 Jun 2018 at 09:34, Alexey Romanenko 
> wrote:
>
>> +1 for having Wiki for devs and users.
>>
>> Even though editing interface is not so native and obvious (comparing to
>> Google docs), but, at least, it will be already put in one place and should
>> be much more easy to search and discover.
>>
>> The only my concern about Wiki (based on using it in other different
>> projects) that, in course of time, the information becomes outdated and
>> weak structured which makes this not so valuable and even deceptive.
>>
>> WBR,
>> Alexey
>>
>> On 12 Jun 2018, at 18:01, Robert Bradshaw  wrote:
>>
>> On Mon, Jun 11, 2018 at 2:40 PM Kenneth Knowles  wrote:
>>
>>> OK, yea, that all makes sense to me. Like this?
>>>
>>>  - site/documentation: writing just for users
>>>  - site/contribute: basic stuff as-is, writing for users to entice them,
>>> links to the next...
>>>  - wiki/contributors: contributors writing just for each other
>>>
>>> And you also have
>>>
>>>  - wiki/users: users writing for users
>>>
>>> That's interesting.
>>>
>>
>> Yep. We don't have to start wiki/users right away, but it could be
>> useful down the line.
>>
>>
>>
>>> On Mon, Jun 11, 2018 at 2:30 PM Robert Bradshaw 
>>> wrote:
>>>
 On Fri, Jun 8, 2018 at 2:18 PM Kenneth Knowles  wrote:


> I disagree strongly here - I don't think the wiki will have
> appropriate polish for users. Even if carefully polished I don't think the
> presentation style is right, and it is not flexible. Power users will find
> it, of course.
>

 I wasn't imagining a wiki as a platform for developers to author
 documentation, rather a place for users to author content for other users
 (tips and tricks, handy PTransforms, etc.) at a much lower bar than
 expecting users to go in and update our documentation. I agree with the
 goal of not (further) fragmenting our documentation.

 As for mixing contributor vs. user information on the same site, I
 think it's valuable to have some integration and treat the two as a
 continuum (after all, our (direct) users are already developers) and
 consider it an asset to have a "contribute" heading right in the main site.
 (Perhaps, if it's confusing, we could move it all the way to the right.) I
 don't think we'll be doing ourselves a favor by blinding copying all the
 existing docs to a wiki. That being said I think it makes sense to start
 playing with using a wiki, and see how much value that adds on top of what
 we already have.


>
>
>> On Fri, Jun 8, 2018 at 12:05 PM Thomas Weise  wrote:
>>
>>> +1 most of the contributor material could live on Wiki and there it
>>> will be easier to maintain (perhaps the lower bar for updates will lead 
>>> to
>>> more information and increased maintenance). Contribution policy related
>>> material should remain on the website and go through proper
>>> review/versioning.
>>>
>>>
>>> On Fri, Jun 8, 2018 at 11:44 AM, Udi Meiri  wrote:
>>>
 (a) Yes.

Re: Permissions for confluence

[Mikhail Gryzykhin]

I suggest to close this thread for now. With that being said, I'll do
following:

1. I'll prepare documentation in .md format and stash it in temporary
folder in code branch to make it available.
2. I'll revive original thread.

--Mikhail

Have feedback ?


On Fri, Jul 13, 2018 at 4:41 PM Anton Kedin  wrote:

> I may be mistaken, but there was no final conclusion reached, so probably
> guidance from PMC will be needed where specifically to put things. I
> personally think that this kind of documentation is a right thing to put
> under cwiki/contributors.
>
> From the last thread I think Kenn, Jean-Baptiste (j...@nanthrax.net), and
> Daniel (dk...@apache.org) had permissions.
>
> On Fri, Jul 13, 2018 at 4:28 PM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I believe that last month we decided to keep developers documentation on
>> Apache confluence wiki (https://cwiki.apache.org/confluence/display/BEAM
>> ).
>>
>> I have some documentation regarding spinning up Jenkins via Docker.
>>
>> Can you help me with following:
>> 1) Is confluence a correct place to put this documentation?
>> 2) How do I get edit permissions for confluence?
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Proof-of-concept Beam PR dashboard (based off of Spark's PR dashboard) to improve discoverability

[Holden Karau]

Took me waaay longer than planed, and the regexes and components could use
some work, but I've got a quick Beam PR dashboard up at
https://boos-demo-projects-are-rad.appspot.com/. The code is a fork of the
Spark one, and its at
https://github.com/holdenk/spark-pr-dashboard/tree/support-beam in the beam
support branch. I don't how useful this will be for folks, but given the
discussion going on around CODEOWNERS I figured people were feeling the
pain of trying to keep on top of reviews.

I'm still working on trying to get mentionbot working (its being a bit
frustrating to upgrade to recent version of dependencies as a non-JS
programmer), but hopefully I can do something there too.

If anyone has thoughts about what good tags would be for the review
dashboard let me know, I just kicked it off with some tabs which I
personally care about.

Twitter: https://twitter.com/holdenkarau

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Ahmet Altay]

Update:  https://issues.apache.org/jira/browse/BEAM-4784 is not a release
blocker, details in the JIRA issue.

On Fri, Jul 13, 2018 at 11:12 AM, Thomas Weise  wrote:

> Can one of our Python experts please take a look at
> https://issues.apache.org/jira/browse/BEAM-4784 and advise if this should
> be addressed for the release?
>
> Thanks,
> Thomas
>
>
> On Fri, Jul 13, 2018 at 11:02 AM Ahmet Altay  wrote:
>
>>
>>
>> On Fri, Jul 13, 2018 at 10:48 AM, Pablo Estrada 
>> wrote:
>>
>>> Hi all,
>>> I've triaged most issues marked for 2.6.0 release. I've localized two
>>> that need a decision / attention:
>>>
>>> - https://issues.apache.org/jira/browse/BEAM-4417 - Bigquery IO Numeric
>>> Datatype Support. Cham is not available to fix this at the moment, but this
>>> is a critical issue. Is anyone able to tackle this / should we bump this to
>>> next release?
>>>
>>
>> I bumped this to the next release. I think Cham will be the best person
>> to address it when he is back. And with the regular release cadence, it
>> would not be delayed by much.
>>
>>
>>>
>>> - https://issues.apache.org/jira/browse/BEAM-4750 - Performance
>>> degradation due to some safeguards in beam-sdks-java-core. JB, are you
>>> looking to fix this? Should we bump? I had the impression that it was an
>>> easy fix, but I'm not sure.
>>>
>>> If you're aware of any other issue that needs to be included as a
>>> release blocker, please report it to me.
>>> Best
>>> -P.
>>>
>>> On Thu, Jul 12, 2018 at 2:15 AM Etienne Chauchot 
>>> wrote:
>>>
 +1,

 Thanks for volunteering Pablo, thanks also to have caught tickets that
 I forgot to close :)

 Etienne

 Le mercredi 11 juillet 2018 à 12:55 -0700, Alan Myrvold a écrit :

 +1 Thanks for volunteering, Pablo

 On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
 wrote:

 +1 sounds great

 On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:

 +1

 Thanks for volunteering, Pablo!

 On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
 wrote:

 +1

 I planned to send the proposal as well ;)

 Regards
 JB

 On 09/07/2018 23:16, Pablo Estrada wrote:
 > Hello everyone!
 >
 > As per the previously agreed-upon schedule for Beam releases, the
 > process for the 2.6.0 Beam release should start on July 17th.
 >
 > I volunteer to perform this release.
 >
 > Here is the schedule that I have in mind:
 >
 > - We start triaging JIRA issues this week.
 > - I will cut a release branch on July 17.
 > - After July 17, any blockers will need to be cherry-picked into the
 > release branch.
 > - As soon as tests look good, and blockers have been addressed, I will
 > perform the other release tasks.
 >
 > Does that seem reasonable to the community?
 >
 > Best
 > -P.
 > --
 > Got feedback? go/pabloem-feedback
 

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com



 --
>>> Got feedback? go/pabloem-feedback
>>> 
>>>
>>
>>

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

Mention bot looks cool, as it tries to guess the reviewer using blame.
I've written a quick and dirty script that uses only CODEOWNERS.

Its output looks like:
$ python suggest_reviewers.py --pr 5940
INFO:root:Selected reviewer @lukecwik for:
/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
(path_pattern: /runners/core-construction-java*)
INFO:root:Selected reviewer @lukecwik for:
/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
(path_pattern: /runners/core-construction-java*)
INFO:root:Selected reviewer @echauchot for:
/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
(path_pattern: /runners/core-java*)
INFO:root:Selected reviewer @lukecwik for: /runners/flink/build.gradle
(path_pattern: */build.gradle*)
INFO:root:Selected reviewer @lukecwik for:
/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
(path_pattern: *.java)
INFO:root:Selected reviewer @pabloem for:
/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
(path_pattern: /runners/google-cloud-dataflow-java*)
INFO:root:Selected reviewer @lukecwik for:
/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
(path_pattern: /sdks/java/core*)
Suggested reviewers: @echauchot, @lukecwik, @pabloem

Script is in: https://github.com/apache/beam/pull/5951


What does the community think? Do you prefer blame-based or rules-based
reviewer suggestions?

On Fri, Jul 13, 2018 at 11:13 AM Holden Karau  wrote:

> I'm looking at something similar in the Spark project, and while it's now
> archived by FB it seems like something like
> https://github.com/facebookarchive/mention-bot might do what we want. I'm
> going to spin up a version on my K8 cluster and see if I can ask infra to
> add a webhook and if it works for Spark we could ask INFRA to add a second
> webhook for Beam. (Or if the Beam folks are more interested in
> experimenting I can do Beam first as a smaller project and roll with that).
>
> Let me know :)
>
> On Fri, Jul 13, 2018 at 10:53 AM, Eugene Kirpichov 
> wrote:
>
>> Sounds reasonable for now, thanks!
>> It's unfortunate that Github's CODEOWNERS feature appears to be
>> effectively unusable for Beam but I'd hope that Github might pay attention
>> and fix things if we submit feedback, with us being one of the most active
>> Apache projects - did anyone do this yet / planning to?
>>
>> On Fri, Jul 13, 2018 at 10:23 AM Udi Meiri  wrote:
>>
>>> While I like the idea of having a CODEOWNERS file, the Github
>>> implementation is lacking:
>>> 1. Reviewers are automatically assigned at each push.
>>> 2. Reviewer assignment can be excessive (e.g. 5 reviewers in Eugene's PR
>>> 5940).
>>> 3. Non-committers aren't assigned as reviewers.
>>> 4. Non-committers can't change the list of reviewers.
>>>
>>> I propose renaming the file to disable the auto-reviewer assignment
>>> feature.
>>> In its place I'll add a script that suggests reviewers.
>>>
>>> On Fri, Jul 13, 2018 at 9:09 AM Udi Meiri  wrote:
>>>
 Hi Etienne,

 Yes you could be as precise as you want. The paths I listed are just
 suggestions. :)


 On Fri, Jul 13, 2018 at 1:12 AM Jean-Baptiste Onofré 
 wrote:

> Hi,
>
> I think it's already do-able just providing the expected path.
>
> It's a good idea especially for the core.
>
> Regards
> JB
>
> On 13/07/2018 09:51, Etienne Chauchot wrote:
> > Hi Udi,
> >
> > I also have a question, related to what Eugene asked : I see that the
> > code paths are the ones of the modules. Can we be more precise than
> that
> > to assign reviewers ? As an example, I added myself to runner/core
> > because I wanted to take a look at the PRs related to
> > runner/core/metrics but I'm getting assigned to all runner-core PRs.
> Can
> > we specify paths like
> > runners/core-java/src/main/java/org/apache/beam/runners/core/metrics
> ?
> > I know it is a bit too precise so a bit risky, but in that particular
> > case, I doubt that the path will change.
> >
> > Etienne
> >
> > Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
> >> Hi Udi,
> >>
> >> I see that the PR was merged - thanks! However it seems to have some
> >> unintended effects.
> >>
> >> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
> >> reviewer manually, but the moment I pushed a new commit, it
> >> auto-assigned a lot of other people to it, and I had to remove them.
> >> This seems like a big inconvenience to me, is there a way to
> disable this?
> >>
> >> Thanks.
> >>
> >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri  >> > wrote:
> >>> :/ That makes it a 

Re: Permissions for confluence

[Anton Kedin]

I may be mistaken, but there was no final conclusion reached, so probably
guidance from PMC will be needed where specifically to put things. I
personally think that this kind of documentation is a right thing to put
under cwiki/contributors.

>From the last thread I think Kenn, Jean-Baptiste (j...@nanthrax.net), and
Daniel (dk...@apache.org) had permissions.

On Fri, Jul 13, 2018 at 4:28 PM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> I believe that last month we decided to keep developers documentation on
> Apache confluence wiki (https://cwiki.apache.org/confluence/display/BEAM).
>
> I have some documentation regarding spinning up Jenkins via Docker.
>
> Can you help me with following:
> 1) Is confluence a correct place to put this documentation?
> 2) How do I get edit permissions for confluence?
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Re: Permissions for confluence

[Kai Jiang]

Same question. I think only committers have permission to edit page. Also,
looking forward to working on some pages.

On Fri, Jul 13, 2018, 16:28 Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> I believe that last month we decided to keep developers documentation on
> Apache confluence wiki (https://cwiki.apache.org/confluence/display/BEAM).
>
> I have some documentation regarding spinning up Jenkins via Docker.
>
> Can you help me with following:
> 1) Is confluence a correct place to put this documentation?
> 2) How do I get edit permissions for confluence?
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Permissions for confluence

[Mikhail Gryzykhin]

Hi everyone,

I believe that last month we decided to keep developers documentation on
Apache confluence wiki (https://cwiki.apache.org/confluence/display/BEAM).

I have some documentation regarding spinning up Jenkins via Docker.

Can you help me with following:
1) Is confluence a correct place to put this documentation?
2) How do I get edit permissions for confluence?

Regards,
--Mikhail

Have feedback ?

Re: Live coding & reviewing adventures

[Holden Karau]

Thats a great idea! I did something like that earlier focused on the Go SDK
only (see - https://www.youtube.com/watch?v=g0Iq4np-WVk &
https://www.youtube.com/watch?v=P4jIfhPTKQo ), and I'll try and do some
more general ones later on as we get more stuff working on the portability
framework with Flink :)

On Fri, Jul 13, 2018 at 12:39 PM, Ankur Goenka  wrote:

> Thanks Holden for doing this.
> Looking forward to attend the live session.
> Suggestion: It will super useful to do a live session for beam setup for
> users and another for contributors.
>
>
> On Fri, Jul 13, 2018 at 12:33 PM Innocent Djiofack 
> wrote:
>
>> Thanks I think this will be super useful. I will tune in.
>>
>> On Fri, Jul 13, 2018 at 2:54 PM Holden Karau 
>> wrote:
>>
>>> Hi folks! I've been doing some live coding in my other projects and I
>>> figured I'd do some with Apache Beam as well.
>>>
>>> Today @ 3pm pacific I'm going be doing some impromptu exploration better
>>> review tooling possibilities (looking at forking spark-pr-dashboard for
>>> other projects like beam and setting up mentionbot to work with ASF infra)
>>> - https://www.youtube.com/watch?v=ff8_jbzC8JI
>>>
>>> Next week (Thursday the 19th at 2pm pacific) I'm going to be working on
>>> trying to get easier dependency management for the Python portable runner
>>> in place - https://www.youtube.com/watch?v=Sv0XhS2pYqA
>>>
>>> If your interested in seeing more of the development process I hope you
>>> will join me :)
>>>
>>> P.S.
>>>
>>> You can also follow on twitch which does a better job of notifications
>>> https://www.twitch.tv/holdenkarau
>>>
>>> Also one of the other thing I do is "live reviews" of PRs but they are
>>> generally opt-in and I don't have enough opt-ins from the Beam community to
>>> do live reviews in Beam, if you work on Beam and would be OK with me doing
>>> a live streamed review of your PRs let me know (if your curious to what
>>> they look like you can see some of them here in Spark land
>>> 
>>> ).
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>> --
>>
>> *DJIOFACK INNOCENT*
>> *"Be better than the day before!" -*
>> *+1 404 751 8024*
>>
>


-- 
Twitter: https://twitter.com/holdenkarau

Re: Live coding & reviewing adventures

[Ankur Goenka]

Thanks Holden for doing this.
Looking forward to attend the live session.
Suggestion: It will super useful to do a live session for beam setup for
users and another for contributors.


On Fri, Jul 13, 2018 at 12:33 PM Innocent Djiofack 
wrote:

> Thanks I think this will be super useful. I will tune in.
>
> On Fri, Jul 13, 2018 at 2:54 PM Holden Karau  wrote:
>
>> Hi folks! I've been doing some live coding in my other projects and I
>> figured I'd do some with Apache Beam as well.
>>
>> Today @ 3pm pacific I'm going be doing some impromptu exploration better
>> review tooling possibilities (looking at forking spark-pr-dashboard for
>> other projects like beam and setting up mentionbot to work with ASF infra)
>> - https://www.youtube.com/watch?v=ff8_jbzC8JI
>>
>> Next week (Thursday the 19th at 2pm pacific) I'm going to be working on
>> trying to get easier dependency management for the Python portable runner
>> in place - https://www.youtube.com/watch?v=Sv0XhS2pYqA
>>
>> If your interested in seeing more of the development process I hope you
>> will join me :)
>>
>> P.S.
>>
>> You can also follow on twitch which does a better job of notifications
>> https://www.twitch.tv/holdenkarau
>>
>> Also one of the other thing I do is "live reviews" of PRs but they are
>> generally opt-in and I don't have enough opt-ins from the Beam community to
>> do live reviews in Beam, if you work on Beam and would be OK with me doing
>> a live streamed review of your PRs let me know (if your curious to what
>> they look like you can see some of them here in Spark land
>> 
>> ).
>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>>
> --
>
> *DJIOFACK INNOCENT*
> *"Be better than the day before!" -*
> *+1 404 751 8024*
>

Re: Live coding & reviewing adventures

[Innocent Djiofack]

Thanks I think this will be super useful. I will tune in.

On Fri, Jul 13, 2018 at 2:54 PM Holden Karau  wrote:

> Hi folks! I've been doing some live coding in my other projects and I
> figured I'd do some with Apache Beam as well.
>
> Today @ 3pm pacific I'm going be doing some impromptu exploration better
> review tooling possibilities (looking at forking spark-pr-dashboard for
> other projects like beam and setting up mentionbot to work with ASF infra)
> - https://www.youtube.com/watch?v=ff8_jbzC8JI
>
> Next week (Thursday the 19th at 2pm pacific) I'm going to be working on
> trying to get easier dependency management for the Python portable runner
> in place - https://www.youtube.com/watch?v=Sv0XhS2pYqA
>
> If your interested in seeing more of the development process I hope you
> will join me :)
>
> P.S.
>
> You can also follow on twitch which does a better job of notifications
> https://www.twitch.tv/holdenkarau
>
> Also one of the other thing I do is "live reviews" of PRs but they are
> generally opt-in and I don't have enough opt-ins from the Beam community to
> do live reviews in Beam, if you work on Beam and would be OK with me doing
> a live streamed review of your PRs let me know (if your curious to what
> they look like you can see some of them here in Spark land
> 
> ).
>
>
> --
> Twitter: https://twitter.com/holdenkarau
>
-- 

*DJIOFACK INNOCENT*
*"Be better than the day before!" -*
*+1 404 751 8024*

Live coding & reviewing adventures

[Holden Karau]

Hi folks! I've been doing some live coding in my other projects and I
figured I'd do some with Apache Beam as well.

Today @ 3pm pacific I'm going be doing some impromptu exploration better
review tooling possibilities (looking at forking spark-pr-dashboard for
other projects like beam and setting up mentionbot to work with ASF infra)
- https://www.youtube.com/watch?v=ff8_jbzC8JI

Next week (Thursday the 19th at 2pm pacific) I'm going to be working on
trying to get easier dependency management for the Python portable runner
in place - https://www.youtube.com/watch?v=Sv0XhS2pYqA

If your interested in seeing more of the development process I hope you
will join me :)

P.S.

You can also follow on twitch which does a better job of notifications
https://www.twitch.tv/holdenkarau

Also one of the other thing I do is "live reviews" of PRs but they are
generally opt-in and I don't have enough opt-ins from the Beam community to
do live reviews in Beam, if you work on Beam and would be OK with me doing
a live streamed review of your PRs let me know (if your curious to what
they look like you can see some of them here in Spark land

).

-- 
Twitter: https://twitter.com/holdenkarau

Re: CODEOWNERS for apache/beam repo

[Holden Karau]

I'm looking at something similar in the Spark project, and while it's now
archived by FB it seems like something like
https://github.com/facebookarchive/mention-bot might do what we want. I'm
going to spin up a version on my K8 cluster and see if I can ask infra to
add a webhook and if it works for Spark we could ask INFRA to add a second
webhook for Beam. (Or if the Beam folks are more interested in
experimenting I can do Beam first as a smaller project and roll with that).

Let me know :)

On Fri, Jul 13, 2018 at 10:53 AM, Eugene Kirpichov 
wrote:

> Sounds reasonable for now, thanks!
> It's unfortunate that Github's CODEOWNERS feature appears to be
> effectively unusable for Beam but I'd hope that Github might pay attention
> and fix things if we submit feedback, with us being one of the most active
> Apache projects - did anyone do this yet / planning to?
>
> On Fri, Jul 13, 2018 at 10:23 AM Udi Meiri  wrote:
>
>> While I like the idea of having a CODEOWNERS file, the Github
>> implementation is lacking:
>> 1. Reviewers are automatically assigned at each push.
>> 2. Reviewer assignment can be excessive (e.g. 5 reviewers in Eugene's PR
>> 5940).
>> 3. Non-committers aren't assigned as reviewers.
>> 4. Non-committers can't change the list of reviewers.
>>
>> I propose renaming the file to disable the auto-reviewer assignment
>> feature.
>> In its place I'll add a script that suggests reviewers.
>>
>> On Fri, Jul 13, 2018 at 9:09 AM Udi Meiri  wrote:
>>
>>> Hi Etienne,
>>>
>>> Yes you could be as precise as you want. The paths I listed are just
>>> suggestions. :)
>>>
>>>
>>> On Fri, Jul 13, 2018 at 1:12 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Hi,

 I think it's already do-able just providing the expected path.

 It's a good idea especially for the core.

 Regards
 JB

 On 13/07/2018 09:51, Etienne Chauchot wrote:
 > Hi Udi,
 >
 > I also have a question, related to what Eugene asked : I see that the
 > code paths are the ones of the modules. Can we be more precise than
 that
 > to assign reviewers ? As an example, I added myself to runner/core
 > because I wanted to take a look at the PRs related to
 > runner/core/metrics but I'm getting assigned to all runner-core PRs.
 Can
 > we specify paths like
 > runners/core-java/src/main/java/org/apache/beam/runners/core/metrics
 ?
 > I know it is a bit too precise so a bit risky, but in that particular
 > case, I doubt that the path will change.
 >
 > Etienne
 >
 > Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
 >> Hi Udi,
 >>
 >> I see that the PR was merged - thanks! However it seems to have some
 >> unintended effects.
 >>
 >> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
 >> reviewer manually, but the moment I pushed a new commit, it
 >> auto-assigned a lot of other people to it, and I had to remove them.
 >> This seems like a big inconvenience to me, is there a way to disable
 this?
 >>
 >> Thanks.
 >>
 >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri >>> >> > wrote:
 >>> :/ That makes it a little less useful.
 >>>
 >>> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson
 >>> mailto:timrobertson...@gmail.com>>
 wrote:
  Hi Udi
 
  I asked the GH helpdesk and they confirmed that only people with
  write access will actually be automatically chosen.
 
  It don't expect it should stop us using it, but we should be aware
  that there are non-committers also willing to review.
 
  Thanks,
  Tim
 
  On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin
  mailto:mig...@google.com>> wrote:
 > Idea looks good in general.
 >
 > Did you look into ways to keep this file up-to-date? For example
 we
 > can run monthly job to see if owner was active during this period.
 >
 > --Mikhail
 >
 > Have feedback ?
 >
 >
 > On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri >>> > > wrote:
 >> Thanks all!
 >> I'll try to get the file merged today and see how it works out.
 >> Please surface any issues, such as with auto-assignment, here or
 >> in JIRA.
 >>
 >> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot
 >> mailto:echauc...@apache.org>> wrote:
 >>> Hi,
 >>>
 >>> I added myself as a reviewer for some modules.
 >>>
 >>> Etienne
 >>>
 >>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
  Hi everyone,
 
  I'm proposing to add auto-reviewer-assignment using Github's
  CODEOWNERS mechanism.
  Initial version is
  

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Thomas Weise]

Can one of our Python experts please take a look at
https://issues.apache.org/jira/browse/BEAM-4784 and advise if this should
be addressed for the release?

Thanks,
Thomas


On Fri, Jul 13, 2018 at 11:02 AM Ahmet Altay  wrote:

>
>
> On Fri, Jul 13, 2018 at 10:48 AM, Pablo Estrada 
> wrote:
>
>> Hi all,
>> I've triaged most issues marked for 2.6.0 release. I've localized two
>> that need a decision / attention:
>>
>> - https://issues.apache.org/jira/browse/BEAM-4417 - Bigquery IO Numeric
>> Datatype Support. Cham is not available to fix this at the moment, but this
>> is a critical issue. Is anyone able to tackle this / should we bump this to
>> next release?
>>
>
> I bumped this to the next release. I think Cham will be the best person to
> address it when he is back. And with the regular release cadence, it would
> not be delayed by much.
>
>
>>
>> - https://issues.apache.org/jira/browse/BEAM-4750 - Performance
>> degradation due to some safeguards in beam-sdks-java-core. JB, are you
>> looking to fix this? Should we bump? I had the impression that it was an
>> easy fix, but I'm not sure.
>>
>> If you're aware of any other issue that needs to be included as a release
>> blocker, please report it to me.
>> Best
>> -P.
>>
>> On Thu, Jul 12, 2018 at 2:15 AM Etienne Chauchot 
>> wrote:
>>
>>> +1,
>>>
>>> Thanks for volunteering Pablo, thanks also to have caught tickets that I
>>> forgot to close :)
>>>
>>> Etienne
>>>
>>> Le mercredi 11 juillet 2018 à 12:55 -0700, Alan Myrvold a écrit :
>>>
>>> +1 Thanks for volunteering, Pablo
>>>
>>> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
>>> wrote:
>>>
>>> +1 sounds great
>>>
>>> On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
>>>
>>> +1
>>>
>>> Thanks for volunteering, Pablo!
>>>
>>> On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
>>> wrote:
>>>
>>> +1
>>>
>>> I planned to send the proposal as well ;)
>>>
>>> Regards
>>> JB
>>>
>>> On 09/07/2018 23:16, Pablo Estrada wrote:
>>> > Hello everyone!
>>> >
>>> > As per the previously agreed-upon schedule for Beam releases, the
>>> > process for the 2.6.0 Beam release should start on July 17th.
>>> >
>>> > I volunteer to perform this release.
>>> >
>>> > Here is the schedule that I have in mind:
>>> >
>>> > - We start triaging JIRA issues this week.
>>> > - I will cut a release branch on July 17.
>>> > - After July 17, any blockers will need to be cherry-picked into the
>>> > release branch.
>>> > - As soon as tests look good, and blockers have been addressed, I will
>>> > perform the other release tasks.
>>> >
>>> > Does that seem reasonable to the community?
>>> >
>>> > Best
>>> > -P.
>>> > --
>>> > Got feedback? go/pabloem-feedback
>>> 
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>>
>>> --
>> Got feedback? go/pabloem-feedback
>> 
>>
>
>

Re: Automatically create JIRA tickets for failing post-commit tests

[Ahmet Altay]

This proposal looks good to me.

On Fri, Jul 13, 2018 at 10:49 AM, Jason Kuster 
wrote:

> I like the idea of auto-creating bugs. I think that if we do this we
> should make sure to set a time period after which we will evaluate whether
> it has succeeded, i.e. whether bugs filed as a result of this have been
> triaged, owned, fixed, and closed.
>
> On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin 
> wrote:
>
>> Can we get some PMC opinions on this?
>>
>> According to https://www.apache.org/dev/infra-contact#request-checklist,
>> it is highly recommended to have your opinions when filing a request.
>>
>
Do you need a specific comment or a LGTM is good enough?


>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:
>>
>>> I think this looks good, we should enable the plugin and try it out.
>>> Concrete details of the follow-up tasks (auto-assignment, triage, and
>>> dashboarding) will probably depend on how functional the plugin is and what
>>> the test failures data looks like.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 @Yifan Zou 

 I believe that we should test-drive the system with tickets + PR first
 and decide on email notification later. We already have tests failure
 emails sent to commits@, I believe most people filter out or not
 signed up for that list though.

 It creates only one ticket, and keeps it for recurring test failures.

 @Andrew Pilloud 
 Thank you for the suggestion. I'll add it to design doc.

 --Mikhail



 On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:

> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
> ticket being ignored and no actions being taken.
>
> I can see the challenges on ticket assignment. Like Mikhail mentioned,
> the plugin does not support dynamic assignments. We have to implement
> custom script to determine the assignees and do some tricks to the jenkins
> job. Also, the post-commits tests usually cover tons of stuffs that it is
> difficult to find which part was broken and ask the right person to look
> into within the Auto JIRA process. Some naive thoughts: Are we able to 
> send
> emails to the dev@ to ask people to take care of the JIRA issues? Are
> we able to find component leads and ask them triage the test failure
> tickets?
>
> Another nitpick comment. Does the jenkins job file the JIRA issue in
> every test failure? Sometimes the test continuously fails in a time period
> due to the same reason. In this case, we will get some duplicate issues
> filed by Jenkins. I think it could be better if we can avoid filing issues
> if the previous one has not been resolved.
>
> Thanks.
> Yifan
>
>
> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
> wrote:
>
>> That sounds great. You should add this detail to the doc.
>>
>> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> We already have component for this purpose: "test-failures". All
>>> tickets created will go to that component. As an option, we can add 
>>> link to
>>> view list of open JIRA tickets to PR template.
>>>
>>> We also would want to create graph on dashboard with amount of
>>> unassigned and assigned bugs.
>>>
>>> I believe that we can also add counter of unassigned bugs to PR
>>> template. This way it will be easier for everyone to know when there's 
>>> some
>>> tests issue not attended.
>>>
>>> --Mikhail
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>>> wrote:
>>>
 So it sounds like you will want to create a component for untriaged
 issues so they are easy to find. I like the idea of distributing the 
 work
 of triaging post commit failures to new PR authors as a condition of
 merging. I feel like we will just be filling JIRA with spam if the 
 issues
 are automatically created without a plan for triage.

 Andrew

 On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:

> Maybe this is also a good thread to start the discussion that if
> we want to enforce postcommit test for every PR.
>
> Can we afford the cost of longer waiting time to catch potential
> bugs?
>
> -Rui
>
> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <
> mig...@google.com> wrote:
>
>> That's a valid point.
>>
>> Unfortunately, the JiraTestResultReporter plugin does not have
>> features to dynamically assign owners. Additionally, I don't think 
>> it is
>> always easy to find proper owner for post-commit tests at first 
>> glance,

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Ahmet Altay]

On Fri, Jul 13, 2018 at 10:48 AM, Pablo Estrada  wrote:

> Hi all,
> I've triaged most issues marked for 2.6.0 release. I've localized two that
> need a decision / attention:
>
> - https://issues.apache.org/jira/browse/BEAM-4417 - Bigquery IO Numeric
> Datatype Support. Cham is not available to fix this at the moment, but this
> is a critical issue. Is anyone able to tackle this / should we bump this to
> next release?
>

I bumped this to the next release. I think Cham will be the best person to
address it when he is back. And with the regular release cadence, it would
not be delayed by much.


>
> - https://issues.apache.org/jira/browse/BEAM-4750 - Performance
> degradation due to some safeguards in beam-sdks-java-core. JB, are you
> looking to fix this? Should we bump? I had the impression that it was an
> easy fix, but I'm not sure.
>
> If you're aware of any other issue that needs to be included as a release
> blocker, please report it to me.
> Best
> -P.
>
> On Thu, Jul 12, 2018 at 2:15 AM Etienne Chauchot 
> wrote:
>
>> +1,
>>
>> Thanks for volunteering Pablo, thanks also to have caught tickets that I
>> forgot to close :)
>>
>> Etienne
>>
>> Le mercredi 11 juillet 2018 à 12:55 -0700, Alan Myrvold a écrit :
>>
>> +1 Thanks for volunteering, Pablo
>>
>> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
>> wrote:
>>
>> +1 sounds great
>>
>> On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
>>
>> +1
>>
>> Thanks for volunteering, Pablo!
>>
>> On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
>> wrote:
>>
>> +1
>>
>> I planned to send the proposal as well ;)
>>
>> Regards
>> JB
>>
>> On 09/07/2018 23:16, Pablo Estrada wrote:
>> > Hello everyone!
>> >
>> > As per the previously agreed-upon schedule for Beam releases, the
>> > process for the 2.6.0 Beam release should start on July 17th.
>> >
>> > I volunteer to perform this release.
>> >
>> > Here is the schedule that I have in mind:
>> >
>> > - We start triaging JIRA issues this week.
>> > - I will cut a release branch on July 17.
>> > - After July 17, any blockers will need to be cherry-picked into the
>> > release branch.
>> > - As soon as tests look good, and blockers have been addressed, I will
>> > perform the other release tasks.
>> >
>> > Does that seem reasonable to the community?
>> >
>> > Best
>> > -P.
>> > --
>> > Got feedback? go/pabloem-feedback
>> 
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>
>>
>> --
> Got feedback? go/pabloem-feedback
> 
>

Re: CODEOWNERS for apache/beam repo

[Eugene Kirpichov]

Sounds reasonable for now, thanks!
It's unfortunate that Github's CODEOWNERS feature appears to be effectively
unusable for Beam but I'd hope that Github might pay attention and fix
things if we submit feedback, with us being one of the most active Apache
projects - did anyone do this yet / planning to?

On Fri, Jul 13, 2018 at 10:23 AM Udi Meiri  wrote:

> While I like the idea of having a CODEOWNERS file, the Github
> implementation is lacking:
> 1. Reviewers are automatically assigned at each push.
> 2. Reviewer assignment can be excessive (e.g. 5 reviewers in Eugene's PR
> 5940).
> 3. Non-committers aren't assigned as reviewers.
> 4. Non-committers can't change the list of reviewers.
>
> I propose renaming the file to disable the auto-reviewer assignment
> feature.
> In its place I'll add a script that suggests reviewers.
>
> On Fri, Jul 13, 2018 at 9:09 AM Udi Meiri  wrote:
>
>> Hi Etienne,
>>
>> Yes you could be as precise as you want. The paths I listed are just
>> suggestions. :)
>>
>>
>> On Fri, Jul 13, 2018 at 1:12 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi,
>>>
>>> I think it's already do-able just providing the expected path.
>>>
>>> It's a good idea especially for the core.
>>>
>>> Regards
>>> JB
>>>
>>> On 13/07/2018 09:51, Etienne Chauchot wrote:
>>> > Hi Udi,
>>> >
>>> > I also have a question, related to what Eugene asked : I see that the
>>> > code paths are the ones of the modules. Can we be more precise than
>>> that
>>> > to assign reviewers ? As an example, I added myself to runner/core
>>> > because I wanted to take a look at the PRs related to
>>> > runner/core/metrics but I'm getting assigned to all runner-core PRs.
>>> Can
>>> > we specify paths like
>>> > runners/core-java/src/main/java/org/apache/beam/runners/core/metrics ?
>>> > I know it is a bit too precise so a bit risky, but in that particular
>>> > case, I doubt that the path will change.
>>> >
>>> > Etienne
>>> >
>>> > Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
>>> >> Hi Udi,
>>> >>
>>> >> I see that the PR was merged - thanks! However it seems to have some
>>> >> unintended effects.
>>> >>
>>> >> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
>>> >> reviewer manually, but the moment I pushed a new commit, it
>>> >> auto-assigned a lot of other people to it, and I had to remove them.
>>> >> This seems like a big inconvenience to me, is there a way to disable
>>> this?
>>> >>
>>> >> Thanks.
>>> >>
>>> >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri >> >> > wrote:
>>> >>> :/ That makes it a little less useful.
>>> >>>
>>> >>> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson
>>> >>> mailto:timrobertson...@gmail.com>>
>>> wrote:
>>>  Hi Udi
>>> 
>>>  I asked the GH helpdesk and they confirmed that only people with
>>>  write access will actually be automatically chosen.
>>> 
>>>  It don't expect it should stop us using it, but we should be aware
>>>  that there are non-committers also willing to review.
>>> 
>>>  Thanks,
>>>  Tim
>>> 
>>>  On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin
>>>  mailto:mig...@google.com>> wrote:
>>> > Idea looks good in general.
>>> >
>>> > Did you look into ways to keep this file up-to-date? For example we
>>> > can run monthly job to see if owner was active during this period.
>>> >
>>> > --Mikhail
>>> >
>>> > Have feedback ?
>>> >
>>> >
>>> > On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri >> > > wrote:
>>> >> Thanks all!
>>> >> I'll try to get the file merged today and see how it works out.
>>> >> Please surface any issues, such as with auto-assignment, here or
>>> >> in JIRA.
>>> >>
>>> >> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot
>>> >> mailto:echauc...@apache.org>> wrote:
>>> >>> Hi,
>>> >>>
>>> >>> I added myself as a reviewer for some modules.
>>> >>>
>>> >>> Etienne
>>> >>>
>>> >>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>>>  Hi everyone,
>>> 
>>>  I'm proposing to add auto-reviewer-assignment using Github's
>>>  CODEOWNERS mechanism.
>>>  Initial version is
>>>  here: _https://github.com/apache/beam/pull/5909/files_
>>> 
>>>  I need help from the community in determining owners for each
>>>  component.
>>>  Feel free to directly edit the PR (if you have permission) or
>>>  add a comment.
>>> 
>>> 
>>>  Background
>>>  The idea is to:
>>>  1. Document good review candidates for each component.
>>>  2. Help choose reviewers using the auto-assignment mechanism.
>>>  The suggestion is in no way binding.
>>> 
>>> 
>>> 
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - 

Re: Automatically create JIRA tickets for failing post-commit tests

[Jason Kuster]

I like the idea of auto-creating bugs. I think that if we do this we should
make sure to set a time period after which we will evaluate whether it has
succeeded, i.e. whether bugs filed as a result of this have been triaged,
owned, fixed, and closed.

On Fri, Jul 13, 2018 at 10:20 AM Mikhail Gryzykhin 
wrote:

> Can we get some PMC opinions on this?
>
> According to https://www.apache.org/dev/infra-contact#request-checklist,
> it is highly recommended to have your opinions when filing a request.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:
>
>> I think this looks good, we should enable the plugin and try it out.
>> Concrete details of the follow-up tasks (auto-assignment, triage, and
>> dashboarding) will probably depend on how functional the plugin is and what
>> the test failures data looks like.
>>
>> Regards,
>> Anton
>>
>> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> @Yifan Zou 
>>>
>>> I believe that we should test-drive the system with tickets + PR first
>>> and decide on email notification later. We already have tests failure
>>> emails sent to commits@, I believe most people filter out or not signed
>>> up for that list though.
>>>
>>> It creates only one ticket, and keeps it for recurring test failures.
>>>
>>> @Andrew Pilloud 
>>> Thank you for the suggestion. I'll add it to design doc.
>>>
>>> --Mikhail
>>>
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:
>>>
 +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
 ticket being ignored and no actions being taken.

 I can see the challenges on ticket assignment. Like Mikhail mentioned,
 the plugin does not support dynamic assignments. We have to implement
 custom script to determine the assignees and do some tricks to the jenkins
 job. Also, the post-commits tests usually cover tons of stuffs that it is
 difficult to find which part was broken and ask the right person to look
 into within the Auto JIRA process. Some naive thoughts: Are we able to send
 emails to the dev@ to ask people to take care of the JIRA issues? Are
 we able to find component leads and ask them triage the test failure
 tickets?

 Another nitpick comment. Does the jenkins job file the JIRA issue in
 every test failure? Sometimes the test continuously fails in a time period
 due to the same reason. In this case, we will get some duplicate issues
 filed by Jenkins. I think it could be better if we can avoid filing issues
 if the previous one has not been resolved.

 Thanks.
 Yifan


 On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
 wrote:

> That sounds great. You should add this detail to the doc.
>
> On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
> wrote:
>
>> We already have component for this purpose: "test-failures". All
>> tickets created will go to that component. As an option, we can add link 
>> to
>> view list of open JIRA tickets to PR template.
>>
>> We also would want to create graph on dashboard with amount of
>> unassigned and assigned bugs.
>>
>> I believe that we can also add counter of unassigned bugs to PR
>> template. This way it will be easier for everyone to know when there's 
>> some
>> tests issue not attended.
>>
>> --Mikhail
>>
>>
>> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
>> wrote:
>>
>>> So it sounds like you will want to create a component for untriaged
>>> issues so they are easy to find. I like the idea of distributing the 
>>> work
>>> of triaging post commit failures to new PR authors as a condition of
>>> merging. I feel like we will just be filling JIRA with spam if the 
>>> issues
>>> are automatically created without a plan for triage.
>>>
>>> Andrew
>>>
>>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>>
 Maybe this is also a good thread to start the discussion that if we
 want to enforce postcommit test for every PR.

 Can we afford the cost of longer waiting time to catch potential
 bugs?

 -Rui

 On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin <
 mig...@google.com> wrote:

> That's a valid point.
>
> Unfortunately, the JiraTestResultReporter plugin does not have
> features to dynamically assign owners. Additionally, I don't think it 
> is
> always easy to find proper owner for post-commit tests at first 
> glance,
> since they usually cover broad specter of issues.
>
> My assumption is that we need someone to triage new issues.
>
> Ideally, any contributor, who sees failing test, should check
> unassigned tickets and either do triage, or assign them to someone 

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Pablo Estrada]

Hi all,
I've triaged most issues marked for 2.6.0 release. I've localized two that
need a decision / attention:

- https://issues.apache.org/jira/browse/BEAM-4417 - Bigquery IO Numeric
Datatype Support. Cham is not available to fix this at the moment, but this
is a critical issue. Is anyone able to tackle this / should we bump this to
next release?

- https://issues.apache.org/jira/browse/BEAM-4750 - Performance degradation
due to some safeguards in beam-sdks-java-core. JB, are you looking to fix
this? Should we bump? I had the impression that it was an easy fix, but I'm
not sure.

If you're aware of any other issue that needs to be included as a release
blocker, please report it to me.
Best
-P.

On Thu, Jul 12, 2018 at 2:15 AM Etienne Chauchot 
wrote:

> +1,
>
> Thanks for volunteering Pablo, thanks also to have caught tickets that I
> forgot to close :)
>
> Etienne
>
> Le mercredi 11 juillet 2018 à 12:55 -0700, Alan Myrvold a écrit :
>
> +1 Thanks for volunteering, Pablo
>
> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
> wrote:
>
> +1 sounds great
>
> On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
>
> +1
>
> Thanks for volunteering, Pablo!
>
> On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
> wrote:
>
> +1
>
> I planned to send the proposal as well ;)
>
> Regards
> JB
>
> On 09/07/2018 23:16, Pablo Estrada wrote:
> > Hello everyone!
> >
> > As per the previously agreed-upon schedule for Beam releases, the
> > process for the 2.6.0 Beam release should start on July 17th.
> >
> > I volunteer to perform this release.
> >
> > Here is the schedule that I have in mind:
> >
> > - We start triaging JIRA issues this week.
> > - I will cut a release branch on July 17.
> > - After July 17, any blockers will need to be cherry-picked into the
> > release branch.
> > - As soon as tests look good, and blockers have been addressed, I will
> > perform the other release tasks.
> >
> > Does that seem reasonable to the community?
> >
> > Best
> > -P.
> > --
> > Got feedback? go/pabloem-feedback
> 
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
>
> --
Got feedback? go/pabloem-feedback

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

While I like the idea of having a CODEOWNERS file, the Github
implementation is lacking:
1. Reviewers are automatically assigned at each push.
2. Reviewer assignment can be excessive (e.g. 5 reviewers in Eugene's PR
5940).
3. Non-committers aren't assigned as reviewers.
4. Non-committers can't change the list of reviewers.

I propose renaming the file to disable the auto-reviewer assignment feature.
In its place I'll add a script that suggests reviewers.

On Fri, Jul 13, 2018 at 9:09 AM Udi Meiri  wrote:

> Hi Etienne,
>
> Yes you could be as precise as you want. The paths I listed are just
> suggestions. :)
>
>
> On Fri, Jul 13, 2018 at 1:12 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi,
>>
>> I think it's already do-able just providing the expected path.
>>
>> It's a good idea especially for the core.
>>
>> Regards
>> JB
>>
>> On 13/07/2018 09:51, Etienne Chauchot wrote:
>> > Hi Udi,
>> >
>> > I also have a question, related to what Eugene asked : I see that the
>> > code paths are the ones of the modules. Can we be more precise than that
>> > to assign reviewers ? As an example, I added myself to runner/core
>> > because I wanted to take a look at the PRs related to
>> > runner/core/metrics but I'm getting assigned to all runner-core PRs. Can
>> > we specify paths like
>> > runners/core-java/src/main/java/org/apache/beam/runners/core/metrics ?
>> > I know it is a bit too precise so a bit risky, but in that particular
>> > case, I doubt that the path will change.
>> >
>> > Etienne
>> >
>> > Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
>> >> Hi Udi,
>> >>
>> >> I see that the PR was merged - thanks! However it seems to have some
>> >> unintended effects.
>> >>
>> >> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
>> >> reviewer manually, but the moment I pushed a new commit, it
>> >> auto-assigned a lot of other people to it, and I had to remove them.
>> >> This seems like a big inconvenience to me, is there a way to disable
>> this?
>> >>
>> >> Thanks.
>> >>
>> >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri > >> > wrote:
>> >>> :/ That makes it a little less useful.
>> >>>
>> >>> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson
>> >>> mailto:timrobertson...@gmail.com>> wrote:
>>  Hi Udi
>> 
>>  I asked the GH helpdesk and they confirmed that only people with
>>  write access will actually be automatically chosen.
>> 
>>  It don't expect it should stop us using it, but we should be aware
>>  that there are non-committers also willing to review.
>> 
>>  Thanks,
>>  Tim
>> 
>>  On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin
>>  mailto:mig...@google.com>> wrote:
>> > Idea looks good in general.
>> >
>> > Did you look into ways to keep this file up-to-date? For example we
>> > can run monthly job to see if owner was active during this period.
>> >
>> > --Mikhail
>> >
>> > Have feedback ?
>> >
>> >
>> > On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri > > > wrote:
>> >> Thanks all!
>> >> I'll try to get the file merged today and see how it works out.
>> >> Please surface any issues, such as with auto-assignment, here or
>> >> in JIRA.
>> >>
>> >> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot
>> >> mailto:echauc...@apache.org>> wrote:
>> >>> Hi,
>> >>>
>> >>> I added myself as a reviewer for some modules.
>> >>>
>> >>> Etienne
>> >>>
>> >>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>>  Hi everyone,
>> 
>>  I'm proposing to add auto-reviewer-assignment using Github's
>>  CODEOWNERS mechanism.
>>  Initial version is
>>  here: _https://github.com/apache/beam/pull/5909/files_
>> 
>>  I need help from the community in determining owners for each
>>  component.
>>  Feel free to directly edit the PR (if you have permission) or
>>  add a comment.
>> 
>> 
>>  Background
>>  The idea is to:
>>  1. Document good review candidates for each component.
>>  2. Help choose reviewers using the auto-assignment mechanism.
>>  The suggestion is in no way binding.
>> 
>> 
>> 
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Automatically create JIRA tickets for failing post-commit tests

[Mikhail Gryzykhin]

Can we get some PMC opinions on this?

According to https://www.apache.org/dev/infra-contact#request-checklist, it
is highly recommended to have your opinions when filing a request.

--Mikhail

Have feedback ?


On Wed, Jul 11, 2018 at 6:01 PM Anton Kedin  wrote:

> I think this looks good, we should enable the plugin and try it out.
> Concrete details of the follow-up tasks (auto-assignment, triage, and
> dashboarding) will probably depend on how functional the plugin is and what
> the test failures data looks like.
>
> Regards,
> Anton
>
> On Wed, Jul 11, 2018 at 5:00 PM Mikhail Gryzykhin 
> wrote:
>
>> @Yifan Zou 
>>
>> I believe that we should test-drive the system with tickets + PR first
>> and decide on email notification later. We already have tests failure
>> emails sent to commits@, I believe most people filter out or not signed
>> up for that list though.
>>
>> It creates only one ticket, and keeps it for recurring test failures.
>>
>> @Andrew Pilloud 
>> Thank you for the suggestion. I'll add it to design doc.
>>
>> --Mikhail
>>
>>
>>
>> On Wed, Jul 11, 2018 at 4:52 PM Yifan Zou  wrote:
>>
>>> +1 to Andrew's concerns. Leaving the tickets unassigned will cause the
>>> ticket being ignored and no actions being taken.
>>>
>>> I can see the challenges on ticket assignment. Like Mikhail mentioned,
>>> the plugin does not support dynamic assignments. We have to implement
>>> custom script to determine the assignees and do some tricks to the jenkins
>>> job. Also, the post-commits tests usually cover tons of stuffs that it is
>>> difficult to find which part was broken and ask the right person to look
>>> into within the Auto JIRA process. Some naive thoughts: Are we able to send
>>> emails to the dev@ to ask people to take care of the JIRA issues? Are
>>> we able to find component leads and ask them triage the test failure
>>> tickets?
>>>
>>> Another nitpick comment. Does the jenkins job file the JIRA issue in
>>> every test failure? Sometimes the test continuously fails in a time period
>>> due to the same reason. In this case, we will get some duplicate issues
>>> filed by Jenkins. I think it could be better if we can avoid filing issues
>>> if the previous one has not been resolved.
>>>
>>> Thanks.
>>> Yifan
>>>
>>>
>>> On Wed, Jul 11, 2018 at 4:37 PM Andrew Pilloud 
>>> wrote:
>>>
 That sounds great. You should add this detail to the doc.

 On Wed, Jul 11, 2018 at 4:29 PM Mikhail Gryzykhin 
 wrote:

> We already have component for this purpose: "test-failures". All
> tickets created will go to that component. As an option, we can add link 
> to
> view list of open JIRA tickets to PR template.
>
> We also would want to create graph on dashboard with amount of
> unassigned and assigned bugs.
>
> I believe that we can also add counter of unassigned bugs to PR
> template. This way it will be easier for everyone to know when there's 
> some
> tests issue not attended.
>
> --Mikhail
>
>
> On Wed, Jul 11, 2018 at 4:24 PM Andrew Pilloud 
> wrote:
>
>> So it sounds like you will want to create a component for untriaged
>> issues so they are easy to find. I like the idea of distributing the work
>> of triaging post commit failures to new PR authors as a condition of
>> merging. I feel like we will just be filling JIRA with spam if the issues
>> are automatically created without a plan for triage.
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 4:12 PM Rui Wang  wrote:
>>
>>> Maybe this is also a good thread to start the discussion that if we
>>> want to enforce postcommit test for every PR.
>>>
>>> Can we afford the cost of longer waiting time to catch potential
>>> bugs?
>>>
>>> -Rui
>>>
>>> On Wed, Jul 11, 2018 at 4:04 PM Mikhail Gryzykhin 
>>> wrote:
>>>
 That's a valid point.

 Unfortunately, the JiraTestResultReporter plugin does not have
 features to dynamically assign owners. Additionally, I don't think it 
 is
 always easy to find proper owner for post-commit tests at first glance,
 since they usually cover broad specter of issues.

 My assumption is that we need someone to triage new issues.

 Ideally, any contributor, who sees failing test, should check
 unassigned tickets and either do triage, or assign them to someone who 
 can.
 I strongly encourage this approach.

 We have couple other ready-made options to consider:
 1. We can configure JIRA component owner who would be assigned to
 created tickets.
 2. JiraTestReporterPlugin can assign tickets to specific user. This
 is configured per Jenkins job. We can utilize this if someone 
 volunteers.
 3. Dynamic assignment will most likely require custom solution.

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

Hi Etienne,

Yes you could be as precise as you want. The paths I listed are just
suggestions. :)


On Fri, Jul 13, 2018 at 1:12 AM Jean-Baptiste Onofré 
wrote:

> Hi,
>
> I think it's already do-able just providing the expected path.
>
> It's a good idea especially for the core.
>
> Regards
> JB
>
> On 13/07/2018 09:51, Etienne Chauchot wrote:
> > Hi Udi,
> >
> > I also have a question, related to what Eugene asked : I see that the
> > code paths are the ones of the modules. Can we be more precise than that
> > to assign reviewers ? As an example, I added myself to runner/core
> > because I wanted to take a look at the PRs related to
> > runner/core/metrics but I'm getting assigned to all runner-core PRs. Can
> > we specify paths like
> > runners/core-java/src/main/java/org/apache/beam/runners/core/metrics ?
> > I know it is a bit too precise so a bit risky, but in that particular
> > case, I doubt that the path will change.
> >
> > Etienne
> >
> > Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
> >> Hi Udi,
> >>
> >> I see that the PR was merged - thanks! However it seems to have some
> >> unintended effects.
> >>
> >> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
> >> reviewer manually, but the moment I pushed a new commit, it
> >> auto-assigned a lot of other people to it, and I had to remove them.
> >> This seems like a big inconvenience to me, is there a way to disable
> this?
> >>
> >> Thanks.
> >>
> >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri  >> > wrote:
> >>> :/ That makes it a little less useful.
> >>>
> >>> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson
> >>> mailto:timrobertson...@gmail.com>> wrote:
>  Hi Udi
> 
>  I asked the GH helpdesk and they confirmed that only people with
>  write access will actually be automatically chosen.
> 
>  It don't expect it should stop us using it, but we should be aware
>  that there are non-committers also willing to review.
> 
>  Thanks,
>  Tim
> 
>  On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin
>  mailto:mig...@google.com>> wrote:
> > Idea looks good in general.
> >
> > Did you look into ways to keep this file up-to-date? For example we
> > can run monthly job to see if owner was active during this period.
> >
> > --Mikhail
> >
> > Have feedback ?
> >
> >
> > On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  > > wrote:
> >> Thanks all!
> >> I'll try to get the file merged today and see how it works out.
> >> Please surface any issues, such as with auto-assignment, here or
> >> in JIRA.
> >>
> >> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot
> >> mailto:echauc...@apache.org>> wrote:
> >>> Hi,
> >>>
> >>> I added myself as a reviewer for some modules.
> >>>
> >>> Etienne
> >>>
> >>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>  Hi everyone,
> 
>  I'm proposing to add auto-reviewer-assignment using Github's
>  CODEOWNERS mechanism.
>  Initial version is
>  here: _https://github.com/apache/beam/pull/5909/files_
> 
>  I need help from the community in determining owners for each
>  component.
>  Feel free to directly edit the PR (if you have permission) or
>  add a comment.
> 
> 
>  Background
>  The idea is to:
>  1. Document good review candidates for each component.
>  2. Help choose reviewers using the auto-assignment mechanism.
>  The suggestion is in no way binding.
> 
> 
> 
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Łukasz Gajowy]

@Etienne: Nice to see the graphs! :)

@Ismael: Good idea, there's no document yet. I think we could create a
small google doc with instructions on how to do this.

pt., 13 lip 2018 o 10:46 Etienne Chauchot  napisał(a):

> Hi,
>
> @Andrew, this is because I did not find a way to set 2 scales on the Y
> axis on the perfkit graphs. Indeed numResults varies from 1 to 100 000 and
> runtimeSec is usually bellow 10s.
>
> Etienne
>
> Le jeudi 12 juillet 2018 à 12:04 -0700, Andrew Pilloud a écrit :
>
> This is great, should make performance work much easier! I'm going to get
> the Beam SQL Nexmark jobs publishing as well. (Opened
> https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might take
> on the Dataflow runner as well if no one else volunteers.
>
> I am curious as to why you have two separate graphs for runtime and count
> rather then graphing runtime/count to get the throughput rate for each run?
> Or should that be a third graph? Looks like it would just be a small tweak
> to the query in perfkit.
>
>
>
> Andrew
>
> On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada  wrote:
>
> This is really cool Etienne : ) thanks for working on this.
> Our of curiosity, do you know how often the tests run on each runner?
>
> Best
> -P.
>
> On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau 
> wrote:
>
> Awesome Etienne, this is really important for the (user) community to have
> that visibility since it is one of the most important aspect of the Beam's
> quality, kudo!
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
>
> Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a
> écrit :
>
> It's really great to have these dashboards and integration in Jenkins !
>
> Thanks Etienne for driving this !
>
> Regards
> JB
>
> On 11/07/2018 15:13, Etienne Chauchot wrote:
> >
> > Hi guys,
> >
> > I'm glad to announce that the CI of Beam has much improved ! Indeed
> > Nexmark is now included in the perfkit dashboards.
> >
> > At each commit on master, nexmark suites are run and plots are created
> > on the graphs.
> >
> > I've created 2 kind of dashboards:
> > - one for performances (run times of the queries)
> > - one for the size of the output PCollection (which should be constant)
> >
> > There are dashboards for these runners:
> > - spark
> > - flink
> > - direct runner
> >
> > Each dashboard contains:
> > - graphs in batch mode
> > - graphs in streaming mode
> > - graphs for the 13 queries.
> >
> > That gives more than a hundred of graphs (my right finger hurts after so
> > many clics on the mouse :) ). It is detailed that much so that anyone
> > can focus on the area they have interest in.
> > Feel free to also create new dashboards with more aggregated data.
> >
> > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
> > perfkit dashboards.
> >
> > Dashboards are there:
> >
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> > <
> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> >
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> >
> >
> > Enjoy,
> >
> > Etienne
> >
> >
>
>

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Etienne Chauchot]

Hi, 
@Andrew, this is because I did not find a way to set 2 scales on the Y axis on 
the perfkit graphs. Indeed numResults
varies from 1 to  100 000 and runtimeSec is usually bellow 10s.
Etienne
Le jeudi 12 juillet 2018 à 12:04 -0700, Andrew Pilloud a écrit :
> This is great, should make performance work much easier! I'm going to get the 
> Beam SQL Nexmark jobs publishing as
> well. (Opened https://issues.apache.org/jira/browse/BEAM-4774 to track.) I 
> might take on the Dataflow runner as well
> if no one else volunteers.
> 
> I am curious as to why you have two separate graphs for runtime and count 
> rather then graphing runtime/count to get
> the throughput rate for each run? Or should that be a third graph? Looks like 
> it would just be a small tweak to the
> query in perfkit.
> Andrew
> On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada  wrote:
> > This is really cool Etienne : ) thanks for working on this.Our of 
> > curiosity, do you know how often the tests run on
> > each runner?
> > 
> > Best
> > -P.
> > 
> > On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau  
> > wrote:
> > > Awesome Etienne, this is really important for the (user) community to 
> > > have that visibility since it is one of the
> > > most important aspect of the Beam's quality, kudo!
> > > 
> > > 
> > > Romain Manni-Bucau
> > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > > 
> > > Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a 
> > > écrit :
> > > > It's really great to have these dashboards and integration in Jenkins !
> > > > 
> > > > 
> > > > 
> > > > Thanks Etienne for driving this !
> > > > 
> > > > 
> > > > 
> > > > Regards
> > > > 
> > > > JB
> > > > 
> > > > 
> > > > 
> > > > On 11/07/2018 15:13, Etienne Chauchot wrote:
> > > > 
> > > > > 
> > > > 
> > > > > Hi guys,
> > > > 
> > > > > 
> > > > 
> > > > > I'm glad to announce that the CI of Beam has much improved ! Indeed
> > > > 
> > > > > Nexmark is now included in the perfkit dashboards.
> > > > 
> > > > > 
> > > > 
> > > > > At each commit on master, nexmark suites are run and plots are created
> > > > 
> > > > > on the graphs.
> > > > 
> > > > > 
> > > > 
> > > > > I've created 2 kind of dashboards:
> > > > 
> > > > > - one for performances (run times of the queries)
> > > > 
> > > > > - one for the size of the output PCollection (which should be 
> > > > > constant)
> > > > 
> > > > > 
> > > > 
> > > > > There are dashboards for these runners:
> > > > 
> > > > > - spark
> > > > 
> > > > > - flink
> > > > 
> > > > > - direct runner
> > > > 
> > > > > 
> > > > 
> > > > > Each dashboard contains:
> > > > 
> > > > > - graphs in batch mode 
> > > > 
> > > > > - graphs in streaming mode
> > > > 
> > > > > - graphs for the 13 queries.
> > > > 
> > > > > 
> > > > 
> > > > > That gives more than a hundred of graphs (my right finger hurts after 
> > > > > so
> > > > 
> > > > > many clics on the mouse :) ). It is detailed that much so that anyone
> > > > 
> > > > > can focus on the area they have interest in.
> > > > 
> > > > > Feel free to also create new dashboards with more aggregated data.
> > > > 
> > > > > 
> > > > 
> > > > > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
> > > > 
> > > > > perfkit dashboards.
> > > > 
> > > > > 
> > > > 
> > > > > Dashboards are there:
> > > > 
> > > > > 
> > > > 
> > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> > > > 
> > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> > > > 
> > > > > https://apache-beam-testing.appspo
> > > > t.com/explore?dashboard=5138380291571712
> > > > 
> > > > > 
> > > > 
> > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> > > > 
> > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> > > > 
> > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> > > > 
> > > > > 
> > > > 
> > > > > 
> > > > 
> > > > > Enjoy, 
> > > > 
> > > > > 
> > > > 
> > > > > Etienne
> > > > 
> > > > > 
> > > > 
> > > > > 
> > > > 
> > > > 
> > > > 

Re: CODEOWNERS for apache/beam repo

[Jean-Baptiste Onofré]

Hi,

I think it's already do-able just providing the expected path.

It's a good idea especially for the core.

Regards
JB

On 13/07/2018 09:51, Etienne Chauchot wrote:
> Hi Udi,
> 
> I also have a question, related to what Eugene asked : I see that the
> code paths are the ones of the modules. Can we be more precise than that
> to assign reviewers ? As an example, I added myself to runner/core
> because I wanted to take a look at the PRs related to
> runner/core/metrics but I'm getting assigned to all runner-core PRs. Can
> we specify paths like
> runners/core-java/src/main/java/org/apache/beam/runners/core/metrics ?
> I know it is a bit too precise so a bit risky, but in that particular
> case, I doubt that the path will change.
> 
> Etienne
> 
> Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
>> Hi Udi,
>>
>> I see that the PR was merged - thanks! However it seems to have some
>> unintended effects.
>>
>> On my PR https://github.com/apache/beam/pull/5940 , I assigned a
>> reviewer manually, but the moment I pushed a new commit, it
>> auto-assigned a lot of other people to it, and I had to remove them.
>> This seems like a big inconvenience to me, is there a way to disable this?
>>
>> Thanks.
>>
>> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri > > wrote:
>>> :/ That makes it a little less useful.
>>>
>>> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson
>>> mailto:timrobertson...@gmail.com>> wrote:
 Hi Udi

 I asked the GH helpdesk and they confirmed that only people with
 write access will actually be automatically chosen.

 It don't expect it should stop us using it, but we should be aware
 that there are non-committers also willing to review.

 Thanks,
 Tim

 On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin
 mailto:mig...@google.com>> wrote:
> Idea looks good in general. 
>
> Did you look into ways to keep this file up-to-date? For example we
> can run monthly job to see if owner was active during this period.
>
> --Mikhail
>
> Have feedback ? 
>
>
> On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  > wrote:
>> Thanks all!
>> I'll try to get the file merged today and see how it works out.
>> Please surface any issues, such as with auto-assignment, here or
>> in JIRA.
>>
>> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot
>> mailto:echauc...@apache.org>> wrote:
>>> Hi,
>>>
>>> I added myself as a reviewer for some modules.
>>>
>>> Etienne
>>>
>>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
 Hi everyone,

 I'm proposing to add auto-reviewer-assignment using Github's
 CODEOWNERS mechanism.
 Initial version is
 here: _https://github.com/apache/beam/pull/5909/files_

 I need help from the community in determining owners for each
 component.
 Feel free to directly edit the PR (if you have permission) or
 add a comment.


 Background
 The idea is to:
 1. Document good review candidates for each component.
 2. Help choose reviewers using the auto-assignment mechanism.
 The suggestion is in no way binding.




-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Etienne Chauchot]

Hi Pablo,
yes they run at each commit on master (post-commit jenkins script)
Etienne
Le jeudi 12 juillet 2018 à 11:40 -0700, Pablo Estrada a écrit :
> This is really cool Etienne : ) thanks for working on this.Our of curiosity, 
> do you know how often the tests run on
> each runner?
> 
> Best
> -P.
> 
> On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau  
> wrote:
> > Awesome Etienne, this is really important for the (user) community to have 
> > that visibility since it is one of the
> > most important aspect of the Beam's quality, kudo!
> > 
> > 
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > 
> > Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a 
> > écrit :
> > > It's really great to have these dashboards and integration in Jenkins !
> > > 
> > > 
> > > 
> > > Thanks Etienne for driving this !
> > > 
> > > 
> > > 
> > > Regards
> > > 
> > > JB
> > > 
> > > 
> > > 
> > > On 11/07/2018 15:13, Etienne Chauchot wrote:
> > > 
> > > > 
> > > 
> > > > Hi guys,
> > > 
> > > > 
> > > 
> > > > I'm glad to announce that the CI of Beam has much improved ! Indeed
> > > 
> > > > Nexmark is now included in the perfkit dashboards.
> > > 
> > > > 
> > > 
> > > > At each commit on master, nexmark suites are run and plots are created
> > > 
> > > > on the graphs.
> > > 
> > > > 
> > > 
> > > > I've created 2 kind of dashboards:
> > > 
> > > > - one for performances (run times of the queries)
> > > 
> > > > - one for the size of the output PCollection (which should be constant)
> > > 
> > > > 
> > > 
> > > > There are dashboards for these runners:
> > > 
> > > > - spark
> > > 
> > > > - flink
> > > 
> > > > - direct runner
> > > 
> > > > 
> > > 
> > > > Each dashboard contains:
> > > 
> > > > - graphs in batch mode 
> > > 
> > > > - graphs in streaming mode
> > > 
> > > > - graphs for the 13 queries.
> > > 
> > > > 
> > > 
> > > > That gives more than a hundred of graphs (my right finger hurts after so
> > > 
> > > > many clics on the mouse :) ). It is detailed that much so that anyone
> > > 
> > > > can focus on the area they have interest in.
> > > 
> > > > Feel free to also create new dashboards with more aggregated data.
> > > 
> > > > 
> > > 
> > > > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
> > > 
> > > > perfkit dashboards.
> > > 
> > > > 
> > > 
> > > > Dashboards are there:
> > > 
> > > > 
> > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> > > 
> > > > https://apache-beam-testing.appspot.
> > > com/explore?dashboard=5138380291571712
> > > 
> > > > 
> > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> > > 
> > > > 
> > > 
> > > > 
> > > 
> > > > Enjoy, 
> > > 
> > > > 
> > > 
> > > > Etienne
> > > 
> > > > 
> > > 
> > > > 
> > > 
> > > 
> > > 

Re: CODEOWNERS for apache/beam repo

[Etienne Chauchot]

Hi Udi,
I also have a question, related to what Eugene asked : I see that the code 
paths are the ones of the modules. Can we be
more precise than that to assign reviewers ? As an example, I added myself to 
runner/core because I wanted to take a
look at the PRs related to runner/core/metrics but I'm getting assigned to all 
runner-core PRs. Can we specify paths
like runners/core-java/src/main/java/org/apache/beam/runners/core/metrics ?I 
know it is a bit too precise so a bit
risky, but in that particular case, I doubt that the path will change.
Etienne
Le jeudi 12 juillet 2018 à 16:49 -0700, Eugene Kirpichov a écrit :
> Hi Udi,
> I see that the PR was merged - thanks! However it seems to have some 
> unintended effects.
> 
> On my PR https://github.com/apache/beam/pull/594040 , I assigned a reviewer 
> manually, but the moment I pushed a new
> commit, it auto-assigned a lot of other people to it, and I had to remove 
> them. This seems like a big inconvenience to
> me, is there a way to disable this?
> 
> Thanks.
> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri  wrote:
> > :/ That makes it a little less useful.
> > On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson  
> > wrote:
> > > Hi Udi
> > > I asked the GH helpdesk and they confirmed that only people with write 
> > > access will actually be automatically
> > > chosen.
> > > 
> > > It don't expect it should stop us using it, but we should be aware that 
> > > there are non-committers also willing to
> > > review.
> > > 
> > > Thanks,
> > > Tim
> > > On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin  
> > > wrote:
> > > > Idea looks good in general. 
> > > > Did you look into ways to keep this file up-to-date? For example we can 
> > > > run monthly job to see if owner was
> > > > active during this period.
> > > > --Mikhail
> > > > Have feedback? 
> > > > 
> > > > On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  wrote:
> > > > > Thanks all!I'll try to get the file merged today and see how it works 
> > > > > out.
> > > > > Please surface any issues, such as with auto-assignment, here or in 
> > > > > JIRA.
> > > > > On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
> > > > >  wrote:
> > > > > > Hi,
> > > > > > I added myself as a reviewer for some modules.
> > > > > > Etienne
> > > > > > Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
> > > > > > > Hi everyone,
> > > > > > > I'm proposing to add auto-reviewer-assignment using Github's 
> > > > > > > CODEOWNERS mechanism.
> > > > > > > Initial version is here: 
> > > > > > > https://github.com/apache/beam/pull/5909/files
> > > > > > > I need help from the community in determining owners for each 
> > > > > > > component.
> > > > > > > Feel free to directly edit the PR (if you have permission) or add 
> > > > > > > a comment.
> > > > > > > 
> > > > > > > 
> > > > > > > Background
> > > > > > > The idea is to:
> > > > > > > 1. Document good review candidates for each component.
> > > > > > > 2. Help choose reviewers using the auto-assignment mechanism. The 
> > > > > > > suggestion is in no way binding.
> > > > > > > 
> > > > > > > 
> > > > > > >