Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Udi Meiri]

+1

On Fri, Jun 1, 2018 at 4:27 PM Lukasz Cwik  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 2:53 PM Thomas Weise  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 2:17 PM, Robert Bradshaw 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 1:43 PM Andrew Pilloud 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 1:31 PM Huygaa Batsaikhan 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> +1 (non-binding).
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner 
 wrote:

> +1 (non-binding)
>
> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
>> wrote:
>>
>>> +1 (non-binding): automating policy ensures it is applied fairly
>>> and evenly and lessens the load on project maintainers; hearty 
>>> agreement.
>>>
>>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
>>> wrote:
>>>
 +1 (non-binding) I updated the pull request to be 60 days
 (instead of 90) to match the contribute policy.

 On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Following the discussion, please vote on the move to activate
> probot/stale [3] to notify authors of stale PRs per current
> policy and then close them after a 7 day grace period.
>
> For more details, see:
>
>  - our stale PR policy [1]
>  - the discussion thread [2]
>  - Probot stale [3]
>  - BEAM ticket summarizing discussion [4]
>  - INFRA ticket to activate probot/stale [5]
>  - Example PR that would activate it [6]
>
> Please vote:
> [ ] +1, Approve that we activate probot/stale
> [ ] -1, Do not approve (please provide specific comments)
>
> Kenn
>
> [1] https://beam.apache.org/contribute/#stale-pull-requests
> [2]
> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
> [3] https://github.com/probot/stale
> [4] https://issues.apache.org/jira/browse/BEAM-4423
> [5] https://issues.apache.org/jira/browse/INFRA-16589
> [6] https://github.com/apache/beam/pull/5532
>

>>>
>>> --
>>> ---
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>> See something? Say something. go/jasonkuster-feedback
>>> 
>>>
>>
>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [VOTE] Code Review Process

[Scott Wegner]

+1

On Fri, Jun 1, 2018 at 3:44 PM Pablo Estrada  wrote:

> +1 :) glad that we had this discussion
>
> On Fri, Jun 1, 2018, 3:38 PM Udi Meiri  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 1:46 PM Andrew Pilloud 
>> wrote:
>>
>>> +1 - I hope this doesn't reduce the urgency to fix the root cause: not
>>> having enough committers.
>>>
>>> On Fri, Jun 1, 2018 at 1:18 PM Henning Rohde  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 12:27 PM Dan Halperin 
 wrote:

> +1 -- this is encoding what I previously thought the process was and
> what, in practice, I think was often the behavior of committers anyway.
>
> On Fri, Jun 1, 2018 at 12:21 PM, Yifan Zou 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 +1

 Thanks,
 Cham

 On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster <
 jasonkus...@google.com> wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles <
> k...@google.com> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
>> wrote:
>>
>>> As we seem to largely have consensus in "Reducing Committer
>>> Load for Code Reviews"[1], this is a vote to change the Beam 
>>> policy on Code
>>> Reviews to require that
>>>
>>> (1) At least one committer is involved with the code review,
>>> as either a reviewer or as the author
>>> (2) A contributor has approved the change
>>>
>>> prior to merging any change.
>>>
>>> This changes our policy from its current requirement that at
>>> least one committer *who is not the author* has approved the 
>>> change prior
>>> to merging. We believe that changing this process will improve 
>>> code review
>>> throughput, reduce committer load, and engage more of the 
>>> community in the
>>> code review process.
>>>
>>> Please vote:
>>> [ ] +1: Accept the above proposal to change the Beam code
>>> review/merge policy
>>> [ ] -1: Leave the Code Review policy unchanged
>>>
>>> Thanks,
>>>
>>> Thomas
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>>
>>
>

>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

> --
> Got feedback? go/pabloem-feedback
> 
>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Lukasz Cwik]

+1

On Fri, Jun 1, 2018 at 2:53 PM Thomas Weise  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 2:17 PM, Robert Bradshaw 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 1:43 PM Andrew Pilloud 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 1:31 PM Huygaa Batsaikhan 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> +1 (non-binding).
>>
>> Thanks,
>> Cham
>>
>> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner 
>>> wrote:
>>>
 +1 (non-binding)

 On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay 
 wrote:

> +1
>
> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
> wrote:
>
>> +1 (non-binding): automating policy ensures it is applied fairly
>> and evenly and lessens the load on project maintainers; hearty 
>> agreement.
>>
>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
>> wrote:
>>
>>> +1 (non-binding) I updated the pull request to be 60 days
>>> (instead of 90) to match the contribute policy.
>>>
>>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Following the discussion, please vote on the move to activate
 probot/stale [3] to notify authors of stale PRs per current
 policy and then close them after a 7 day grace period.

 For more details, see:

  - our stale PR policy [1]
  - the discussion thread [2]
  - Probot stale [3]
  - BEAM ticket summarizing discussion [4]
  - INFRA ticket to activate probot/stale [5]
  - Example PR that would activate it [6]

 Please vote:
 [ ] +1, Approve that we activate probot/stale
 [ ] -1, Do not approve (please provide specific comments)

 Kenn

 [1] https://beam.apache.org/contribute/#stale-pull-requests
 [2]
 https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
 [3] https://github.com/probot/stale
 [4] https://issues.apache.org/jira/browse/BEAM-4423
 [5] https://issues.apache.org/jira/browse/INFRA-16589
 [6] https://github.com/apache/beam/pull/5532

>>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>
>

Re: [VOTE] Code Review Process

[Pablo Estrada]

+1 :) glad that we had this discussion

On Fri, Jun 1, 2018, 3:38 PM Udi Meiri  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 1:46 PM Andrew Pilloud  wrote:
>
>> +1 - I hope this doesn't reduce the urgency to fix the root cause: not
>> having enough committers.
>>
>> On Fri, Jun 1, 2018 at 1:18 PM Henning Rohde  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 12:27 PM Dan Halperin 
>>> wrote:
>>>
 +1 -- this is encoding what I previously thought the process was and
 what, in practice, I think was often the behavior of committers anyway.

 On Fri, Jun 1, 2018 at 12:21 PM, Yifan Zou  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> +1
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles <
 k...@google.com> wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
> wrote:
>
>> As we seem to largely have consensus in "Reducing Committer
>> Load for Code Reviews"[1], this is a vote to change the Beam 
>> policy on Code
>> Reviews to require that
>>
>> (1) At least one committer is involved with the code review,
>> as either a reviewer or as the author
>> (2) A contributor has approved the change
>>
>> prior to merging any change.
>>
>> This changes our policy from its current requirement that at
>> least one committer *who is not the author* has approved the 
>> change prior
>> to merging. We believe that changing this process will improve 
>> code review
>> throughput, reduce committer load, and engage more of the 
>> community in the
>> code review process.
>>
>> Please vote:
>> [ ] +1: Accept the above proposal to change the Beam code
>> review/merge policy
>> [ ] -1: Leave the Code Review policy unchanged
>>
>> Thanks,
>>
>> Thomas
>>
>> [1]
>> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>
>

>>>

 --
 ---
 Jason Kuster
 Apache Beam / Google Cloud Dataflow

 See something? Say something. go/jasonkuster-feedback
 

>>>
 --
Got feedback? go/pabloem-feedback

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Thomas Weise]

+1

On Fri, Jun 1, 2018 at 2:17 PM, Robert Bradshaw  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 1:43 PM Andrew Pilloud  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 1:31 PM Huygaa Batsaikhan 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath <
 chamik...@google.com> wrote:

> +1 (non-binding).
>
> Thanks,
> Cham
>
> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner 
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>>>
 +1

 On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
 wrote:

> +1 (non-binding): automating policy ensures it is applied fairly
> and evenly and lessens the load on project maintainers; hearty 
> agreement.
>
> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
> wrote:
>
>> +1 (non-binding) I updated the pull request to be 60 days
>> (instead of 90) to match the contribute policy.
>>
>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Following the discussion, please vote on the move to activate
>>> probot/stale [3] to notify authors of stale PRs per current
>>> policy and then close them after a 7 day grace period.
>>>
>>> For more details, see:
>>>
>>>  - our stale PR policy [1]
>>>  - the discussion thread [2]
>>>  - Probot stale [3]
>>>  - BEAM ticket summarizing discussion [4]
>>>  - INFRA ticket to activate probot/stale [5]
>>>  - Example PR that would activate it [6]
>>>
>>> Please vote:
>>> [ ] +1, Approve that we activate probot/stale
>>> [ ] -1, Do not approve (please provide specific comments)
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>>> [2] https://lists.apache.org/thread.html/
>>> bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%
>>> 3Cdev.beam.apache.org%3E
>>> [3] https://github.com/probot/stale
>>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>>> [6] https://github.com/apache/beam/pull/5532
>>>
>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: Managing outdated dependencies

[Chamikara Jayalath]

Thanks. I left these ideas bit detailed for clarification. I'll rewrite
them in a more concise form when we have enough feedback.

- Cham

On Fri, Jun 1, 2018 at 1:58 PM Kenneth Knowles  wrote:

> This does seem really useful. I appreciate the detailed explanations. If
> we formalize it into policy, I'd love to make it a bit more concise, and
> with appropriate room for human contestation of the guidelines.
>
> On Fri, Jun 1, 2018 at 1:47 PM Scott Wegner  wrote:
>
>> Thanks Cham. Overall this seems like a useful hygiene improvement for the
>> project. I've left some comments in the doc.
>>
>> On Fri, Jun 1, 2018 at 10:48 AM Chamikara Jayalath 
>> wrote:
>>
>>> Hi All,
>>>
>>> I've copied ideas proposed in the doc below for more visibility. Any
>>> comments are welcome.
>>>
>>>
>>>
>>> * - Human readable per-SDK reports on status of Beam dependencies are
>>> generated weekly and shared with the Beam community through the dev list.
>>> These reports should be concise and should highlight the cases where the
>>> community has to act on. See [4] for more details on this.- Beam Components
>>> (IO connectors, runners, etc) should always try to use versions of
>>> dependencies that are defined at the top level. Per-component dependency
>>> version overrides should only be performed in rare cases and should come
>>> with clear warnings for users.- Upgrading a dependency with an outdated
>>> major version becomes a blocker for next major version release of Beam and
>>> for any minor version releases after next immediate minor version release.
>>> For example, if a dependency is identified to be outdated while the latest
>>> release is x.y.z, upgrading this dependency becomes a blocker for releases
>>> (x+1).0.0 and x.(y+2).0 of Beam. Additionally, upgrading to a major version
>>> of a dependency will only be enforced if the new major version of the
>>> dependency can be adapted without a significant rewrite to any Beam
>>> component. Note that this policy intentionally allows one of the minor
>>> version releases to proceed without upgrading the dependency which I
>>> believe will give Beam community enough breathing room to upgrade
>>> dependencies without significantly affecting the release frequency.-
>>> Upgrading a dependency with a significantly outdated minor version (based
>>> on methodology defined in [4]) becomes a blocker for next major version
>>> release of Beam and for any minor version releases of Beam after next
>>> immediate minor version release. Note that this policy does not force Beam
>>> to adapt every minor version release of a dependency.- When performing a
>>> release, release manager should make sure that blockers identified through
>>> above process are resolved before the release candidate is cut.-
>>> Optionally, dependency declarations may have comments that identify owners
>>> that should be responsible for upgrading the respective dependencies.
>>> Release manager may choose to assign a blocking JIRA for a dependency to
>>> its owner.*
>>> Thanks,
>>> Cham
>>>
>>> On Thu, May 31, 2018 at 7:11 PM Chamikara Jayalath 
>>> wrote:
>>>
 Hi All,

 We recently ran into many issues due to Beam dependencies being
 significantly out of date. For example see [1], [2], and [3].

 Yifan Zou recently introduced a proposal [4] that would allow us to
 identify outdated dependencies. But to really make sure that this helps the
 Beam project and community I believe we should adapt several small policy
 changes to our development and release process.

 To this end, I have created following short document that identifies
 the dependency issue and proposes several policy changes. I greatly
 appreciate if you can take a look and comment.


 https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing

 Thanks,
 Cham

 [1] https://issues.apache.org/jira/browse/BEAM-3098
 [2] https://issues.apache.org/jira/browse/BEAM-3991
 [3] https://issues.apache.org/jira/browse/BEAM-4229
 [4]
 https://lists.apache.org/thread.html/758625106a6cfe9ba23d7b39625da20e050c6279b138b18b3f0013e7@%3Cdev.beam.apache.org%3E

>>>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Robert Bradshaw]

+1

On Fri, Jun 1, 2018 at 1:43 PM Andrew Pilloud  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 1:31 PM Huygaa Batsaikhan 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath 
>>> wrote:
>>>
 +1 (non-binding).

 Thanks,
 Cham

 On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner 
> wrote:
>
>> +1 (non-binding)
>>
>> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
>>> wrote:
>>>
 +1 (non-binding): automating policy ensures it is applied fairly
 and evenly and lessens the load on project maintainers; hearty 
 agreement.

 On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
 wrote:

> +1 (non-binding) I updated the pull request to be 60 days (instead
> of 90) to match the contribute policy.
>
> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Following the discussion, please vote on the move to activate
>> probot/stale [3] to notify authors of stale PRs per current
>> policy and then close them after a 7 day grace period.
>>
>> For more details, see:
>>
>>  - our stale PR policy [1]
>>  - the discussion thread [2]
>>  - Probot stale [3]
>>  - BEAM ticket summarizing discussion [4]
>>  - INFRA ticket to activate probot/stale [5]
>>  - Example PR that would activate it [6]
>>
>> Please vote:
>> [ ] +1, Approve that we activate probot/stale
>> [ ] -1, Do not approve (please provide specific comments)
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>> [2]
>> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
>> [3] https://github.com/probot/stale
>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>> [6] https://github.com/apache/beam/pull/5532
>>
>

 --
 ---
 Jason Kuster
 Apache Beam / Google Cloud Dataflow

 See something? Say something. go/jasonkuster-feedback
 

>>>

Re: Managing outdated dependencies

[Kenneth Knowles]

This does seem really useful. I appreciate the detailed explanations. If we
formalize it into policy, I'd love to make it a bit more concise, and with
appropriate room for human contestation of the guidelines.

On Fri, Jun 1, 2018 at 1:47 PM Scott Wegner  wrote:

> Thanks Cham. Overall this seems like a useful hygiene improvement for the
> project. I've left some comments in the doc.
>
> On Fri, Jun 1, 2018 at 10:48 AM Chamikara Jayalath 
> wrote:
>
>> Hi All,
>>
>> I've copied ideas proposed in the doc below for more visibility. Any
>> comments are welcome.
>>
>>
>>
>> * - Human readable per-SDK reports on status of Beam dependencies are
>> generated weekly and shared with the Beam community through the dev list.
>> These reports should be concise and should highlight the cases where the
>> community has to act on. See [4] for more details on this.- Beam Components
>> (IO connectors, runners, etc) should always try to use versions of
>> dependencies that are defined at the top level. Per-component dependency
>> version overrides should only be performed in rare cases and should come
>> with clear warnings for users.- Upgrading a dependency with an outdated
>> major version becomes a blocker for next major version release of Beam and
>> for any minor version releases after next immediate minor version release.
>> For example, if a dependency is identified to be outdated while the latest
>> release is x.y.z, upgrading this dependency becomes a blocker for releases
>> (x+1).0.0 and x.(y+2).0 of Beam. Additionally, upgrading to a major version
>> of a dependency will only be enforced if the new major version of the
>> dependency can be adapted without a significant rewrite to any Beam
>> component. Note that this policy intentionally allows one of the minor
>> version releases to proceed without upgrading the dependency which I
>> believe will give Beam community enough breathing room to upgrade
>> dependencies without significantly affecting the release frequency.-
>> Upgrading a dependency with a significantly outdated minor version (based
>> on methodology defined in [4]) becomes a blocker for next major version
>> release of Beam and for any minor version releases of Beam after next
>> immediate minor version release. Note that this policy does not force Beam
>> to adapt every minor version release of a dependency.- When performing a
>> release, release manager should make sure that blockers identified through
>> above process are resolved before the release candidate is cut.-
>> Optionally, dependency declarations may have comments that identify owners
>> that should be responsible for upgrading the respective dependencies.
>> Release manager may choose to assign a blocking JIRA for a dependency to
>> its owner.*
>> Thanks,
>> Cham
>>
>> On Thu, May 31, 2018 at 7:11 PM Chamikara Jayalath 
>> wrote:
>>
>>> Hi All,
>>>
>>> We recently ran into many issues due to Beam dependencies being
>>> significantly out of date. For example see [1], [2], and [3].
>>>
>>> Yifan Zou recently introduced a proposal [4] that would allow us to
>>> identify outdated dependencies. But to really make sure that this helps the
>>> Beam project and community I believe we should adapt several small policy
>>> changes to our development and release process.
>>>
>>> To this end, I have created following short document that identifies the
>>> dependency issue and proposes several policy changes. I greatly appreciate
>>> if you can take a look and comment.
>>>
>>>
>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-3098
>>> [2] https://issues.apache.org/jira/browse/BEAM-3991
>>> [3] https://issues.apache.org/jira/browse/BEAM-4229
>>> [4]
>>> https://lists.apache.org/thread.html/758625106a6cfe9ba23d7b39625da20e050c6279b138b18b3f0013e7@%3Cdev.beam.apache.org%3E
>>>
>>

Re: Managing outdated dependencies

[Scott Wegner]

Thanks Cham. Overall this seems like a useful hygiene improvement for the
project. I've left some comments in the doc.

On Fri, Jun 1, 2018 at 10:48 AM Chamikara Jayalath 
wrote:

> Hi All,
>
> I've copied ideas proposed in the doc below for more visibility. Any
> comments are welcome.
>
>
>
> * - Human readable per-SDK reports on status of Beam dependencies are
> generated weekly and shared with the Beam community through the dev list.
> These reports should be concise and should highlight the cases where the
> community has to act on. See [4] for more details on this.- Beam Components
> (IO connectors, runners, etc) should always try to use versions of
> dependencies that are defined at the top level. Per-component dependency
> version overrides should only be performed in rare cases and should come
> with clear warnings for users.- Upgrading a dependency with an outdated
> major version becomes a blocker for next major version release of Beam and
> for any minor version releases after next immediate minor version release.
> For example, if a dependency is identified to be outdated while the latest
> release is x.y.z, upgrading this dependency becomes a blocker for releases
> (x+1).0.0 and x.(y+2).0 of Beam. Additionally, upgrading to a major version
> of a dependency will only be enforced if the new major version of the
> dependency can be adapted without a significant rewrite to any Beam
> component. Note that this policy intentionally allows one of the minor
> version releases to proceed without upgrading the dependency which I
> believe will give Beam community enough breathing room to upgrade
> dependencies without significantly affecting the release frequency.-
> Upgrading a dependency with a significantly outdated minor version (based
> on methodology defined in [4]) becomes a blocker for next major version
> release of Beam and for any minor version releases of Beam after next
> immediate minor version release. Note that this policy does not force Beam
> to adapt every minor version release of a dependency.- When performing a
> release, release manager should make sure that blockers identified through
> above process are resolved before the release candidate is cut.-
> Optionally, dependency declarations may have comments that identify owners
> that should be responsible for upgrading the respective dependencies.
> Release manager may choose to assign a blocking JIRA for a dependency to
> its owner.*
> Thanks,
> Cham
>
> On Thu, May 31, 2018 at 7:11 PM Chamikara Jayalath 
> wrote:
>
>> Hi All,
>>
>> We recently ran into many issues due to Beam dependencies being
>> significantly out of date. For example see [1], [2], and [3].
>>
>> Yifan Zou recently introduced a proposal [4] that would allow us to
>> identify outdated dependencies. But to really make sure that this helps the
>> Beam project and community I believe we should adapt several small policy
>> changes to our development and release process.
>>
>> To this end, I have created following short document that identifies the
>> dependency issue and proposes several policy changes. I greatly appreciate
>> if you can take a look and comment.
>>
>>
>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>>
>> Thanks,
>> Cham
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-3098
>> [2] https://issues.apache.org/jira/browse/BEAM-3991
>> [3] https://issues.apache.org/jira/browse/BEAM-4229
>> [4]
>> https://lists.apache.org/thread.html/758625106a6cfe9ba23d7b39625da20e050c6279b138b18b3f0013e7@%3Cdev.beam.apache.org%3E
>>
>

Re: [VOTE] Code Review Process

[Andrew Pilloud]

+1 - I hope this doesn't reduce the urgency to fix the root cause: not
having enough committers.

On Fri, Jun 1, 2018 at 1:18 PM Henning Rohde  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 12:27 PM Dan Halperin  wrote:
>
>> +1 -- this is encoding what I previously thought the process was and
>> what, in practice, I think was often the behavior of committers anyway.
>>
>> On Fri, Jun 1, 2018 at 12:21 PM, Yifan Zou  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath <
 chamik...@google.com> wrote:

> +1
>
> Thanks,
> Cham
>
> On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
>>> wrote:
>>>
 As we seem to largely have consensus in "Reducing Committer
 Load for Code Reviews"[1], this is a vote to change the Beam 
 policy on Code
 Reviews to require that

 (1) At least one committer is involved with the code review, as
 either a reviewer or as the author
 (2) A contributor has approved the change

 prior to merging any change.

 This changes our policy from its current requirement that at
 least one committer *who is not the author* has approved the 
 change prior
 to merging. We believe that changing this process will improve 
 code review
 throughput, reduce committer load, and engage more of the 
 community in the
 code review process.

 Please vote:
 [ ] +1: Accept the above proposal to change the Beam code
 review/merge policy
 [ ] -1: Leave the Code Review policy unchanged

 Thanks,

 Thomas

 [1]
 https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E

>>>
>>
>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>
>>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Andrew Pilloud]

+1

On Fri, Jun 1, 2018 at 1:31 PM Huygaa Batsaikhan  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath 
>> wrote:
>>
>>> +1 (non-binding).
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner  wrote:

> +1 (non-binding)
>
> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
>> wrote:
>>
>>> +1 (non-binding): automating policy ensures it is applied fairly and
>>> evenly and lessens the load on project maintainers; hearty agreement.
>>>
>>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
>>> wrote:
>>>
 +1 (non-binding) I updated the pull request to be 60 days (instead
 of 90) to match the contribute policy.

 On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Following the discussion, please vote on the move to activate
> probot/stale [3] to notify authors of stale PRs per current
> policy and then close them after a 7 day grace period.
>
> For more details, see:
>
>  - our stale PR policy [1]
>  - the discussion thread [2]
>  - Probot stale [3]
>  - BEAM ticket summarizing discussion [4]
>  - INFRA ticket to activate probot/stale [5]
>  - Example PR that would activate it [6]
>
> Please vote:
> [ ] +1, Approve that we activate probot/stale
> [ ] -1, Do not approve (please provide specific comments)
>
> Kenn
>
> [1] https://beam.apache.org/contribute/#stale-pull-requests
> [2]
> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
> [3] https://github.com/probot/stale
> [4] https://issues.apache.org/jira/browse/BEAM-4423
> [5] https://issues.apache.org/jira/browse/INFRA-16589
> [6] https://github.com/apache/beam/pull/5532
>

>>>
>>> --
>>> ---
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>> See something? Say something. go/jasonkuster-feedback
>>> 
>>>
>>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Huygaa Batsaikhan]

+1

On Fri, Jun 1, 2018 at 1:17 PM Henning Rohde  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath 
> wrote:
>
>> +1 (non-binding).
>>
>> Thanks,
>> Cham
>>
>> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner  wrote:
>>>
 +1 (non-binding)

 On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:

> +1
>
> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
> wrote:
>
>> +1 (non-binding): automating policy ensures it is applied fairly and
>> evenly and lessens the load on project maintainers; hearty agreement.
>>
>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
>> wrote:
>>
>>> +1 (non-binding) I updated the pull request to be 60 days (instead
>>> of 90) to match the contribute policy.
>>>
>>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Following the discussion, please vote on the move to activate
 probot/stale [3] to notify authors of stale PRs per current policy
 and then close them after a 7 day grace period.

 For more details, see:

  - our stale PR policy [1]
  - the discussion thread [2]
  - Probot stale [3]
  - BEAM ticket summarizing discussion [4]
  - INFRA ticket to activate probot/stale [5]
  - Example PR that would activate it [6]

 Please vote:
 [ ] +1, Approve that we activate probot/stale
 [ ] -1, Do not approve (please provide specific comments)

 Kenn

 [1] https://beam.apache.org/contribute/#stale-pull-requests
 [2]
 https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
 [3] https://github.com/probot/stale
 [4] https://issues.apache.org/jira/browse/BEAM-4423
 [5] https://issues.apache.org/jira/browse/INFRA-16589
 [6] https://github.com/apache/beam/pull/5532

>>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: [VOTE] Code Review Process

[Henning Rohde]

+1

On Fri, Jun 1, 2018 at 12:27 PM Dan Halperin  wrote:

> +1 -- this is encoding what I previously thought the process was and what,
> in practice, I think was often the behavior of committers anyway.
>
> On Fri, Jun 1, 2018 at 12:21 PM, Yifan Zou  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath 
>>> wrote:
>>>
 +1

 Thanks,
 Cham

 On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
>> wrote:
>>
>>> As we seem to largely have consensus in "Reducing Committer Load
>>> for Code Reviews"[1], this is a vote to change the Beam policy on 
>>> Code
>>> Reviews to require that
>>>
>>> (1) At least one committer is involved with the code review, as
>>> either a reviewer or as the author
>>> (2) A contributor has approved the change
>>>
>>> prior to merging any change.
>>>
>>> This changes our policy from its current requirement that at
>>> least one committer *who is not the author* has approved the change 
>>> prior
>>> to merging. We believe that changing this process will improve code 
>>> review
>>> throughput, reduce committer load, and engage more of the community 
>>> in the
>>> code review process.
>>>
>>> Please vote:
>>> [ ] +1: Accept the above proposal to change the Beam code
>>> review/merge policy
>>> [ ] -1: Leave the Code Review policy unchanged
>>>
>>> Thanks,
>>>
>>> Thomas
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>>
>>
>

>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Henning Rohde]

+1

On Fri, Jun 1, 2018 at 10:16 AM Chamikara Jayalath 
wrote:

> +1 (non-binding).
>
> Thanks,
> Cham
>
> On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner  wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>>>
 +1

 On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
 wrote:

> +1 (non-binding): automating policy ensures it is applied fairly and
> evenly and lessens the load on project maintainers; hearty agreement.
>
> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
> wrote:
>
>> +1 (non-binding) I updated the pull request to be 60 days (instead of
>> 90) to match the contribute policy.
>>
>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Following the discussion, please vote on the move to activate
>>> probot/stale [3] to notify authors of stale PRs per current policy
>>> and then close them after a 7 day grace period.
>>>
>>> For more details, see:
>>>
>>>  - our stale PR policy [1]
>>>  - the discussion thread [2]
>>>  - Probot stale [3]
>>>  - BEAM ticket summarizing discussion [4]
>>>  - INFRA ticket to activate probot/stale [5]
>>>  - Example PR that would activate it [6]
>>>
>>> Please vote:
>>> [ ] +1, Approve that we activate probot/stale
>>> [ ] -1, Do not approve (please provide specific comments)
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>>> [2]
>>> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
>>> [3] https://github.com/probot/stale
>>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>>> [6] https://github.com/apache/beam/pull/5532
>>>
>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: [VOTE] Code Review Process

[Dan Halperin]

+1 -- this is encoding what I previously thought the process was and what,
in practice, I think was often the behavior of committers anyway.

On Fri, Jun 1, 2018 at 12:21 PM, Yifan Zou  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath 
>> wrote:
>>
>>> +1
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
> wrote:
>
>> As we seem to largely have consensus in "Reducing Committer Load
>> for Code Reviews"[1], this is a vote to change the Beam policy on 
>> Code
>> Reviews to require that
>>
>> (1) At least one committer is involved with the code review, as
>> either a reviewer or as the author
>> (2) A contributor has approved the change
>>
>> prior to merging any change.
>>
>> This changes our policy from its current requirement that at
>> least one committer *who is not the author* has approved the change 
>> prior
>> to merging. We believe that changing this process will improve code 
>> review
>> throughput, reduce committer load, and engage more of the community 
>> in the
>> code review process.
>>
>> Please vote:
>> [ ] +1: Accept the above proposal to change the Beam code
>> review/merge policy
>> [ ] -1: Leave the Code Review policy unchanged
>>
>> Thanks,
>>
>> Thomas
>>
>> [1] https://lists.apache.org/thread.html/7c1fde3884fbefacc25
>> 2b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>
>

>>>

 --
 ---
 Jason Kuster
 Apache Beam / Google Cloud Dataflow

 See something? Say something. go/jasonkuster-feedback
 

>>>

Re: [VOTE] Code Review Process

[Yifan Zou]

+1

On Fri, Jun 1, 2018 at 12:10 PM Robert Bradshaw  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath 
> wrote:
>
>> +1
>>
>> Thanks,
>> Cham
>>
>> On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
 wrote:

> As we seem to largely have consensus in "Reducing Committer Load
> for Code Reviews"[1], this is a vote to change the Beam policy on Code
> Reviews to require that
>
> (1) At least one committer is involved with the code review, as
> either a reviewer or as the author
> (2) A contributor has approved the change
>
> prior to merging any change.
>
> This changes our policy from its current requirement that at least
> one committer *who is not the author* has approved the change prior to
> merging. We believe that changing this process will improve code 
> review
> throughput, reduce committer load, and engage more of the community 
> in the
> code review process.
>
> Please vote:
> [ ] +1: Accept the above proposal to change the Beam code
> review/merge policy
> [ ] -1: Leave the Code Review policy unchanged
>
> Thanks,
>
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>

>>>
>>
>>>
>>> --
>>> ---
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>> See something? Say something. go/jasonkuster-feedback
>>> 
>>>
>>

[Call for items] Beam June Newsletter

[Griselda Cuevas]

Hi Everyone,

Here's

[1] the template for the June Beam Newsletter.

*Add the updates you want to share with the community by 6/5 11:59 p.m.*
*Pacific Time.*

I'll edit and send the final version through the users mailing list on 6/7.

Thank you!
Gris

[1]
https://docs.google.com/document/d/1BwRhOu-uDd3SLB_Om_Beke5RoGKos4hj7Ljh7zM2YIo/edit

Re: [VOTE] Code Review Process

[Robert Bradshaw]

+1

On Fri, Jun 1, 2018 at 12:06 PM Chamikara Jayalath 
wrote:

> +1
>
> Thanks,
> Cham
>
> On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh 
>>> wrote:
>>>
 As we seem to largely have consensus in "Reducing Committer Load
 for Code Reviews"[1], this is a vote to change the Beam policy on Code
 Reviews to require that

 (1) At least one committer is involved with the code review, as
 either a reviewer or as the author
 (2) A contributor has approved the change

 prior to merging any change.

 This changes our policy from its current requirement that at least
 one committer *who is not the author* has approved the change prior to
 merging. We believe that changing this process will improve code review
 throughput, reduce committer load, and engage more of the community in 
 the
 code review process.

 Please vote:
 [ ] +1: Accept the above proposal to change the Beam code
 review/merge policy
 [ ] -1: Leave the Code Review policy unchanged

 Thanks,

 Thomas

 [1]
 https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E

>>>
>>
>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: [VOTE] Code Review Process

[Chamikara Jayalath]

+1

Thanks,
Cham

On Fri, Jun 1, 2018 at 11:36 AM Jason Kuster  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev 
>>> wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:
>>
>>> As we seem to largely have consensus in "Reducing Committer Load for
>>> Code Reviews"[1], this is a vote to change the Beam policy on Code 
>>> Reviews
>>> to require that
>>>
>>> (1) At least one committer is involved with the code review, as
>>> either a reviewer or as the author
>>> (2) A contributor has approved the change
>>>
>>> prior to merging any change.
>>>
>>> This changes our policy from its current requirement that at least
>>> one committer *who is not the author* has approved the change prior to
>>> merging. We believe that changing this process will improve code review
>>> throughput, reduce committer load, and engage more of the community in 
>>> the
>>> code review process.
>>>
>>> Please vote:
>>> [ ] +1: Accept the above proposal to change the Beam code
>>> review/merge policy
>>> [ ] -1: Leave the Code Review policy unchanged
>>>
>>> Thanks,
>>>
>>> Thomas
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>>
>>
>

>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: [VOTE] Code Review Process

[Ankur Goenka]

+1

On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:

> As we seem to largely have consensus in "Reducing Committer Load for
> Code Reviews"[1], this is a vote to change the Beam policy on Code Reviews
> to require that
>
> (1) At least one committer is involved with the code review, as either
> a reviewer or as the author
> (2) A contributor has approved the change
>
> prior to merging any change.
>
> This changes our policy from its current requirement that at least one
> committer *who is not the author* has approved the change prior to 
> merging.
> We believe that changing this process will improve code review throughput,
> reduce committer load, and engage more of the community in the code review
> process.
>
> Please vote:
> [ ] +1: Accept the above proposal to change the Beam code review/merge
> policy
> [ ] -1: Leave the Code Review policy unchanged
>
> Thanks,
>
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>

>>>
>>

Re: [VOTE] Code Review Process

[Jason Kuster]

+1

On Fri, Jun 1, 2018 at 11:36 AM Ankur Goenka  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles 
 wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:
>
>> As we seem to largely have consensus in "Reducing Committer Load for
>> Code Reviews"[1], this is a vote to change the Beam policy on Code 
>> Reviews
>> to require that
>>
>> (1) At least one committer is involved with the code review, as
>> either a reviewer or as the author
>> (2) A contributor has approved the change
>>
>> prior to merging any change.
>>
>> This changes our policy from its current requirement that at least
>> one committer *who is not the author* has approved the change prior to
>> merging. We believe that changing this process will improve code review
>> throughput, reduce committer load, and engage more of the community in 
>> the
>> code review process.
>>
>> Please vote:
>> [ ] +1: Accept the above proposal to change the Beam code
>> review/merge policy
>> [ ] -1: Leave the Code Review policy unchanged
>>
>> Thanks,
>>
>> Thomas
>>
>> [1]
>> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>
>

>>>

-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Re: [VOTE] Code Review Process

[Huygaa Batsaikhan]

+1. This will allow non-committers to be actively involved in code reviews
and reduce committer load.

On Fri, Jun 1, 2018 at 11:28 AM Charles Chen  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev 
> wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles  wrote:
>>>
 +1

 On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:

> As we seem to largely have consensus in "Reducing Committer Load for
> Code Reviews"[1], this is a vote to change the Beam policy on Code Reviews
> to require that
>
> (1) At least one committer is involved with the code review, as either
> a reviewer or as the author
> (2) A contributor has approved the change
>
> prior to merging any change.
>
> This changes our policy from its current requirement that at least one
> committer *who is not the author* has approved the change prior to 
> merging.
> We believe that changing this process will improve code review throughput,
> reduce committer load, and engage more of the community in the code review
> process.
>
> Please vote:
> [ ] +1: Accept the above proposal to change the Beam code review/merge
> policy
> [ ] -1: Leave the Code Review policy unchanged
>
> Thanks,
>
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>

>>>
>>

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Huygaa Batsaikhan]

Congrats!

On Fri, Jun 1, 2018 at 10:26 AM Thomas Groh  wrote:

> Congrats, you three!
>
> On Thu, May 31, 2018 at 7:09 PM Davor Bonaci  wrote:
>
>> Please join me and the rest of Beam PMC in welcoming the following
>> contributors as our newest committers. They have significantly contributed
>> to the project in different ways, and we look forward to many more
>> contributions in the future.
>>
>> * Griselda Cuevas
>> * Pablo Estrada
>> * Jason Kuster
>>
>> (Apologizes for a delayed announcement, and the lack of the usual
>> paragraph summarizing individual contributions.)
>>
>> Congratulations to all three! Welcome!
>>
>

Re: [VOTE] Code Review Process

[Charles Chen]

+1

On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev 
wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:
>>>
 As we seem to largely have consensus in "Reducing Committer Load for
 Code Reviews"[1], this is a vote to change the Beam policy on Code Reviews
 to require that

 (1) At least one committer is involved with the code review, as either
 a reviewer or as the author
 (2) A contributor has approved the change

 prior to merging any change.

 This changes our policy from its current requirement that at least one
 committer *who is not the author* has approved the change prior to merging.
 We believe that changing this process will improve code review throughput,
 reduce committer load, and engage more of the community in the code review
 process.

 Please vote:
 [ ] +1: Accept the above proposal to change the Beam code review/merge
 policy
 [ ] -1: Leave the Code Review policy unchanged

 Thanks,

 Thomas

 [1]
 https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E

>>>
>>
>

Re: [VOTE] Code Review Process

[Valentyn Tymofieiev]

+1

On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:
>>
>>> As we seem to largely have consensus in "Reducing Committer Load for
>>> Code Reviews"[1], this is a vote to change the Beam policy on Code Reviews
>>> to require that
>>>
>>> (1) At least one committer is involved with the code review, as either a
>>> reviewer or as the author
>>> (2) A contributor has approved the change
>>>
>>> prior to merging any change.
>>>
>>> This changes our policy from its current requirement that at least one
>>> committer *who is not the author* has approved the change prior to merging.
>>> We believe that changing this process will improve code review throughput,
>>> reduce committer load, and engage more of the community in the code review
>>> process.
>>>
>>> Please vote:
>>> [ ] +1: Accept the above proposal to change the Beam code review/merge
>>> policy
>>> [ ] -1: Leave the Code Review policy unchanged
>>>
>>> Thanks,
>>>
>>> Thomas
>>>
>>> [1] https://lists.apache.org/thread.html/7c1fde3884fbefacc25
>>> 2b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>>
>>
>

Re: Managing outdated dependencies

[Chamikara Jayalath]

Hi All,

I've copied ideas proposed in the doc below for more visibility. Any
comments are welcome.



* - Human readable per-SDK reports on status of Beam dependencies are
generated weekly and shared with the Beam community through the dev list.
These reports should be concise and should highlight the cases where the
community has to act on. See [4] for more details on this.- Beam Components
(IO connectors, runners, etc) should always try to use versions of
dependencies that are defined at the top level. Per-component dependency
version overrides should only be performed in rare cases and should come
with clear warnings for users.- Upgrading a dependency with an outdated
major version becomes a blocker for next major version release of Beam and
for any minor version releases after next immediate minor version release.
For example, if a dependency is identified to be outdated while the latest
release is x.y.z, upgrading this dependency becomes a blocker for releases
(x+1).0.0 and x.(y+2).0 of Beam. Additionally, upgrading to a major version
of a dependency will only be enforced if the new major version of the
dependency can be adapted without a significant rewrite to any Beam
component. Note that this policy intentionally allows one of the minor
version releases to proceed without upgrading the dependency which I
believe will give Beam community enough breathing room to upgrade
dependencies without significantly affecting the release frequency.-
Upgrading a dependency with a significantly outdated minor version (based
on methodology defined in [4]) becomes a blocker for next major version
release of Beam and for any minor version releases of Beam after next
immediate minor version release. Note that this policy does not force Beam
to adapt every minor version release of a dependency.- When performing a
release, release manager should make sure that blockers identified through
above process are resolved before the release candidate is cut.-
Optionally, dependency declarations may have comments that identify owners
that should be responsible for upgrading the respective dependencies.
Release manager may choose to assign a blocking JIRA for a dependency to
its owner.*
Thanks,
Cham

On Thu, May 31, 2018 at 7:11 PM Chamikara Jayalath 
wrote:

> Hi All,
>
> We recently ran into many issues due to Beam dependencies being
> significantly out of date. For example see [1], [2], and [3].
>
> Yifan Zou recently introduced a proposal [4] that would allow us to
> identify outdated dependencies. But to really make sure that this helps the
> Beam project and community I believe we should adapt several small policy
> changes to our development and release process.
>
> To this end, I have created following short document that identifies the
> dependency issue and proposes several policy changes. I greatly appreciate
> if you can take a look and comment.
>
>
> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing
>
> Thanks,
> Cham
>
> [1] https://issues.apache.org/jira/browse/BEAM-3098
> [2] https://issues.apache.org/jira/browse/BEAM-3991
> [3] https://issues.apache.org/jira/browse/BEAM-4229
> [4]
> https://lists.apache.org/thread.html/758625106a6cfe9ba23d7b39625da20e050c6279b138b18b3f0013e7@%3Cdev.beam.apache.org%3E
>

Re: [SQL] Unsupported features

[Kai Jiang]

Sounds a good idea! I will file the major problems later and use a task
issue to track.

Best,
Kai
ᐧ

On Fri, Jun 1, 2018 at 10:10 AM Anton Kedin  wrote:

> This looks very helpful, thank you.
>
> Can you file Jiras for the major problems? Or maybe a single jira for the
> whole thing with sub-tasks for specific problems.
>
> Regards,
> Anton
>
> On Wed, May 30, 2018 at 9:12 AM Kenneth Knowles  wrote:
>
>> This is extremely useful. Thanks for putting so much information together!
>>
>> Kenn
>>
>> On Wed, May 30, 2018 at 8:19 AM Kai Jiang  wrote:
>>
>>> Hi all,
>>>
>>> Based on pull/5481 , I
>>> manually did a coverage test with TPC-ds queries (65%) and TPC-h queries
>>> (100%) and want to see what features Beam SQL is currently not supporting.
>>> Test was running on DirectRunner.
>>>
>>> I want to share the result.​
>>>  TPC-DS queries on Beam
>>> 
>>> ​
>>> TL;DR:
>>>
>>>1. aggregation function (stddev) missing or calculation of
>>>aggregation functions combination.
>>>2. nested beamjoinrel(condition=[true], joinType=[inner]) / cross
>>>join error
>>>3. date type casting/ calculation and other types casting.
>>>4. LIKE operator in String / alias for substring function
>>>5. order by w/o limit clause.
>>>6. OR operator is supported in join condition
>>>7. Syntax: exist/ not exist (errors) .rank() over (partition by)
>>>/ view (unsupported)
>>>
>>>
>>> Best,
>>> Kai
>>> ᐧ
>>>
>>

Re: [VOTE] Code Review Process

[Ahmet Altay]

+1

On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:
>
>> As we seem to largely have consensus in "Reducing Committer Load for Code
>> Reviews"[1], this is a vote to change the Beam policy on Code Reviews to
>> require that
>>
>> (1) At least one committer is involved with the code review, as either a
>> reviewer or as the author
>> (2) A contributor has approved the change
>>
>> prior to merging any change.
>>
>> This changes our policy from its current requirement that at least one
>> committer *who is not the author* has approved the change prior to merging.
>> We believe that changing this process will improve code review throughput,
>> reduce committer load, and engage more of the community in the code review
>> process.
>>
>> Please vote:
>> [ ] +1: Accept the above proposal to change the Beam code review/merge
>> policy
>> [ ] -1: Leave the Code Review policy unchanged
>>
>> Thanks,
>>
>> Thomas
>>
>> [1] https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9
>> a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>>
>

Re: [VOTE] Code Review Process

[Kenneth Knowles]

+1

On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh  wrote:

> As we seem to largely have consensus in "Reducing Committer Load for Code
> Reviews"[1], this is a vote to change the Beam policy on Code Reviews to
> require that
>
> (1) At least one committer is involved with the code review, as either a
> reviewer or as the author
> (2) A contributor has approved the change
>
> prior to merging any change.
>
> This changes our policy from its current requirement that at least one
> committer *who is not the author* has approved the change prior to merging.
> We believe that changing this process will improve code review throughput,
> reduce committer load, and engage more of the community in the code review
> process.
>
> Please vote:
> [ ] +1: Accept the above proposal to change the Beam code review/merge
> policy
> [ ] -1: Leave the Code Review policy unchanged
>
> Thanks,
>
> Thomas
>
> [1]
> https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E
>

Re: Design Proposal: Beam-Site Automation Reliability

[Scott Wegner]

Pre-commit filtering has come up on previous discussions as well and is an
obvious improvement. I've opened BEAM-4445 [1] for this and assigned it to
myself.

[1] https://issues.apache.org/jira/browse/BEAM-4445

On Fri, Jun 1, 2018 at 10:01 AM Kenneth Knowles  wrote:

> +1
>
> Can we separate precommit filtering and get it set up independent from
> this? I think there's a lot of good directions to go once it is the norm.
>
> On Thu, May 31, 2018 at 9:25 PM Thomas Weise  wrote:
>
>> Very nice, enthusiastic +1
>>
>> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner  wrote:
>>
>>> Thanks to everyone who reviewed the doc. I put together a plan based on
>>> the initial feedback to improve website automation reliability. At a
>>> glance, I am proposing to:
>>>
>>> * Migrate website source code to the main apache/beam repository
>>> * Discontinue checking-in generated HTML during the PR workflow
>>> * Align to the existing apache/beam PR process (code review policy,
>>> precommits, generic Git merge)
>>> * Filter pre-commit jobs to only run when necessary
>>> * Add a post-commit Jenkins job to push generated HTML to a separate
>>> publishing branch
>>>
>>> Please take another look at the doc, specifically the new section
>>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>>> I'd like to gather feedback by Monday June 4, and if there is consensus
>>> move forward with the implementation.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner  wrote:
>>>
 I've been looking into the beam-site merge automation reliability, and
 I'd like to get some early feedback on ideas for improvement. Please take a
 look at https://s.apache.org/beam-site-automation:

 > Apache Beam's website is maintained via the beam-site Git repository,
 with a set of automation that manages the workflow from merging a pull
 request to publishing. The automation is centralized in a tool called
 Mergebot, which was built for Beam and donated to the ASF. However, the
 automation has been somewhat unreliable, and when there are issues, very
 few individuals have the necessary permissions and expertise to resolve
 them. Overall, the reliability of Beam-site automation is impeding
 productivity for Beam-site development.

 At this point I'm seeking feedback on a few possible solutions:

 1. Invest in improvements to Mergebot reliability. Make stability
 tweaks for various failure modes, distribute Mergebot expertise and
 operations permissions to more committers.
 2. Deprecate Mergebot and revert to manual process. With the current
 unreliability, some committers choose to forego merge automation anyway.
 3. Generate HTML only during publishing. This seems to be newly
 supported by the Apache GitPubSub workflow. This would eliminate most or
 all of the automation that Mergebot is responsible for.

 Feel free to add comments in the doc.

 Thanks,
 Scott



 Got feedback? tinyurl.com/swegner-feedback

>>>
>>

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Thomas Groh]

Congrats, you three!

On Thu, May 31, 2018 at 7:09 PM Davor Bonaci  wrote:

> Please join me and the rest of Beam PMC in welcoming the following
> contributors as our newest committers. They have significantly contributed
> to the project in different ways, and we look forward to many more
> contributions in the future.
>
> * Griselda Cuevas
> * Pablo Estrada
> * Jason Kuster
>
> (Apologizes for a delayed announcement, and the lack of the usual
> paragraph summarizing individual contributions.)
>
> Congratulations to all three! Welcome!
>

Re: SQL shaded jars don't work. How to test?

[Andrew Pilloud]

For the shadowJar issue see https://issues.apache.org/jira/browse/BEAM-4402.
There are 280 test failures, but many of them are the same issue.

I believe there is significant work on both, but I think there is
significant value as well. I suppose there should be a vote on applying
these rules to the whole project and a fix it day to make it happen.

Andrew

On Fri, Jun 1, 2018 at 9:39 AM Kenneth Knowles  wrote:

> Spotless is slightly off topic and less serious than test coverage
> problems, but see https://issues.apache.org/jira/browse/BEAM-4394.
>
> On Fri, Jun 1, 2018 at 9:30 AM Scott Wegner  wrote:
>
>> Andrew, it sounds like you're suggesting testShadowJar and enableSpotless
>> should also be the default for all modules? Do you have an idea of how much
>> effort is required for migrating? For failOnWarning / ErrorProne, the
>> migration is pretty straightforward, so I created starter bugs for each
>> module [1] with step-by-step instructions that anybody could pick up, and
>> so far it's been fairly successful-- only 13 out of 47 left to go. Do we
>> have anything tracking the work for testShadowJar / spotless?
>>
>> [1] https://issues.apache.org/jira/issues/?jql=labels+%3D+errorprone
>>
>>
>> On Thu, May 31, 2018 at 9:20 AM Andrew Pilloud 
>> wrote:
>>
>>> There is now a testShadowJar option on applyJavaNature, which allows you
>>> to run your tests against the shaded jar. Everyone should turn it on for
>>> their module along with failOnWarning and enableSpotless for maximum
>>> testing.
>>>
>>> On Thu, May 24, 2018 at 1:24 PM Andrew Pilloud 
>>> wrote:
>>>
 I've make the SQL PR able to be merged standalone so I can get back to
 work. I've also opened issues to track the things I found and dumped my
 work in progress into a PR.

 https://issues.apache.org/jira/browse/BEAM-4402
 https://issues.apache.org/jira/browse/BEAM-4403
 https://github.com/apache/beam/pull/5471

 Andrew

 On Thu, May 24, 2018 at 11:54 AM Kenneth Knowles 
 wrote:

> If it is taking days of work we should definitely put it behind a flag
> and do it incrementally, find a way to share the work.
>
> If our tests aren't running on the actual artifacts, then we don't
> really have assurance that they work. That should interest just about
> everyone. (though it is also status quo relative to mvn)
>
> Kenn
>
> On Thu, May 24, 2018 at 10:17 AM Andrew Pilloud 
> wrote:
>
>> I think I'm down to 11 packages with failures, some of which might be
>> precommits that don't run outside of Jenkins. Its not an issue of 
>> figuring
>> them out, it is an issue of time spent doing so.
>>
>> On Thu, May 24, 2018 at 9:31 AM Lukasz Cwik  wrote:
>>
>>> Were you able to figure out how to fix them or still having problems?
>>>
>>> On Thu, May 24, 2018 at 9:27 AM Andrew Pilloud 
>>> wrote:
>>>
 I spent all of yesterday investigating and fixing dependency issues
 outside of SQL. I really regret the decision to write a test for this.
 Would it be acceptable for us to put testing with the output jar 
 behind a
 flag like we do for failOnWarning?

 On Wed, May 23, 2018 at 5:21 PM Kenneth Knowles 
 wrote:

> What's the status of moving it forward? Is it a ton of work / too
> much to do quickly?
>
> On Wed, May 23, 2018 at 9:11 AM Andrew Pilloud <
> apill...@google.com> wrote:
>
>> To loop the list in on discussions going on in
>> https://github.com/apache/beam/pull/5443: our normal tests don't
>> run against the shaded jars. Gradle can run the tests against the 
>> shaded
>> jars, but a bunch fail due to dependency issues. It's not just SQL.
>>
>> Andrew
>>
>> On Mon, May 21, 2018 at 11:35 AM Lukasz Cwik 
>> wrote:
>>
>>> Shading requires two pieces of information:
>>> 1) Which dependencies should be part of the shaded jar
>>> (controlled by includes/excludes)
>>> 2) How to relocate code within those dependencies (controlled by
>>> relocations)
>>>
>>> The reason why the exclude(".*") exists is because typically it
>>> is an error to produce a shaded package with dependencies which are 
>>> not
>>> relocated. When libraries do this, it causes a lot of
>>> NoClassFound/NoMethodFound errors for users since a user can't know 
>>> which
>>> version of a dependency they are actually getting (the one that was 
>>> bundled
>>> part of your jar or the one they depend on as a library). Only 
>>> applications
>>> should ever really do this, libraries should always repackage all 
>>> their
>>> code to prevent such errors.
>>>
>>> Note that in the SQL 

[VOTE] Code Review Process

[Thomas Groh]

As we seem to largely have consensus in "Reducing Committer Load for Code
Reviews"[1], this is a vote to change the Beam policy on Code Reviews to
require that

(1) At least one committer is involved with the code review, as either a
reviewer or as the author
(2) A contributor has approved the change

prior to merging any change.

This changes our policy from its current requirement that at least one
committer *who is not the author* has approved the change prior to merging.
We believe that changing this process will improve code review throughput,
reduce committer load, and engage more of the community in the code review
process.

Please vote:
[ ] +1: Accept the above proposal to change the Beam code review/merge
policy
[ ] -1: Leave the Code Review policy unchanged

Thanks,

Thomas

[1]
https://lists.apache.org/thread.html/7c1fde3884fbefacc252b6d4b434f9a9c2cf024f381654aa3e47df18@%3Cdev.beam.apache.org%3E

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Chamikara Jayalath]

+1 (non-binding).

Thanks,
Cham

On Fri, Jun 1, 2018 at 10:05 AM Kenneth Knowles  wrote:

> +1
>
> On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner  wrote:
>
>> +1 (non-binding)
>>
>> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>>
>>> +1
>>>
>>> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster 
>>> wrote:
>>>
 +1 (non-binding): automating policy ensures it is applied fairly and
 evenly and lessens the load on project maintainers; hearty agreement.

 On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold 
 wrote:

> +1 (non-binding) I updated the pull request to be 60 days (instead of
> 90) to match the contribute policy.
>
> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Following the discussion, please vote on the move to activate
>> probot/stale [3] to notify authors of stale PRs per current policy
>> and then close them after a 7 day grace period.
>>
>> For more details, see:
>>
>>  - our stale PR policy [1]
>>  - the discussion thread [2]
>>  - Probot stale [3]
>>  - BEAM ticket summarizing discussion [4]
>>  - INFRA ticket to activate probot/stale [5]
>>  - Example PR that would activate it [6]
>>
>> Please vote:
>> [ ] +1, Approve that we activate probot/stale
>> [ ] -1, Do not approve (please provide specific comments)
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>> [2]
>> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
>> [3] https://github.com/probot/stale
>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>> [6] https://github.com/apache/beam/pull/5532
>>
>

 --
 ---
 Jason Kuster
 Apache Beam / Google Cloud Dataflow

 See something? Say something. go/jasonkuster-feedback
 

>>>

Re: [SQL] Unsupported features

[Anton Kedin]

This looks very helpful, thank you.

Can you file Jiras for the major problems? Or maybe a single jira for the
whole thing with sub-tasks for specific problems.

Regards,
Anton

On Wed, May 30, 2018 at 9:12 AM Kenneth Knowles  wrote:

> This is extremely useful. Thanks for putting so much information together!
>
> Kenn
>
> On Wed, May 30, 2018 at 8:19 AM Kai Jiang  wrote:
>
>> Hi all,
>>
>> Based on pull/5481 , I
>> manually did a coverage test with TPC-ds queries (65%) and TPC-h queries
>> (100%) and want to see what features Beam SQL is currently not supporting.
>> Test was running on DirectRunner.
>>
>> I want to share the result.​
>>  TPC-DS queries on Beam
>> 
>> ​
>> TL;DR:
>>
>>1. aggregation function (stddev) missing or calculation of
>>aggregation functions combination.
>>2. nested beamjoinrel(condition=[true], joinType=[inner]) / cross
>>join error
>>3. date type casting/ calculation and other types casting.
>>4. LIKE operator in String / alias for substring function
>>5. order by w/o limit clause.
>>6. OR operator is supported in join condition
>>7. Syntax: exist/ not exist (errors) .rank() over (partition by)
>>/ view (unsupported)
>>
>>
>> Best,
>> Kai
>> ᐧ
>>
>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Kenneth Knowles]

+1

On Fri, Jun 1, 2018 at 9:54 AM Scott Wegner  wrote:

> +1 (non-binding)
>
> On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:
>
>> +1
>>
>> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster  wrote:
>>
>>> +1 (non-binding): automating policy ensures it is applied fairly and
>>> evenly and lessens the load on project maintainers; hearty agreement.
>>>
>>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold  wrote:
>>>
 +1 (non-binding) I updated the pull request to be 60 days (instead of
 90) to match the contribute policy.

 On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:

> Hi all,
>
> Following the discussion, please vote on the move to activate
> probot/stale [3] to notify authors of stale PRs per current policy
> and then close them after a 7 day grace period.
>
> For more details, see:
>
>  - our stale PR policy [1]
>  - the discussion thread [2]
>  - Probot stale [3]
>  - BEAM ticket summarizing discussion [4]
>  - INFRA ticket to activate probot/stale [5]
>  - Example PR that would activate it [6]
>
> Please vote:
> [ ] +1, Approve that we activate probot/stale
> [ ] -1, Do not approve (please provide specific comments)
>
> Kenn
>
> [1] https://beam.apache.org/contribute/#stale-pull-requests
> [2]
> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
> [3] https://github.com/probot/stale
> [4] https://issues.apache.org/jira/browse/BEAM-4423
> [5] https://issues.apache.org/jira/browse/INFRA-16589
> [6] https://github.com/apache/beam/pull/5532
>

>>>
>>> --
>>> ---
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>> See something? Say something. go/jasonkuster-feedback
>>> 
>>>
>>

Re: Design Proposal: Beam-Site Automation Reliability

[Kenneth Knowles]

+1

Can we separate precommit filtering and get it set up independent from
this? I think there's a lot of good directions to go once it is the norm.

On Thu, May 31, 2018 at 9:25 PM Thomas Weise  wrote:

> Very nice, enthusiastic +1
>
> On Thu, May 31, 2018 at 3:24 PM, Scott Wegner  wrote:
>
>> Thanks to everyone who reviewed the doc. I put together a plan based on
>> the initial feedback to improve website automation reliability. At a
>> glance, I am proposing to:
>>
>> * Migrate website source code to the main apache/beam repository
>> * Discontinue checking-in generated HTML during the PR workflow
>> * Align to the existing apache/beam PR process (code review policy,
>> precommits, generic Git merge)
>> * Filter pre-commit jobs to only run when necessary
>> * Add a post-commit Jenkins job to push generated HTML to a separate
>> publishing branch
>>
>> Please take another look at the doc, specifically the new section
>> entitled "Proposed Solution": https://s.apache.org/beam-site-automation
>> I'd like to gather feedback by Monday June 4, and if there is consensus
>> move forward with the implementation.
>>
>> Thanks,
>> Scott
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>> On Tue, May 29, 2018 at 4:32 PM Scott Wegner  wrote:
>>
>>> I've been looking into the beam-site merge automation reliability, and
>>> I'd like to get some early feedback on ideas for improvement. Please take a
>>> look at https://s.apache.org/beam-site-automation:
>>>
>>> > Apache Beam's website is maintained via the beam-site Git repository,
>>> with a set of automation that manages the workflow from merging a pull
>>> request to publishing. The automation is centralized in a tool called
>>> Mergebot, which was built for Beam and donated to the ASF. However, the
>>> automation has been somewhat unreliable, and when there are issues, very
>>> few individuals have the necessary permissions and expertise to resolve
>>> them. Overall, the reliability of Beam-site automation is impeding
>>> productivity for Beam-site development.
>>>
>>> At this point I'm seeking feedback on a few possible solutions:
>>>
>>> 1. Invest in improvements to Mergebot reliability. Make stability tweaks
>>> for various failure modes, distribute Mergebot expertise and operations
>>> permissions to more committers.
>>> 2. Deprecate Mergebot and revert to manual process. With the current
>>> unreliability, some committers choose to forego merge automation anyway.
>>> 3. Generate HTML only during publishing. This seems to be newly
>>> supported by the Apache GitPubSub workflow. This would eliminate most or
>>> all of the automation that Mergebot is responsible for.
>>>
>>> Feel free to add comments in the doc.
>>>
>>> Thanks,
>>> Scott
>>>
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>
>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Scott Wegner]

+1 (non-binding)

On Fri, Jun 1, 2018 at 9:39 AM Ahmet Altay  wrote:

> +1
>
> On Fri, Jun 1, 2018, 9:32 AM Jason Kuster  wrote:
>
>> +1 (non-binding): automating policy ensures it is applied fairly and
>> evenly and lessens the load on project maintainers; hearty agreement.
>>
>> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold  wrote:
>>
>>> +1 (non-binding) I updated the pull request to be 60 days (instead of
>>> 90) to match the contribute policy.
>>>
>>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:
>>>
 Hi all,

 Following the discussion, please vote on the move to activate
 probot/stale [3] to notify authors of stale PRs per current policy and
 then close them after a 7 day grace period.

 For more details, see:

  - our stale PR policy [1]
  - the discussion thread [2]
  - Probot stale [3]
  - BEAM ticket summarizing discussion [4]
  - INFRA ticket to activate probot/stale [5]
  - Example PR that would activate it [6]

 Please vote:
 [ ] +1, Approve that we activate probot/stale
 [ ] -1, Do not approve (please provide specific comments)

 Kenn

 [1] https://beam.apache.org/contribute/#stale-pull-requests
 [2]
 https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
 [3] https://github.com/probot/stale
 [4] https://issues.apache.org/jira/browse/BEAM-4423
 [5] https://issues.apache.org/jira/browse/INFRA-16589
 [6] https://github.com/apache/beam/pull/5532

>>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Mark Liu]

Congratulations!

On Fri, Jun 1, 2018 at 9:41 AM Robin Qiu  wrote:

> Congrats to all!
>
> On Fri, Jun 1, 2018 at 8:42 AM Henning Rohde  wrote:
>
>> Congratulations!
>>
>> On Fri, Jun 1, 2018 at 7:03 AM Jesse Anderson 
>> wrote:
>>
>>> Welcome!
>>>
>>> On Fri, Jun 1, 2018, 2:02 AM Etienne Chauchot 
>>> wrote:
>>>
 Congrats to all !
 Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :

 Please join me and the rest of Beam PMC in welcoming the following
 contributors as our newest committers. They have significantly contributed
 to the project in different ways, and we look forward to many more
 contributions in the future.

 * Griselda Cuevas
 * Pablo Estrada
 * Jason Kuster

 (Apologizes for a delayed announcement, and the lack of the usual
 paragraph summarizing individual contributions.)

 Congratulations to all three! Welcome!

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Robin Qiu]

Congrats to all!

On Fri, Jun 1, 2018 at 8:42 AM Henning Rohde  wrote:

> Congratulations!
>
> On Fri, Jun 1, 2018 at 7:03 AM Jesse Anderson 
> wrote:
>
>> Welcome!
>>
>> On Fri, Jun 1, 2018, 2:02 AM Etienne Chauchot 
>> wrote:
>>
>>> Congrats to all !
>>> Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :
>>>
>>> Please join me and the rest of Beam PMC in welcoming the following
>>> contributors as our newest committers. They have significantly contributed
>>> to the project in different ways, and we look forward to many more
>>> contributions in the future.
>>>
>>> * Griselda Cuevas
>>> * Pablo Estrada
>>> * Jason Kuster
>>>
>>> (Apologizes for a delayed announcement, and the lack of the usual
>>> paragraph summarizing individual contributions.)
>>>
>>> Congratulations to all three! Welcome!
>>>
>>>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Ahmet Altay]

+1

On Fri, Jun 1, 2018, 9:32 AM Jason Kuster  wrote:

> +1 (non-binding): automating policy ensures it is applied fairly and
> evenly and lessens the load on project maintainers; hearty agreement.
>
> On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold  wrote:
>
>> +1 (non-binding) I updated the pull request to be 60 days (instead of 90)
>> to match the contribute policy.
>>
>> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Following the discussion, please vote on the move to activate
>>> probot/stale [3] to notify authors of stale PRs per current policy and
>>> then close them after a 7 day grace period.
>>>
>>> For more details, see:
>>>
>>>  - our stale PR policy [1]
>>>  - the discussion thread [2]
>>>  - Probot stale [3]
>>>  - BEAM ticket summarizing discussion [4]
>>>  - INFRA ticket to activate probot/stale [5]
>>>  - Example PR that would activate it [6]
>>>
>>> Please vote:
>>> [ ] +1, Approve that we activate probot/stale
>>> [ ] -1, Do not approve (please provide specific comments)
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>>> [2]
>>> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
>>> [3] https://github.com/probot/stale
>>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>>> [6] https://github.com/apache/beam/pull/5532
>>>
>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
>

Re: SQL shaded jars don't work. How to test?

[Kenneth Knowles]

Spotless is slightly off topic and less serious than test coverage
problems, but see https://issues.apache.org/jira/browse/BEAM-4394.

On Fri, Jun 1, 2018 at 9:30 AM Scott Wegner  wrote:

> Andrew, it sounds like you're suggesting testShadowJar and enableSpotless
> should also be the default for all modules? Do you have an idea of how much
> effort is required for migrating? For failOnWarning / ErrorProne, the
> migration is pretty straightforward, so I created starter bugs for each
> module [1] with step-by-step instructions that anybody could pick up, and
> so far it's been fairly successful-- only 13 out of 47 left to go. Do we
> have anything tracking the work for testShadowJar / spotless?
>
> [1] https://issues.apache.org/jira/issues/?jql=labels+%3D+errorprone
>
>
> On Thu, May 31, 2018 at 9:20 AM Andrew Pilloud 
> wrote:
>
>> There is now a testShadowJar option on applyJavaNature, which allows you
>> to run your tests against the shaded jar. Everyone should turn it on for
>> their module along with failOnWarning and enableSpotless for maximum
>> testing.
>>
>> On Thu, May 24, 2018 at 1:24 PM Andrew Pilloud 
>> wrote:
>>
>>> I've make the SQL PR able to be merged standalone so I can get back to
>>> work. I've also opened issues to track the things I found and dumped my
>>> work in progress into a PR.
>>>
>>> https://issues.apache.org/jira/browse/BEAM-4402
>>> https://issues.apache.org/jira/browse/BEAM-4403
>>> https://github.com/apache/beam/pull/5471
>>>
>>> Andrew
>>>
>>> On Thu, May 24, 2018 at 11:54 AM Kenneth Knowles  wrote:
>>>
 If it is taking days of work we should definitely put it behind a flag
 and do it incrementally, find a way to share the work.

 If our tests aren't running on the actual artifacts, then we don't
 really have assurance that they work. That should interest just about
 everyone. (though it is also status quo relative to mvn)

 Kenn

 On Thu, May 24, 2018 at 10:17 AM Andrew Pilloud 
 wrote:

> I think I'm down to 11 packages with failures, some of which might be
> precommits that don't run outside of Jenkins. Its not an issue of figuring
> them out, it is an issue of time spent doing so.
>
> On Thu, May 24, 2018 at 9:31 AM Lukasz Cwik  wrote:
>
>> Were you able to figure out how to fix them or still having problems?
>>
>> On Thu, May 24, 2018 at 9:27 AM Andrew Pilloud 
>> wrote:
>>
>>> I spent all of yesterday investigating and fixing dependency issues
>>> outside of SQL. I really regret the decision to write a test for this.
>>> Would it be acceptable for us to put testing with the output jar behind 
>>> a
>>> flag like we do for failOnWarning?
>>>
>>> On Wed, May 23, 2018 at 5:21 PM Kenneth Knowles 
>>> wrote:
>>>
 What's the status of moving it forward? Is it a ton of work / too
 much to do quickly?

 On Wed, May 23, 2018 at 9:11 AM Andrew Pilloud 
 wrote:

> To loop the list in on discussions going on in
> https://github.com/apache/beam/pull/5443: our normal tests don't
> run against the shaded jars. Gradle can run the tests against the 
> shaded
> jars, but a bunch fail due to dependency issues. It's not just SQL.
>
> Andrew
>
> On Mon, May 21, 2018 at 11:35 AM Lukasz Cwik 
> wrote:
>
>> Shading requires two pieces of information:
>> 1) Which dependencies should be part of the shaded jar
>> (controlled by includes/excludes)
>> 2) How to relocate code within those dependencies (controlled by
>> relocations)
>>
>> The reason why the exclude(".*") exists is because typically it
>> is an error to produce a shaded package with dependencies which are 
>> not
>> relocated. When libraries do this, it causes a lot of
>> NoClassFound/NoMethodFound errors for users since a user can't know 
>> which
>> version of a dependency they are actually getting (the one that was 
>> bundled
>> part of your jar or the one they depend on as a library). Only 
>> applications
>> should ever really do this, libraries should always repackage all 
>> their
>> code to prevent such errors.
>>
>> Note that in the SQL package, you can provide your own
>> shadowClosure to the applyJavaNature() which means that the default 
>> won't
>> apply. For example:
>> https://github.com/apache/beam/blob/a3ba6a0e8de3ae72b8fc6fc6038eb9dc725f092e/sdks/java/harness/build.gradle#L20
>> and remove the 'DEFAULT_SHADOW_CLOSURE <<'
>>
>>
>> On Mon, May 21, 2018 at 10:26 AM Andrew Pilloud <
>> apill...@google.com> wrote:
>>
>>> The issue SQL is seeing is caused by a default dependency of
>>> exclude(".*") 

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Jason Kuster]

+1 (non-binding): automating policy ensures it is applied fairly and evenly
and lessens the load on project maintainers; hearty agreement.

On Fri, Jun 1, 2018 at 9:25 AM Alan Myrvold  wrote:

> +1 (non-binding) I updated the pull request to be 60 days (instead of 90)
> to match the contribute policy.
>
> On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Following the discussion, please vote on the move to activate
>> probot/stale [3] to notify authors of stale PRs per current policy and
>> then close them after a 7 day grace period.
>>
>> For more details, see:
>>
>>  - our stale PR policy [1]
>>  - the discussion thread [2]
>>  - Probot stale [3]
>>  - BEAM ticket summarizing discussion [4]
>>  - INFRA ticket to activate probot/stale [5]
>>  - Example PR that would activate it [6]
>>
>> Please vote:
>> [ ] +1, Approve that we activate probot/stale
>> [ ] -1, Do not approve (please provide specific comments)
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/#stale-pull-requests
>> [2]
>> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
>> [3] https://github.com/probot/stale
>> [4] https://issues.apache.org/jira/browse/BEAM-4423
>> [5] https://issues.apache.org/jira/browse/INFRA-16589
>> [6] https://github.com/apache/beam/pull/5532
>>
>

-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Re: SQL shaded jars don't work. How to test?

[Scott Wegner]

Andrew, it sounds like you're suggesting testShadowJar and enableSpotless
should also be the default for all modules? Do you have an idea of how much
effort is required for migrating? For failOnWarning / ErrorProne, the
migration is pretty straightforward, so I created starter bugs for each
module [1] with step-by-step instructions that anybody could pick up, and
so far it's been fairly successful-- only 13 out of 47 left to go. Do we
have anything tracking the work for testShadowJar / spotless?

[1] https://issues.apache.org/jira/issues/?jql=labels+%3D+errorprone


On Thu, May 31, 2018 at 9:20 AM Andrew Pilloud  wrote:

> There is now a testShadowJar option on applyJavaNature, which allows you
> to run your tests against the shaded jar. Everyone should turn it on for
> their module along with failOnWarning and enableSpotless for maximum
> testing.
>
> On Thu, May 24, 2018 at 1:24 PM Andrew Pilloud 
> wrote:
>
>> I've make the SQL PR able to be merged standalone so I can get back to
>> work. I've also opened issues to track the things I found and dumped my
>> work in progress into a PR.
>>
>> https://issues.apache.org/jira/browse/BEAM-4402
>> https://issues.apache.org/jira/browse/BEAM-4403
>> https://github.com/apache/beam/pull/5471
>>
>> Andrew
>>
>> On Thu, May 24, 2018 at 11:54 AM Kenneth Knowles  wrote:
>>
>>> If it is taking days of work we should definitely put it behind a flag
>>> and do it incrementally, find a way to share the work.
>>>
>>> If our tests aren't running on the actual artifacts, then we don't
>>> really have assurance that they work. That should interest just about
>>> everyone. (though it is also status quo relative to mvn)
>>>
>>> Kenn
>>>
>>> On Thu, May 24, 2018 at 10:17 AM Andrew Pilloud 
>>> wrote:
>>>
 I think I'm down to 11 packages with failures, some of which might be
 precommits that don't run outside of Jenkins. Its not an issue of figuring
 them out, it is an issue of time spent doing so.

 On Thu, May 24, 2018 at 9:31 AM Lukasz Cwik  wrote:

> Were you able to figure out how to fix them or still having problems?
>
> On Thu, May 24, 2018 at 9:27 AM Andrew Pilloud 
> wrote:
>
>> I spent all of yesterday investigating and fixing dependency issues
>> outside of SQL. I really regret the decision to write a test for this.
>> Would it be acceptable for us to put testing with the output jar behind a
>> flag like we do for failOnWarning?
>>
>> On Wed, May 23, 2018 at 5:21 PM Kenneth Knowles 
>> wrote:
>>
>>> What's the status of moving it forward? Is it a ton of work / too
>>> much to do quickly?
>>>
>>> On Wed, May 23, 2018 at 9:11 AM Andrew Pilloud 
>>> wrote:
>>>
 To loop the list in on discussions going on in
 https://github.com/apache/beam/pull/5443: our normal tests don't
 run against the shaded jars. Gradle can run the tests against the 
 shaded
 jars, but a bunch fail due to dependency issues. It's not just SQL.

 Andrew

 On Mon, May 21, 2018 at 11:35 AM Lukasz Cwik 
 wrote:

> Shading requires two pieces of information:
> 1) Which dependencies should be part of the shaded jar (controlled
> by includes/excludes)
> 2) How to relocate code within those dependencies (controlled by
> relocations)
>
> The reason why the exclude(".*") exists is because typically it is
> an error to produce a shaded package with dependencies which are not
> relocated. When libraries do this, it causes a lot of
> NoClassFound/NoMethodFound errors for users since a user can't know 
> which
> version of a dependency they are actually getting (the one that was 
> bundled
> part of your jar or the one they depend on as a library). Only 
> applications
> should ever really do this, libraries should always repackage all 
> their
> code to prevent such errors.
>
> Note that in the SQL package, you can provide your own
> shadowClosure to the applyJavaNature() which means that the default 
> won't
> apply. For example:
> https://github.com/apache/beam/blob/a3ba6a0e8de3ae72b8fc6fc6038eb9dc725f092e/sdks/java/harness/build.gradle#L20
> and remove the 'DEFAULT_SHADOW_CLOSURE <<'
>
>
> On Mon, May 21, 2018 at 10:26 AM Andrew Pilloud <
> apill...@google.com> wrote:
>
>> The issue SQL is seeing is caused by a default dependency of
>> exclude(".*") added in build_rules.gradle. This breaks the normal 
>> method of
>> building shadow jars as everything must be explicitly included. SQL
>> explicitly added calcite to the jar, but not calcite's dependencies. 
>> I've
>> been told this is the desired behavior as we want to ensure 
>> 

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Alan Myrvold]

+1 (non-binding) I updated the pull request to be 60 days (instead of 90)
to match the contribute policy.

On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:

> Hi all,
>
> Following the discussion, please vote on the move to activate
> probot/stale [3] to notify authors of stale PRs per current policy and
> then close them after a 7 day grace period.
>
> For more details, see:
>
>  - our stale PR policy [1]
>  - the discussion thread [2]
>  - Probot stale [3]
>  - BEAM ticket summarizing discussion [4]
>  - INFRA ticket to activate probot/stale [5]
>  - Example PR that would activate it [6]
>
> Please vote:
> [ ] +1, Approve that we activate probot/stale
> [ ] -1, Do not approve (please provide specific comments)
>
> Kenn
>
> [1] https://beam.apache.org/contribute/#stale-pull-requests
> [2]
> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
> [3] https://github.com/probot/stale
> [4] https://issues.apache.org/jira/browse/BEAM-4423
> [5] https://issues.apache.org/jira/browse/INFRA-16589
> [6] https://github.com/apache/beam/pull/5532
>

[VOTE] Use probot/stale to automatically manage stale pull requests

[Kenneth Knowles]

Hi all,

Following the discussion, please vote on the move to activate probot/stale
[3] to notify authors of stale PRs per current policy and then close them
after a 7 day grace period.

For more details, see:

 - our stale PR policy [1]
 - the discussion thread [2]
 - Probot stale [3]
 - BEAM ticket summarizing discussion [4]
 - INFRA ticket to activate probot/stale [5]
 - Example PR that would activate it [6]

Please vote:
[ ] +1, Approve that we activate probot/stale
[ ] -1, Do not approve (please provide specific comments)

Kenn

[1] https://beam.apache.org/contribute/#stale-pull-requests
[2]
https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
[3] https://github.com/probot/stale
[4] https://issues.apache.org/jira/browse/BEAM-4423
[5] https://issues.apache.org/jira/browse/INFRA-16589
[6] https://github.com/apache/beam/pull/5532

Re: Closing (automatically?) inactive pull requests

[Alan Myrvold]

A vote makes sense.

The proposed config is at https://github.com/apache/beam/pull/5532/files
and will mark pull requests as stale after 90 days and close them 7 days
later.
Issues are not affected.

On Fri, Jun 1, 2018 at 9:14 AM Kenneth Knowles  wrote:

> Ismael had mentioned to vote on this, so let's do that now to make sure no
> one missed this discussion.
>
> On Thu, May 31, 2018 at 9:33 PM Alan Myrvold  wrote:
>
>> Thanks. I can look into adding the stale.yaml file for old pull requests/
>>
>> On Thu, May 31, 2018 at 8:07 PM Kenneth Knowles  wrote:
>>
>>> Update: you brought the information needed, and it is now enabled.
>>> Thanks for the follow-through!
>>>
>>> Since you dug into probot's details, I took the liberty of assigning
>>> BEAM-4423 to you, in case throwing together the needed configs is fresh in
>>> your mind and you are in the mood to continue. (if not, pass the hot
>>> potato, back to me is OK too :-)
>>>
>>> Kenn
>>>
>>> On Thu, May 31, 2018 at 4:50 PM Alan Myrvold 
>>> wrote:
>>>
 INFRA-16589 got closed asking to clarify that the probot-stale app
 would not have permissions to merge automatically.
 From my reading of the permissions documentation, it would not. I added
 a comment to INFRA-16589

 On Tue, May 29, 2018 at 10:05 AM Lukasz Cwik  wrote:

> I opened up INFRA-16589
>  for Apache infra
> to install Stale but they denied a similar request INFRA-16394
>  because of
> permissions issues. I tried clarifying the permissions requested.
>
> On Tue, May 29, 2018 at 9:39 AM Scott Wegner 
> wrote:
>
>> +1. I've opened BEAM-4423 to capture the discussion here:
>> https://issues.apache.org/jira/browse/BEAM-4423
>>
>> On Thu, May 24, 2018 at 5:34 PM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> +1 for automatically closing. Currently, contribution guide mentions
>>> that pull requests without responses to actionable comments become stale
>>> (and closed) after two months but last I checked there were many pull
>>> requests that met this criteria but had not been closed after many 
>>> months.
>>> So seems like humans are reluctant to act on the established policy :)
>>>
>>> - Cham
>>>
>>> On Wed, May 23, 2018 at 11:25 AM Kenneth Knowles 
>>> wrote:
>>>
 That makes sense, to just focus on Beam's decision. It seems the
 tool is already built. I thought we just had to deploy it, but maybe 
 not
 even that, if we can just activate it:
 https://github.com/apps/stale

 Kenn

 On Wed, May 23, 2018 at 9:31 AM Ismaël Mejía 
 wrote:

> Given that reaching consensus in both communities seems like a
> harder task
> than just deciding our policy. in the Beam side Why don't we just
> go ahead
> and vote around this + build the tool, and if the Flink guys are
> interested
> they can take it, no?
>
> in the future we can share that code.
> On Wed, May 16, 2018 at 3:31 PM Piotr Nowojski <
> pi...@data-artisans.com>
> wrote:
>
> > The question is what would such tool offer on top of over a
> Github’s view
> of PR sorted by “least recently updated”:
>
>
>
> https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-asc
> <
> https://github.com/apache/flink/pulls?q=is:pr+is:open+sort:updated-asc
> >
>
> > ? Maybe it would be good enough to have a policy about waiting x
> months/weeks for a contributor to respond. If he doesn’t, we
> either take
> over PR we or close it. Having “clean” view of oldest PRs would be
> beneficial for reviewing PRs in ~FIFO order as part of community
> work.
>
> > Having
>
> > > On 16 May 2018, at 10:57, Fabian Hueske 
> wrote:
> > >
> > > Hi,
> > >
> > > I'm not objecting closing stale PRs.
> > > We have quite a few PRs with very little chance of being
> merged and I
> would
> > > certainly appreciate cleaning up those.
> > > However, I think we should not automate closing PRs for the
> reasons I
> gave
> > > before.
> > >
> > > A tool that reminds us of state PRs as proposed by Till seems
> like a
> good
> > > idea and might actually help to re-adjust priorities from time
> to time.
> > >
> > > @Yazdan: Right now there is no assignment happening.
> Committers decide
> > > which PRs they want to review and merge.
> > >
> > > Best, Fabian
> > >
> > > 

Re: Closing (automatically?) inactive pull requests

[Kenneth Knowles]

Ismael had mentioned to vote on this, so let's do that now to make sure no
one missed this discussion.

On Thu, May 31, 2018 at 9:33 PM Alan Myrvold  wrote:

> Thanks. I can look into adding the stale.yaml file for old pull requests/
>
> On Thu, May 31, 2018 at 8:07 PM Kenneth Knowles  wrote:
>
>> Update: you brought the information needed, and it is now enabled. Thanks
>> for the follow-through!
>>
>> Since you dug into probot's details, I took the liberty of assigning
>> BEAM-4423 to you, in case throwing together the needed configs is fresh in
>> your mind and you are in the mood to continue. (if not, pass the hot
>> potato, back to me is OK too :-)
>>
>> Kenn
>>
>> On Thu, May 31, 2018 at 4:50 PM Alan Myrvold  wrote:
>>
>>> INFRA-16589 got closed asking to clarify that the probot-stale app would
>>> not have permissions to merge automatically.
>>> From my reading of the permissions documentation, it would not. I added
>>> a comment to INFRA-16589
>>>
>>> On Tue, May 29, 2018 at 10:05 AM Lukasz Cwik  wrote:
>>>
 I opened up INFRA-16589
  for Apache infra
 to install Stale but they denied a similar request INFRA-16394
  because of
 permissions issues. I tried clarifying the permissions requested.

 On Tue, May 29, 2018 at 9:39 AM Scott Wegner 
 wrote:

> +1. I've opened BEAM-4423 to capture the discussion here:
> https://issues.apache.org/jira/browse/BEAM-4423
>
> On Thu, May 24, 2018 at 5:34 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> +1 for automatically closing. Currently, contribution guide mentions
>> that pull requests without responses to actionable comments become stale
>> (and closed) after two months but last I checked there were many pull
>> requests that met this criteria but had not been closed after many 
>> months.
>> So seems like humans are reluctant to act on the established policy :)
>>
>> - Cham
>>
>> On Wed, May 23, 2018 at 11:25 AM Kenneth Knowles 
>> wrote:
>>
>>> That makes sense, to just focus on Beam's decision. It seems the
>>> tool is already built. I thought we just had to deploy it, but maybe not
>>> even that, if we can just activate it: https://github.com/apps/stale
>>>
>>> Kenn
>>>
>>> On Wed, May 23, 2018 at 9:31 AM Ismaël Mejía 
>>> wrote:
>>>
 Given that reaching consensus in both communities seems like a
 harder task
 than just deciding our policy. in the Beam side Why don't we just
 go ahead
 and vote around this + build the tool, and if the Flink guys are
 interested
 they can take it, no?

 in the future we can share that code.
 On Wed, May 16, 2018 at 3:31 PM Piotr Nowojski <
 pi...@data-artisans.com>
 wrote:

 > The question is what would such tool offer on top of over a
 Github’s view
 of PR sorted by “least recently updated”:



 https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-asc
 <
 https://github.com/apache/flink/pulls?q=is:pr+is:open+sort:updated-asc
 >

 > ? Maybe it would be good enough to have a policy about waiting x
 months/weeks for a contributor to respond. If he doesn’t, we either
 take
 over PR we or close it. Having “clean” view of oldest PRs would be
 beneficial for reviewing PRs in ~FIFO order as part of community
 work.

 > Having

 > > On 16 May 2018, at 10:57, Fabian Hueske 
 wrote:
 > >
 > > Hi,
 > >
 > > I'm not objecting closing stale PRs.
 > > We have quite a few PRs with very little chance of being merged
 and I
 would
 > > certainly appreciate cleaning up those.
 > > However, I think we should not automate closing PRs for the
 reasons I
 gave
 > > before.
 > >
 > > A tool that reminds us of state PRs as proposed by Till seems
 like a
 good
 > > idea and might actually help to re-adjust priorities from time
 to time.
 > >
 > > @Yazdan: Right now there is no assignment happening. Committers
 decide
 > > which PRs they want to review and merge.
 > >
 > > Best, Fabian
 > >
 > > 2018-05-16 4:26 GMT+02:00 Yaz Sh :
 > >
 > >> I have questions in this regard (you guys might have addressed
 it in
 this
 > >> email chain):
 > >> how PRs get assigned to a reviewer apart of a contributor tag
 someone?
 > >> what if PR never gets a reviewer attention and it became
 in-active due
 to
 > >> long review 

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Henning Rohde]

Congratulations!

On Fri, Jun 1, 2018 at 7:03 AM Jesse Anderson 
wrote:

> Welcome!
>
> On Fri, Jun 1, 2018, 2:02 AM Etienne Chauchot 
> wrote:
>
>> Congrats to all !
>> Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :
>>
>> Please join me and the rest of Beam PMC in welcoming the following
>> contributors as our newest committers. They have significantly contributed
>> to the project in different ways, and we look forward to many more
>> contributions in the future.
>>
>> * Griselda Cuevas
>> * Pablo Estrada
>> * Jason Kuster
>>
>> (Apologizes for a delayed announcement, and the lack of the usual
>> paragraph summarizing individual contributions.)
>>
>> Congratulations to all three! Welcome!
>>
>>

Re: [VOTE] Go SDK

[Henning Rohde]

Thanks, Davor!

On Thu, May 31, 2018 at 7:10 PM Davor Bonaci  wrote:

> The IP clearance document has been filed into Foundation records, and is
> currently under review. No further action necessary, unless we hear back.
>
> On Fri, May 25, 2018 at 10:31 AM, Henning Rohde 
> wrote:
>
>> Thanks a lot, Davor! Much appreciated.
>>
>> Thanks,
>>  Henning
>>
>> On Fri, May 25, 2018 at 10:26 AM Davor Bonaci  wrote:
>>
>>> ETA: weekend.
>>>
>>> On Fri, May 25, 2018 at 9:35 AM Henning Rohde 
>>> wrote:
>>>
 RESULT: the vote passed with only +1s! Thanks you all for the kind
 comments.

 The only pending item is the IP clearance form (draft:
 https://web.tresorit.com/l#nUkKlgi3cBYxYAOyhCMXIw
 ).
 Are there any ASF members who can help getting it recorded?

 Thanks,
  Henning


 On Wed, May 23, 2018 at 2:45 PM Henning Rohde 
 wrote:

> Thanks Davor! I filled out the form to the best of my ability and
> placed it here (avoiding attachments on the list):
>
> https://web.tresorit.com/l#nUkKlgi3cBYxYAOyhCMXIw
> 
>
> Please take a look and let me know if you need anything more from me.
>
> Thanks,
>  Henning
>
> On Wed, May 23, 2018 at 8:51 AM Thomas Groh  wrote:
>
>> +1!
>>
>> I, for one, could not be more excited about our glorious portable
>> future.
>>
>> On Mon, May 21, 2018 at 6:03 PM Henning Rohde 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Now that the remaining issues have been resolved as discussed, I'd
>>> like to propose a formal vote on accepting the Go SDK into master. The 
>>> main
>>> practical difference is that the Go SDK would be part of the Apache Beam
>>> release going forward.
>>>
>>> Highlights of the Go SDK:
>>>  * Go user experience with natively-typed DoFns with (simulated)
>>> generic types
>>>  * Covers most of the Beam model: ParDo, GBK, CoGBK, Flatten,
>>> Combine, Windowing, ..
>>>  * Includes several IO connectors: Datastore, BigQuery, PubSub,
>>> extensible textio.
>>>  * Supports the portability framework for both batch and streaming,
>>> notably the upcoming portable Flink runner
>>>  * Supports a direct runner for small batch workloads and testing.
>>>  * Includes pre-commit tests and post-commit integration tests.
>>>
>>> And last but not least
>>>  *  includes contributions from several independent users and
>>> developers, notably an IO connector for Datastore!
>>>
>>> Website: https://beam.apache.org/documentation/sdks/go/
>>> Code: https://github.com/apache/beam/tree/master/sdks/go
>>> Design: https://s.apache.org/beam-go-sdk-design-rfc
>>>
>>> Please vote:
>>> [ ] +1, Approve that the Go SDK becomes an official part of Beam
>>> [ ] -1, Do not approve (please provide specific comments)
>>>
>>> Thanks,
>>>  The Gophers of Apache Beam
>>>
>>>
>>>
>

Re: Documenting Github PR jenkins trigger phrases

[Alan Myrvold]

This is tracked by https://issues.apache.org/jira/browse/BEAM-3068

On Fri, Jun 1, 2018 at 7:38 AM Ismaël Mejía  wrote:

> Any progress on this ? Is there a JIRA to track it ?
> On Fri, May 11, 2018 at 6:02 PM Jean-Baptiste Onofré 
> wrote:
> >
> > Agree, I thought there's already a PR about that.
> >
> > Regards
> > JB
> >
> > On 11/05/2018 16:11, Alexey Romanenko wrote:
> > > +1 to add such reference guide of Jenkins commands to Testing guide. It
> > > should be extremely useful, especially for those who were not aware
> > > about this before.
> > >
> > > WBR,
> > > Alexey
> > >
> > >> On 11 May 2018, at 02:44, Ankur Goenka  > >> > wrote:
> > >>
> > >> In my experience affect of white space in commit is inconsistent for
> > >> certain commands they do matter while for others they don't.
> > >>
> > >> On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev
> > >> mailto:valen...@google.com>> wrote:
> > >>
> > >> +1 to writing a Beam Jenkins spellbook.
> > >> I have observed that Jenkins commands sometimes don't work for the
> > >> first time, why could this be? Do end of lines at the end of
> > >> command matter?
> > >>
> > >>
> > >> On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud
> > >> mailto:apill...@google.com>> wrote:
> > >>
> > >> It would be great to have the set of "Run {Java,Python,Go}
> > >> PreCommit" documented in the contributors guide as well. Those
> > >> match up to the jobs auto run on every PR and are the ones I
> > >> use most. There is no security, anyone can run them including
> > >> 'Run Seed Job'. That one seems like a good one to document in
> > >> testing, because it is the one that loads changes to the rest.
> > >>
> > >> Andrew
> > >>
> > >> On Thu, May 10, 2018 at 12:27 PM Huygaa Batsaikhan
> > >> mailto:bat...@google.com>> wrote:
> > >>
> > >> Hi devs,
> > >>
> > >> We can run various jenkins commands (precommit,
> > >> postcommit, performance tests) directly from Github Pull
> > >> Request UI by commenting phrases such as "retest this
> > >> please". Unfortunately, this tool is not documented. I am
> > >> adding a brief documentation in
> > >> https://beam.apache.org/contribute/testing/ and I need
> > >> some help.
> > >>
> > >>  1. What are the most common phrases used?
> > >>  2. Can anyone run these commands? Are there any
> > >> permission issues?
> > >>  3. Does it make sense to categorize the commands as
> > >> Performance tests, Precommit, Postcommit, and Release
> > >> Validation?
> > >>
> > >> Let me know what you think,
> > >>
> > >> Thanks,
> > >> Huygaa
> > >>
> > >>
> > >
>

Re: [Proposal] Apache Beam Swag Store

[Matthias Baetens]

We are very close to launching the store --- we should have news next week!

On Fri, 1 Jun 2018 at 15:43 Ismaël Mejía  wrote:

> Any news on this ?
> On Tue, Nov 7, 2017 at 1:46 AM Matthias Baetens
>  wrote:
> >
> > +1
> > Awesome stuff, thanks for the effort Gris!
> >
> > On Mon, Nov 6, 2017 at 5:31 AM, Griselda Cuevas  >
> > wrote:
> >
> > > Excellent - we're getting approval from the trademark@ folks. I'll
> update
> > > the thread when I have news.
> > >
> > >
> > >
> > > On 5 November 2017 at 14:04, Ted Yu  wrote:
> > >
> > > > +1
> > > >  Original message From: Jacob Marble <
> > > jmar...@kochava.com>
> > > > Date: 11/5/17  1:50 PM  (GMT-08:00) To: dev@beam.apache.org
> Subject: Re:
> > > > [Proposal] Apache Beam Swag Store
> > > > I think this is a great idea, ready to order mine. :)
> > > >
> > > > Jacob
> > > >
> > > > On Sat, Oct 28, 2017 at 11:19 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > > > wrote:
> > > >
> > > > > It sounds good. Please let us know trademark update.
> > > > >
> > > > > Thanks
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Oct 28, 2017, 20:15, at 20:15, Griselda Cuevas
> > > > 
> > > > > wrote:
> > > > > >Thanks for the feedback all, I'll send this idea to the trademark@
> > > > > >folks
> > > > > >and wait for validation. Once we have it I'll look into building
> the
> > > > > >store
> > > > > >possibly embedded in the website.
> > > > > >
> > > > > >Enjoy the weekend.
> > > > > >G
> > > > > >
> > > > > >5
> > > > > >
> > > > > >
> > > > > >
> > > > > >On 27 October 2017 at 11:53, Tyler Akidau
>  > > >
> > > > > >wrote:
> > > > > >
> > > > > >> One additional note: for the logos w/ the name below them,
> would be
> > > > > >nice to
> > > > > >> not have quite so much whitespace between the logo and the name.
> > > > > >Otherwise,
> > > > > >> trademark validation aside, this looks great.
> > > > > >>
> > > > > >> On Fri, Oct 27, 2017 at 10:15 AM Griselda Cuevas
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Dan - thanks for bringing this up to my attention. I
> haven't
> > > > > >raised
> > > > > >> this
> > > > > >> > up to the tradema...@apache.org people. Can I just reach out
> to
> > > > > >them to
> > > > > >> > get
> > > > > >> > the proposal or should one of the PMCs do this?
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > Gris Cuevas Zambrano
> > > > > >> >
> > > > > >> > g...@google.com
> > > > > >> >
> > > > > >> > Open Source Strategy
> > > > > >> >
> > > > > >> > 345 Spear Street, San Francisco, 94105
> > > > > >> >  > > > > >> +94105=gmail=g>
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On 26 October 2017 at 18:30, Daniel Kulp 
> > > wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > > Have you run this through tradema...@apache.org yet?
> > > > > >> > >
> > > > > >> > > I bring this up for two reasons:
> > > > > >> > >
> > > > > >> > > 1) We would need to make sure the appearance and such of the
> > > logo
> > > > > >is
> > > > > >> > > “correct”
> > > > > >> > >
> > > > > >> > > 2).A few years ago, Apache did have a partnership with a
> company
> > > > > >that
> > > > > >> > > would produce various swag things and then donate a portion
> back
> > > > > >to
> > > > > >> > > Apache.   I don’t know what the state of that agreement is
> and
> > > > > >whether
> > > > > >> > that
> > > > > >> > > would restrict going to another vendor or something.
> > > > > >> > >
> > > > > >> > > Dan
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > > On Oct 25, 2017, at 8:51 AM, Griselda Cuevas
> > > > > > > > > > >> >
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > Hi Everyone,
> > > > > >> > > >
> > > > > >> > > > I'd like to propose the creation of an online swag store
> for
> > > > > >Apache
> > > > > >> > Beam
> > > > > >> > > where anyone could order swag from a wide selection and get
> it
> > > > > >deliver
> > > > > >> to
> > > > > >> > > their home or office. I got in touch with a provider who
> could
> > > > > >include
> > > > > >> > this
> > > > > >> > > service as part of an order I'm placing to send swag for the
> > > > > >Meetups
> > > > > >> > we're
> > > > > >> > > organizing this year. What do you think?
> > > > > >> > > >
> > > > > >> > > > I'd also like to get your feedback on the swag I'm
> requesting
> > > > > >(you
> > > > > >> can
> > > > > >> > > see it in the pdf I attached), what do you think of the
> colors,
> > > > > >design,
> > > > > >> > > etc.?
> > > > > >> > > >
> > > > > >> > > > Lastly, I'll be ordering swag for Meetup organized this
> year
> > > so
> > > > > >if
> > > > > >> > > you're hosting one or speaking at one get in touch with me
> to
> > > > > >send
> > > > > >> some!
> > > > > >> > > >
> > > > > >> > > > Cheers,
> > > > > >> > > > G
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Daniel Kulp
> > > > > >> > > dk...@apache.org - http://dankulp.com/blog
> > > > > >> > > Talend 

Re: [Proposal] Apache Beam Swag Store

[Ismaël Mejía]

Any news on this ?
On Tue, Nov 7, 2017 at 1:46 AM Matthias Baetens
 wrote:
>
> +1
> Awesome stuff, thanks for the effort Gris!
>
> On Mon, Nov 6, 2017 at 5:31 AM, Griselda Cuevas 
> wrote:
>
> > Excellent - we're getting approval from the trademark@ folks. I'll update
> > the thread when I have news.
> >
> >
> >
> > On 5 November 2017 at 14:04, Ted Yu  wrote:
> >
> > > +1
> > >  Original message From: Jacob Marble <
> > jmar...@kochava.com>
> > > Date: 11/5/17  1:50 PM  (GMT-08:00) To: dev@beam.apache.org Subject: Re:
> > > [Proposal] Apache Beam Swag Store
> > > I think this is a great idea, ready to order mine. :)
> > >
> > > Jacob
> > >
> > > On Sat, Oct 28, 2017 at 11:19 AM, Jean-Baptiste Onofré 
> > > wrote:
> > >
> > > > It sounds good. Please let us know trademark update.
> > > >
> > > > Thanks
> > > > Regards
> > > > JB
> > > >
> > > > On Oct 28, 2017, 20:15, at 20:15, Griselda Cuevas
> > > 
> > > > wrote:
> > > > >Thanks for the feedback all, I'll send this idea to the trademark@
> > > > >folks
> > > > >and wait for validation. Once we have it I'll look into building the
> > > > >store
> > > > >possibly embedded in the website.
> > > > >
> > > > >Enjoy the weekend.
> > > > >G
> > > > >
> > > > >5
> > > > >
> > > > >
> > > > >
> > > > >On 27 October 2017 at 11:53, Tyler Akidau  > >
> > > > >wrote:
> > > > >
> > > > >> One additional note: for the logos w/ the name below them, would be
> > > > >nice to
> > > > >> not have quite so much whitespace between the logo and the name.
> > > > >Otherwise,
> > > > >> trademark validation aside, this looks great.
> > > > >>
> > > > >> On Fri, Oct 27, 2017 at 10:15 AM Griselda Cuevas
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Dan - thanks for bringing this up to my attention. I haven't
> > > > >raised
> > > > >> this
> > > > >> > up to the tradema...@apache.org people. Can I just reach out to
> > > > >them to
> > > > >> > get
> > > > >> > the proposal or should one of the PMCs do this?
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Gris Cuevas Zambrano
> > > > >> >
> > > > >> > g...@google.com
> > > > >> >
> > > > >> > Open Source Strategy
> > > > >> >
> > > > >> > 345 Spear Street, San Francisco, 94105
> > > > >> >  > > > >> +94105=gmail=g>
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On 26 October 2017 at 18:30, Daniel Kulp 
> > wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > Have you run this through tradema...@apache.org yet?
> > > > >> > >
> > > > >> > > I bring this up for two reasons:
> > > > >> > >
> > > > >> > > 1) We would need to make sure the appearance and such of the
> > logo
> > > > >is
> > > > >> > > “correct”
> > > > >> > >
> > > > >> > > 2).A few years ago, Apache did have a partnership with a company
> > > > >that
> > > > >> > > would produce various swag things and then donate a portion back
> > > > >to
> > > > >> > > Apache.   I don’t know what the state of that agreement is and
> > > > >whether
> > > > >> > that
> > > > >> > > would restrict going to another vendor or something.
> > > > >> > >
> > > > >> > > Dan
> > > > >> > >
> > > > >> > >
> > > > >> > > > On Oct 25, 2017, at 8:51 AM, Griselda Cuevas
> > > > > > > > >> >
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > Hi Everyone,
> > > > >> > > >
> > > > >> > > > I'd like to propose the creation of an online swag store for
> > > > >Apache
> > > > >> > Beam
> > > > >> > > where anyone could order swag from a wide selection and get it
> > > > >deliver
> > > > >> to
> > > > >> > > their home or office. I got in touch with a provider who could
> > > > >include
> > > > >> > this
> > > > >> > > service as part of an order I'm placing to send swag for the
> > > > >Meetups
> > > > >> > we're
> > > > >> > > organizing this year. What do you think?
> > > > >> > > >
> > > > >> > > > I'd also like to get your feedback on the swag I'm requesting
> > > > >(you
> > > > >> can
> > > > >> > > see it in the pdf I attached), what do you think of the colors,
> > > > >design,
> > > > >> > > etc.?
> > > > >> > > >
> > > > >> > > > Lastly, I'll be ordering swag for Meetup organized this year
> > so
> > > > >if
> > > > >> > > you're hosting one or speaking at one get in touch with me to
> > > > >send
> > > > >> some!
> > > > >> > > >
> > > > >> > > > Cheers,
> > > > >> > > > G
> > > > >> > >
> > > > >> > > --
> > > > >> > > Daniel Kulp
> > > > >> > > dk...@apache.org - http://dankulp.com/blog
> > > > >> > > Talend Community Coder - http://coders.talend.com
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
>
>
> --
>
>
> *Matthias Baetens*
>
>
> *datatonic | data power unleashed*
>
> office +44 203 668 3680  |  mobile +44 74 918 20646
>
> Level24 | 1 Canada Square | Canary Wharf | E14 5AB London
>
>
> We've been announced
> 
> as
> one of the top global Google Cloud 

Re: [PROPOSAL] Preparing 2.5.0 release next week

[Jasper Wang]

Hi JB and team,

Very excited to hear that the release process of python SDK 2.5 has
started. Can’t wait to use it for streaming on Dataflow.

Just a silly question though - how long the release process typically takes?

Thx in advance.

Jasper

On Thu, 31 May 2018 at 11:14 pm, Jean-Baptiste Onofré 
wrote:

> Yes I confirm: I already checked that last week. Thanks for the double
> check !
>
> Regards
> JB
> Le 31 mai 2018, à 15:13, Etienne Chauchot  a écrit:
>>
>> Hi,
>> I did some tests on the maven artifacts produced by the gradle build:
>>
>> I published maven artifacts to local maven repo using :
>>
>>  ./gradlew publishToMavenLocal -PisRelease --no-parallel -x test
>>
>> then used beam samples project (maven based) and did a
>>
>> mvn dependency:tree
>>
>> => transitive dependencies of all beam related artifacts are well resolved
>>
>> Le jeudi 31 mai 2018 à 14:43 +0200, Jean-Baptiste Onofré a écrit :
>>
>> Agree, I gonna start the release process tonight (my time).
>>
>> Regards
>> JB
>> Le 31 mai 2018, à 14:29, Kenneth Knowles < k...@google.com> a écrit:
>>
>> Yea - the branch should be cut before trying to finish the burndown.
>>
>> Kenn
>>
>> On Thu, May 31, 2018 at 2:09 AM Robert Bradshaw < rober...@google.com>
>> wrote:
>>
>> I think it makes sense to cut the release and get the ball rolling, and
>> iff the  ParquetIO/S3 issue turns out to be simple, we cherry-pick,
>> otherwise we add a note.
>>
>> On Thu, May 31, 2018 at 1:56 AM Jean-Baptiste Onofré < j...@nanthrax.net>
>> wrote:
>>
>> Hi,
>>
>> Regarding RabbitMqIO, Eugene provided new feedback last night that I
>> would like to implement. However, it's not a release blocker, so I will
>> move forward with 2.5.0 release without RabbitMqIO  (I will include in
>> 2.6.0).
>>
>> Regarding ParquetIO, I tested HDFS successfully as well (I had an issue
>> on my namenode). S3 is important, but I think we can just add a note in
>> the RELEASE NOTE  and move forward with 2.5.0. Thoughts ?
>>
>> So, please, let me know if I can start the release process today (I
>> would like to  move forward asap).
>>
>> Thanks,
>> Regards
>> JB
>>
>>

Re: Documenting Github PR jenkins trigger phrases

[Ismaël Mejía]

Any progress on this ? Is there a JIRA to track it ?
On Fri, May 11, 2018 at 6:02 PM Jean-Baptiste Onofré  wrote:
>
> Agree, I thought there's already a PR about that.
>
> Regards
> JB
>
> On 11/05/2018 16:11, Alexey Romanenko wrote:
> > +1 to add such reference guide of Jenkins commands to Testing guide. It
> > should be extremely useful, especially for those who were not aware
> > about this before.
> >
> > WBR,
> > Alexey
> >
> >> On 11 May 2018, at 02:44, Ankur Goenka  >> > wrote:
> >>
> >> In my experience affect of white space in commit is inconsistent for
> >> certain commands they do matter while for others they don't.
> >>
> >> On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev
> >> mailto:valen...@google.com>> wrote:
> >>
> >> +1 to writing a Beam Jenkins spellbook.
> >> I have observed that Jenkins commands sometimes don't work for the
> >> first time, why could this be? Do end of lines at the end of
> >> command matter?
> >>
> >>
> >> On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud
> >> mailto:apill...@google.com>> wrote:
> >>
> >> It would be great to have the set of "Run {Java,Python,Go}
> >> PreCommit" documented in the contributors guide as well. Those
> >> match up to the jobs auto run on every PR and are the ones I
> >> use most. There is no security, anyone can run them including
> >> 'Run Seed Job'. That one seems like a good one to document in
> >> testing, because it is the one that loads changes to the rest.
> >>
> >> Andrew
> >>
> >> On Thu, May 10, 2018 at 12:27 PM Huygaa Batsaikhan
> >> mailto:bat...@google.com>> wrote:
> >>
> >> Hi devs,
> >>
> >> We can run various jenkins commands (precommit,
> >> postcommit, performance tests) directly from Github Pull
> >> Request UI by commenting phrases such as "retest this
> >> please". Unfortunately, this tool is not documented. I am
> >> adding a brief documentation in
> >> https://beam.apache.org/contribute/testing/ and I need
> >> some help.
> >>
> >>  1. What are the most common phrases used?
> >>  2. Can anyone run these commands? Are there any
> >> permission issues?
> >>  3. Does it make sense to categorize the commands as
> >> Performance tests, Precommit, Postcommit, and Release
> >> Validation?
> >>
> >> Let me know what you think,
> >>
> >> Thanks,
> >> Huygaa
> >>
> >>
> >

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Jesse Anderson]

Welcome!

On Fri, Jun 1, 2018, 2:02 AM Etienne Chauchot  wrote:

> Congrats to all !
> Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :
>
> Please join me and the rest of Beam PMC in welcoming the following
> contributors as our newest committers. They have significantly contributed
> to the project in different ways, and we look forward to many more
> contributions in the future.
>
> * Griselda Cuevas
> * Pablo Estrada
> * Jason Kuster
>
> (Apologizes for a delayed announcement, and the lack of the usual
> paragraph summarizing individual contributions.)
>
> Congratulations to all three! Welcome!
>
>

Re: Survey: what is everyone working on that you want to share?

[Etienne Chauchot]

Hi Kenn,Sorry for the late answer, for myself, what I'm currently working on:
1. Add Nexmark to continuous integration as postcommits and export response 
times to performance dashboards: ticket: htt
ps://issues.apache.org/jira/browse/BEAM-
4225PR1:https://github.com/apache/beam/pull/5464PR2:https://github.com/apache/beam/pull/4976
2. Push metrics to a backend in an runner agnostic wayticket: 
https://issues.apache.org/jira/browse/BEAM-3310PR: already
merged, current work is integration with Dataflow (Flink and Spark are already 
done)
BestEtienne

Le lundi 14 mai 2018 à 22:27 -0700, Kenneth Knowles a écrit :
> Hi all,
> TL;DR: for anyone who is willing & able to answer: What big-picture thing are 
> you working on? I want to highlight it
> here:
> https://beam.apache.org/contribute/#works-in-progress
> 
> (I want each of these to have a nice description, too! And a saved JIRA 
> search for starter tasks would be clever...)
>  context 
> 
> Following feedback from the Beam summit and other discussions, I've been 
> working with Melissa and looping in folks to
> revamp the website to have a more welcoming tone and to make it easier for 
> newcomers to get started.
> 
> Part of that is making the site and guide more concise and making the entry 
> points more prominent and interesting.
> Starter tasks are OK, but to get a lasting engagement the idea is for starter 
> tasks connect a newcomer with ongoing
> projects. Most importantly, they are more likely to have exciting 
> interactions with experienced contributors, and to
> have follow-up work to the starter task.
> 
> Kenn

Re: parquet/beam

[Reuven Lax]

This is an interesting direction. I had looked at Arrow as a default Coder
replacement for schema PCollections, and that didn't seem fruitful as we
would need to create Arrow batches of size 1. However using Arrow to encode
batches over the FnAPI might indeed be an interesting approach. Streaming
small batches instead of streaming individual elements may also prove more
efficient.

On Thu, May 31, 2018 at 8:55 PM Lukasz Cwik  wrote:

> It really needs someone to take a deep dive and look into whether Arrow is
> a good fit now considering all the use cases that Apache Beam has. I did a
> look about a year ago when designing the Fn Data API and concluded at that
> point in time it wasn't great for several reasons but mainly due to the
> fact that our systems seem to be about pushing/streaming bytes from one
> place to the next. It could become a better fit as Apache Beam improves.
> For example, migrating PCollections to have schemas would be a big push
> towards using something like Arrow as a transport as they would better
> represent each other.
>
> Anyone should feel free to write up a doc and/or prototype/develop an
> Apache Arrow transport.
>
>
> On Thu, May 31, 2018 at 10:43 AM Lukasz Cwik  wrote:
>
>> Kenn, it can be done but requires explicit flow control communication
>> between the Runner -> SDK and SDK -> Runner to be developed to support
>> sub-bundle groupings.
>>
>> Transports and in memory layouts are related but improving our coders to
>> use in memory layouts would give us most of the benefit. For example, if we
>> used flatbuffers or an equivalent technology to do that work for us. Which
>> leads us to rethinking how our coders are modelled as encoding and decoding
>> streams of bytes and whether they would be better suited as something else.
>>
>> On Thu, May 31, 2018 at 9:16 AM Kenneth Knowles  wrote:
>>
>>> For the latter, can we have the Fn API data plane transmit sub-bundle
>>> groupings to benefit from the memory layout? On input the runner controls,
>>> on output the SDK controls (spilling)? Just random thoughts.
>>>
>>> Kenn
>>>
>>> On Thu, May 31, 2018 at 8:21 AM Lukasz Cwik  wrote:
>>>
 Tyler and I had reached out to Arrow folks[1] asking about how could we
 support the KV> when the iterable of values is beyond
 memory size limits. There is an open JIRA about adding support for large
 byte[] and strings and list types in ARROW-750[2]. Robert had pointed out
 that we could do the same thing we are planning to do when using the Beam
 Fn Data API when handling really large values over the Beam Fn State API as
 described here[3].

 The other issue that hasn't yet been discussed is that Arrow
 materializes and stores the data on memory (or disk) while the Beam Fn Data
 API is more about "streaming" data between two actors. This allows us to
 process very large bundles and also allow for arbitrary blow up in output
 from a single element (a runner can effectively control how large a bundle
 is that is sent to an SDK harness but can't guarantee that the SDK will not
 take a single element and produce lots and lots of data from it).

 1:
 https://lists.apache.org/thread.html/ce36c311e34af8bea230c89e7ada38923e6845d6bc875ccfbc003cfe@%3Cdev.arrow.apache.org%3E
 2: https://issues.apache.org/jira/browse/ARROW-750
 3:
 https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.y6e78jyiwn50

 On Thu, May 31, 2018 at 7:56 AM Reuven Lax  wrote:

> I've looked at arrow, and there's some trickiness. Beam has a record
> model and arrow works best with large batches of records. We could do per
> record encoding, but that might be inefficient in arrow.
>
> On Thu, May 31, 2018, 5:50 PM Ismaël Mejía  wrote:
>
>> If I understand correctly Arrow allows a common multi language
>> in-memory data representation, so basically it is a columnar data
>> format that you can use to transfer data betweeen libraries in python
>> (pandas, numpy, etc), Java and other languages. This avoids the
>> round-trip to disk to do so. So we should maybe take a look to it
>> because it could be a pretty efficient way to transfer data in
>> multi-language pipelines (useful for portability). They even seem to
>> be working in a full platform based on it with streaming capabilities:
>> https://blog.rstudio.com/2018/04/19/arrow-and-beyond/
>>
>> There is also a serialized version of it called feather. I suppose
>> that an extension to support this format can make sense.
>> https://github.com/wesm/feather
>>
>> Maybe Holden can give some other ideas on possible valid uses on Beam
>> (or correct me if I say something incorrect) because this seems to be
>> important in the python on Spark world at this moment.
>> On Thu, May 31, 2018 at 2:01 AM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>> >

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Etienne Chauchot]

Congrats to all !
Le jeudi 31 mai 2018 à 19:08 -0700, Davor Bonaci a écrit :
> Please join me and the rest of Beam PMC in welcoming the following 
> contributors as our newest committers. They have
> significantly contributed to the project in different ways, and we look 
> forward to many more contributions in the
> future.
> 
> * Griselda Cuevas
> * Pablo Estrada
> * Jason Kuster
> 
> (Apologizes for a delayed announcement, and the lack of the usual paragraph 
> summarizing individual contributions.)
> 
> Congratulations to all three! Welcome!
> 

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Matthias Baetens]

Great news, congrats all!

On Fri, 1 Jun 2018 at 08:32 Holden Karau  wrote:

> Congrats all!
>
> On Fri, Jun 1, 2018 at 12:12 AM Ismaël Mejía  wrote:
>
>> Congratulations!
>>
>> On Fri, Jun 1, 2018 at 8:26 AM Pei HE  wrote:
>> >
>> > Congrats!
>> >
>> > On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen  wrote:
>> > > Congratulations everyone!
>> > >
>> > >
>> > > On Thu, May 31, 2018, 10:14 PM Pablo Estrada 
>> wrote:
>> > >>
>> > >> Thanks to the PMC! Very humbled and excited to keep taking part in
>> this
>> > >> great community.
>> > >> :)
>> > >> -P.
>> > >>
>> > >>
>> > >> On Thu, May 31, 2018, 10:10 PM Tim 
>> wrote:
>> > >>>
>> > >>> Congratulations!
>> > >>>
>> > >>>
>> > >>> Tim
>> > >>>
>> > >>> On 1 Jun 2018, at 07:05, Andrew Psaltis 
>> wrote:
>> > >>>
>> > >>> Congrats!
>> > >>>
>> > >>> On Fri, Jun 1, 2018 at 12:26 AM, Thomas Weise 
>> wrote:
>> > 
>> >  Congrats!
>> > 
>> > 
>> >  On Thu, May 31, 2018 at 9:25 PM, Alan Myrvold > >
>> >  wrote:
>> > >
>> > > Congrats Gris+Pablo+Jason. Well deserved.
>> > >
>> > > On Thu, May 31, 2018 at 9:15 PM Jason Kuster <
>> jasonkus...@google.com>
>> > > wrote:
>> > >>
>> > >> Thank you to Davor and the PMC; I'm excited to be able to help
>> Beam in
>> > >> this new capacity. Bring on the PRs. :D
>> > >>
>> > >> On Thu, May 31, 2018 at 8:55 PM Xin Wang > >
>> > >> wrote:
>> > >>>
>> > >>> Congrats!
>> > >>>
>> > >>> - Xin Wang
>> > >>>
>> > >>> 2018-06-01 11:52 GMT+08:00 Rui Wang :
>> > 
>> >  Congrats!
>> > 
>> >  -Rui
>> > 
>> >  On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré
>> >   wrote:
>> > >
>> > > Congrats !
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On 01/06/2018 04:08, Davor Bonaci wrote:
>> > > > Please join me and the rest of Beam PMC in welcoming the
>> > > > following
>> > > > contributors as our newest committers. They have
>> significantly
>> > > > contributed to the project in different ways, and we look
>> forward
>> > > > to
>> > > > many more contributions in the future.
>> > > >
>> > > > * Griselda Cuevas
>> > > > * Pablo Estrada
>> > > > * Jason Kuster
>> > > >
>> > > > (Apologizes for a delayed announcement, and the lack of the
>> usual
>> > > > paragraph summarizing individual contributions.)
>> > > >
>> > > > Congratulations to all three! Welcome!
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Thanks,
>> > >>> Xin
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> ---
>> > >> Jason Kuster
>> > >> Apache Beam / Google Cloud Dataflow
>> > >>
>> > >> See something? Say something. go/jasonkuster-feedback
>> > 
>> > 
>> > >>>
>> > >> --
>> > >> Got feedback? go/pabloem-feedback
>>
> --
> Twitter: https://twitter.com/holdenkarau
>
--

Re: "Radically modular data ingestion APIs in Apache Beam" @ Strata - slides available

[Matthias Baetens]

Hey Ismaël,

That totally makes sense, I will work on this on the weekend and send a PR.

Cheers,
Matthias

On Thu, 31 May 2018 at 09:45 Ismaël Mejía  wrote:

> Great !
>
> Matthias I think it makes sense to have these guidelines in the beam
> site better than a google doc. Can you please submit a PR for this?
>
> On Thu, May 31, 2018 at 8:03 AM Matthias Baetens
>  wrote:
> >
> > Hey Eugene, hi all!
> >
> > Happy to say your talk is now on the Beam YouTube channel and can be
> watched here.
> > It'd be great to see more of these on the channel so we can start
> sharing this on meetups, conferences and other places and see this grow, so
> don't hesitate to reach out to me (directly) if you want you video on the
> channel. Find more about the process / guidelines here.
> >
> > Best,
> > Matthias
> >
> > On Mon, 14 May 2018 at 22:19 Eugene Kirpichov 
> wrote:
> >>
> >> Hi Matthias,
> >>
> >> Thank you. Here is the raw file on Google Drive:
> https://drive.google.com/file/d/1gUoe6UrpNNO3ijYSgTvD5GBy-14Zx6dT/view?usp=sharing
> >> And yes, I have permission from O'Reilly/Strata to use this file
> whichever way I want, so it's ok to share on YouTube.
> >>
> >> On Fri, May 11, 2018 at 12:13 PM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
> >>>
> >>> Hey Eugene,
> >>>
> >>> Apologies for picking this up so late, but I could help uploading your
> video to the Beam channel.
> >>> Are you able to send me the raw file and do you have sign-off to go
> ahead with sharing it on YouTube?
> >>>
> >>> Thanks.
> >>> Matthias
> >>>
> >>> On Sat, 14 Apr 2018 at 21:45 Eugene Kirpichov 
> wrote:
> 
>  Hi all,
> 
>  The video is now available. I got it from my Strata account and I
> have permission to use and share it freely, so I published it on my own
> YouTube page (where there's nothing else...). Perhaps it makes sense to add
> to the Beam YouTube channel, but AFAIK only a PMC member can do that.
> 
>  https://www.youtube.com/watch?v=NIn9E5TVoCA
> 
> 
>  On Tue, Mar 13, 2018 at 3:33 AM James  wrote:
> >
> > Very informative, thanks!
> >
> > On Fri, Mar 9, 2018 at 4:49 PM Etienne Chauchot <
> echauc...@apache.org> wrote:
> >>
> >> Great !
> >>
> >> Thanks for sharing.
> >>
> >> Etienne
> >>
> >>
> >> Le jeudi 08 mars 2018 à 19:49 +, Eugene Kirpichov a écrit :
> >>
> >> Hey all,
> >>
> >> The slides for my yesterday's talk at Strata San Jose
> https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63696
> have been posted on the talk page. They may be of interest both to users
> and IO authors.
> >>
> >> Thanks.
> >>>
> >>> --
> >>>
> >
> > --
> >
>
--

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Holden Karau]

Congrats all!

On Fri, Jun 1, 2018 at 12:12 AM Ismaël Mejía  wrote:

> Congratulations!
>
> On Fri, Jun 1, 2018 at 8:26 AM Pei HE  wrote:
> >
> > Congrats!
> >
> > On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen  wrote:
> > > Congratulations everyone!
> > >
> > >
> > > On Thu, May 31, 2018, 10:14 PM Pablo Estrada 
> wrote:
> > >>
> > >> Thanks to the PMC! Very humbled and excited to keep taking part in
> this
> > >> great community.
> > >> :)
> > >> -P.
> > >>
> > >>
> > >> On Thu, May 31, 2018, 10:10 PM Tim  wrote:
> > >>>
> > >>> Congratulations!
> > >>>
> > >>>
> > >>> Tim
> > >>>
> > >>> On 1 Jun 2018, at 07:05, Andrew Psaltis 
> wrote:
> > >>>
> > >>> Congrats!
> > >>>
> > >>> On Fri, Jun 1, 2018 at 12:26 AM, Thomas Weise 
> wrote:
> > 
> >  Congrats!
> > 
> > 
> >  On Thu, May 31, 2018 at 9:25 PM, Alan Myrvold 
> >  wrote:
> > >
> > > Congrats Gris+Pablo+Jason. Well deserved.
> > >
> > > On Thu, May 31, 2018 at 9:15 PM Jason Kuster <
> jasonkus...@google.com>
> > > wrote:
> > >>
> > >> Thank you to Davor and the PMC; I'm excited to be able to help
> Beam in
> > >> this new capacity. Bring on the PRs. :D
> > >>
> > >> On Thu, May 31, 2018 at 8:55 PM Xin Wang 
> > >> wrote:
> > >>>
> > >>> Congrats!
> > >>>
> > >>> - Xin Wang
> > >>>
> > >>> 2018-06-01 11:52 GMT+08:00 Rui Wang :
> > 
> >  Congrats!
> > 
> >  -Rui
> > 
> >  On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré
> >   wrote:
> > >
> > > Congrats !
> > >
> > > Regards
> > > JB
> > >
> > > On 01/06/2018 04:08, Davor Bonaci wrote:
> > > > Please join me and the rest of Beam PMC in welcoming the
> > > > following
> > > > contributors as our newest committers. They have
> significantly
> > > > contributed to the project in different ways, and we look
> forward
> > > > to
> > > > many more contributions in the future.
> > > >
> > > > * Griselda Cuevas
> > > > * Pablo Estrada
> > > > * Jason Kuster
> > > >
> > > > (Apologizes for a delayed announcement, and the lack of the
> usual
> > > > paragraph summarizing individual contributions.)
> > > >
> > > > Congratulations to all three! Welcome!
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Thanks,
> > >>> Xin
> > >>
> > >>
> > >>
> > >> --
> > >> ---
> > >> Jason Kuster
> > >> Apache Beam / Google Cloud Dataflow
> > >>
> > >> See something? Say something. go/jasonkuster-feedback
> > 
> > 
> > >>>
> > >> --
> > >> Got feedback? go/pabloem-feedback
>
-- 
Twitter: https://twitter.com/holdenkarau

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Ismaël Mejía]

Congratulations!

On Fri, Jun 1, 2018 at 8:26 AM Pei HE  wrote:
>
> Congrats!
>
> On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen  wrote:
> > Congratulations everyone!
> >
> >
> > On Thu, May 31, 2018, 10:14 PM Pablo Estrada  wrote:
> >>
> >> Thanks to the PMC! Very humbled and excited to keep taking part in this
> >> great community.
> >> :)
> >> -P.
> >>
> >>
> >> On Thu, May 31, 2018, 10:10 PM Tim  wrote:
> >>>
> >>> Congratulations!
> >>>
> >>>
> >>> Tim
> >>>
> >>> On 1 Jun 2018, at 07:05, Andrew Psaltis  wrote:
> >>>
> >>> Congrats!
> >>>
> >>> On Fri, Jun 1, 2018 at 12:26 AM, Thomas Weise  wrote:
> 
>  Congrats!
> 
> 
>  On Thu, May 31, 2018 at 9:25 PM, Alan Myrvold 
>  wrote:
> >
> > Congrats Gris+Pablo+Jason. Well deserved.
> >
> > On Thu, May 31, 2018 at 9:15 PM Jason Kuster 
> > wrote:
> >>
> >> Thank you to Davor and the PMC; I'm excited to be able to help Beam in
> >> this new capacity. Bring on the PRs. :D
> >>
> >> On Thu, May 31, 2018 at 8:55 PM Xin Wang 
> >> wrote:
> >>>
> >>> Congrats!
> >>>
> >>> - Xin Wang
> >>>
> >>> 2018-06-01 11:52 GMT+08:00 Rui Wang :
> 
>  Congrats!
> 
>  -Rui
> 
>  On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré
>   wrote:
> >
> > Congrats !
> >
> > Regards
> > JB
> >
> > On 01/06/2018 04:08, Davor Bonaci wrote:
> > > Please join me and the rest of Beam PMC in welcoming the
> > > following
> > > contributors as our newest committers. They have significantly
> > > contributed to the project in different ways, and we look forward
> > > to
> > > many more contributions in the future.
> > >
> > > * Griselda Cuevas
> > > * Pablo Estrada
> > > * Jason Kuster
> > >
> > > (Apologizes for a delayed announcement, and the lack of the usual
> > > paragraph summarizing individual contributions.)
> > >
> > > Congratulations to all three! Welcome!
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Xin
> >>
> >>
> >>
> >> --
> >> ---
> >> Jason Kuster
> >> Apache Beam / Google Cloud Dataflow
> >>
> >> See something? Say something. go/jasonkuster-feedback
> 
> 
> >>>
> >> --
> >> Got feedback? go/pabloem-feedback

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Pei HE]

Congrats!

On Fri, Jun 1, 2018 at 2:12 PM, Charles Chen  wrote:
> Congratulations everyone!
>
>
> On Thu, May 31, 2018, 10:14 PM Pablo Estrada  wrote:
>>
>> Thanks to the PMC! Very humbled and excited to keep taking part in this
>> great community.
>> :)
>> -P.
>>
>>
>> On Thu, May 31, 2018, 10:10 PM Tim  wrote:
>>>
>>> Congratulations!
>>>
>>>
>>> Tim
>>>
>>> On 1 Jun 2018, at 07:05, Andrew Psaltis  wrote:
>>>
>>> Congrats!
>>>
>>> On Fri, Jun 1, 2018 at 12:26 AM, Thomas Weise  wrote:

 Congrats!


 On Thu, May 31, 2018 at 9:25 PM, Alan Myrvold 
 wrote:
>
> Congrats Gris+Pablo+Jason. Well deserved.
>
> On Thu, May 31, 2018 at 9:15 PM Jason Kuster 
> wrote:
>>
>> Thank you to Davor and the PMC; I'm excited to be able to help Beam in
>> this new capacity. Bring on the PRs. :D
>>
>> On Thu, May 31, 2018 at 8:55 PM Xin Wang 
>> wrote:
>>>
>>> Congrats!
>>>
>>> - Xin Wang
>>>
>>> 2018-06-01 11:52 GMT+08:00 Rui Wang :

 Congrats!

 -Rui

 On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré
  wrote:
>
> Congrats !
>
> Regards
> JB
>
> On 01/06/2018 04:08, Davor Bonaci wrote:
> > Please join me and the rest of Beam PMC in welcoming the
> > following
> > contributors as our newest committers. They have significantly
> > contributed to the project in different ways, and we look forward
> > to
> > many more contributions in the future.
> >
> > * Griselda Cuevas
> > * Pablo Estrada
> > * Jason Kuster
> >
> > (Apologizes for a delayed announcement, and the lack of the usual
> > paragraph summarizing individual contributions.)
> >
> > Congratulations to all three! Welcome!
>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Xin
>>
>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback


>>>
>> --
>> Got feedback? go/pabloem-feedback

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Charles Chen]

Congratulations everyone!

On Thu, May 31, 2018, 10:14 PM Pablo Estrada  wrote:

> Thanks to the PMC! Very humbled and excited to keep taking part in this
> great community.
> :)
> -P.
>
>
> On Thu, May 31, 2018, 10:10 PM Tim  wrote:
>
>> Congratulations!
>>
>>
>> Tim
>>
>> On 1 Jun 2018, at 07:05, Andrew Psaltis  wrote:
>>
>> Congrats!
>>
>> On Fri, Jun 1, 2018 at 12:26 AM, Thomas Weise  wrote:
>>
>>> Congrats!
>>>
>>>
>>> On Thu, May 31, 2018 at 9:25 PM, Alan Myrvold 
>>> wrote:
>>>
 Congrats Gris+Pablo+Jason. Well deserved.

 On Thu, May 31, 2018 at 9:15 PM Jason Kuster 
 wrote:

> Thank you to Davor and the PMC; I'm excited to be able to help Beam in
> this new capacity. Bring on the PRs. :D
>
> On Thu, May 31, 2018 at 8:55 PM Xin Wang 
> wrote:
>
>> Congrats!
>>
>> - Xin Wang
>>
>> 2018-06-01 11:52 GMT+08:00 Rui Wang :
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré <
>>> j...@nanthrax.net> wrote:
>>>
 Congrats !

 Regards
 JB

 On 01/06/2018 04:08, Davor Bonaci wrote:
 > Please join me and the rest of Beam PMC in welcoming the following
 > contributors as our newest committers. They have significantly
 > contributed to the project in different ways, and we look forward
 to
 > many more contributions in the future.
 >
 > * Griselda Cuevas
 > * Pablo Estrada
 > * Jason Kuster
 >
 > (Apologizes for a delayed announcement, and the lack of the usual
 > paragraph summarizing individual contributions.)
 >
 > Congratulations to all three! Welcome!

>>>
>>
>>
>> --
>> Thanks,
>> Xin
>>
>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

>>>
>> --
> Got feedback? go/pabloem-feedback
> 
>