Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Reza Rokni]

+1

On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli  wrote:

> +1
>
> On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi  wrote:
>
>> +1
>>
>> On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang 
>> wrote:
>>
>>> +1
>>>
>>> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett <
>>> whatwouldausti...@gmail.com> wrote:
>>>
 +1

 On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Hi everyone,
>
> Please vote whether to sign a pledge on behalf of Apache Beam to
> sunset Beam Python 2 offering (in new releases) in 2020 on
> http://python3stament.org as follows:
>
> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in
> 2020.
> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in
> Beam in 2020.
>
> The motivation and details for this vote were discussed in [1, 2].
> Please follow up in [2] if you have any questions.
>
> This is a procedural vote [3] that will follow the majority approval
> rules and will be open for at least 72 hours.
>
> Thanks,
> Valentyn
>
> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
> [3] https://www.apache.org/foundation/voting.html
>
>

-- 

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Tanay Tummalapalli]

+1

On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi  wrote:

> +1
>
> On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang 
> wrote:
>
>> +1
>>
>> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett <
>> whatwouldausti...@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
>>> wrote:
>>>
 Hi everyone,

 Please vote whether to sign a pledge on behalf of Apache Beam to sunset
 Beam Python 2 offering (in new releases) in 2020 on
 http://python3stament.org as follows:

 [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in
 2020.
 [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam
 in 2020.

 The motivation and details for this vote were discussed in [1, 2].
 Please follow up in [2] if you have any questions.

 This is a procedural vote [3] that will follow the majority approval
 rules and will be open for at least 72 hours.

 Thanks,
 Valentyn

 [1]
 https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
 [2]
 https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
 [3] https://www.apache.org/foundation/voting.html

Re: Introduction + Support in Comms for Beam!

[Lukasz Cwik]

Welcome to the community.

On Mon, Sep 30, 2019 at 3:15 PM María Cruz  wrote:

> Hi everyone,
> my name is María Cruz, I am from Buenos Aires but I live in the Bay Area.
> I recently became acquainted with Apache Beam project, and I got a chance
> to meet some of the Beam community at Apache Con North America this past
> September. I'm testing out a communications framework
> 
> for Open Source communities. I'm emailing the list now because I'd like to
> work on a communications strategy for Beam, to make the most of the content
> you produce during Beam Summits.
>
> A little bit more about me. I am a communications strategist with 11 years
> of experience in the field, 8 of which are in the non-profit sector. I
> started working in Open Source in 2013, when I joined Wikimedia, the social
> movement behind Wikipedia. I now work to support Google Open Source
> projects, and I also volunteer in the communications team of the Apache
> Software Foundation, working closely with Sally (for those of you who know
> her).
>
> I will be sending the list a proposal in the coming days. Looking forward
> to hearing from you!
>
> Best,
>
> María
>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Suneel Marthi]

+1

On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang  wrote:

> +1
>
> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett 
> wrote:
>
>> +1
>>
>> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
>>> Beam Python 2 offering (in new releases) in 2020 on
>>> http://python3stament.org as follows:
>>>
>>> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
>>> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam
>>> in 2020.
>>>
>>> The motivation and details for this vote were discussed in [1, 2].
>>> Please follow up in [2] if you have any questions.
>>>
>>> This is a procedural vote [3] that will follow the majority approval
>>> rules and will be open for at least 72 hours.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> [2]
>>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>>> [3] https://www.apache.org/foundation/voting.html
>>>
>>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Manu Zhang]

+1

On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett 
wrote:

> +1
>
> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi everyone,
>>
>> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
>> Beam Python 2 offering (in new releases) in 2020 on
>> http://python3stament.org as follows:
>>
>> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
>> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam
>> in 2020.
>>
>> The motivation and details for this vote were discussed in [1, 2]. Please
>> follow up in [2] if you have any questions.
>>
>> This is a procedural vote [3] that will follow the majority approval
>> rules and will be open for at least 72 hours.
>>
>> Thanks,
>> Valentyn
>>
>> [1]
>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> [2]
>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>> [3] https://www.apache.org/foundation/voting.html
>>
>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Austin Bennett]

+1

On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
wrote:

> Hi everyone,
>
> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
> Beam Python 2 offering (in new releases) in 2020 on
> http://python3stament.org as follows:
>
> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam in
> 2020.
>
> The motivation and details for this vote were discussed in [1, 2]. Please
> follow up in [2] if you have any questions.
>
> This is a procedural vote [3] that will follow the majority approval rules
> and will be open for at least 72 hours.
>
> Thanks,
> Valentyn
>
> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
> [3] https://www.apache.org/foundation/voting.html
>
>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Robert Bradshaw]

+1

On Mon, Sep 30, 2019 at 5:35 PM David Cavazos  wrote:
>
> +1
>
> On Mon, Sep 30, 2019 at 5:27 PM Ahmet Altay  wrote:
>>
>> +1
>>
>> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev  
>> wrote:
>>>
>>> Hi everyone,
>>>
>>> Please vote whether to sign a pledge on behalf of Apache Beam to sunset 
>>> Beam Python 2 offering (in new releases) in 2020 on 
>>> http://python3stament.org as follows:
>>>
>>> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
>>> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam in 
>>> 2020.
>>>
>>> The motivation and details for this vote were discussed in [1, 2]. Please 
>>> follow up in [2] if you have any questions.
>>>
>>> This is a procedural vote [3] that will follow the majority approval rules 
>>> and will be open for at least 72 hours.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> [1] 
>>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>>> [2] 
>>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>>> [3] https://www.apache.org/foundation/voting.html
>>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[David Cavazos]

+1

On Mon, Sep 30, 2019 at 5:27 PM Ahmet Altay  wrote:

> +1
>
> On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi everyone,
>>
>> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
>> Beam Python 2 offering (in new releases) in 2020 on
>> http://python3stament.org as follows:
>>
>> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
>> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam
>> in 2020.
>>
>> The motivation and details for this vote were discussed in [1, 2]. Please
>> follow up in [2] if you have any questions.
>>
>> This is a procedural vote [3] that will follow the majority approval
>> rules and will be open for at least 72 hours.
>>
>> Thanks,
>> Valentyn
>>
>> [1]
>> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
>> [2]
>> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
>> [3] https://www.apache.org/foundation/voting.html
>>
>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Ahmet Altay]

+1

On Mon, Sep 30, 2019 at 5:22 PM Valentyn Tymofieiev 
wrote:

> Hi everyone,
>
> Please vote whether to sign a pledge on behalf of Apache Beam to sunset
> Beam Python 2 offering (in new releases) in 2020 on
> http://python3stament.org as follows:
>
> [ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
> [ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam in
> 2020.
>
> The motivation and details for this vote were discussed in [1, 2]. Please
> follow up in [2] if you have any questions.
>
> This is a procedural vote [3] that will follow the majority approval rules
> and will be open for at least 72 hours.
>
> Thanks,
> Valentyn
>
> [1]
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
> [3] https://www.apache.org/foundation/voting.html
>
>

[VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

[Valentyn Tymofieiev]

Hi everyone,

Please vote whether to sign a pledge on behalf of Apache Beam to sunset
Beam Python 2 offering (in new releases) in 2020 on
http://python3stament.org as follows:

[ ] +1: Sign a pledge to discontinue support of Python 2 in Beam in 2020.
[ ] -1: Do not sign a pledge to discontinue support of Python 2 in Beam in
2020.

The motivation and details for this vote were discussed in [1, 2]. Please
follow up in [2] if you have any questions.

This is a procedural vote [3] that will follow the majority approval rules
and will be open for at least 72 hours.

Thanks,
Valentyn

[1]
https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
[2]
https://lists.apache.org/thread.html/456631fe1a696c537ef8ebfee42cd3ea8121bf7c639c52da5f7032e7@%3Cdev.beam.apache.org%3E
[3] https://www.apache.org/foundation/voting.html

Re: Plan for dropping python 2 support

[Valentyn Tymofieiev]

As suggested and enthusiastically supported by several folks in this
thread, I will send a vote to sign a pledge on http://python3statement.org
on behalf of Apache Beam to discontinue Python 2 support in or before 2020.

The motivation for signing the pledge is:
- to provide another signal to Beam users, and projects that depend on Beam
that Beam Python 2 offering will soon sunset;
- to facilitate adoption of Python 3 by Beam users, developers, and runner
maintainers;
- to facilitate adoption of Python 3 in wider Python ecosystem.

See also http://python3stament.org for background behind this pledge and
the list of projects which have already signed it.

On Mon, Sep 23, 2019 at 4:45 PM Kyle Weaver  wrote:

> Re feedback collection, we already print a message:
> "You are using Apache Beam with Python 2. New releases of Apache Beam will
> soon support Python 3 only."
> When users run Python 2 pipelines. This might be a good place to provide
> additional info along with a place to send feedback (probably user@).
> While I'm sure not everyone out there reads their logs, I imagine this is a
> sure and easy way of reaching at least some Python 2 users.
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
>
>
> On Fri, Sep 20, 2019 at 10:28 AM Valentyn Tymofieiev 
> wrote:
>
>> Thank you, Chad, for refreshing this conversation and adding the
>> perspective of Python 2 users of Beam who have not(yet) completed the
>> migration. My thoughts below.
>>
>> - It is in the best interest of everyone to ensure a smooth migration for
>> Beam users. However a migration needs to happen since Python ecosystem is
>> moving off of Python 2.
>> - Beam has a couple of dozen dependencies, and we cannot have an
>> expectation that Python 2 versions of these dependencies will be maintained
>> in 2020.
>> - BEAM-1251 should be closed, since it may communicate a signal that Beam
>> does not support Python 3, while it does. Beam has first announced support
>> of Python 3 in Beam 2.11.0, admittedly later than many mainstream libraries
>> in Python ecosystem.
>> - I think Python 2 LTS release (if we continue them) may have critical
>> bug fixes, but not new features, so we won't be backporting new features.
>> - Beam portability allows users to customize usercode runtime
>> environment, and it should be possible for users to supply a Python 2 SDK
>> harness container, should they have no other option. This would require a
>> backported user-supplied version of Beam SDK that works on Python 2,
>> although such SDK may become difficult/impractical to maintain for most
>> users.
>> - There are several open issues related to Python 3, but they are
>> improvements in nature, and we are steadily closing them off. I am not
>> aware of any adoption blockers for Beam Python 3, specific to Beam.
>> - I have not heard of users reports who attempted but were not able to
>> use Beam on Python 3.
>> - This does not mean that our offering is perfect, there may be errors
>> and omissions that are yet to be discovered. However, it would be in the
>> best interest of the Beam community to discover these issues earlier. A
>> message that Beam will discontinue Python 2 support will encourage users to
>> migrate, therefore I also support Beam signing
>> https://python3statement.org.
>> - Having more usage statistics and feedback closer to 2020 can help us be
>> more confident in deciding when to stop Python 2 support.
>>
>> On Thu, Sep 19, 2019 at 6:05 PM Ahmet Altay  wrote:
>>
>>> Thanks a lot for sharing your thoughts, I completely agree that we need
>>> to minimize the burden on our users as much as possible. Especially in this
>>> case when we are offering a robust python 3 solution just now. However I do
>>> share the same concerns related to dependencies and tool chains, It will be
>>> increasingly difficult for us to keep our code base compatible with python2
>>> and python3 overtime. (To be very explicit, one of those dependencies is
>>> Dataflow's python pre-portability workers.)
>>>
>>> On Thu, Sep 19, 2019 at 5:17 PM Maximilian Michels 
>>> wrote:
>>>
 Granted that we just have finalized the Python 3 support, we should
 allow time for it to mature and for users to make the switch.

 > Oh, and one more thing, I think it'd make sense for Apache Beam to
 > sign https://python3statement.org/. The promise is that we'd
 > discontinue Python 2 support *in* 2020, which is not committing us to
 > January if we're not ready. Worth a vote?

 +1

>>>
>>> +1
>>>
>>>

 On 19.09.19 15:59, Robert Bradshaw wrote:
 > Oh, and one more thing, I think it'd make sense for Apache Beam to
 > sign https://python3statement.org/. The promise is that we'd
 > discontinue Python 2 support *in* 2020, which is not committing us to
 > January if we're not ready. Worth a vote?
 >
 >
 > On Thu, Sep 19, 2019 at 3:58 PM Robert Bradshaw 
 wrote:
 >>
 >> Exactly how long we s

Re: Multiple iterations after GroupByKey with SparkRunner

[Kenneth Knowles]

In the car analogy, you have something this:

Iterable: car
Iterator: taxi ride

They are related, but not as variations of a common concept.

In the discussion of Combine vs RSBK, if the reducer is required to be an
associative and commutative operator, then it is the same thing under a
different name. If the reducer can be non-associative or non-commutative,
then it admits fewer transformations/optimizations.

If you introduce a GroupIteratorsByKey and implement GroupByKey as a
transform that combines the iterator by concatenation, I think you do get
an internally consistent system. To execute efficiently, you need to always
identify and replace the GroupByKey operation with a primitive one. It does
make some sense to expose the weakest primitives for the sake of DSLs. But
they are very poorly suited for end-users, and for GBK on most runners you
get the more powerful one for free.

Kenn

On Mon, Sep 30, 2019 at 2:02 AM Jan Lukavský  wrote:

> > The fact that the annotation on the ParDo "changes" the GroupByKey
> implementation is very specific to the Spark runner implementation.
>
> I don't quite agree. It is not very specific to Spark, it is specific to
> generally all runners, that produce grouped elements in a way that is not
> reiterable. That is the key property. The example you gave with HDFS does
> not satisfy this condition (files on HDFS are certainly reiterable), and
> that's why no change to the GBK is needed (it actually already has the
> required property). A quick look at what FlinkRunner (at least non portable
> does) is that it implements GBK using reducing elements into List. That is
> going to crash on big PCollection, which is even nicely documented:
>
>* For internal use to translate {@link GroupByKey}. For a large {@link 
> PCollection} this is
>* expected to crash!
>
>
> If this is fixed, then it is likely to start behave the same as Spark. So
> actually I think the opposite is true - Dataflow is a special case, because
> of how its internal shuffle service works.
>
> > In general I sympathize with the worry about non-local effects. Beam is
> already full of them (e.g. a Window.into statement effects downstream
> GroupByKeys). In each case where they were added there was extensive debate
> and discussion (Windowing semantics were debated for many months), exactly
> because there was concern over adding these non-local effects. In every
> case, no other good solution could be found. For the case of windowing for
> example, it was often easy to propose simple local APIs (e.g. just pass the
> window fn as a parameter to GroupByKey), however all of these local
> solutions ended up not working for important use cases when we analyzed
> them more deeply.
>
> That is very interesting. Could you elaborate more about some examples of
> the use cases which didn't work? I'd like to try to match it against how
> Euphoria is structured, it should be more resistant to this non-local
> effects, because it very often bundles together multiple Beam's primitives
> to single transform - ReduceByKey is one example of this, if is actually
> mix of Window.into() + GBK + ParDo, Although it might look like if this
> transform can be broken down to something else, then it is not primitive
> (euphoria has no native equivalent of GBK itself), but it has several other
> nice implications - that is that Combine now becomes a special case of RBK.
> It now becomes only a question of where and how you can "run" the reduce
> function. The logic is absolutely equal. This can be worked in more detail
> and actually show, that even Combine and RBK can be decribed by a more
> general stateful operation (ReduceStateByKey), and so finally Euphoria
> actually has only two really "primitive" operations - these are FlatMap
> (basically stateless ParDo) and RSBK. As I already mentioned on some other
> thread, when stateful ParDo would support merging windows, it can be shown
> that both Combine and GBK become special cases of this.
>
> > As you mentioned below, I do think it's perfectly reasonable for a DSL
> to impose its own semantics. Scio already does this - the raw Beam API is
> used by a DSL as a substrate, but the DSL does not need to blindly mirror
> the semantics of the raw Beam API - at least in my opinion!
>
> Sure, but currently, there is no way for DSL to "hook" into runner, so it
> has to use raw Beam SDK, and so this will fail in cases like this - where
> Beam actually has stronger guarantees than it is required by the DSL. It
> would be cool if we could find a way to do that - this pretty much aligns
> with another question raised on ML, about the possibility to override a
> default implementation of a PTransform for specific pipeline.
>
> Jan
>
>
> On 9/29/19 7:46 PM, Reuven Lax wrote:
>
> Jan,
>
> The fact that the annotation on the ParDo "changes" the GroupByKey
> implementation is very specific to the Spark runner implementation. You can
> imagine another runner that simply writes out fil

Introduction + Support in Comms for Beam!

[María Cruz]

Hi everyone,
my name is María Cruz, I am from Buenos Aires but I live in the Bay Area. I
recently became acquainted with Apache Beam project, and I got a chance to
meet some of the Beam community at Apache Con North America this past
September. I'm testing out a communications framework

for Open Source communities. I'm emailing the list now because I'd like to
work on a communications strategy for Beam, to make the most of the content
you produce during Beam Summits.

A little bit more about me. I am a communications strategist with 11 years
of experience in the field, 8 of which are in the non-profit sector. I
started working in Open Source in 2013, when I joined Wikimedia, the social
movement behind Wikipedia. I now work to support Google Open Source
projects, and I also volunteer in the communications team of the Apache
Software Foundation, working closely with Sally (for those of you who know
her).

I will be sending the list a proposal in the coming days. Looking forward
to hearing from you!

Best,

María

Re: Heads-up: Python IT test breakage is investigated in BEAM-8324

[Valentyn Tymofieiev]

Should be now fixed by https://github.com/apache/beam/pull/9695.

On Mon, Sep 30, 2019 at 1:32 PM Maximilian Michels  wrote:

> Thanks for letting us know. Was wondering, why the tests suddenly broke.
> See https://builds.apache.org/job/beam_PreCommit_Python_Cron/ and
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/
>
> On 30.09.19 10:45, Valentyn Tymofieiev wrote:
> >
>

Re: Heads-up: Python IT test breakage is investigated in BEAM-8324

[Maximilian Michels]

Thanks for letting us know. Was wondering, why the tests suddenly broke. 
See https://builds.apache.org/job/beam_PreCommit_Python_Cron/ and 
https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/


On 30.09.19 10:45, Valentyn Tymofieiev wrote:

Re: [ANNOUNCE] New committer: Alan Myrvold

[Udi Meiri]

Congrats Alan!

On Mon, Sep 30, 2019 at 11:12 AM Alan Myrvold  wrote:

> Thanks!! Looking forward to making more impact to Apache Beam
>
> On Mon, Sep 30, 2019 at 10:56 AM Mikhail Gryzykhin 
> wrote:
>
>> Congratulations!
>>
>> On Mon, Sep 30, 2019 at 9:47 AM David Cavazos 
>> wrote:
>>
>>> Congratulations Alan!
>>>
>>> On Mon, Sep 30, 2019 at 7:57 AM Connell O'Callaghan 
>>> wrote:
>>>
 Congratulations Alan - well done!!! Ahmet thank you for sharing this
 great news!!!

 On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy 
 wrote:

> Congratulations :)
>
> pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):
>
>> Woohoo Congratulations!
>>
>> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía 
>>> wrote:
>>>
 Congrats Alan!

 On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli <
 ttanay...@gmail.com> wrote:

> Congratulations, Alan!
>
>
> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Alan. Well deserved.
>>>
>>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 Congrats Alan!!

 On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
 wrote:

> Congrats Alan!
> On 9/27/19 10:22 PM, Mark Liu wrote:
>
> Congratulations Alan!!!
>
> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
> wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka <
>> goe...@google.com> wrote:
>>
>>> Congratulations Alan!
>>>
>>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang <
>>> zyi...@google.com> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu <
 robi...@google.com> wrote:

> Congrats, Alan!
>
> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
>> ruo...@google.com> wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang <
>>> ruw...@google.com> wrote:
>>>
 Congrats!

 -Rui

 On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
 pabl...@google.com> wrote:

> Yooh! : D
>
> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
> yifan...@google.com> wrote:
>
>> Congratulations, Alan!
>>
>> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
>> al...@google.com> wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in
>>> welcoming a new
>>> committer: Alan Myrvold
>>>
>>> Alan has been a long time Beam contributor. His
>>> contributions made Beam more productive and friendlier 
>>> [1] for all
>>> contributors with significant improvements to Beam 
>>> release process,
>>> automation, and infrastructure.
>>>
>>> In consideration of Alan's contributions, the Beam
>>> PMC trusts him
>>> with the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Alan, for your contributions and looking
>>> forward to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>>> [2] https://beam.apache.org/contribute/become-a-
>>> committer/#an-apache-beam-committer
>>>
>>
>>>
>

Re: [ANNOUNCE] New committer: Alan Myrvold

[Alan Myrvold]

Thanks!! Looking forward to making more impact to Apache Beam

On Mon, Sep 30, 2019 at 10:56 AM Mikhail Gryzykhin 
wrote:

> Congratulations!
>
> On Mon, Sep 30, 2019 at 9:47 AM David Cavazos  wrote:
>
>> Congratulations Alan!
>>
>> On Mon, Sep 30, 2019 at 7:57 AM Connell O'Callaghan 
>> wrote:
>>
>>> Congratulations Alan - well done!!! Ahmet thank you for sharing this
>>> great news!!!
>>>
>>> On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy 
>>> wrote:
>>>
 Congratulations :)

 pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):

> Woohoo Congratulations!
>
> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía 
>> wrote:
>>
>>> Congrats Alan!
>>>
>>> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli <
>>> ttanay...@gmail.com> wrote:
>>>
 Congratulations, Alan!


 On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
 wrote:

> Congratulations!
>
> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Alan. Well deserved.
>>
>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> Congrats Alan!!
>>>
>>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
>>> wrote:
>>>
 Congrats Alan!
 On 9/27/19 10:22 PM, Mark Liu wrote:

 Congratulations Alan!!!

 On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
 wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka <
> goe...@google.com> wrote:
>
>> Congratulations Alan!
>>
>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang <
>> zyi...@google.com> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu <
>>> robi...@google.com> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
 hannahji...@google.com> wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
> ruo...@google.com> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang <
>> ruw...@google.com> wrote:
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
>>> pabl...@google.com> wrote:
>>>
 Yooh! : D

 On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
 yifan...@google.com> wrote:

> Congratulations, Alan!
>
> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
> al...@google.com> wrote:
>
>> Hi,
>>
>> Please join me and the rest of the Beam PMC in
>> welcoming a new
>> committer: Alan Myrvold
>>
>> Alan has been a long time Beam contributor. His
>> contributions made Beam more productive and friendlier 
>> [1] for all
>> contributors with significant improvements to Beam 
>> release process,
>> automation, and infrastructure.
>>
>> In consideration of Alan's contributions, the Beam
>> PMC trusts him
>> with the responsibilities of a Beam committer [2].
>>
>> Thank you, Alan, for your contributions and looking
>> forward to many more!
>>
>> Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>> [2] https://beam.apache.org/contribute/become-a-
>> committer/#an-apache-beam-committer
>>
>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>
>
> --
> Cheers,
> Gleb
>

>
> --
>
>>

[DISCUSS] Beam SQL filter push-down

[Kirill Kozlov]

The objective is to create a universal way for Beam SQL IO APIs to support
filter/project push-down.
A proposed way to achieve that is by introducing an interface
responsible for identifying what portion(s) of a Calc can be moved down to
IO layer. Also, adding following methods to a BeamSqlTable interface to
pass necessary parameters to IO APIs:
- BeamSqlTableFilter supportsFilter(RexNode program, RexNode filter)
- Boolean supportsProjects()
- PCollection buildIOReader(PBegin begin, BeamSqlTableFilter
filters, List fieldNames)

Please feel free to provide feedback and suggestions on this proposal.
Thank you!

Here is a more complete design doc:
https://docs.google.com/document/d/1-ysD7U7qF3MAmSfkbXZO_5PLJBevAL9bktlLCerd_jE/edit?usp=sharing

--
Kirill Kozlov

Re: [ANNOUNCE] New committer: Alan Myrvold

[Mikhail Gryzykhin]

Congratulations!

On Mon, Sep 30, 2019 at 9:47 AM David Cavazos  wrote:

> Congratulations Alan!
>
> On Mon, Sep 30, 2019 at 7:57 AM Connell O'Callaghan 
> wrote:
>
>> Congratulations Alan - well done!!! Ahmet thank you for sharing this
>> great news!!!
>>
>> On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy  wrote:
>>
>>> Congratulations :)
>>>
>>> pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):
>>>
 Woohoo Congratulations!

 On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:

> Congratulations, Alan!
>
>
> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía 
> wrote:
>
>> Congrats Alan!
>>
>> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli <
>> ttanay...@gmail.com> wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
>>> wrote:
>>>
 Congratulations!

 On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Alan. Well deserved.
>
> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> Congrats Alan!!
>>
>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
>> wrote:
>>
>>> Congrats Alan!
>>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>>
>>> Congratulations Alan!!!
>>>
>>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
>>> wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka <
 goe...@google.com> wrote:

> Congratulations Alan!
>
> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang <
> zyi...@google.com> wrote:
>
>> Congrats, Alan!
>>
>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu <
>> robi...@google.com> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>>> hannahji...@google.com> wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
 ruo...@google.com> wrote:

> Congratulations, Alan!
>
>
> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang <
> ruw...@google.com> wrote:
>
>> Congrats!
>>
>> -Rui
>>
>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
>> pabl...@google.com> wrote:
>>
>>> Yooh! : D
>>>
>>> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
>>> yifan...@google.com> wrote:
>>>
 Congratulations, Alan!

 On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
 al...@google.com> wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in
> welcoming a new
> committer: Alan Myrvold
>
> Alan has been a long time Beam contributor. His
> contributions made Beam more productive and friendlier 
> [1] for all
> contributors with significant improvements to Beam 
> release process,
> automation, and infrastructure.
>
> In consideration of Alan's contributions, the Beam PMC
> trusts him
> with the responsibilities of a Beam committer [2].
>
> Thank you, Alan, for your contributions and looking
> forward to many more!
>
> Ahmet, on behalf of the Apache Beam PMC
>
> [1]
> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
> [2] https://beam.apache.org/contribute/become-a-
> committer/#an-apache-beam-committer
>

>
> --
> 
> Ruoyun  Huang
>
>

 --
 Cheers,
 Gleb

>>>

 --

 This email may be confidential and privileged. If you received this
 communication by mistake, please don't forward it to anyone else, please
 erase all copies and attachments, and please let me know that it has gone
 to the wrong person.

 The above terms re

Heads-up: Python IT test breakage is investigated in BEAM-8324

[Valentyn Tymofieiev]

Re: [ANNOUNCE] New committer: Alan Myrvold

[David Cavazos]

Congratulations Alan!

On Mon, Sep 30, 2019 at 7:57 AM Connell O'Callaghan 
wrote:

> Congratulations Alan - well done!!! Ahmet thank you for sharing this great
> news!!!
>
> On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy  wrote:
>
>> Congratulations :)
>>
>> pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):
>>
>>> Woohoo Congratulations!
>>>
>>> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>>>
 Congratulations, Alan!


 On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía  wrote:

> Congrats Alan!
>
> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congratulations, Alan. Well deserved.

 On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
 chamik...@google.com> wrote:

> Congrats Alan!!
>
> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
> wrote:
>
>> Congrats Alan!
>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>
>> Congratulations Alan!!!
>>
>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
>> wrote:
>>
>>> Congrats Alan!
>>>
>>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
>>> wrote:
>>>
 Congratulations Alan!

 On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
 wrote:

> Congrats, Alan!
>
> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
> wrote:
>
>> Congrats, Alan!
>>
>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>> hannahji...@google.com> wrote:
>>
>>> Congrats Alan!
>>>
>>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
>>> ruo...@google.com> wrote:
>>>
 Congratulations, Alan!


 On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
 wrote:

> Congrats!
>
> -Rui
>
> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
> pabl...@google.com> wrote:
>
>> Yooh! : D
>>
>> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
>> yifan...@google.com> wrote:
>>
>>> Congratulations, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
>>> al...@google.com> wrote:
>>>
 Hi,

 Please join me and the rest of the Beam PMC in
 welcoming a new
 committer: Alan Myrvold

 Alan has been a long time Beam contributor. His
 contributions made Beam more productive and friendlier [1] 
 for all
 contributors with significant improvements to Beam release 
 process,
 automation, and infrastructure.

 In consideration of Alan's contributions, the Beam PMC
 trusts him
 with the responsibilities of a Beam committer [2].

 Thank you, Alan, for your contributions and looking
 forward to many more!

 Ahmet, on behalf of the Apache Beam PMC

 [1]
 https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
 [2] https://beam.apache.org/contribute/become-a-
 committer/#an-apache-beam-committer

>>>

 --
 
 Ruoyun  Huang


>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
>>>
>>> --
>>>
>>> This email may be confidential and privileged. If you received this
>>> communication by mistake, please don't forward it to anyone else, please
>>> erase all copies and attachments, and please let me know that it has gone
>>> to the wrong person.
>>>
>>> The above terms reflect a potential business arrangement, are provided
>>> solely as a basis for further discussion, and are not intended to be and do
>>> not constitute a legally binding obligation. No legally binding obligations
>>> will be created, implied, or inferred until an agreement in final form is
>>> executed in writing by all parties involved.

Re: Why is there no standard boolean coder?

[Maximilian Michels]

+1

On 29.09.19 13:44, Chad Dombrova wrote:
I’m planning on porting the existing Java coder to Python. Any 
objections to that?


-chad


On Sun, Sep 29, 2019 at 1:02 PM Robert Burke > wrote:


+1

I'm happy to whip together the Go SDK version once the encoding has
been concretely decided.

On Fri, Sep 27, 2019, 6:07 PM Chad Dombrova mailto:chad...@gmail.com>> wrote:


It would still be a standard coder - the distinction I'm
proposing is that there are certain coders that _must_ be
implemented by a new runner/sdk (for example windowedvalue,
varint, kv, ...) since they are important for SDK - runner
communication, but now we're starting to standardize coders
that are useful for cross-language and schemas.

Got it.  Sounds good.

-chad

Re: [ANNOUNCE] New committer: Alan Myrvold

[Connell O'Callaghan]

Congratulations Alan - well done!!! Ahmet thank you for sharing this great
news!!!

On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy  wrote:

> Congratulations :)
>
> pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):
>
>> Woohoo Congratulations!
>>
>> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía  wrote:
>>>
 Congrats Alan!

 On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
 wrote:

> Congratulations, Alan!
>
>
> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Alan. Well deserved.
>>>
>>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 Congrats Alan!!

 On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
 wrote:

> Congrats Alan!
> On 9/27/19 10:22 PM, Mark Liu wrote:
>
> Congratulations Alan!!!
>
> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
> wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
>> wrote:
>>
>>> Congratulations Alan!
>>>
>>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
>>> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
 wrote:

> Congrats, Alan!
>
> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
>> ruo...@google.com> wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
>>> wrote:
>>>
 Congrats!

 -Rui

 On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
 pabl...@google.com> wrote:

> Yooh! : D
>
> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
> yifan...@google.com> wrote:
>
>> Congratulations, Alan!
>>
>> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
>> al...@google.com> wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming
>>> a new
>>> committer: Alan Myrvold
>>>
>>> Alan has been a long time Beam contributor. His
>>> contributions made Beam more productive and friendlier [1] 
>>> for all
>>> contributors with significant improvements to Beam release 
>>> process,
>>> automation, and infrastructure.
>>>
>>> In consideration of Alan's contributions, the Beam PMC
>>> trusts him
>>> with the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Alan, for your contributions and looking
>>> forward to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>>> [2] https://beam.apache.org/contribute/become-a-
>>> committer/#an-apache-beam-committer
>>>
>>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>>
>> The above terms reflect a potential business arrangement, are provided
>> solely as a basis for further discussion, and are not intended to be and do
>> not constitute a legally binding obligation. No legally binding obligations
>> will be created, implied, or inferred until an agreement in final form is
>> executed in writing by all parties involved.
>>
>

Re: [ANNOUNCE] New committer: Alan Myrvold

[Łukasz Gajowy]

Congratulations :)

pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):

> Woohoo Congratulations!
>
> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía  wrote:
>>
>>> Congrats Alan!
>>>
>>> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
>>> wrote:
>>>
 Congratulations, Alan!


 On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov  wrote:

> Congratulations!
>
> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Alan. Well deserved.
>>
>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> Congrats Alan!!
>>>
>>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
>>> wrote:
>>>
 Congrats Alan!
 On 9/27/19 10:22 PM, Mark Liu wrote:

 Congratulations Alan!!!

 On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
 wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
> wrote:
>
>> Congratulations Alan!
>>
>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
>> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
>>> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
 hannahji...@google.com> wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
> ruo...@google.com> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
>> wrote:
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
>>> pabl...@google.com> wrote:
>>>
 Yooh! : D

 On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
 yifan...@google.com> wrote:

> Congratulations, Alan!
>
> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
> al...@google.com> wrote:
>
>> Hi,
>>
>> Please join me and the rest of the Beam PMC in welcoming
>> a new
>> committer: Alan Myrvold
>>
>> Alan has been a long time Beam contributor. His
>> contributions made Beam more productive and friendlier [1] 
>> for all
>> contributors with significant improvements to Beam release 
>> process,
>> automation, and infrastructure.
>>
>> In consideration of Alan's contributions, the Beam PMC
>> trusts him
>> with the responsibilities of a Beam committer [2].
>>
>> Thank you, Alan, for your contributions and looking
>> forward to many more!
>>
>> Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>> [2] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>
>
> --
> Cheers,
> Gleb
>

>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>

Re: [ANNOUNCE] New committer: Alan Myrvold

[Reza Rokni]

Woohoo Congratulations!

On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:

> Congratulations, Alan!
>
>
> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía  wrote:
>
>> Congrats Alan!
>>
>> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
>> wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov  wrote:
>>>
 Congratulations!

 On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Alan. Well deserved.
>
> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> Congrats Alan!!
>>
>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský  wrote:
>>
>>> Congrats Alan!
>>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>>
>>> Congratulations Alan!!!
>>>
>>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang  wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
 wrote:

> Congratulations Alan!
>
> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
> wrote:
>
>> Congrats, Alan!
>>
>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
>> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>>> hannahji...@google.com> wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang 
 wrote:

> Congratulations, Alan!
>
>
> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
> wrote:
>
>> Congrats!
>>
>> -Rui
>>
>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
>> pabl...@google.com> wrote:
>>
>>> Yooh! : D
>>>
>>> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
>>> yifan...@google.com> wrote:
>>>
 Congratulations, Alan!

 On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
 al...@google.com> wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in welcoming a
> new
> committer: Alan Myrvold
>
> Alan has been a long time Beam contributor. His
> contributions made Beam more productive and friendlier [1] 
> for all
> contributors with significant improvements to Beam release 
> process,
> automation, and infrastructure.
>
> In consideration of Alan's contributions, the Beam PMC
> trusts him
> with the responsibilities of a Beam committer [2].
>
> Thank you, Alan, for your contributions and looking
> forward to many more!
>
> Ahmet, on behalf of the Apache Beam PMC
>
> [1]
> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
> [2] https://beam.apache.org/contribute/become-a-committer
> /#an-apache-beam-committer
>

>
> --
> 
> Ruoyun  Huang
>
>

 --
 Cheers,
 Gleb

>>>

-- 

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

Re: [ANNOUNCE] New committer: Alan Myrvold

[Thomas Weise]

Congratulations, Alan!


On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía  wrote:

> Congrats Alan!
>
> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov  wrote:
>>
>>> Congratulations!
>>>
>>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congratulations, Alan. Well deserved.

 On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
 chamik...@google.com> wrote:

> Congrats Alan!!
>
> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský  wrote:
>
>> Congrats Alan!
>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>
>> Congratulations Alan!!!
>>
>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang  wrote:
>>
>>> Congrats Alan!
>>>
>>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
>>> wrote:
>>>
 Congratulations Alan!

 On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
 wrote:

> Congrats, Alan!
>
> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
> wrote:
>
>> Congrats, Alan!
>>
>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>> hannahji...@google.com> wrote:
>>
>>> Congrats Alan!
>>>
>>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang 
>>> wrote:
>>>
 Congratulations, Alan!


 On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
 wrote:

> Congrats!
>
> -Rui
>
> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
> pabl...@google.com> wrote:
>
>> Yooh! : D
>>
>> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou <
>> yifan...@google.com> wrote:
>>
>>> Congratulations, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay <
>>> al...@google.com> wrote:
>>>
 Hi,

 Please join me and the rest of the Beam PMC in welcoming a
 new
 committer: Alan Myrvold

 Alan has been a long time Beam contributor. His
 contributions made Beam more productive and friendlier [1] for 
 all
 contributors with significant improvements to Beam release 
 process,
 automation, and infrastructure.

 In consideration of Alan's contributions, the Beam PMC
 trusts him
 with the responsibilities of a Beam committer [2].

 Thank you, Alan, for your contributions and looking forward
 to many more!

 Ahmet, on behalf of the Apache Beam PMC

 [1]
 https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
 [2] https://beam.apache.org/contribute/become-a-committer
 /#an-apache-beam-committer

>>>

 --
 
 Ruoyun  Huang


>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Beam Dependency Check Report (2019-09-30)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
Sphinx
1.8.5
2.2.0
2019-05-20
2019-08-19BEAM-7370
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.25.0
2019-02-11
2019-09-16BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
org.conscrypt:conscrypt-openjdk
1.1.3
2.2.1
2018-06-04
2019-08-08BEAM-5748
org.eclipse.jetty:jetty-server
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5753
Gradle:
5.2.1
5.6.2
2019-08-19
2019-09-09BEAM-8002

 A dependency update is high priority if it satisfies one of following criteria: 

 It has major versions update available, e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0; 


 It is over 3 minor versions behind the latest version, e.g. org.tukaani:xz 1.5 -> 1.8; 


 The current version is behind the later version for over 180 days, e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11. 

 In Beam, we make a best-effort attempt at keeping all dependencies up-to-date.
 In the future, issues will be filed and tracked for these automatically,
 but in the meantime you can search for existing issues or open a new one.

 For more information:  Beam Dependency Guide  

Beam KinesisIO Migration V1 to V2

[Cam Mach]

Hello Beam Dev,

I have discussed with a couple of Beam dev regarding this topic. We found
something interesting in the new AWS Kinesis SDK and Libraries V2, so like
to propose a design for this migration.

Here is the design doc:
https://docs.google.com/document/d/1XeIVbiDHBReZY8rEI2OWA3cTEQuaR7RPdwGAup6S1DM


I would love to hear from you, your feedback and comments

Thanks,
Cam

Re: [ANNOUNCE] New committer: Alan Myrvold

[Ismaël Mejía]

Congrats Alan!

On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli 
wrote:

> Congratulations, Alan!
>
>
> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov  wrote:
>
>> Congratulations!
>>
>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> Congratulations, Alan. Well deserved.
>>>
>>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath 
>>> wrote:
>>>
 Congrats Alan!!

 On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský  wrote:

> Congrats Alan!
> On 9/27/19 10:22 PM, Mark Liu wrote:
>
> Congratulations Alan!!!
>
> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang  wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
>> wrote:
>>
>>> Congratulations Alan!
>>>
>>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
>>> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
 wrote:

> Congrats, Alan!
>
> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Congrats Alan!
>>
>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang 
>> wrote:
>>
>>> Congratulations, Alan!
>>>
>>>
>>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
>>> wrote:
>>>
 Congrats!

 -Rui

 On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
 pabl...@google.com> wrote:

> Yooh! : D
>
> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou 
> wrote:
>
>> Congratulations, Alan!
>>
>> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay 
>> wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new
>>> committer: Alan Myrvold
>>>
>>> Alan has been a long time Beam contributor. His
>>> contributions made Beam more productive and friendlier [1] for 
>>> all
>>> contributors with significant improvements to Beam release 
>>> process,
>>> automation, and infrastructure.
>>>
>>> In consideration of Alan's contributions, the Beam PMC
>>> trusts him
>>> with the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Alan, for your contributions and looking forward
>>> to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>>> [2] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>

Re: [ANNOUNCE] New committer: Alan Myrvold

[Tanay Tummalapalli]

Congratulations, Alan!


On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov  wrote:

> Congratulations!
>
> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev 
> wrote:
>
>> Congratulations, Alan. Well deserved.
>>
>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath 
>> wrote:
>>
>>> Congrats Alan!!
>>>
>>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský  wrote:
>>>
 Congrats Alan!
 On 9/27/19 10:22 PM, Mark Liu wrote:

 Congratulations Alan!!!

 On Fri, Sep 27, 2019 at 12:55 PM Ning Kang  wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
> wrote:
>
>> Congratulations Alan!
>>
>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
>> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
>>> wrote:
>>>
 Congrats, Alan!

 On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
 hannahji...@google.com> wrote:

> Congrats Alan!
>
> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang 
> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
>> wrote:
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada <
>>> pabl...@google.com> wrote:
>>>
 Yooh! : D

 On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou 
 wrote:

> Congratulations, Alan!
>
> On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay 
> wrote:
>
>> Hi,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new
>> committer: Alan Myrvold
>>
>> Alan has been a long time Beam contributor. His contributions
>> made Beam more productive and friendlier [1] for all 
>> contributors with
>> significant improvements to Beam release process, automation, and
>> infrastructure.
>>
>> In consideration of Alan's contributions, the Beam PMC trusts
>> him
>> with the responsibilities of a Beam committer [2].
>>
>> Thank you, Alan, for your contributions and looking forward
>> to many more!
>>
>> Ahmet, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
>> [2] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>
>
> --
> Cheers,
> Gleb
>

Re: Multiple iterations after GroupByKey with SparkRunner

[Jan Lukavský]

> The fact that the annotation on the ParDo "changes" the GroupByKey 
implementation is very specific to the Spark runner implementation.


I don't quite agree. It is not very specific to Spark, it is specific to 
generally all runners, that produce grouped elements in a way that is 
not reiterable. That is the key property. The example you gave with HDFS 
does not satisfy this condition (files on HDFS are certainly 
reiterable), and that's why no change to the GBK is needed (it actually 
already has the required property). A quick look at what FlinkRunner (at 
least non portable does) is that it implements GBK using reducing 
elements into List. That is going to crash on big PCollection, which is 
even nicely documented:


* For internal use to translate {@link GroupByKey}. For a large {@link 
PCollection} this is
* expected to crash!

If this is fixed, then it is likely to start behave the same as Spark. 
So actually I think the opposite is true - Dataflow is a special case, 
because of how its internal shuffle service works.


 In general I sympathize with the worry about non-local effects. Beam 
is already full of them (e.g. a Window.into statement effects downstream 
GroupByKeys). In each case where they were added there was extensive 
debate and discussion (Windowing semantics were debated for many 
months), exactly because there was concern over adding these non-local 
effects. In every case, no other good solution could be found. For the 
case of windowing for example, it was often easy to propose simple local 
APIs (e.g. just pass the window fn as a parameter to GroupByKey), 
however all of these local solutions ended up not working for important 
use cases when we analyzed them more deeply.


That is very interesting. Could you elaborate more about some examples 
of the use cases which didn't work? I'd like to try to match it against 
how Euphoria is structured, it should be more resistant to this 
non-local effects, because it very often bundles together multiple 
Beam's primitives to single transform - ReduceByKey is one example of 
this, if is actually mix of Window.into() + GBK + ParDo, Although it 
might look like if this transform can be broken down to something else, 
then it is not primitive (euphoria has no native equivalent of GBK 
itself), but it has several other nice implications - that is that 
Combine now becomes a special case of RBK. It now becomes only a 
question of where and how you can "run" the reduce function. The logic 
is absolutely equal. This can be worked in more detail and actually 
show, that even Combine and RBK can be decribed by a more general 
stateful operation (ReduceStateByKey), and so finally Euphoria actually 
has only two really "primitive" operations - these are FlatMap 
(basically stateless ParDo) and RSBK. As I already mentioned on some 
other thread, when stateful ParDo would support merging windows, it can 
be shown that both Combine and GBK become special cases of this.


 As you mentioned below, I do think it's perfectly reasonable for a DSL 
to impose its own semantics. Scio already does this - the raw Beam API 
is used by a DSL as a substrate, but the DSL does not need to blindly 
mirror the semantics of the raw Beam API - at least in my opinion!


Sure, but currently, there is no way for DSL to "hook" into runner, so 
it has to use raw Beam SDK, and so this will fail in cases like this - 
where Beam actually has stronger guarantees than it is required by the 
DSL. It would be cool if we could find a way to do that - this pretty 
much aligns with another question raised on ML, about the possibility to 
override a default implementation of a PTransform for specific pipeline.


Jan


On 9/29/19 7:46 PM, Reuven Lax wrote:

Jan,

The fact that the annotation on the ParDo "changes" the GroupByKey 
implementation is very specific to the Spark runner implementation. 
You can imagine another runner that simply writes out files in HDFS to 
implement a GroupByKey - this GroupByKey implementation is agnostic 
whether the result will be reiterated or not; in this case it is very 
much the ParDo implementation that changes to implement a reiterable. 
vI think you don't like the fact that an annotation on the ParDo will 
have a non-local effect on the implementation of the GroupByKey 
upstream. However arguably the non-local effect is just a quirk of how 
the Spark runner is implemented - other runners might have a local effect.


In general I sympathize with the worry about non-local effects. Beam 
is already full of them (e.g. a Window.into statement effects 
downstream GroupByKeys). In each case where they were added there was 
extensive debate and discussion (Windowing semantics were debated for 
many months), exactly because there was concern over adding these 
non-local effects. In every case, no other good solution could be 
found. For the case of windowing for example, it was often easy to 
propose simple local APIs (e.g. just pass the window fn 

Re: [ANNOUNCE] New committer: Alan Myrvold

[Gleb Kanterov]

Congratulations!

On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev 
wrote:

> Congratulations, Alan. Well deserved.
>
> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath 
> wrote:
>
>> Congrats Alan!!
>>
>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský  wrote:
>>
>>> Congrats Alan!
>>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>>
>>> Congratulations Alan!!!
>>>
>>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang  wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka 
 wrote:

> Congratulations Alan!
>
> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang 
> wrote:
>
>> Congrats, Alan!
>>
>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu 
>> wrote:
>>
>>> Congrats, Alan!
>>>
>>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>>> hannahji...@google.com> wrote:
>>>
 Congrats Alan!

 On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang 
 wrote:

> Congratulations, Alan!
>
>
> On Fri, Sep 27, 2019 at 9:55 AM Rui Wang 
> wrote:
>
>> Congrats!
>>
>> -Rui
>>
>> On Fri, Sep 27, 2019 at 9:54 AM Pablo Estrada 
>> wrote:
>>
>>> Yooh! : D
>>>
>>> On Fri, Sep 27, 2019 at 9:53 AM Yifan Zou 
>>> wrote:
>>>
 Congratulations, Alan!

 On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay 
 wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Alan Myrvold
>
> Alan has been a long time Beam contributor. His contributions
> made Beam more productive and friendlier [1] for all contributors 
> with
> significant improvements to Beam release process, automation, and
> infrastructure.
>
> In consideration of Alan's contributions, the Beam PMC trusts
> him
> with the responsibilities of a Beam committer [2].
>
> Thank you, Alan, for your contributions and looking forward to
> many more!
>
> Ahmet, on behalf of the Apache Beam PMC
>
> [1]
> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
> [2] https://beam.apache.org/contribute/become-a-committer
> /#an-apache-beam-committer
>

>
> --
> 
> Ruoyun  Huang
>
>

-- 
Cheers,
Gleb