Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Valentyn Tymofieiev]

Thanks everyone for sharing your perspectives so far. It sounds like we can
mitigate the cost of test infrastructure by having:
- a selection of (fast) tests that we will want to run against all Python
versions we support.
- high priority Python versions, which we will test extensively.
- infrequent postcommit test that exercise low-priority versions.
We will need test infrastructure improvements to have the flexibility of
designating versions of high-pri/low-pri and minimizing efforts requiring
adopting a new version.

There is still a question of how long we want to support old Py3.x
versions. As mentioned above, I think we should not support them beyond EOL
(5 years after a release). I wonder if that is still too long. The cost of
supporting a version may include:
 - Developing against older Python version
 - Release overhead (building & storing containers, wheels, doing release
validation)
 - Complexity / development cost to support the quirks of the minor
versions.

We can decide to drop support, after, say, 4 years, or after usage drops
below a threshold, or decide on a case-by-case basis. Thoughts? Also asked
for feedback on user@ [1]

[1]
https://lists.apache.org/thread.html/r630a3b55aa8e75c68c8252ea6f824c3ab231ad56e18d916dfb84d9e8%40%3Cuser.beam.apache.org%3E

On Wed, Feb 26, 2020 at 5:27 PM Robert Bradshaw  wrote:

> On Wed, Feb 26, 2020 at 5:21 PM Valentyn Tymofieiev 
> wrote:
> >
> > > +1 to consulting users.
> > I will message user@ as well and point to this thread.
> >
> > > I would propose getting in warnings about 3.5 EoL well ahead of time.
> > I think we should document on our website, and  in the code (warnings)
> that users should not expect SDKs to be supported in Beam beyond the EOL.
> If we want to have flexibility to drop support earlier than EOL, we need to
> be more careful with messaging because users might otherwise expect that
> support will last until EOL, if we mention EOL date.
>
> +1
>
> > I am hoping that we can establish a consensus for when we will be
> dropping support for a version, so that we don't have to discuss it on a
> case by case basis in the future.
> >
> > > I think it would makes sense to add support for 3.8 right away (or at
> least get a good sense of what work needs to be done and what our
> dependency situation is like)
> > https://issues.apache.org/jira/browse/BEAM-8494 is a starting point. I
> tried 3.8 a while ago some dependencies were not able to install, checked
> again just now. SDK is "installable" after minor changes. Some tests don't
> pass. BEAM-8494 does not have an owner atm, and if anyone is interested I'm
> happy to give further pointers and help get started.
> >
> > > For the 3.x series, I think we will get the most signal out of the
> lowest and highest version, and can get by with smoke tests +
> > infrequent post-commits for the ones between.
> >
> > > I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.
> >
> > These are good ideas. Do you think we will want to have an ability  to
> run some (inexpensive) tests for all versions  frequently (on presubmits),
> or this is extra complexity that can be avoided? I am thinking about type
> inference for example. Afaik inference logic is very sensitive to the
> version. Would it be acceptable to catch  errors there in infrequent
> postcommits or an early signal will be preferred?
>
> This is a good example--the type inference tests are sensitive to
> version (due to using internal details and relying on the
> still-evolving typing module) but also run in ~15 seconds. I think
> these should be in precommits. We just don't need to run every test
> for every version.
>
> > On Wed, Feb 26, 2020 at 5:17 PM Kyle Weaver  wrote:
> >>
> >> Oh, I didn't see Robert's earlier email:
> >>
> >> > Currently 3.5 downloads sit at 3.7%, or about
> >> > 20% of all Python 3 downloads.
> >>
> >> Where did these numbers come from?
> >>
> >> On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver 
> wrote:
> >>>
> >>> > I agree with having low-frequency tests for low-priority versions.
> >>> > Low-priority versions could be determined according to least usage.
> >>>
> >>> +1. While the difference may not be as great between, say, 3.6 and
> 3.7, I think that if we had to choose, it would be more useful to test the
> versions folks are actually using the most. 3.5 only has about a third of
> the Docker pulls of 3.6 or 3.7 [1]. Does anyone have other usage statistics
> we can consult?
> >>>
> >>> [1] https://hub.docker.com/search?q=apachebeam%2Fpython=image
> >>>
> >>> On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang 
> wrote:
> 
>  I feel 4+ versions take too long to run anything.
> 
>  would vote for lowest + highest,  2 versions.
> 
>  On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
> >
> > I agree with having low-frequency tests for low-priority versions.
> > Low-priority versions could be determined 

Re: Jenkins problems: javaPreCommitPortabilityApiJava11 and No Space left

[Kyle Weaver]

It looks like apache-beam-jenkins-5 is back to normal now:
https://builds.apache.org/computer/apache-beam-jenkins-5/builds

> Java PreCommit also seems to be failing due to a couple of errors in
'BigQureyIO" and "SpannerIO".

Judging by the log output, the job failed because of a timeout, possibly
(probably?) related to Rabbit mq test:

*10:52:08* >* Task
:sdks:java:testing:test-utils:buildDependents**12:11:39* Build timed
out (after 120 minutes). Marking the build as aborted.*12:11:39* Build
was aborted*12:11:39* Recording test results*12:11:41* >* Task
:sdks:java:io:rabbitmq:test* FAILED*12:11:41* *12:11:41* FAILURE:
Build failed with an exception.



On Wed, Feb 26, 2020 at 5:22 PM Pulasthi Supun Wickramasinghe <
pulasthi...@gmail.com> wrote:

> I got the same build issues for my pull request as well. Java PreCommit
> also seems to be failing due to a couple of errors in 'BigQureyIO" and
> "SpannerIO".
>
> [1]
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/10160/java/fixed/
>
> Best Regards,
> Pulasthi
>
> On Wed, Feb 26, 2020 at 1:27 PM Alex Van Boxel  wrote:
>
>> I see 2 problems with Jenkins popping up. The root beam project is
>> missing:
>>
>>
>> JavaPortabilityApiJava11 ("Run JavaPortabilityApiJava11 PreCommit")  is
>> running, but:
>>
>> Task 'javaPreCommitPortabilityApiJava11' not found in root project 'beam'
>>
>> And we have no space left on device:
>>
>> Example:
>>
>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/9287/console
>>
>>
>>
>>  _/
>> _/ Alex Van Boxel
>>
>
>
> --
> Pulasthi S. Wickramasinghe
> PhD Candidate  | Research Assistant
> School of Informatics and Computing | Digital Science Center
> Indiana University, Bloomington
> cell: 224-386-9035 <(224)%20386-9035>
>

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Robert Bradshaw]

On Wed, Feb 26, 2020 at 5:21 PM Valentyn Tymofieiev  wrote:
>
> > +1 to consulting users.
> I will message user@ as well and point to this thread.
>
> > I would propose getting in warnings about 3.5 EoL well ahead of time.
> I think we should document on our website, and  in the code (warnings) that 
> users should not expect SDKs to be supported in Beam beyond the EOL. If we 
> want to have flexibility to drop support earlier than EOL, we need to be more 
> careful with messaging because users might otherwise expect that support will 
> last until EOL, if we mention EOL date.

+1

> I am hoping that we can establish a consensus for when we will be dropping 
> support for a version, so that we don't have to discuss it on a case by case 
> basis in the future.
>
> > I think it would makes sense to add support for 3.8 right away (or at least 
> > get a good sense of what work needs to be done and what our dependency 
> > situation is like)
> https://issues.apache.org/jira/browse/BEAM-8494 is a starting point. I tried 
> 3.8 a while ago some dependencies were not able to install, checked again 
> just now. SDK is "installable" after minor changes. Some tests don't pass. 
> BEAM-8494 does not have an owner atm, and if anyone is interested I'm happy 
> to give further pointers and help get started.
>
> > For the 3.x series, I think we will get the most signal out of the lowest 
> > and highest version, and can get by with smoke tests +
> infrequent post-commits for the ones between.
>
> > I agree with having low-frequency tests for low-priority versions. 
> > Low-priority versions could be determined according to least usage.
>
> These are good ideas. Do you think we will want to have an ability  to run 
> some (inexpensive) tests for all versions  frequently (on presubmits), or 
> this is extra complexity that can be avoided? I am thinking about type 
> inference for example. Afaik inference logic is very sensitive to the 
> version. Would it be acceptable to catch  errors there in infrequent 
> postcommits or an early signal will be preferred?

This is a good example--the type inference tests are sensitive to
version (due to using internal details and relying on the
still-evolving typing module) but also run in ~15 seconds. I think
these should be in precommits. We just don't need to run every test
for every version.

> On Wed, Feb 26, 2020 at 5:17 PM Kyle Weaver  wrote:
>>
>> Oh, I didn't see Robert's earlier email:
>>
>> > Currently 3.5 downloads sit at 3.7%, or about
>> > 20% of all Python 3 downloads.
>>
>> Where did these numbers come from?
>>
>> On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver  wrote:
>>>
>>> > I agree with having low-frequency tests for low-priority versions.
>>> > Low-priority versions could be determined according to least usage.
>>>
>>> +1. While the difference may not be as great between, say, 3.6 and 3.7, I 
>>> think that if we had to choose, it would be more useful to test the 
>>> versions folks are actually using the most. 3.5 only has about a third of 
>>> the Docker pulls of 3.6 or 3.7 [1]. Does anyone have other usage statistics 
>>> we can consult?
>>>
>>> [1] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>>>
>>> On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang  wrote:

 I feel 4+ versions take too long to run anything.

 would vote for lowest + highest,  2 versions.

 On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
>
> I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.
>
>
>
> On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw  
> wrote:
>>
>> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
>> >
>> > Are these divergent enough that they all need to consume testing 
>> > resources? For example can lower priority versions be daily runs or 
>> > some such?
>>
>> For the 3.x series, I think we will get the most signal out of the
>> lowest and highest version, and can get by with smoke tests +
>> infrequent post-commits for the ones between.
>>
>> > Kenn
>> >
>> > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw  
>> > wrote:
>> >>
>> >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
>> >> 20% of all Python 3 downloads.
>> >>
>> >> I would propose getting in warnings about 3.5 EoL well ahead of time,
>> >> at the very least as part of the 2.7 warning.
>> >>
>> >> Fortunately, supporting multiple 3.x versions is significantly easier
>> >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
>> >> dropping 3.5 and adding 3.8 but consider their merits independently.
>> >>
>> >>
>> >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver  
>> >> wrote:
>> >> >
>> >> > 5 versions is too many IMO. We've had issues with Python precommit 
>> >> > resource usage in the past, and 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Robert Bradshaw]

On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver  wrote:
>
> > I agree with having low-frequency tests for low-priority versions.
> > Low-priority versions could be determined according to least usage.
>
> +1. While the difference may not be as great between, say, 3.6 and 3.7, I 
> think that if we had to choose, it would be more useful to test the versions 
> folks are actually using the most. 3.5 only has about a third of the Docker 
> pulls of 3.6 or 3.7 [1]. Does anyone have other usage statistics we can 
> consult?

While usage is important, we should also prioritize testing resources
according to the signal we get from them. If we only tested 3.6 and
3.7 it's very easy to break 3.5 due to accidentally using new features
introduced in 3.6. (It's happened to me, fortunately we had tests :).
On the other hand, if something works in  both 3.5 and 3.7, I would be
quite surprised if it does not work in 3.6.

Pypi download stats from https://pypistats.org/packages/apache-beam

> [1] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>
> On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang  wrote:
>>
>> I feel 4+ versions take too long to run anything.
>>
>> would vote for lowest + highest,  2 versions.
>>
>> On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
>>>
>>> I agree with having low-frequency tests for low-priority versions.
>>> Low-priority versions could be determined according to least usage.
>>>
>>>
>>>
>>> On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw  wrote:

 On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
 >
 > Are these divergent enough that they all need to consume testing 
 > resources? For example can lower priority versions be daily runs or some 
 > such?

 For the 3.x series, I think we will get the most signal out of the
 lowest and highest version, and can get by with smoke tests +
 infrequent post-commits for the ones between.

 > Kenn
 >
 > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw  
 > wrote:
 >>
 >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
 >> 20% of all Python 3 downloads.
 >>
 >> I would propose getting in warnings about 3.5 EoL well ahead of time,
 >> at the very least as part of the 2.7 warning.
 >>
 >> Fortunately, supporting multiple 3.x versions is significantly easier
 >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
 >> dropping 3.5 and adding 3.8 but consider their merits independently.
 >>
 >>
 >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver  wrote:
 >> >
 >> > 5 versions is too many IMO. We've had issues with Python precommit 
 >> > resource usage in the past, and adding another version would surely 
 >> > exacerbate those issues. And we have also already had to leave out 
 >> > certain features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 
 >> > before adding 3.8. After dropping Python 2 and adding 3.8, that will 
 >> > leave us with the latest three minor versions (3.6, 3.7, 3.8), which 
 >> > I think is closer to the "sweet spot." Though I would be interested 
 >> > in hearing if there are any users who would prefer we continue 
 >> > supporting 3.5.
 >> >
 >> > [1] 
 >> > https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
 >> >
 >> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev 
 >> >  wrote:
 >> >>
 >> >> I would like to start a discussion about identifying a guideline for 
 >> >> answering questions like:
 >> >>
 >> >> 1. When will Beam support a new Python version (say, Python 3.8)?
 >> >> 2. When will Beam drop support for an old Python version (say, 
 >> >> Python 3.5)?
 >> >> 3. How many Python versions should we aim to support concurrently 
 >> >> (investigate issues, have continuous integration tests)?
 >> >> 4. What comes first: adding support for a new version (3.8) or 
 >> >> deprecating older one (3.5)? This may affect the max load our test 
 >> >> infrastructure needs to sustain.
 >> >>
 >> >> We are already getting requests for supporting Python 3.8 and there 
 >> >> were some good reasons[1] to drop support for Python 3.5 (at least, 
 >> >> early versions of 3.5). Answering these questions would help set 
 >> >> expectations in Beam user community, Beam dev community, and  may 
 >> >> help us establish resource requirements for test infrastructure and 
 >> >> plan efforts.
 >> >>
 >> >> PEP-0602 [2] establishes a yearly release cycle for Python versions 
 >> >> starting from 3.9. Each release is a long-term support release and 
 >> >> is supported for 5 years: first 1.5 years allow for general bug fix 
 >> >> support, remaining 3.5 years have security fix support.
 >> >>
 >> >> At every point, there may be up to 5 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Valentyn Tymofieiev]

> +1 to consulting users.
I will message user@ as well and point to this thread.

> I would propose getting in warnings about 3.5 EoL well ahead of time.
I think we should document on our website, and  in the code (warnings) that
users should not expect SDKs to be supported in Beam beyond the EOL. If we
want to have flexibility to drop support earlier than EOL, we need to be
more careful with messaging because users might otherwise expect that
support will last until EOL, if we mention EOL date.

I am hoping that we can establish a consensus for when we will be dropping
support for a version, so that we don't have to discuss it on a case by
case basis in the future.

> I think it would makes sense to add support for 3.8 right away (or at
least get a good sense of what work needs to be done and what our
dependency situation is like)
https://issues.apache.org/jira/browse/BEAM-8494 is a starting point. I
tried 3.8 a while ago some dependencies were not able to install, checked
again just now. SDK is "installable" after minor changes. Some tests don't
pass. BEAM-8494 does not have an owner atm, and if anyone is interested I'm
happy to give further pointers and help get started.

> For the 3.x series, I think we will get the most signal out of the lowest
and highest version, and can get by with smoke tests +
infrequent post-commits for the ones between.

> I agree with having low-frequency tests for low-priority versions.
Low-priority versions could be determined according to least usage.

These are good ideas. Do you think we will want to have an ability  to run
some (inexpensive) tests for all versions  frequently (on presubmits), or
this is extra complexity that can be avoided? I am thinking about type
inference for example. Afaik inference logic is very sensitive to the
version. Would it be acceptable to catch  errors there in infrequent
postcommits or an early signal will be preferred?

On Wed, Feb 26, 2020 at 5:17 PM Kyle Weaver  wrote:

> Oh, I didn't see Robert's earlier email:
>
> > Currently 3.5 downloads sit at 3.7%, or about
> > 20% of all Python 3 downloads.
>
> Where did these numbers come from?
>
> On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver  wrote:
>
>> > I agree with having low-frequency tests for low-priority versions.
>> > Low-priority versions could be determined according to least usage.
>>
>> +1. While the difference may not be as great between, say, 3.6 and 3.7, I
>> think that if we had to choose, it would be more useful to test the
>> versions folks are actually using the most. 3.5 only has about a third
>> of the Docker pulls of 3.6 or 3.7 [1]. Does anyone have other usage
>> statistics we can consult?
>>
>> [1] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>>
>> On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang  wrote:
>>
>>> I feel 4+ versions take too long to run anything.
>>>
>>> would vote for lowest + highest,  2 versions.
>>>
>>> On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
>>>
 I agree with having low-frequency tests for low-priority versions.
 Low-priority versions could be determined according to least usage.



 On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw 
 wrote:

> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles 
> wrote:
> >
> > Are these divergent enough that they all need to consume testing
> resources? For example can lower priority versions be daily runs or some
> such?
>
> For the 3.x series, I think we will get the most signal out of the
> lowest and highest version, and can get by with smoke tests +
> infrequent post-commits for the ones between.
>
> > Kenn
> >
> > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw 
> wrote:
> >>
> >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or
> about
> >> 20% of all Python 3 downloads.
> >>
> >> I would propose getting in warnings about 3.5 EoL well ahead of
> time,
> >> at the very least as part of the 2.7 warning.
> >>
> >> Fortunately, supporting multiple 3.x versions is significantly
> easier
> >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
> >> dropping 3.5 and adding 3.8 but consider their merits independently.
> >>
> >>
> >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver 
> wrote:
> >> >
> >> > 5 versions is too many IMO. We've had issues with Python
> precommit resource usage in the past, and adding another version would
> surely exacerbate those issues. And we have also already had to leave out
> certain features on 3.5 [1]. Therefore, I am in favor of dropping 3.5
> before adding 3.8. After dropping Python 2 and adding 3.8, that will leave
> us with the latest three minor versions (3.6, 3.7, 3.8), which I think is
> closer to the "sweet spot." Though I would be interested in hearing if
> there are any users who would prefer we continue supporting 3.5.
> >> >
> >> > 

Re: Jenkins problems: javaPreCommitPortabilityApiJava11 and No Space left

[Pulasthi Supun Wickramasinghe]

I got the same build issues for my pull request as well. Java PreCommit
also seems to be failing due to a couple of errors in 'BigQureyIO" and
"SpannerIO".

[1]
https://builds.apache.org/job/beam_PreCommit_Java_Commit/10160/java/fixed/

Best Regards,
Pulasthi

On Wed, Feb 26, 2020 at 1:27 PM Alex Van Boxel  wrote:

> I see 2 problems with Jenkins popping up. The root beam project is
> missing:
>
>
> JavaPortabilityApiJava11 ("Run JavaPortabilityApiJava11 PreCommit")  is
> running, but:
>
> Task 'javaPreCommitPortabilityApiJava11' not found in root project 'beam'
>
> And we have no space left on device:
>
> Example:
>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/9287/console
>
>
>
>  _/
> _/ Alex Van Boxel
>


-- 
Pulasthi S. Wickramasinghe
PhD Candidate  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Kyle Weaver]

Oh, I didn't see Robert's earlier email:

> Currently 3.5 downloads sit at 3.7%, or about
> 20% of all Python 3 downloads.

Where did these numbers come from?

On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver  wrote:

> > I agree with having low-frequency tests for low-priority versions.
> > Low-priority versions could be determined according to least usage.
>
> +1. While the difference may not be as great between, say, 3.6 and 3.7, I
> think that if we had to choose, it would be more useful to test the
> versions folks are actually using the most. 3.5 only has about a third
> of the Docker pulls of 3.6 or 3.7 [1]. Does anyone have other usage
> statistics we can consult?
>
> [1] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>
> On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang  wrote:
>
>> I feel 4+ versions take too long to run anything.
>>
>> would vote for lowest + highest,  2 versions.
>>
>> On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
>>
>>> I agree with having low-frequency tests for low-priority versions.
>>> Low-priority versions could be determined according to least usage.
>>>
>>>
>>>
>>> On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw 
>>> wrote:
>>>
 On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles 
 wrote:
 >
 > Are these divergent enough that they all need to consume testing
 resources? For example can lower priority versions be daily runs or some
 such?

 For the 3.x series, I think we will get the most signal out of the
 lowest and highest version, and can get by with smoke tests +
 infrequent post-commits for the ones between.

 > Kenn
 >
 > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw 
 wrote:
 >>
 >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
 >> 20% of all Python 3 downloads.
 >>
 >> I would propose getting in warnings about 3.5 EoL well ahead of time,
 >> at the very least as part of the 2.7 warning.
 >>
 >> Fortunately, supporting multiple 3.x versions is significantly easier
 >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
 >> dropping 3.5 and adding 3.8 but consider their merits independently.
 >>
 >>
 >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver 
 wrote:
 >> >
 >> > 5 versions is too many IMO. We've had issues with Python precommit
 resource usage in the past, and adding another version would surely
 exacerbate those issues. And we have also already had to leave out certain
 features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before adding
 3.8. After dropping Python 2 and adding 3.8, that will leave us with the
 latest three minor versions (3.6, 3.7, 3.8), which I think is closer to the
 "sweet spot." Though I would be interested in hearing if there are any
 users who would prefer we continue supporting 3.5.
 >> >
 >> > [1]
 https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
 >> >
 >> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 >> >>
 >> >> I would like to start a discussion about identifying a guideline
 for answering questions like:
 >> >>
 >> >> 1. When will Beam support a new Python version (say, Python 3.8)?
 >> >> 2. When will Beam drop support for an old Python version (say,
 Python 3.5)?
 >> >> 3. How many Python versions should we aim to support concurrently
 (investigate issues, have continuous integration tests)?
 >> >> 4. What comes first: adding support for a new version (3.8) or
 deprecating older one (3.5)? This may affect the max load our test
 infrastructure needs to sustain.
 >> >>
 >> >> We are already getting requests for supporting Python 3.8 and
 there were some good reasons[1] to drop support for Python 3.5 (at least,
 early versions of 3.5). Answering these questions would help set
 expectations in Beam user community, Beam dev community, and  may help us
 establish resource requirements for test infrastructure and plan efforts.
 >> >>
 >> >> PEP-0602 [2] establishes a yearly release cycle for Python
 versions starting from 3.9. Each release is a long-term support release and
 is supported for 5 years: first 1.5 years allow for general bug fix
 support, remaining 3.5 years have security fix support.
 >> >>
 >> >> At every point, there may be up to 5 Python minor versions that
 did not yet reach EOL, see "Release overlap with 12 month diagram" [3]. We
 can try to support all of them, but that may come at a cost of velocity: we
 will have more tests to maintain, and we will have to develop Beam against
 a lower version for a longer period. Supporting less versions will have
 implications for user experience. It also may be difficult to ensure
 support of the most 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Kyle Weaver]

> I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.

+1. While the difference may not be as great between, say, 3.6 and 3.7, I
think that if we had to choose, it would be more useful to test the
versions folks are actually using the most. 3.5 only has about a third
of the Docker pulls of 3.6 or 3.7 [1]. Does anyone have other usage
statistics we can consult?

[1] https://hub.docker.com/search?q=apachebeam%2Fpython=image

On Wed, Feb 26, 2020 at 5:00 PM Ruoyun Huang  wrote:

> I feel 4+ versions take too long to run anything.
>
> would vote for lowest + highest,  2 versions.
>
> On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:
>
>> I agree with having low-frequency tests for low-priority versions.
>> Low-priority versions could be determined according to least usage.
>>
>>
>>
>> On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw 
>> wrote:
>>
>>> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
>>> >
>>> > Are these divergent enough that they all need to consume testing
>>> resources? For example can lower priority versions be daily runs or some
>>> such?
>>>
>>> For the 3.x series, I think we will get the most signal out of the
>>> lowest and highest version, and can get by with smoke tests +
>>> infrequent post-commits for the ones between.
>>>
>>> > Kenn
>>> >
>>> > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw 
>>> wrote:
>>> >>
>>> >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
>>> >> 20% of all Python 3 downloads.
>>> >>
>>> >> I would propose getting in warnings about 3.5 EoL well ahead of time,
>>> >> at the very least as part of the 2.7 warning.
>>> >>
>>> >> Fortunately, supporting multiple 3.x versions is significantly easier
>>> >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
>>> >> dropping 3.5 and adding 3.8 but consider their merits independently.
>>> >>
>>> >>
>>> >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver 
>>> wrote:
>>> >> >
>>> >> > 5 versions is too many IMO. We've had issues with Python precommit
>>> resource usage in the past, and adding another version would surely
>>> exacerbate those issues. And we have also already had to leave out certain
>>> features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before adding
>>> 3.8. After dropping Python 2 and adding 3.8, that will leave us with the
>>> latest three minor versions (3.6, 3.7, 3.8), which I think is closer to the
>>> "sweet spot." Though I would be interested in hearing if there are any
>>> users who would prefer we continue supporting 3.5.
>>> >> >
>>> >> > [1]
>>> https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
>>> >> >
>>> >> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>> >> >>
>>> >> >> I would like to start a discussion about identifying a guideline
>>> for answering questions like:
>>> >> >>
>>> >> >> 1. When will Beam support a new Python version (say, Python 3.8)?
>>> >> >> 2. When will Beam drop support for an old Python version (say,
>>> Python 3.5)?
>>> >> >> 3. How many Python versions should we aim to support concurrently
>>> (investigate issues, have continuous integration tests)?
>>> >> >> 4. What comes first: adding support for a new version (3.8) or
>>> deprecating older one (3.5)? This may affect the max load our test
>>> infrastructure needs to sustain.
>>> >> >>
>>> >> >> We are already getting requests for supporting Python 3.8 and
>>> there were some good reasons[1] to drop support for Python 3.5 (at least,
>>> early versions of 3.5). Answering these questions would help set
>>> expectations in Beam user community, Beam dev community, and  may help us
>>> establish resource requirements for test infrastructure and plan efforts.
>>> >> >>
>>> >> >> PEP-0602 [2] establishes a yearly release cycle for Python
>>> versions starting from 3.9. Each release is a long-term support release and
>>> is supported for 5 years: first 1.5 years allow for general bug fix
>>> support, remaining 3.5 years have security fix support.
>>> >> >>
>>> >> >> At every point, there may be up to 5 Python minor versions that
>>> did not yet reach EOL, see "Release overlap with 12 month diagram" [3]. We
>>> can try to support all of them, but that may come at a cost of velocity: we
>>> will have more tests to maintain, and we will have to develop Beam against
>>> a lower version for a longer period. Supporting less versions will have
>>> implications for user experience. It also may be difficult to ensure
>>> support of the most recent version early, since our  dependencies (e.g.
>>> picklers) may not be supporting them yet.
>>> >> >>
>>> >> >> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>>> >> >>
>>> >> >> Is 4 versions a sweet spot? Too much? Too little? What do you
>>> think?
>>> >> >>
>>> >> >> [1]
>>> 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Ruoyun Huang]

I feel 4+ versions take too long to run anything.

would vote for lowest + highest,  2 versions.

On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri  wrote:

> I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.
>
>
>
> On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw 
> wrote:
>
>> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
>> >
>> > Are these divergent enough that they all need to consume testing
>> resources? For example can lower priority versions be daily runs or some
>> such?
>>
>> For the 3.x series, I think we will get the most signal out of the
>> lowest and highest version, and can get by with smoke tests +
>> infrequent post-commits for the ones between.
>>
>> > Kenn
>> >
>> > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw 
>> wrote:
>> >>
>> >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
>> >> 20% of all Python 3 downloads.
>> >>
>> >> I would propose getting in warnings about 3.5 EoL well ahead of time,
>> >> at the very least as part of the 2.7 warning.
>> >>
>> >> Fortunately, supporting multiple 3.x versions is significantly easier
>> >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
>> >> dropping 3.5 and adding 3.8 but consider their merits independently.
>> >>
>> >>
>> >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver 
>> wrote:
>> >> >
>> >> > 5 versions is too many IMO. We've had issues with Python precommit
>> resource usage in the past, and adding another version would surely
>> exacerbate those issues. And we have also already had to leave out certain
>> features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before adding
>> 3.8. After dropping Python 2 and adding 3.8, that will leave us with the
>> latest three minor versions (3.6, 3.7, 3.8), which I think is closer to the
>> "sweet spot." Though I would be interested in hearing if there are any
>> users who would prefer we continue supporting 3.5.
>> >> >
>> >> > [1]
>> https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
>> >> >
>> >> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> >> >>
>> >> >> I would like to start a discussion about identifying a guideline
>> for answering questions like:
>> >> >>
>> >> >> 1. When will Beam support a new Python version (say, Python 3.8)?
>> >> >> 2. When will Beam drop support for an old Python version (say,
>> Python 3.5)?
>> >> >> 3. How many Python versions should we aim to support concurrently
>> (investigate issues, have continuous integration tests)?
>> >> >> 4. What comes first: adding support for a new version (3.8) or
>> deprecating older one (3.5)? This may affect the max load our test
>> infrastructure needs to sustain.
>> >> >>
>> >> >> We are already getting requests for supporting Python 3.8 and there
>> were some good reasons[1] to drop support for Python 3.5 (at least, early
>> versions of 3.5). Answering these questions would help set expectations in
>> Beam user community, Beam dev community, and  may help us establish
>> resource requirements for test infrastructure and plan efforts.
>> >> >>
>> >> >> PEP-0602 [2] establishes a yearly release cycle for Python versions
>> starting from 3.9. Each release is a long-term support release and is
>> supported for 5 years: first 1.5 years allow for general bug fix support,
>> remaining 3.5 years have security fix support.
>> >> >>
>> >> >> At every point, there may be up to 5 Python minor versions that did
>> not yet reach EOL, see "Release overlap with 12 month diagram" [3]. We can
>> try to support all of them, but that may come at a cost of velocity: we
>> will have more tests to maintain, and we will have to develop Beam against
>> a lower version for a longer period. Supporting less versions will have
>> implications for user experience. It also may be difficult to ensure
>> support of the most recent version early, since our  dependencies (e.g.
>> picklers) may not be supporting them yet.
>> >> >>
>> >> >> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>> >> >>
>> >> >> Is 4 versions a sweet spot? Too much? Too little? What do you think?
>> >> >>
>> >> >> [1]
>> https://github.com/apache/beam/pull/10821#issuecomment-590167711
>> >> >> [2] https://www.python.org/dev/peps/pep-0602/
>> >> >> [3] https://www.python.org/dev/peps/pep-0602/#id17
>>
>

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Udi Meiri]

I agree with having low-frequency tests for low-priority versions.
Low-priority versions could be determined according to least usage.



On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw  wrote:

> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
> >
> > Are these divergent enough that they all need to consume testing
> resources? For example can lower priority versions be daily runs or some
> such?
>
> For the 3.x series, I think we will get the most signal out of the
> lowest and highest version, and can get by with smoke tests +
> infrequent post-commits for the ones between.
>
> > Kenn
> >
> > On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw 
> wrote:
> >>
> >> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
> >> 20% of all Python 3 downloads.
> >>
> >> I would propose getting in warnings about 3.5 EoL well ahead of time,
> >> at the very least as part of the 2.7 warning.
> >>
> >> Fortunately, supporting multiple 3.x versions is significantly easier
> >> than spanning 2.7 and 3.x. I would rather not impose an ordering on
> >> dropping 3.5 and adding 3.8 but consider their merits independently.
> >>
> >>
> >> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver 
> wrote:
> >> >
> >> > 5 versions is too many IMO. We've had issues with Python precommit
> resource usage in the past, and adding another version would surely
> exacerbate those issues. And we have also already had to leave out certain
> features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before adding
> 3.8. After dropping Python 2 and adding 3.8, that will leave us with the
> latest three minor versions (3.6, 3.7, 3.8), which I think is closer to the
> "sweet spot." Though I would be interested in hearing if there are any
> users who would prefer we continue supporting 3.5.
> >> >
> >> > [1]
> https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
> >> >
> >> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >> >>
> >> >> I would like to start a discussion about identifying a guideline for
> answering questions like:
> >> >>
> >> >> 1. When will Beam support a new Python version (say, Python 3.8)?
> >> >> 2. When will Beam drop support for an old Python version (say,
> Python 3.5)?
> >> >> 3. How many Python versions should we aim to support concurrently
> (investigate issues, have continuous integration tests)?
> >> >> 4. What comes first: adding support for a new version (3.8) or
> deprecating older one (3.5)? This may affect the max load our test
> infrastructure needs to sustain.
> >> >>
> >> >> We are already getting requests for supporting Python 3.8 and there
> were some good reasons[1] to drop support for Python 3.5 (at least, early
> versions of 3.5). Answering these questions would help set expectations in
> Beam user community, Beam dev community, and  may help us establish
> resource requirements for test infrastructure and plan efforts.
> >> >>
> >> >> PEP-0602 [2] establishes a yearly release cycle for Python versions
> starting from 3.9. Each release is a long-term support release and is
> supported for 5 years: first 1.5 years allow for general bug fix support,
> remaining 3.5 years have security fix support.
> >> >>
> >> >> At every point, there may be up to 5 Python minor versions that did
> not yet reach EOL, see "Release overlap with 12 month diagram" [3]. We can
> try to support all of them, but that may come at a cost of velocity: we
> will have more tests to maintain, and we will have to develop Beam against
> a lower version for a longer period. Supporting less versions will have
> implications for user experience. It also may be difficult to ensure
> support of the most recent version early, since our  dependencies (e.g.
> picklers) may not be supporting them yet.
> >> >>
> >> >> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
> >> >>
> >> >> Is 4 versions a sweet spot? Too much? Too little? What do you think?
> >> >>
> >> >> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
> >> >> [2] https://www.python.org/dev/peps/pep-0602/
> >> >> [3] https://www.python.org/dev/peps/pep-0602/#id17
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Robert Bradshaw]

On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles  wrote:
>
> Are these divergent enough that they all need to consume testing resources? 
> For example can lower priority versions be daily runs or some such?

For the 3.x series, I think we will get the most signal out of the
lowest and highest version, and can get by with smoke tests +
infrequent post-commits for the ones between.

> Kenn
>
> On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw  wrote:
>>
>> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
>> 20% of all Python 3 downloads.
>>
>> I would propose getting in warnings about 3.5 EoL well ahead of time,
>> at the very least as part of the 2.7 warning.
>>
>> Fortunately, supporting multiple 3.x versions is significantly easier
>> than spanning 2.7 and 3.x. I would rather not impose an ordering on
>> dropping 3.5 and adding 3.8 but consider their merits independently.
>>
>>
>> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver  wrote:
>> >
>> > 5 versions is too many IMO. We've had issues with Python precommit 
>> > resource usage in the past, and adding another version would surely 
>> > exacerbate those issues. And we have also already had to leave out certain 
>> > features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before 
>> > adding 3.8. After dropping Python 2 and adding 3.8, that will leave us 
>> > with the latest three minor versions (3.6, 3.7, 3.8), which I think is 
>> > closer to the "sweet spot." Though I would be interested in hearing if 
>> > there are any users who would prefer we continue supporting 3.5.
>> >
>> > [1] 
>> > https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
>> >
>> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev  
>> > wrote:
>> >>
>> >> I would like to start a discussion about identifying a guideline for 
>> >> answering questions like:
>> >>
>> >> 1. When will Beam support a new Python version (say, Python 3.8)?
>> >> 2. When will Beam drop support for an old Python version (say, Python 
>> >> 3.5)?
>> >> 3. How many Python versions should we aim to support concurrently 
>> >> (investigate issues, have continuous integration tests)?
>> >> 4. What comes first: adding support for a new version (3.8) or 
>> >> deprecating older one (3.5)? This may affect the max load our test 
>> >> infrastructure needs to sustain.
>> >>
>> >> We are already getting requests for supporting Python 3.8 and there were 
>> >> some good reasons[1] to drop support for Python 3.5 (at least, early 
>> >> versions of 3.5). Answering these questions would help set expectations 
>> >> in Beam user community, Beam dev community, and  may help us establish 
>> >> resource requirements for test infrastructure and plan efforts.
>> >>
>> >> PEP-0602 [2] establishes a yearly release cycle for Python versions 
>> >> starting from 3.9. Each release is a long-term support release and is 
>> >> supported for 5 years: first 1.5 years allow for general bug fix support, 
>> >> remaining 3.5 years have security fix support.
>> >>
>> >> At every point, there may be up to 5 Python minor versions that did not 
>> >> yet reach EOL, see "Release overlap with 12 month diagram" [3]. We can 
>> >> try to support all of them, but that may come at a cost of velocity: we 
>> >> will have more tests to maintain, and we will have to develop Beam 
>> >> against a lower version for a longer period. Supporting less versions 
>> >> will have implications for user experience. It also may be difficult to 
>> >> ensure support of the most recent version early, since our  dependencies 
>> >> (e.g. picklers) may not be supporting them yet.
>> >>
>> >> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>> >>
>> >> Is 4 versions a sweet spot? Too much? Too little? What do you think?
>> >>
>> >> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
>> >> [2] https://www.python.org/dev/peps/pep-0602/
>> >> [3] https://www.python.org/dev/peps/pep-0602/#id17

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Kenneth Knowles]

Are these divergent enough that they all need to consume testing resources?
For example can lower priority versions be daily runs or some such?

Kenn

On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw  wrote:

> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
> 20% of all Python 3 downloads.
>
> I would propose getting in warnings about 3.5 EoL well ahead of time,
> at the very least as part of the 2.7 warning.
>
> Fortunately, supporting multiple 3.x versions is significantly easier
> than spanning 2.7 and 3.x. I would rather not impose an ordering on
> dropping 3.5 and adding 3.8 but consider their merits independently.
>
>
> On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver  wrote:
> >
> > 5 versions is too many IMO. We've had issues with Python precommit
> resource usage in the past, and adding another version would surely
> exacerbate those issues. And we have also already had to leave out certain
> features on 3.5 [1]. Therefore, I am in favor of dropping 3.5 before adding
> 3.8. After dropping Python 2 and adding 3.8, that will leave us with the
> latest three minor versions (3.6, 3.7, 3.8), which I think is closer to the
> "sweet spot." Though I would be interested in hearing if there are any
> users who would prefer we continue supporting 3.5.
> >
> > [1]
> https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
> >
> > On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev 
> wrote:
> >>
> >> I would like to start a discussion about identifying a guideline for
> answering questions like:
> >>
> >> 1. When will Beam support a new Python version (say, Python 3.8)?
> >> 2. When will Beam drop support for an old Python version (say, Python
> 3.5)?
> >> 3. How many Python versions should we aim to support concurrently
> (investigate issues, have continuous integration tests)?
> >> 4. What comes first: adding support for a new version (3.8) or
> deprecating older one (3.5)? This may affect the max load our test
> infrastructure needs to sustain.
> >>
> >> We are already getting requests for supporting Python 3.8 and there
> were some good reasons[1] to drop support for Python 3.5 (at least, early
> versions of 3.5). Answering these questions would help set expectations in
> Beam user community, Beam dev community, and  may help us establish
> resource requirements for test infrastructure and plan efforts.
> >>
> >> PEP-0602 [2] establishes a yearly release cycle for Python versions
> starting from 3.9. Each release is a long-term support release and is
> supported for 5 years: first 1.5 years allow for general bug fix support,
> remaining 3.5 years have security fix support.
> >>
> >> At every point, there may be up to 5 Python minor versions that did not
> yet reach EOL, see "Release overlap with 12 month diagram" [3]. We can try
> to support all of them, but that may come at a cost of velocity: we will
> have more tests to maintain, and we will have to develop Beam against a
> lower version for a longer period. Supporting less versions will have
> implications for user experience. It also may be difficult to ensure
> support of the most recent version early, since our  dependencies (e.g.
> picklers) may not be supporting them yet.
> >>
> >> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
> >>
> >> Is 4 versions a sweet spot? Too much? Too little? What do you think?
> >>
> >> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
> >> [2] https://www.python.org/dev/peps/pep-0602/
> >> [3] https://www.python.org/dev/peps/pep-0602/#id17
>

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Robert Bradshaw]

+1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
20% of all Python 3 downloads.

I would propose getting in warnings about 3.5 EoL well ahead of time,
at the very least as part of the 2.7 warning.

Fortunately, supporting multiple 3.x versions is significantly easier
than spanning 2.7 and 3.x. I would rather not impose an ordering on
dropping 3.5 and adding 3.8 but consider their merits independently.


On Wed, Feb 26, 2020 at 3:16 PM Kyle Weaver  wrote:
>
> 5 versions is too many IMO. We've had issues with Python precommit resource 
> usage in the past, and adding another version would surely exacerbate those 
> issues. And we have also already had to leave out certain features on 3.5 
> [1]. Therefore, I am in favor of dropping 3.5 before adding 3.8. After 
> dropping Python 2 and adding 3.8, that will leave us with the latest three 
> minor versions (3.6, 3.7, 3.8), which I think is closer to the "sweet spot." 
> Though I would be interested in hearing if there are any users who would 
> prefer we continue supporting 3.5.
>
> [1] 
> https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55
>
> On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev  
> wrote:
>>
>> I would like to start a discussion about identifying a guideline for 
>> answering questions like:
>>
>> 1. When will Beam support a new Python version (say, Python 3.8)?
>> 2. When will Beam drop support for an old Python version (say, Python 3.5)?
>> 3. How many Python versions should we aim to support concurrently 
>> (investigate issues, have continuous integration tests)?
>> 4. What comes first: adding support for a new version (3.8) or deprecating 
>> older one (3.5)? This may affect the max load our test infrastructure needs 
>> to sustain.
>>
>> We are already getting requests for supporting Python 3.8 and there were 
>> some good reasons[1] to drop support for Python 3.5 (at least, early 
>> versions of 3.5). Answering these questions would help set expectations in 
>> Beam user community, Beam dev community, and  may help us establish resource 
>> requirements for test infrastructure and plan efforts.
>>
>> PEP-0602 [2] establishes a yearly release cycle for Python versions starting 
>> from 3.9. Each release is a long-term support release and is supported for 5 
>> years: first 1.5 years allow for general bug fix support, remaining 3.5 
>> years have security fix support.
>>
>> At every point, there may be up to 5 Python minor versions that did not yet 
>> reach EOL, see "Release overlap with 12 month diagram" [3]. We can try to 
>> support all of them, but that may come at a cost of velocity: we will have 
>> more tests to maintain, and we will have to develop Beam against a lower 
>> version for a longer period. Supporting less versions will have implications 
>> for user experience. It also may be difficult to ensure support of the most 
>> recent version early, since our  dependencies (e.g. picklers) may not be 
>> supporting them yet.
>>
>> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>>
>> Is 4 versions a sweet spot? Too much? Too little? What do you think?
>>
>> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
>> [2] https://www.python.org/dev/peps/pep-0602/
>> [3] https://www.python.org/dev/peps/pep-0602/#id17

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Kyle Weaver]

5 versions is too many IMO. We've had issues with Python precommit resource
usage in the past, and adding another version would surely exacerbate those
issues. And we have also already had to leave out certain features on 3.5
[1]. Therefore, I am in favor of dropping 3.5 before adding 3.8. After
dropping Python 2 and adding 3.8, that will leave us with the latest three
minor versions (3.6, 3.7, 3.8), which I think is closer to the "sweet
spot." Though I would be interested in hearing if there are any users who
would prefer we continue supporting 3.5.

[1]
https://github.com/apache/beam/blob/8658b95545352e51f35959f38334f3c7df8b48eb/sdks/python/apache_beam/runners/portability/flink_runner.py#L55

On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev 
wrote:

> I would like to start a discussion about identifying a guideline for
> answering questions like:
>
> 1. When will Beam support a new Python version (say, Python 3.8)?
> 2. When will Beam drop support for an old Python version (say, Python
> 3.5)?
> 3. How many Python versions should we aim to support concurrently
> (investigate issues, have continuous integration tests)?
> 4. What comes first: adding support for a new version (3.8) or deprecating
> older one (3.5)? This may affect the max load our test infrastructure needs
> to sustain.
>
> We are already getting requests for supporting Python 3.8 and there were
> some good reasons[1] to drop support for Python 3.5 (at least, early
> versions of 3.5). Answering these questions would help set expectations in
> Beam user community, Beam dev community, and  may help us establish
> resource requirements for test infrastructure and plan efforts.
>
> PEP-0602 [2] establishes a yearly release cycle for Python versions
> starting from 3.9. Each release is a long-term support release and is
> supported for 5 years: first 1.5 years allow for general bug fix support,
> remaining 3.5 years have security fix support.
>
> At every point, there may be up to 5 Python minor versions that did not
> yet reach EOL, see "Release overlap with 12 month diagram" [3]. We can try
> to support all of them, but that may come at a cost of velocity: we will
> have more tests to maintain, and we will have to develop Beam against a
> lower version for a longer period. Supporting less versions will have
> implications for user experience. It also may be difficult to ensure
> support of the most recent version early, since our  dependencies (e.g.
> picklers) may not be supporting them yet.
>
> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>
> Is 4 versions a sweet spot? Too much? Too little? What do you think?
>
> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
> [2] https://www.python.org/dev/peps/pep-0602/
> [3] https://www.python.org/dev/peps/pep-0602/#id17
>

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Robert Bradshaw]

Thanks for bringing this up. I've actually been thinking about the
same thing (specifically with regards to 3.5 and 3.8).

I think it would makes sense to add support for 3.8 right away (or at
least get a good sense of what work needs to be done and what our
dependency situation is like), and to drop support for 3.5 soonish (in
particular, Python 3.5 is EOL 2020-09-13 [1], and also it'd be good if
users porting from Python 2.7 don't port to it).

As for testing and infrastructure, we probably don't need to test all
permutations on all releases (though exactly what the correct
subsetting should be is TBD). Likely we could run complete suites for
the oldest and newest supported version, slimmed down, cheaper
versions for the middle ones, and maybe full post-commits (which would
give us signals into what our slimmed-down coverage is).

[1] https://devguide.python.org/#status-of-python-branches

On Wed, Feb 26, 2020 at 3:00 PM Valentyn Tymofieiev  wrote:
>
> I would like to start a discussion about identifying a guideline for 
> answering questions like:
>
> 1. When will Beam support a new Python version (say, Python 3.8)?
> 2. When will Beam drop support for an old Python version (say, Python 3.5)?
> 3. How many Python versions should we aim to support concurrently 
> (investigate issues, have continuous integration tests)?
> 4. What comes first: adding support for a new version (3.8) or deprecating 
> older one (3.5)? This may affect the max load our test infrastructure needs 
> to sustain.
>
> We are already getting requests for supporting Python 3.8 and there were some 
> good reasons[1] to drop support for Python 3.5 (at least, early versions of 
> 3.5). Answering these questions would help set expectations in Beam user 
> community, Beam dev community, and  may help us establish resource 
> requirements for test infrastructure and plan efforts.
>
> PEP-0602 [2] establishes a yearly release cycle for Python versions starting 
> from 3.9. Each release is a long-term support release and is supported for 5 
> years: first 1.5 years allow for general bug fix support, remaining 3.5 years 
> have security fix support.
>
> At every point, there may be up to 5 Python minor versions that did not yet 
> reach EOL, see "Release overlap with 12 month diagram" [3]. We can try to 
> support all of them, but that may come at a cost of velocity: we will have 
> more tests to maintain, and we will have to develop Beam against a lower 
> version for a longer period. Supporting less versions will have implications 
> for user experience. It also may be difficult to ensure support of the most 
> recent version early, since our  dependencies (e.g. picklers) may not be 
> supporting them yet.
>
> Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).
>
> Is 4 versions a sweet spot? Too much? Too little? What do you think?
>
> [1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
> [2] https://www.python.org/dev/peps/pep-0602/
> [3] https://www.python.org/dev/peps/pep-0602/#id17

[DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Valentyn Tymofieiev]

I would like to start a discussion about identifying a guideline for
answering questions like:

1. When will Beam support a new Python version (say, Python 3.8)?
2. When will Beam drop support for an old Python version (say, Python 3.5)?
3. How many Python versions should we aim to support concurrently
(investigate issues, have continuous integration tests)?
4. What comes first: adding support for a new version (3.8) or deprecating
older one (3.5)? This may affect the max load our test infrastructure needs
to sustain.

We are already getting requests for supporting Python 3.8 and there were
some good reasons[1] to drop support for Python 3.5 (at least, early
versions of 3.5). Answering these questions would help set expectations in
Beam user community, Beam dev community, and  may help us establish
resource requirements for test infrastructure and plan efforts.

PEP-0602 [2] establishes a yearly release cycle for Python versions
starting from 3.9. Each release is a long-term support release and is
supported for 5 years: first 1.5 years allow for general bug fix support,
remaining 3.5 years have security fix support.

At every point, there may be up to 5 Python minor versions that did not yet
reach EOL, see "Release overlap with 12 month diagram" [3]. We can try to
support all of them, but that may come at a cost of velocity: we will have
more tests to maintain, and we will have to develop Beam against a lower
version for a longer period. Supporting less versions will have
implications for user experience. It also may be difficult to ensure
support of the most recent version early, since our  dependencies (e.g.
picklers) may not be supporting them yet.

Currently we support 4 Python versions (2.7, 3.5, 3.6, 3.7).

Is 4 versions a sweet spot? Too much? Too little? What do you think?

[1] https://github.com/apache/beam/pull/10821#issuecomment-590167711
[2] https://www.python.org/dev/peps/pep-0602/
[3] https://www.python.org/dev/peps/pep-0602/#id17

Re: [EXT] Re: using avro instead of json for BigQueryIO.Write

[Pablo Estrada]

This is great. I'll take a look today.

On Wed, Feb 26, 2020 at 9:42 AM Chuck Yang  wrote:

> Hi Devs,
>
> I was able to get around to working on Avro file loads to BigQuery in
> Python SDK and now have a PR available at
> https://github.com/apache/beam/pull/10979 . Comments appreciated :)
>
> Thanks,
> Chuck
>
> On Wed, Nov 27, 2019 at 10:10 AM Chuck Yang 
> wrote:
> >
> > I would love to fix this, but not sure if I have the bandwidth at the
> > moment. Anyway, created the jira here:
> > https://jira.apache.org/jira/browse/BEAM-8841
> >
> > Thanks!
> > Chuck
>
> --
>
>
> *Confidentiality Note:* We care about protecting our proprietary
> information, confidential material, and trade secrets. This message may
> contain some or all of those things. Cruise will suffer material harm if
> anyone other than the intended recipient disseminates or takes any action
> based on this message. If you have received this message (including any
> attachments) in error, please delete it immediately and notify the sender
> promptly.
>

Re: Can any contributor trigger PreCommit tests?

[Ismaël Mejía]

done

On Wed, Feb 26, 2020 at 10:43 PM Wenbing Bai 
wrote:

> Hi Liu,
>
> Here is my PR https://github.com/apache/beam/pull/10901. Pablo is helping
> on this PR.
>
> Thank you for the quick response.
>
> Wenbing
>
> On Tue, Feb 25, 2020 at 1:16 PM Liu Wang  wrote:
>
>> How can I become a committer?
>>
>> On 2020/02/25 19:14:28, Hannah Jiang  wrote:
>> > Committers can trigger the tests. Can you share your PRs here so people
>> who
>> > have permission can trigger tests for your PRs?
>> > Please refer to this thread
>> > <
>> https://lists.apache.org/thread.html/27c1482cb02bdf03e8ea7ad48fe6d2c170527507ed13bed8dea87b7e%40%3Cdev.beam.apache.org%3E
>> >
>> > for more details of the issue.
>> >
>> >
>> >
>> >
>> > On Tue, Feb 25, 2020 at 10:53 AM Wenbing Bai > >
>> > wrote:
>> >
>> > > I have the same issue. Follow this thread.
>> > >
>> > > Wenbing
>> > >
>> > > On Tue, Feb 25, 2020 at 9:43 AM Liu Wang  wrote:
>> > >
>> > >> Hi,
>> > >>
>> > >> I'm a contributor but can't trigger tests for my PRs. Do I need to
>> get
>> > >> any permission for this?
>> > >> Is it like any contributor can apply for and get the permission?
>> > >>
>> > >> Thanks,
>> > >> Liu
>> > >>
>> > >
>> > >
>> > > --
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Wenbing Bai
>> > >
>> > > Senior Software Engineer, MLP
>> > >
>> > > Cruise
>> > >
>> > > Pronouns: She/Her
>> > >
>> > >
>> > >
>> > > *Confidentiality Note:* We care about protecting our proprietary
>> > > information, confidential material, and trade secrets. This message
>> may
>> > > contain some or all of those things. Cruise will suffer material harm
>> if
>> > > anyone other than the intended recipient disseminates or takes any
>> action
>> > > based on this message. If you have received this message (including
>> any
>> > > attachments) in error, please delete it immediately and notify the
>> sender
>> > > promptly.
>> >
>>
>
>
> --
>
>
>
>
>
> Wenbing Bai
>
> Senior Software Engineer, MLP
>
> Cruise
>
> Pronouns: She/Her
>
>
>
> *Confidentiality Note:* We care about protecting our proprietary
> information, confidential material, and trade secrets. This message may
> contain some or all of those things. Cruise will suffer material harm if
> anyone other than the intended recipient disseminates or takes any action
> based on this message. If you have received this message (including any
> attachments) in error, please delete it immediately and notify the sender
> promptly.

Re: Can any contributor trigger PreCommit tests?

[Wenbing Bai]

Hi Liu,

Here is my PR https://github.com/apache/beam/pull/10901. Pablo is helping
on this PR.

Thank you for the quick response.

Wenbing

On Tue, Feb 25, 2020 at 1:16 PM Liu Wang  wrote:

> How can I become a committer?
>
> On 2020/02/25 19:14:28, Hannah Jiang  wrote:
> > Committers can trigger the tests. Can you share your PRs here so people
> who
> > have permission can trigger tests for your PRs?
> > Please refer to this thread
> > <
> https://lists.apache.org/thread.html/27c1482cb02bdf03e8ea7ad48fe6d2c170527507ed13bed8dea87b7e%40%3Cdev.beam.apache.org%3E
> >
> > for more details of the issue.
> >
> >
> >
> >
> > On Tue, Feb 25, 2020 at 10:53 AM Wenbing Bai 
> > wrote:
> >
> > > I have the same issue. Follow this thread.
> > >
> > > Wenbing
> > >
> > > On Tue, Feb 25, 2020 at 9:43 AM Liu Wang  wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm a contributor but can't trigger tests for my PRs. Do I need to get
> > >> any permission for this?
> > >> Is it like any contributor can apply for and get the permission?
> > >>
> > >> Thanks,
> > >> Liu
> > >>
> > >
> > >
> > > --
> > >
> > >
> > >
> > >
> > >
> > > Wenbing Bai
> > >
> > > Senior Software Engineer, MLP
> > >
> > > Cruise
> > >
> > > Pronouns: She/Her
> > >
> > >
> > >
> > > *Confidentiality Note:* We care about protecting our proprietary
> > > information, confidential material, and trade secrets. This message may
> > > contain some or all of those things. Cruise will suffer material harm
> if
> > > anyone other than the intended recipient disseminates or takes any
> action
> > > based on this message. If you have received this message (including any
> > > attachments) in error, please delete it immediately and notify the
> sender
> > > promptly.
> >
>


-- 





Wenbing Bai

Senior Software Engineer, MLP

Cruise

Pronouns: She/Her

-- 


*Confidentiality Note:* We care about protecting our proprietary 
information, confidential material, and trade secrets. This message may 
contain some or all of those things. Cruise will suffer material harm if 
anyone other than the intended recipient disseminates or takes any action 
based on this message. If you have received this message (including any 
attachments) in error, please delete it immediately and notify the sender 
promptly.

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

[Robert Bradshaw]

+1 (binding)

On Wed, Feb 26, 2020 at 1:11 PM Pablo Estrada  wrote:
>
> +1 (binding)
> Verified hashes.
> Thank you Ismael!
>
> On Wed, Feb 26, 2020 at 11:30 AM Luke Cwik  wrote:
>>
>> +1 (binding)
>>
>> Verified signatures and contents of jar to not contain module-info.class
>>
>> On Wed, Feb 26, 2020 at 10:45 AM Kai Jiang  wrote:
>>>
>>> +1 (non-binding)
>>>
>>> On Wed, Feb 26, 2020 at 01:23 Ismaël Mejía  wrote:

 Please review the release of the following artifacts that we vendor:
  * beam-vendor-bytebuddy-1_10_8

 Hi everyone,
 Please review and vote on the release candidate #2 for the version 0.1, as 
 follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)

 The complete staging area is available for your review, which includes:
 * the official Apache source release to be deployed to dist.apache.org 
 [1], which is signed with the key with fingerprint 
 3415631729E15B33051ADB670A9DAF6713B86349 [2],
 * all artifacts to be deployed to the Maven Central Repository [3],
 * commit hash "63492776f154464f67533a6059f162e6b8cf7315" [4],

 The vote will be open for at least 72 hours. It is adopted by majority 
 approval, with at least 3 PMC affirmative votes.

 Thanks,
 Release Manager

 [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
 [2] https://dist.apache.org/repos/dist/release/beam/KEYS
 [3] https://repository.apache.org/content/repositories/orgapachebeam-1094/
 [4] 
 https://github.com/apache/beam/commit/63492776f154464f67533a6059f162e6b8cf7315

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

[Pablo Estrada]

+1 (binding)
Verified hashes.
Thank you Ismael!

On Wed, Feb 26, 2020 at 11:30 AM Luke Cwik  wrote:

> +1 (binding)
>
> Verified signatures and contents of jar to not contain module-info.class
>
> On Wed, Feb 26, 2020 at 10:45 AM Kai Jiang  wrote:
>
>> +1 (non-binding)
>>
>> On Wed, Feb 26, 2020 at 01:23 Ismaël Mejía  wrote:
>>
>>> Please review the release of the following artifacts that we vendor:
>>>  * beam-vendor-bytebuddy-1_10_8
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #2 for the version 0.1,
>>> as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>> The complete staging area is available for your review, which includes:
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [1], which is signed with the key with fingerprint
>>> 3415631729E15B33051ADB670A9DAF6713B86349 [2],
>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>> * commit hash "63492776f154464f67533a6059f162e6b8cf7315" [4],
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [3]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1094/
>>> [4]
>>> https://github.com/apache/beam/commit/63492776f154464f67533a6059f162e6b8cf7315
>>>
>>>

Re: [VOTE][BIP-1] Beam Schema Options

[Maximilian Michels]

+1 (binding)

Looks like a useful feature to have in schemas and fields. Thank you for 
the good write-up.


-Max

On 26.02.20 19:35, Alex Van Boxel wrote:
Nico, thanks for the addition to SQL/MED. I could use another PMC vote 
to conclude the vote.


  _/
_/ Alex Van Boxel


On Mon, Feb 24, 2020 at 11:25 PM Kenneth Knowles > wrote:


+1 (binding)

Added a link to the wiki proposal for SQL/MED (Management of
External Data) which treats some of the same ideas.

Kenn

On Fri, Feb 21, 2020 at 3:02 PM Brian Hulette mailto:bhule...@google.com>> wrote:

+1 (non-binding) thanks for all your work on this Alex :)

On Fri, Feb 21, 2020 at 6:50 AM Alex Van Boxel mailto:a...@vanboxel.be>> wrote:

+1 (non-binding)

I assume I can vote on my own proposal :-)

  _/
_/ Alex Van Boxel


On Fri, Feb 21, 2020 at 6:36 AM Jean-Baptiste Onofre
mailto:j...@nanthrax.net>> wrote:

+1 (binding)

Very interesting. It remembers me when we start the
discussion about schema support ;)

Regards
JB


Le 20 févr. 2020 à 08:36, Alex Van Boxel
mailto:a...@vanboxel.be>> a écrit :

Hi all,

let's do a vote on the very first Beam Improvement
Proposal. If you have a -1 or -1 (binding) please add
your concern to the open issues section to the wiki.
Thanks.

This is the proposal:

https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options

Can I have your votes.

 _/
_/ Alex Van Boxel

Re: [DISCUSSION] Use github actions for python wheels ?

[Ismaël Mejía]

+1 I have been migrating multiple projects into github actions recently and
even if for some tasks it is not as mature and polished as travis it has
proven to be way more reliable.


On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay  wrote:

> I created https://issues.apache.org/jira/browse/BEAM-9388 to explore
> this. To be explicit and not to do cookie licking, I would not be able to
> work on this at the moment. If anyone is interested please take it.
> Otherwise I will try to come back and explore this when I can.
>
> On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw 
> wrote:
>
>> I'd be in favor of this, assuming it actually simplifies things.
>
>
> This is also my concern. I do think that it will simplify things, but I am
> not certain as I am not very familiar with the github actions.
>
>
>> (Note
>> that the wheels are for several variants of linux, presumably we could
>> do cross-compiles. Also, manylinux is a "minimal" linux specifically
>> built as to produce shared object libraries compatible with a wide
>> variety of distributions--we can't just assume that a shared object
>> library built on one modern linux will just work on another. (But
>> maybe it's sufficient to do this within a docker environment?)
>>
>
> There will be no change in this area. Both in Both Travis and github
> actions offer a comparable set of options.
>
>
>>
>> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles  wrote:
>> >
>> > +1 to exploring this.
>> >
>> > On bui...@apache.org there is lots of discussion and general approval
>> for trying it. It is enabled and used by some projects. Calcite uses it to
>> build their website, for example.
>>
>
> Great.
>
>
>> >
>> > Kenn
>> >
>> >
>> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay  wrote:
>> >>
>> >> Hi all,
>> >>
>> >> I recently had a chance to look at the documentation for github
>> actions. I think we could use github actions instead of travis to for
>> building python wheels during releases. This will have the following
>> advantages:
>> >>
>> >> - We will eliminate one repo. (If you don't know, we have
>> https://github.com/apache/beam-wheels for the sole purpose of building
>> wheels file.)
>> >> - Workflow will be stored in the same repo. This will prevent bit rot
>> that is only discovered at release times. (happened a few times, although
>> usually easy to fix.)
>> >> - github actions supports ubuntu, mac, windows environments. We could
>> try to build wheels for windows as well. (Travis also supports the same
>> environments but we only use linux and mac environments. Maybe there are
>> other blockers for building wheels for Windows.)
>> >> - We could do more, like daily python builds.
>> >>
>> >> Downsides would be:
>> >> - I do not know if github actions will require some special set of
>> permissions that require an approval from infra.
>> >> - Travis works fine most of the time. This might be unnecessary work.
>> >>
>> >> What do you think? Is this feasible, would this be useful?
>> >>
>> >> Ahmet
>>
>

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

[Luke Cwik]

+1 (binding)

Verified signatures and contents of jar to not contain module-info.class

On Wed, Feb 26, 2020 at 10:45 AM Kai Jiang  wrote:

> +1 (non-binding)
>
> On Wed, Feb 26, 2020 at 01:23 Ismaël Mejía  wrote:
>
>> Please review the release of the following artifacts that we vendor:
>>  * beam-vendor-bytebuddy-1_10_8
>>
>> Hi everyone,
>> Please review and vote on the release candidate #2 for the version 0.1,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>> * the official Apache source release to be deployed to dist.apache.org
>> [1], which is signed with the key with fingerprint
>> 3415631729E15B33051ADB670A9DAF6713B86349 [2],
>> * all artifacts to be deployed to the Maven Central Repository [3],
>> * commit hash "63492776f154464f67533a6059f162e6b8cf7315" [4],
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [3]
>> https://repository.apache.org/content/repositories/orgapachebeam-1094/
>> [4]
>> https://github.com/apache/beam/commit/63492776f154464f67533a6059f162e6b8cf7315
>>
>>

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

[Kai Jiang]

+1 (non-binding)

On Wed, Feb 26, 2020 at 01:23 Ismaël Mejía  wrote:

> Please review the release of the following artifacts that we vendor:
>  * beam-vendor-bytebuddy-1_10_8
>
> Hi everyone,
> Please review and vote on the release candidate #2 for the version 0.1, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * the official Apache source release to be deployed to dist.apache.org
> [1], which is signed with the key with fingerprint
> 3415631729E15B33051ADB670A9DAF6713B86349 [2],
> * all artifacts to be deployed to the Maven Central Repository [3],
> * commit hash "63492776f154464f67533a6059f162e6b8cf7315" [4],
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
> [3] https://repository.apache.org/content/repositories/orgapachebeam-1094/
> [4]
> https://github.com/apache/beam/commit/63492776f154464f67533a6059f162e6b8cf7315
>
>

Re: [VOTE][BIP-1] Beam Schema Options

[Alex Van Boxel]

Nico, thanks for the addition to SQL/MED. I could use another PMC vote to
conclude the vote.

 _/
_/ Alex Van Boxel


On Mon, Feb 24, 2020 at 11:25 PM Kenneth Knowles  wrote:

> +1 (binding)
>
> Added a link to the wiki proposal for SQL/MED (Management of External
> Data) which treats some of the same ideas.
>
> Kenn
>
> On Fri, Feb 21, 2020 at 3:02 PM Brian Hulette  wrote:
>
>> +1 (non-binding) thanks for all your work on this Alex :)
>>
>> On Fri, Feb 21, 2020 at 6:50 AM Alex Van Boxel  wrote:
>>
>>> +1 (non-binding)
>>>
>>> I assume I can vote on my own proposal :-)
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>>
>>>
>>> On Fri, Feb 21, 2020 at 6:36 AM Jean-Baptiste Onofre 
>>> wrote:
>>>
 +1 (binding)

 Very interesting. It remembers me when we start the discussion about
 schema support ;)

 Regards
 JB

 Le 20 févr. 2020 à 08:36, Alex Van Boxel  a écrit :

 Hi all,

 let's do a vote on the very first Beam Improvement Proposal. If
 you have a -1 or -1 (binding) please add your concern to the open issues
 section to the wiki. Thanks.

 This is the proposal:

 https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options

 Can I have your votes.

  _/
 _/ Alex Van Boxel

Jenkins problems: javaPreCommitPortabilityApiJava11 and No Space left

[Alex Van Boxel]

I see 2 problems with Jenkins popping up. The root beam project is missing:


JavaPortabilityApiJava11 ("Run JavaPortabilityApiJava11 PreCommit")  is
running, but:

Task 'javaPreCommitPortabilityApiJava11' not found in root project 'beam'

And we have no space left on device:

Example:
https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/9287/console



 _/
_/ Alex Van Boxel

Re: [DISCUSSION] Use github actions for python wheels ?

[Ahmet Altay]

I created https://issues.apache.org/jira/browse/BEAM-9388 to explore this.
To be explicit and not to do cookie licking, I would not be able to work on
this at the moment. If anyone is interested please take it. Otherwise I
will try to come back and explore this when I can.

On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw  wrote:

> I'd be in favor of this, assuming it actually simplifies things.


This is also my concern. I do think that it will simplify things, but I am
not certain as I am not very familiar with the github actions.


> (Note
> that the wheels are for several variants of linux, presumably we could
> do cross-compiles. Also, manylinux is a "minimal" linux specifically
> built as to produce shared object libraries compatible with a wide
> variety of distributions--we can't just assume that a shared object
> library built on one modern linux will just work on another. (But
> maybe it's sufficient to do this within a docker environment?)
>

There will be no change in this area. Both in Both Travis and github
actions offer a comparable set of options.


>
> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles  wrote:
> >
> > +1 to exploring this.
> >
> > On bui...@apache.org there is lots of discussion and general approval
> for trying it. It is enabled and used by some projects. Calcite uses it to
> build their website, for example.
>

Great.


> >
> > Kenn
> >
> >
> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay  wrote:
> >>
> >> Hi all,
> >>
> >> I recently had a chance to look at the documentation for github
> actions. I think we could use github actions instead of travis to for
> building python wheels during releases. This will have the following
> advantages:
> >>
> >> - We will eliminate one repo. (If you don't know, we have
> https://github.com/apache/beam-wheels for the sole purpose of building
> wheels file.)
> >> - Workflow will be stored in the same repo. This will prevent bit rot
> that is only discovered at release times. (happened a few times, although
> usually easy to fix.)
> >> - github actions supports ubuntu, mac, windows environments. We could
> try to build wheels for windows as well. (Travis also supports the same
> environments but we only use linux and mac environments. Maybe there are
> other blockers for building wheels for Windows.)
> >> - We could do more, like daily python builds.
> >>
> >> Downsides would be:
> >> - I do not know if github actions will require some special set of
> permissions that require an approval from infra.
> >> - Travis works fine most of the time. This might be unnecessary work.
> >>
> >> What do you think? Is this feasible, would this be useful?
> >>
> >> Ahmet
>

Re: [EXT] Re: using avro instead of json for BigQueryIO.Write

[Chuck Yang]

Hi Devs,

I was able to get around to working on Avro file loads to BigQuery in
Python SDK and now have a PR available at
https://github.com/apache/beam/pull/10979 . Comments appreciated :)

Thanks,
Chuck

On Wed, Nov 27, 2019 at 10:10 AM Chuck Yang  wrote:
>
> I would love to fix this, but not sure if I have the bandwidth at the
> moment. Anyway, created the jira here:
> https://jira.apache.org/jira/browse/BEAM-8841
>
> Thanks!
> Chuck

-- 


*Confidentiality Note:* We care about protecting our proprietary 
information, confidential material, and trade secrets. This message may 
contain some or all of those things. Cruise will suffer material harm if 
anyone other than the intended recipient disseminates or takes any action 
based on this message. If you have received this message (including any 
attachments) in error, please delete it immediately and notify the sender 
promptly.

Re: python Multiprocessing started with in a do function

[Robert Bradshaw]

I suspect this may be due to long-standing bugs regarding forking a
process that has grpc channels. See, e.g.
https://github.com/grpc/grpc/issues/18321

On Wed, Feb 26, 2020 at 9:02 AM laxman reddy  wrote:
>
> Hello Team,
> i am using beam for experimenting for my project usecase
> i have specific steps for a pipeline. for each step to process i am using do 
> function in pardo invocation in pipeline
> in one of the steps i had implemented multiprocessing implemented
>
> with out multiprocessing the step is working fine
> when i introduce multiprocessing in pardo transform step is blocking execution
>
>
> if use beam supported multiprocessing option vis direct runner number of 
> workers it is not recognizing my module function
> if use multi threading it is again blocking infinitely
>
>
> can you suggest me how to resolve issue with in the pardo do fn
> or any other suggestion when multiprocessing
>
>
>
>
>

python Multiprocessing started with in a do function

[laxman reddy]

Hello Team,
i am using beam for experimenting for my project usecase 
i have specific steps for a pipeline. for each step to process i am using do 
function in pardo invocation in pipeline
in one of the steps i had implemented multiprocessing implemented 

with out multiprocessing the step is working fine 
when i introduce multiprocessing in pardo transform step is blocking execution


if use beam supported multiprocessing option vis direct runner number of 
workers it is not recognizing my module function
if use multi threading it is again blocking infinitely


can you suggest me how to resolve issue with in the pardo do fn
or any other suggestion when multiprocessing 

Re: KafkaIO to read from regex topic

[Alexey Romanenko]

Yes, that was a point in that Jira's discussion. I have more concerns about SDF 
support in different runners.

> On 26 Feb 2020, at 17:41, Reuven Lax  wrote:
> 
> You can of course read a dynamic set of topics today, but what you can't do 
> is have that set of topics change during execution of the pipeline. For that 
> you'd need SDF.
> 
> On Wed, Feb 26, 2020 at 8:35 AM Alexey Romanenko  > wrote:
> Great! 
> 
> I’m interested in having a support of adding/removing new topics/partitions 
> in KafkaIO. If there is no other easier solution than SDF, then we should 
> consider to develop it.
>  
> In the mean time, since, afaik, SDF is not yet supported by all runners 
> completely and current KafkaIO implementation, based on UnboundedSource, is 
> very mature and used by many users, we will need to create a new SDF-based 
> Read implementation aside, as a supplementary one. 
> 
> On 24 Feb 2020, at 18:15, Luke Cwik  > wrote:
>> 
>> I have been working on getting unbounded SDFs working within Beam over 
>> portability so if you are interested in writing an SDF KafkaIO 
>> implementation, I would be interested.
>> 
>> On Mon, Feb 24, 2020 at 7:34 AM Alexey Romanenko > > wrote:
>> Hi Maulik,
>> 
>> For the moment, KafkaIO doesn’t support reading from dynamic topics 
>> (add/remove topics/partitions) without pipeline restart. In a few words, 
>> KafkaIO uses unbounded source API which requires fixed number of splits and 
>> it doesn’t deal well with watermarks for empty splits. One of the potential 
>> solution could be to rewrite KafkaIO Read using SDF but afaik nobody worked 
>> on this. 
>> 
>> There is an open Jira about that [1] with more details if you are 
>> interested. 
>> 
>> [1] https://issues.apache.org/jira/browse/BEAM-5786 
>> 
>> 
>>> On 24 Feb 2020, at 11:27, Maulik Soneji >> > wrote:
>>> 
>>> Hello All,
>>> 
>>> I am currently using Beam version 2.19.0 to read data from Kafka using 
>>> KafkaIO. 
>>> Kafka version: 0.11.0.0
>>> My use case is to read all topics from the kafka cluster.
>>> Below is the code that reads data from kafka using KafkaIO.
>>> KafkaIO.read()
>>> .withBootstrapServers(brokers)
>>> .withTopic(topic)
>>> .withKeyDeserializer(ByteArrayDeserializer.class)
>>> .withValueDeserializer(ByteArrayDeserializer.class)
>>> .updateConsumerProperties(props)
>>> .commitOffsetsInFinalize();
>>> If I provide topic as a regex like topic-.*,the request fails with:
>>> Exception in thread "main" 
>>> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
>>> org.apache.kafka.common.errors.InvalidTopicException: Topic topic-.*  is 
>>> invalid
>>> By looking at the code, I saw that there is a call to fetch partition 
>>> information for the topics
>>> at KafkaUnboundSource here 
>>> .
>>> Because we are fetching partitions for only the topic mentioned in the 
>>> builder, it considers the regex as a topic 
>>> and tries to fetch partition information for it even when it is not a topic 
>>> but a regex.
>>> 
>>> My requirement is that I should read from all topics in kafka cluster and 
>>> if there are new tpoics which are added, 
>>> they should be considered as well dynamically without any process restart.
>>> 
>>> Can someone please share details about how I can read data from multiple 
>>> topics by using a regex.
>>> Thanks in advance.
>>> 
>>> Thanks and regards,
>>> Maulik
>> 
> 

Re: KafkaIO to read from regex topic

[Reuven Lax]

You can of course read a dynamic set of topics today, but what you can't do
is have that set of topics change during execution of the pipeline. For
that you'd need SDF.

On Wed, Feb 26, 2020 at 8:35 AM Alexey Romanenko 
wrote:

> Great!
>
> I’m interested in having a support of adding/removing new
> topics/partitions in KafkaIO. If there is no other easier solution than
> SDF, then we should consider to develop it.
>
> In the mean time, since, afaik, SDF is not yet supported by all runners
> completely and current KafkaIO implementation, based on UnboundedSource, is
> very mature and used by many users, we will need to create a new SDF-based
> Read implementation aside, as a supplementary one.
>
> On 24 Feb 2020, at 18:15, Luke Cwik  wrote:
>
>
> I have been working on getting unbounded SDFs working within Beam over
> portability so if you are interested in writing an SDF KafkaIO
> implementation, I would be interested.
>
> On Mon, Feb 24, 2020 at 7:34 AM Alexey Romanenko 
> wrote:
>
>> Hi Maulik,
>>
>> For the moment, KafkaIO doesn’t support reading from dynamic topics
>> (add/remove topics/partitions) without pipeline restart. In a few words,
>> KafkaIO uses unbounded source API which requires fixed number of splits and
>> it doesn’t deal well with watermarks for empty splits. One of the potential
>> solution could be to rewrite KafkaIO Read using SDF but afaik nobody worked
>> on this.
>>
>> There is an open Jira about that [1] with more details if you are
>> interested.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-5786
>>
>> On 24 Feb 2020, at 11:27, Maulik Soneji  wrote:
>>
>> Hello All,
>>
>> I am currently using Beam version 2.19.0 to read data from Kafka using
>> KafkaIO.
>> Kafka version: 0.11.0.0
>> My use case is to read all topics from the kafka cluster.
>> Below is the code that reads data from kafka using KafkaIO.
>>
>> KafkaIO.read()
>> .withBootstrapServers(brokers)
>> .withTopic(topic)
>> .withKeyDeserializer(ByteArrayDeserializer.class)
>> .withValueDeserializer(ByteArrayDeserializer.class)
>> .updateConsumerProperties(props)
>> .commitOffsetsInFinalize();
>>
>> If I provide topic as a regex like topic-.*,the request fails with:
>>
>> Exception in thread "main"
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>> org.apache.kafka.common.errors.InvalidTopicException: Topic topic-.*  is 
>> invalid
>>
>> By looking at the code, I saw that there is a call to fetch partition
>> information for the topics at KafkaUnboundSource here
>> .
>> Because we are fetching partitions for only the topic mentioned in the
>> builder, it considers the regex as a topic and tries to fetch partition
>> information for it even when it is not a topic but a regex. My requirement
>> is that I should read from all topics in kafka cluster and if there are new
>> tpoics which are added, they should be considered as well dynamically
>> without any process restart. Can someone please share details about how I
>> can read data from multiple topics by using a regex. Thanks in advance.
>> Thanks and regards, Maulik
>>
>>
>>
>

Re: KafkaIO to read from regex topic

[Alexey Romanenko]

Great! 

I’m interested in having a support of adding/removing new topics/partitions in 
KafkaIO. If there is no other easier solution than SDF, then we should consider 
to develop it.
 
In the mean time, since, afaik, SDF is not yet supported by all runners 
completely and current KafkaIO implementation, based on UnboundedSource, is 
very mature and used by many users, we will need to create a new SDF-based Read 
implementation aside, as a supplementary one. 

On 24 Feb 2020, at 18:15, Luke Cwik  wrote:
> 
> I have been working on getting unbounded SDFs working within Beam over 
> portability so if you are interested in writing an SDF KafkaIO 
> implementation, I would be interested.
> 
> On Mon, Feb 24, 2020 at 7:34 AM Alexey Romanenko  > wrote:
> Hi Maulik,
> 
> For the moment, KafkaIO doesn’t support reading from dynamic topics 
> (add/remove topics/partitions) without pipeline restart. In a few words, 
> KafkaIO uses unbounded source API which requires fixed number of splits and 
> it doesn’t deal well with watermarks for empty splits. One of the potential 
> solution could be to rewrite KafkaIO Read using SDF but afaik nobody worked 
> on this. 
> 
> There is an open Jira about that [1] with more details if you are interested. 
> 
> [1] https://issues.apache.org/jira/browse/BEAM-5786 
> 
> 
>> On 24 Feb 2020, at 11:27, Maulik Soneji > > wrote:
>> 
>> Hello All,
>> 
>> I am currently using Beam version 2.19.0 to read data from Kafka using 
>> KafkaIO. 
>> Kafka version: 0.11.0.0
>> My use case is to read all topics from the kafka cluster.
>> Below is the code that reads data from kafka using KafkaIO.
>> KafkaIO.read()
>> .withBootstrapServers(brokers)
>> .withTopic(topic)
>> .withKeyDeserializer(ByteArrayDeserializer.class)
>> .withValueDeserializer(ByteArrayDeserializer.class)
>> .updateConsumerProperties(props)
>> .commitOffsetsInFinalize();
>> If I provide topic as a regex like topic-.*,the request fails with:
>> Exception in thread "main" 
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
>> org.apache.kafka.common.errors.InvalidTopicException: Topic topic-.*  is 
>> invalid
>> By looking at the code, I saw that there is a call to fetch partition 
>> information for the topics
>> at KafkaUnboundSource here 
>> .
>> Because we are fetching partitions for only the topic mentioned in the 
>> builder, it considers the regex as a topic 
>> and tries to fetch partition information for it even when it is not a topic 
>> but a regex.
>> 
>> My requirement is that I should read from all topics in kafka cluster and if 
>> there are new tpoics which are added, 
>> they should be considered as well dynamically without any process restart.
>> 
>> Can someone please share details about how I can read data from multiple 
>> topics by using a regex.
>> Thanks in advance.
>> 
>> Thanks and regards,
>> Maulik
> 

Re: [ANNOUNCE] New committer: Chad Dombrova

[Maximilian Michels]

Congrats! It is good to have you with us.

On 26.02.20 03:57, Valentyn Tymofieiev wrote:

Congratulations, Chad!

On Tue, Feb 25, 2020 at 9:36 AM Chamikara Jayalath > wrote:


Congrats Chad!

On Tue, Feb 25, 2020 at 6:00 AM jincheng sun
mailto:sunjincheng...@gmail.com>> wrote:

Congratulations Chad!

Best,
Jincheng


Jan Lukavský mailto:je...@seznam.cz>> 于2020年
2月25日周二 下午5:05写道：

Congrats Chad!

On 2/25/20 9:48 AM, Gleb Kanterov wrote:

Congratulations!

On Tue, Feb 25, 2020 at 9:44 AM Ismaël Mejía
mailto:ieme...@gmail.com>> wrote:

Congratulations, so well deserved for the lots of
amazing work and new perspectives you have broght into
the project !!!

On Tue, Feb 25, 2020 at 8:24 AM Austin Bennett
mailto:whatwouldausti...@gmail.com>> wrote:

Hooray!

On Mon, Feb 24, 2020, 11:21 PM Alex Van Boxel
mailto:a...@vanboxel.be>> wrote:

Congrats!

 _/
_/ Alex Van Boxel


On Tue, Feb 25, 2020 at 6:21 AM Thomas Weise
mailto:t...@apache.org>> wrote:

Congratulations!


On Mon, Feb 24, 2020, 3:38 PM Ankur Goenka
mailto:goe...@google.com>> wrote:

Congratulations Chad!

On Mon, Feb 24, 2020 at 3:34 PM Ahmet
Altay mailto:al...@google.com>> wrote:

Congratulations!

On Mon, Feb 24, 2020 at 3:25 PM
Sam Bourne mailto:samb...@gmail.com>> wrote:

Nice one Chad. Your typing
efforts are very welcomed.

On Tue, Feb 25, 2020 at 10:16
AM Yichi Zhang
mailto:zyi...@google.com>> wrote:

Congratulations, Chad!

On Mon, Feb 24, 2020 at
3:10 PM Robert Bradshaw
mailto:rober...@google.com>>
wrote:

Well deserved, Chad.
Congratulations!

On Mon, Feb 24, 2020
at 2:43 PM Reza Rokni
mailto:r...@google.com>>
wrote:
>
> Congratulations! :-)
>
> On Tue, Feb 25, 2020
at 6:41 AM Chad
Dombrova
mailto:chad...@gmail.com>>
wrote:
>>
>> Thanks, folks!  I'm
very excited to
"retest this" :)
>>
>> Especially big
thanks to Robert and
Udi for all their hard
work reviewing my PRs.
>>
>> -chad
>>
>>
>> On Mon, Feb 24,
2020 at 1:44 PM Brian
Hulette
mailto:bhule...@google.com>>
wrote:
>>>
>>> Congratulations
Chad! Thanks for all
your contributions :)
>>>
>>> On Mon, Feb 24,

[VOTE] Vendored Dependencies Release Byte Buddy 1.10.8 RC2

[Ismaël Mejía]

Please review the release of the following artifacts that we vendor:
 * beam-vendor-bytebuddy-1_10_8

Hi everyone,
Please review and vote on the release candidate #2 for the version 0.1, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* the official Apache source release to be deployed to dist.apache.org [1],
which is signed with the key with fingerprint
3415631729E15B33051ADB670A9DAF6713B86349 [2],
* all artifacts to be deployed to the Maven Central Repository [3],
* commit hash "63492776f154464f67533a6059f162e6b8cf7315" [4],

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://dist.apache.org/repos/dist/dev/beam/vendor/
[2] https://dist.apache.org/repos/dist/release/beam/KEYS
[3] https://repository.apache.org/content/repositories/orgapachebeam-1094/
[4]
https://github.com/apache/beam/commit/63492776f154464f67533a6059f162e6b8cf7315