I've written a mini doc[1] about how to update python tests to reduce
consumption test resources.
It would be helpful to check this and comment if there are better solutions.
[1]
https://docs.google.com/document/d/1tfCWtMxfqjgsokjRkOGh2I4UAvX8B98ZOys0crzCMiw/edit?usp=sharing
2020年7月31日(金) 9:44
We have added Python 3.8 support in Apache Beam 2.23.0 release[1] and
established the plan to remove Python 2.7 support in 2.25.0 release[2].
I think it is in the interest of the community to reduce the overhead
associated with adding and removing support of Python minor versions in
Beam in the
Thank you, Kyle and Valentyn.
I'll update test codes to treat Python 3.5 and 3.7 as high-priority
versions at this point.
2020年5月12日(火) 2:10 Valentyn Tymofieiev :
>
> I agree with the point echoed earlier that the lowest and the highest of
> supported versions will probably give the most useful
I agree with the point echoed earlier that the lowest and the highest of
supported versions will probably give the most useful test signal for
possible breakages. So 3.5. and 3.7 as high-priority versions SGTM.
This can change later once Beam drops 3.5 support.
On Mon, May 11, 2020 at 10:05 AM
We've since moved our official Docker images here:
https://hub.docker.com/search?q=apache%2Fbeam_python=image
But Docker downloads are not as representative of actual usage as PyPI.
On Mon, May 11, 2020 at 1:05 PM Yoshiki Obata
wrote:
> Hello again,
>
> Test infrastructure update is ongoing
Hello again,
Test infrastructure update is ongoing and then we should determine
which Python versions are high-priority.
According to Pypi downloads stats[1], download proportion of Python
3.5 is almost always greater than one of 3.6 and 3.7.
This situation has not changed since Robert told us
> Not sure how run_pylint.sh is related here - we should run linter on the
> entire codebase.
ah, I mistyped... I meant run_pytest.sh
> I am familiar with beam_PostCommit_PythonXX suites. Is there something
> specific about these suites that you wanted to know?
Test suite runtime will depend on
On Mon, May 4, 2020 at 7:06 PM Yoshiki Obata
wrote:
> Thank you for comment, Valentyn.
>
> > 1) We can seed the smoke test suite with typehints tests, and add more
> tests later if there is a need. We can identify them by the file path or by
> special attributes in test files. Identifying them
Thank you for comment, Valentyn.
> 1) We can seed the smoke test suite with typehints tests, and add more tests
> later if there is a need. We can identify them by the file path or by special
> attributes in test files. Identifying them using filepath seems simpler and
> independent of test
Hi Yoshiki,
Thanks a lot for your help with Python 3 support so far and most recently,
with your work on Python 3.8.
Overall the proposal sounds good to me. I see several aspects here that we
need to address:
1) We can seed the smoke test suite with typehints tests, and add more
tests later if
Hello everyone.
I'm working on Python 3.8 support[1] and now is the time for preparing
test infrastructure.
According to the discussion, I've considered how to prioritize tests.
My plan is as below. I'd like to get your thoughts on this.
- With all low-pri Python, apache_beam.typehints.*_test
https://pypistats.org/packages/apache-beam is an interesting data point.
The good news: Python 3.x more than doubled to nearly 40% of downloads last
month. Interestingly, it looks like a good chunk of this increase was 3.5
(which is now the most popular 3.x version by this metric...)
I agree
Some good news on Python 3.x support: thanks to +David Song
and +Yifan Zou we now have
Python 3.8 on Jenkins, and can start working on adding Python 3.8 support
to Beam (BEAM-8494).
One interesting variable that has not being mentioned is what versions of
> python 3
> are available to users
One interesting variable that has not being mentioned is what versions of
python
3 are available to users via their distribution channels (the linux
distributions they use to develop/run the pipelines).
- RHEL 8 users have python 3.6 available
- RHEL 7 users have python 3.6 available
- Debian
Thanks everyone for sharing your perspectives so far. It sounds like we can
mitigate the cost of test infrastructure by having:
- a selection of (fast) tests that we will want to run against all Python
versions we support.
- high priority Python versions, which we will test extensively.
-
On Wed, Feb 26, 2020 at 5:21 PM Valentyn Tymofieiev wrote:
>
> > +1 to consulting users.
> I will message user@ as well and point to this thread.
>
> > I would propose getting in warnings about 3.5 EoL well ahead of time.
> I think we should document on our website, and in the code (warnings)
On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver wrote:
>
> > I agree with having low-frequency tests for low-priority versions.
> > Low-priority versions could be determined according to least usage.
>
> +1. While the difference may not be as great between, say, 3.6 and 3.7, I
> think that if we had
> +1 to consulting users.
I will message user@ as well and point to this thread.
> I would propose getting in warnings about 3.5 EoL well ahead of time.
I think we should document on our website, and in the code (warnings) that
users should not expect SDKs to be supported in Beam beyond the EOL.
Oh, I didn't see Robert's earlier email:
> Currently 3.5 downloads sit at 3.7%, or about
> 20% of all Python 3 downloads.
Where did these numbers come from?
On Wed, Feb 26, 2020 at 5:15 PM Kyle Weaver wrote:
> > I agree with having low-frequency tests for low-priority versions.
> >
> I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.
+1. While the difference may not be as great between, say, 3.6 and 3.7, I
think that if we had to choose, it would be more useful to test the
versions folks
I feel 4+ versions take too long to run anything.
would vote for lowest + highest, 2 versions.
On Wed, Feb 26, 2020 at 4:52 PM Udi Meiri wrote:
> I agree with having low-frequency tests for low-priority versions.
> Low-priority versions could be determined according to least usage.
>
>
>
> On
I agree with having low-frequency tests for low-priority versions.
Low-priority versions could be determined according to least usage.
On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw wrote:
> On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles wrote:
> >
> > Are these divergent enough that they
On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles wrote:
>
> Are these divergent enough that they all need to consume testing resources?
> For example can lower priority versions be daily runs or some such?
For the 3.x series, I think we will get the most signal out of the
lowest and highest
Are these divergent enough that they all need to consume testing resources?
For example can lower priority versions be daily runs or some such?
Kenn
On Wed, Feb 26, 2020 at 3:25 PM Robert Bradshaw wrote:
> +1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
> 20% of all
+1 to consulting users. Currently 3.5 downloads sit at 3.7%, or about
20% of all Python 3 downloads.
I would propose getting in warnings about 3.5 EoL well ahead of time,
at the very least as part of the 2.7 warning.
Fortunately, supporting multiple 3.x versions is significantly easier
than
5 versions is too many IMO. We've had issues with Python precommit resource
usage in the past, and adding another version would surely exacerbate those
issues. And we have also already had to leave out certain features on 3.5
[1]. Therefore, I am in favor of dropping 3.5 before adding 3.8. After
Thanks for bringing this up. I've actually been thinking about the
same thing (specifically with regards to 3.5 and 3.8).
I think it would makes sense to add support for 3.8 right away (or at
least get a good sense of what work needs to be done and what our
dependency situation is like), and to
27 matches
Mail list logo
