Re: Switching Python sdist format

[Yoshiki Obata]

g prebuilding, install packages directly on 
> the containers with `pip install`, and avoid `pip download` step. Filter out 
> apache_beam from  requirements.txt file when users add it there.
> - Wait until solutions for [11] become available or get involved to help move 
> that forward.
> - Switch to download sdists over a HTTP instead of `pip download` [16], fall 
> back to pip download if not successful.
> - There may be some way to configure PEP-517 properly, that avoids the 
> slowdown, asked on [9].
>
> Thanks,
> Valentyn
>
> [1] 
> https://pypi.org/project/numpy/#copy-hash-modal-89500de9-70e8-4e7e-87f5-12adf0808905
> [2] https://github.com/numpy/numpy/blob/main/pyproject.toml
> [3] https://snarky.ca/clarifying-pep-518
> [4] 
> https://github.com/apache/beam/blob/1ce290bab031192c22f643cac92bd6470788798d/sdks/python/apache_beam/runners/portability/stager.py#L640
> [5] 
> https://github.com/apache/beam/blob/1ce290bab031192c22f643cac92bd6470788798d/sdks/python/apache_beam/runners/portability/stager.py#L238
> [6] https://issues.apache.org/jira/browse/BEAM-4032
> [7] https://github.com/pypa/pip/issues/1884#issuecomment-670364453
> [8] https://github.com/pypa/pip/issues/8387#issuecomment-638118900
> [9] https://github.com/pypa/pip/issues/10195#issuecomment-941811201
> [10] https://github.com/numpy/numpy/pull/14053#issuecomment-637709988
> [11] https://github.com/pypa/pip/issues/1884
> [12] 
> https://discuss.python.org/t/pep-625-file-name-of-a-source-distribution/4686
> [13] https://github.com/pypa/pip/issues/10195
> [14] 
> https://github.com/apache/beam/blob/1ce290bab031192c22f643cac92bd6470788798d/sdks/python/apache_beam/runners/portability/stager.py#L715
> [15] 
> https://github.com/apache/beam/blob/1ce290bab031192c22f643cac92bd6470788798d/sdks/python/apache_beam/options/pipeline_options.py#L1096
> [16] https://github.com/pypa/pip/issues/1884#issuecomment-800483766
>
>
>
>
> On Mon, Oct 11, 2021 at 8:51 AM Robert Bradshaw  wrote:
>>
>> That's fine by me. The only advantage I can think of for .zip is that
>> it's (generally) better supported on Windows, but as far as I know
>> .tar.gz works on Windows just fine for python package distribution.
>>
>> On Mon, Oct 11, 2021 at 5:09 AM Yoshiki Obata  
>> wrote:
>> >
>> > Hello everyone,
>> >
>> > I'm working on BEAM-8954[1] which introduces tox isolated_build for
>> > python tests.
>> > Concerning this issue, I want opinions about using .tar.gz as sdist format.
>> >
>> > Introducing tox isolated_build leads replacement of
>> > build-requirements.txt to pyproject.toml[2] and we should use
>> > pyproject.toml when creating sdist because we install dependencies
>> > with build-requirements.txt before calling "python setup.py sdist"
>> > PEP 517 based build tools like pypa/build will help to do so, but it
>> > does not allow .zip as sdist format[3].
>> > Therefore I think it would be better to switch sdist format to .tar.gz
>> > when starting to use pyproject.toml.
>> >
>> > Are there any obstacles to use .tar.gz?
>> > Please let me know details about adopting .zip as Beam sdist format(I
>> > could not find discussions about this)
>> >
>> > Regards,
>> > yoshiki
>> >
>> > [1] https://issues.apache.org/jira/browse/BEAM-8954
>> > [2] 
>> > https://tox.wiki/en/latest/config.html?highlight=isolated#conf-isolated_build
>> > [3] https://www.python.org/dev/peps/pep-0517/#source-distributions

Switching Python sdist format

[Yoshiki Obata]

Hello everyone,

I'm working on BEAM-8954[1] which introduces tox isolated_build for
python tests.
Concerning this issue, I want opinions about using .tar.gz as sdist format.

Introducing tox isolated_build leads replacement of
build-requirements.txt to pyproject.toml[2] and we should use
pyproject.toml when creating sdist because we install dependencies
with build-requirements.txt before calling "python setup.py sdist"
PEP 517 based build tools like pypa/build will help to do so, but it
does not allow .zip as sdist format[3].
Therefore I think it would be better to switch sdist format to .tar.gz
when starting to use pyproject.toml.

Are there any obstacles to use .tar.gz?
Please let me know details about adopting .zip as Beam sdist format(I
could not find discussions about this)

Regards,
yoshiki

[1] https://issues.apache.org/jira/browse/BEAM-8954
[2] 
https://tox.wiki/en/latest/config.html?highlight=isolated#conf-isolated_build
[3] https://www.python.org/dev/peps/pep-0517/#source-distributions

Re: Should we support VCF IO on Python 3?

[Yoshiki Obata]

Thank you for your reply.

Considering opinions, it would be better to remove VCF IO from the
codebase for the present.
When removing from codebase, removing description from the document
Ahmet commented at https://issues.apache.org/jira/browse/BEAM-5628 is
also needed.

On Wed, Feb 24, 2021 at 2:31 AM Cory McLean  wrote:
>
> +1 to removing from the codebase, and if it becomes of interest again, 
> porting to cyvcf2. But most genomics workflows are not using Beam at the 
> moment.
>
> On Tue, Feb 23, 2021 at 1:12 AM Chamikara Jayalath  
> wrote:
>>
>> Given that we don't support Python 2 anymore, it sounds like this is just 
>> broken code and we cannot expect anybody to be using it (after Beam 2.24.0).
>> If so +1 for removing it from the codebase. If we decide to add it back with 
>> Python3 support, we should be able to refer to (working) 2.24.0 
>> implementation.
>>
>> Thanks,
>> Cham
>>
>> On Mon, Feb 22, 2021 at 5:17 PM Valentyn Tymofieiev  
>> wrote:
>>>
>>> Hi Yoshiki,
>>>
>>> If switching the code to a new version of VCF package is something easy to 
>>> do, I would keep the code, but keep the dependency on vcf packages 
>>> optional, since we know that this code is not in use.  If you decide to try 
>>> this route,  https://issues.apache.org/jira/browse/BEAM-5628 mentions 
>>> cyvcf2 as a possible replacement.
>>>
>>> If replacement is not trivial and/or nobody is interested in making it 
>>> work, I would remove this IO.
>>>
>>> CC'ing a few folks who may have an opinion: +Chamikara Jayalath +Cory 
>>> McLean .
>>>
>>> Thanks for your help with the cleanup!
>>>
>>> On Sun, Feb 21, 2021 at 4:23 AM Yoshiki Obata  
>>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I'm cleaning up Python 2 codepath now and find that VCF IO codes still
>>>> remain though they might not work properly with latest Beam because
>>>> they depend on PyVCF which does not support Python 3.
>>>> According to comments in vcfio.py, migrating to Nucleus is expected,
>>>> but it is concluded that the plan is not the right option at the
>>>> comment of https://issues.apache.org/jira/browse/BEAM-5628
>>>>
>>>> Now, it would be needed to decide which should we do for VCF IO - drop
>>>> support or maintain support using another vcf package.
>>>> Would anyone have a basis for the decision?
>>>>
>>>> Yoshiki

Should we support VCF IO on Python 3?

[Yoshiki Obata]

Hi all,

I'm cleaning up Python 2 codepath now and find that VCF IO codes still
remain though they might not work properly with latest Beam because
they depend on PyVCF which does not support Python 3.
According to comments in vcfio.py, migrating to Nucleus is expected,
but it is concluded that the plan is not the right option at the
comment of https://issues.apache.org/jira/browse/BEAM-5628

Now, it would be needed to decide which should we do for VCF IO - drop
support or maintain support using another vcf package.
Would anyone have a basis for the decision?

Yoshiki

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

I've written a mini doc[1] about how to update python tests to reduce
consumption test resources.
It would be helpful to check this and comment if there are better solutions.

[1] 
https://docs.google.com/document/d/1tfCWtMxfqjgsokjRkOGh2I4UAvX8B98ZOys0crzCMiw/edit?usp=sharing

2020年7月31日(金) 9:44 Valentyn Tymofieiev :
>
> We have added Python 3.8 support in Apache Beam 2.23.0 release[1] and 
> established the plan to remove Python 2.7 support in 2.25.0 release[2].
>
> I think it is in the interest of the community to reduce the overhead 
> associated with adding and removing support of Python minor versions in Beam 
> in the future. To do so, I opened a ticket [3] to document the process of 
> adding/removing a Python version on the Beam website, and would like to recap 
> the discussion on this thread.
>
> It seems that the consensus is to align support of Python versions in Beam 
> with Python annual release cycle[4]. This means:
>
> 1. We will aim to add support for a new Python 3.x version in Beam as soon as 
> it is released.
> 2. After a Python 3.x version reaches the end of support[5], we will remove 
> support for this version in Beam, starting from the first Beam release that 
> is cut after the end-of-support date.
> 3. The rules above are our default course of action, but can be adjusted on a 
> case-by-case basis via a discussion on dev@.
>
> Please let me know if you think this needs further discussion.
>
> A corollary of 1-3 is that:
> - we should plan to remove support for Python 3.5 starting from 2.25.0 
> release, since Python 3.5 reaches[5] end-of-support on 2020-09-13, and we 
> plan to cut 2.25.0 on 2020-09-23 according to our release calendar [6],
> - we can start working on adding Python 3.9 support shortly after.
>
> Thanks,
> Valentyn
>
> [1] https://beam.apache.org/blog/beam-2.23.0/
> [2] 
> https://lists.apache.org/thread.html/r4be18d50ccfc5543a34e083f3e6711f9f370896f109f21f4677c%40%3Cdev.beam.apache.org%3E
> [3] https://issues.apache.org/jira/browse/BEAM-10605
> [4] https://www.python.org/dev/peps/pep-0602/
> [5] https://www.python.org/downloads/
> [6] 
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>
> On Thu, May 14, 2020 at 9:56 AM Yoshiki Obata  wrote:
>>
>> Thank you, Kyle and Valentyn.
>>
>> I'll update test codes to treat Python 3.5 and 3.7 as high-priority
>> versions at this point.
>>
>> 2020年5月12日(火) 2:10 Valentyn Tymofieiev :
>> >
>> > I agree with the point echoed earlier that the lowest and the highest of 
>> > supported versions will probably give the most useful test signal for 
>> > possible breakages. So 3.5. and 3.7 as high-priority versions SGTM.
>> >
>> > This can change later once Beam drops 3.5 support.
>> >
>> > On Mon, May 11, 2020 at 10:05 AM Yoshiki Obata  
>> > wrote:
>> >>
>> >> Hello again,
>> >>
>> >> Test infrastructure update is ongoing and then we should determine
>> >> which Python versions are high-priority.
>> >>
>> >> According to Pypi downloads stats[1], download proportion of Python
>> >> 3.5 is almost always greater than one of 3.6 and 3.7.
>> >> This situation has not changed since Robert told us Python 3.x
>> >> occupies nearly 40% of downloads[2]
>> >>
>> >> On the other hand, according to docker hub[3],
>> >> apachebeam/python3.x_sdk image downloaded the most is one of Python
>> >> 3.7 which was pointed by Kyle[4].
>> >>
>> >> Considering these stats, I think high-priority versions are 3.5 and 3.7.
>> >>
>> >> Is this assumption appropriate?
>> >> I would like to hear your thoughts about this.
>> >>
>> >> [1] https://pypistats.org/packages/apache-beam
>> >> [2] 
>> >> https://lists.apache.org/thread.html/r208c0d11639e790453a17249e511dbfe00a09f91bef8fcd361b4b74a%40%3Cdev.beam.apache.org%3E
>> >> [3] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>> >> [4] 
>> >> https://lists.apache.org/thread.html/r9ca9ad316dae3d60a3bf298eedbe4aeecab2b2664454cc352648abc9%40%3Cdev.beam.apache.org%3E
>> >>
>> >> 2020年5月6日(水) 12:48 Yoshiki Obata :
>> >> >
>> >> > > Not sure how run_pylint.sh is related here - we should run linter on 
>> >> > > the entire codebase.
>> >> > ah, I mistyped... I meant run_pytest.sh
>> >> >
>> >> > > I am familiar with beam_PostCommit_PythonXX suites. Is there 
>> >&g

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

Thank you, Kyle and Valentyn.

I'll update test codes to treat Python 3.5 and 3.7 as high-priority
versions at this point.

2020年5月12日(火) 2:10 Valentyn Tymofieiev :
>
> I agree with the point echoed earlier that the lowest and the highest of 
> supported versions will probably give the most useful test signal for 
> possible breakages. So 3.5. and 3.7 as high-priority versions SGTM.
>
> This can change later once Beam drops 3.5 support.
>
> On Mon, May 11, 2020 at 10:05 AM Yoshiki Obata  
> wrote:
>>
>> Hello again,
>>
>> Test infrastructure update is ongoing and then we should determine
>> which Python versions are high-priority.
>>
>> According to Pypi downloads stats[1], download proportion of Python
>> 3.5 is almost always greater than one of 3.6 and 3.7.
>> This situation has not changed since Robert told us Python 3.x
>> occupies nearly 40% of downloads[2]
>>
>> On the other hand, according to docker hub[3],
>> apachebeam/python3.x_sdk image downloaded the most is one of Python
>> 3.7 which was pointed by Kyle[4].
>>
>> Considering these stats, I think high-priority versions are 3.5 and 3.7.
>>
>> Is this assumption appropriate?
>> I would like to hear your thoughts about this.
>>
>> [1] https://pypistats.org/packages/apache-beam
>> [2] 
>> https://lists.apache.org/thread.html/r208c0d11639e790453a17249e511dbfe00a09f91bef8fcd361b4b74a%40%3Cdev.beam.apache.org%3E
>> [3] https://hub.docker.com/search?q=apachebeam%2Fpython=image
>> [4] 
>> https://lists.apache.org/thread.html/r9ca9ad316dae3d60a3bf298eedbe4aeecab2b2664454cc352648abc9%40%3Cdev.beam.apache.org%3E
>>
>> 2020年5月6日(水) 12:48 Yoshiki Obata :
>> >
>> > > Not sure how run_pylint.sh is related here - we should run linter on the 
>> > > entire codebase.
>> > ah, I mistyped... I meant run_pytest.sh
>> >
>> > > I am familiar with beam_PostCommit_PythonXX suites. Is there something 
>> > > specific about these suites that you wanted to know?
>> > Test suite runtime will depend on the number of  tests in the suite,
>> > how many tests we run in parallel, how long they take to run. To
>> > understand the load on test infrastructure we can monitor Beam test
>> > health metrics [1]. In particular, if time in queue[2] is high, it is
>> > a sign that there are not enough Jenkins slots available to start the
>> > test suite earlier.
>> > Sorry for ambiguous question. I wanted to know how to see the load on
>> > test infrastructure.
>> > The Grafana links you showed serves my purpose. Thank you.
>> >
>> > 2020年5月6日(水) 2:35 Valentyn Tymofieiev :
>> > >
>> > > On Mon, May 4, 2020 at 7:06 PM Yoshiki Obata  
>> > > wrote:
>> > >>
>> > >> Thank you for comment, Valentyn.
>> > >>
>> > >> > 1) We can seed the smoke test suite with typehints tests, and add 
>> > >> > more tests later if there is a need. We can identify them by the file 
>> > >> > path or by special attributes in test files. Identifying them using 
>> > >> > filepath seems simpler and independent of test runner.
>> > >>
>> > >> Yes, making run_pylint.sh allow target test file paths as arguments is
>> > >> good way if could.
>> > >
>> > >
>> > > Not sure how run_pylint.sh is related here - we should run linter on the 
>> > > entire codebase.
>> > >
>> > >>
>> > >> > 3)  We should reduce the code duplication across  
>> > >> > beam/sdks/python/test-suites/$runner/py3*. I think we could move the 
>> > >> > suite definition into a common file like 
>> > >> > beam/sdks/python/test-suites/$runner/build.gradle perhaps, and 
>> > >> > populate individual suites 
>> > >> > (beam/sdks/python/test-suites/$runner/py38/build.gradle) including 
>> > >> > the common file and/or logic from PythonNature [1].
>> > >>
>> > >> Exactly. I'll check it out.
>> > >>
>> > >> > 4) We have some tests that we run only under specific Python 3 
>> > >> > versions, for example: FlinkValidatesRunner test runs using Python 
>> > >> > 3.5: [2]
>> > >> > HDFS Python 3 tests are running only with Python 3.7 [3]. 
>> > >> > Cross-language Py3 tests for Spark are running under Python 3.5[4]: , 
>> > >> > there may be more test 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

Hello again,

Test infrastructure update is ongoing and then we should determine
which Python versions are high-priority.

According to Pypi downloads stats[1], download proportion of Python
3.5 is almost always greater than one of 3.6 and 3.7.
This situation has not changed since Robert told us Python 3.x
occupies nearly 40% of downloads[2]

On the other hand, according to docker hub[3],
apachebeam/python3.x_sdk image downloaded the most is one of Python
3.7 which was pointed by Kyle[4].

Considering these stats, I think high-priority versions are 3.5 and 3.7.

Is this assumption appropriate?
I would like to hear your thoughts about this.

[1] https://pypistats.org/packages/apache-beam
[2] 
https://lists.apache.org/thread.html/r208c0d11639e790453a17249e511dbfe00a09f91bef8fcd361b4b74a%40%3Cdev.beam.apache.org%3E
[3] https://hub.docker.com/search?q=apachebeam%2Fpython=image
[4] 
https://lists.apache.org/thread.html/r9ca9ad316dae3d60a3bf298eedbe4aeecab2b2664454cc352648abc9%40%3Cdev.beam.apache.org%3E

2020年5月6日(水) 12:48 Yoshiki Obata :
>
> > Not sure how run_pylint.sh is related here - we should run linter on the 
> > entire codebase.
> ah, I mistyped... I meant run_pytest.sh
>
> > I am familiar with beam_PostCommit_PythonXX suites. Is there something 
> > specific about these suites that you wanted to know?
> Test suite runtime will depend on the number of  tests in the suite,
> how many tests we run in parallel, how long they take to run. To
> understand the load on test infrastructure we can monitor Beam test
> health metrics [1]. In particular, if time in queue[2] is high, it is
> a sign that there are not enough Jenkins slots available to start the
> test suite earlier.
> Sorry for ambiguous question. I wanted to know how to see the load on
> test infrastructure.
> The Grafana links you showed serves my purpose. Thank you.
>
> 2020年5月6日(水) 2:35 Valentyn Tymofieiev :
> >
> > On Mon, May 4, 2020 at 7:06 PM Yoshiki Obata  
> > wrote:
> >>
> >> Thank you for comment, Valentyn.
> >>
> >> > 1) We can seed the smoke test suite with typehints tests, and add more 
> >> > tests later if there is a need. We can identify them by the file path or 
> >> > by special attributes in test files. Identifying them using filepath 
> >> > seems simpler and independent of test runner.
> >>
> >> Yes, making run_pylint.sh allow target test file paths as arguments is
> >> good way if could.
> >
> >
> > Not sure how run_pylint.sh is related here - we should run linter on the 
> > entire codebase.
> >
> >>
> >> > 3)  We should reduce the code duplication across  
> >> > beam/sdks/python/test-suites/$runner/py3*. I think we could move the 
> >> > suite definition into a common file like 
> >> > beam/sdks/python/test-suites/$runner/build.gradle perhaps, and populate 
> >> > individual suites 
> >> > (beam/sdks/python/test-suites/$runner/py38/build.gradle) including the 
> >> > common file and/or logic from PythonNature [1].
> >>
> >> Exactly. I'll check it out.
> >>
> >> > 4) We have some tests that we run only under specific Python 3 versions, 
> >> > for example: FlinkValidatesRunner test runs using Python 3.5: [2]
> >> > HDFS Python 3 tests are running only with Python 3.7 [3]. Cross-language 
> >> > Py3 tests for Spark are running under Python 3.5[4]: , there may be more 
> >> > test suites that selectively use particular versions.
> >> > We need to correct such suites, so that we do not tie them  to a 
> >> > specific Python version. I see several options here: such tests should 
> >> > run either for all high-priority versions, or run only under the lowest 
> >> > version among the high-priority versions.  We don't have to fix them all 
> >> > at the same time. In general, we should try to make it as easy as 
> >> > possible to configure, whether a suite runs across all  versions, all 
> >> > high-priority versions, or just one version.
> >>
> >> The way of high-priority/low-priority configuration would be useful for 
> >> this.
> >> And which versions to be tested may be related to 5).
> >>
> >> > 5) If postcommit suites (that need to run against all versions) still 
> >> > constitute too much load on the infrastructure, we may need to 
> >> > investigate how to run these suites less frequently.
> >>
> >> That's certainly true, beam_PostCommit_PythonXX and
> >> beam_PostCommit_Python_Chicago_Taxi_(Dataflow|Flink) take about 1
> >&

Re: Jenkins jobs not running for my PR 10438

[Yoshiki Obata]

Thank you Hannah!

And could anyone trigger these checks again in
https://github.com/apache/beam/pull/11656 ?

Run Portable_Python PreCommit
Run Python 3.5 PostCommit
Run Python 3.6 PostCommit
Run Python 3.7 PostCommit

Regards
yoshiki

2020年5月11日(月) 12:54 Hannah Jiang :
>
> It is done. Some more tests were triggered automatically when I commented to 
> the PR.
>
>
> On Sun, May 10, 2020 at 5:37 AM Yoshiki Obata  wrote:
>>
>> Hi Beam committers
>>
>> I would appreciate if you could trigger following 5 tests in
>> https://github.com/apache/beam/pull/11656
>>
>> Run Portable_Python PreCommit
>> Run Python PreCommit
>> Run Python 3.5 PostCommit
>> Run Python 3.6 PostCommit
>> Run Python 3.7 PostCommit
>>
>> Regards
>> yoshiki
>>
>> 2020年5月5日(火) 0:56 Robert Bradshaw :
>> >
>> > Done.
>> >
>> > On Mon, May 4, 2020 at 7:35 AM Rehman Murad Ali
>> >  wrote:
>> > >
>> > > Hi Beam committers,
>> > >
>> > > Would you please trigger the basic checks as well as validatesRunner 
>> > > check for this PR?
>> > > https://github.com/apache/beam/pull/11350
>> > >
>> > >
>> > > Thanks & Regards
>> > >
>> > > Rehman Murad Ali
>> > > Software Engineer
>> > > Mobile: +92 3452076766
>> > > Skype: rehman.muradali
>> > >
>> > >
>> > >
>> > > On Fri, May 1, 2020 at 5:11 PM Ismaël Mejía  wrote:
>> > >>
>> > >> done
>> > >>
>> > >> On Fri, May 1, 2020 at 5:31 AM Tomo Suzuki  wrote:
>> > >> >
>> > >> > Hi Beam committers,
>> > >> >
>> > >> > Would you trigger the precommit checks for this PR?
>> > >> > https://github.com/apache/beam/pull/11586
>> > >> >
>> > >> > Regards,
>> > >> > Tomo

Re: Jenkins jobs not running for my PR 10438

[Yoshiki Obata]

Hi Beam committers

I would appreciate if you could trigger following 5 tests in
https://github.com/apache/beam/pull/11656

Run Portable_Python PreCommit
Run Python PreCommit
Run Python 3.5 PostCommit
Run Python 3.6 PostCommit
Run Python 3.7 PostCommit

Regards
yoshiki

2020年5月5日(火) 0:56 Robert Bradshaw :
>
> Done.
>
> On Mon, May 4, 2020 at 7:35 AM Rehman Murad Ali
>  wrote:
> >
> > Hi Beam committers,
> >
> > Would you please trigger the basic checks as well as validatesRunner check 
> > for this PR?
> > https://github.com/apache/beam/pull/11350
> >
> >
> > Thanks & Regards
> >
> > Rehman Murad Ali
> > Software Engineer
> > Mobile: +92 3452076766
> > Skype: rehman.muradali
> >
> >
> >
> > On Fri, May 1, 2020 at 5:11 PM Ismaël Mejía  wrote:
> >>
> >> done
> >>
> >> On Fri, May 1, 2020 at 5:31 AM Tomo Suzuki  wrote:
> >> >
> >> > Hi Beam committers,
> >> >
> >> > Would you trigger the precommit checks for this PR?
> >> > https://github.com/apache/beam/pull/11586
> >> >
> >> > Regards,
> >> > Tomo

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

> Not sure how run_pylint.sh is related here - we should run linter on the 
> entire codebase.
ah, I mistyped... I meant run_pytest.sh

> I am familiar with beam_PostCommit_PythonXX suites. Is there something 
> specific about these suites that you wanted to know?
Test suite runtime will depend on the number of  tests in the suite,
how many tests we run in parallel, how long they take to run. To
understand the load on test infrastructure we can monitor Beam test
health metrics [1]. In particular, if time in queue[2] is high, it is
a sign that there are not enough Jenkins slots available to start the
test suite earlier.
Sorry for ambiguous question. I wanted to know how to see the load on
test infrastructure.
The Grafana links you showed serves my purpose. Thank you.

2020年5月6日(水) 2:35 Valentyn Tymofieiev :
>
> On Mon, May 4, 2020 at 7:06 PM Yoshiki Obata  wrote:
>>
>> Thank you for comment, Valentyn.
>>
>> > 1) We can seed the smoke test suite with typehints tests, and add more 
>> > tests later if there is a need. We can identify them by the file path or 
>> > by special attributes in test files. Identifying them using filepath seems 
>> > simpler and independent of test runner.
>>
>> Yes, making run_pylint.sh allow target test file paths as arguments is
>> good way if could.
>
>
> Not sure how run_pylint.sh is related here - we should run linter on the 
> entire codebase.
>
>>
>> > 3)  We should reduce the code duplication across  
>> > beam/sdks/python/test-suites/$runner/py3*. I think we could move the suite 
>> > definition into a common file like 
>> > beam/sdks/python/test-suites/$runner/build.gradle perhaps, and populate 
>> > individual suites (beam/sdks/python/test-suites/$runner/py38/build.gradle) 
>> > including the common file and/or logic from PythonNature [1].
>>
>> Exactly. I'll check it out.
>>
>> > 4) We have some tests that we run only under specific Python 3 versions, 
>> > for example: FlinkValidatesRunner test runs using Python 3.5: [2]
>> > HDFS Python 3 tests are running only with Python 3.7 [3]. Cross-language 
>> > Py3 tests for Spark are running under Python 3.5[4]: , there may be more 
>> > test suites that selectively use particular versions.
>> > We need to correct such suites, so that we do not tie them  to a specific 
>> > Python version. I see several options here: such tests should run either 
>> > for all high-priority versions, or run only under the lowest version among 
>> > the high-priority versions.  We don't have to fix them all at the same 
>> > time. In general, we should try to make it as easy as possible to 
>> > configure, whether a suite runs across all  versions, all high-priority 
>> > versions, or just one version.
>>
>> The way of high-priority/low-priority configuration would be useful for this.
>> And which versions to be tested may be related to 5).
>>
>> > 5) If postcommit suites (that need to run against all versions) still 
>> > constitute too much load on the infrastructure, we may need to investigate 
>> > how to run these suites less frequently.
>>
>> That's certainly true, beam_PostCommit_PythonXX and
>> beam_PostCommit_Python_Chicago_Taxi_(Dataflow|Flink) take about 1
>> hour.
>> Does anyone have knowledge about this?
>
>
> I am familiar with beam_PostCommit_PythonXX suites. Is there something 
> specific about these suites that you wanted to know?
> Test suite runtime will depend on the number of  tests in the suite, how many 
> tests we run in parallel, how long they take to run. To understand the load 
> on test infrastructure we can monitor Beam test health metrics [1]. In 
> particular, if time in queue[2] is high, it is a sign that there are not 
> enough Jenkins slots available to start the test suite earlier.
>
> [1] http://104.154.241.245/d/D81lW0pmk/post-commit-test-reliability
> [2] 
> http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1=1588094891600=1588699691600=6
>
>
>>
>> 2020年5月2日(土) 5:18 Valentyn Tymofieiev :
>> >
>> > Hi Yoshiki,
>> >
>> > Thanks a lot for your help with Python 3 support so far and most recently, 
>> > with your work on Python 3.8.
>> >
>> > Overall the proposal sounds good to me. I see several aspects here that we 
>> > need to address:
>> >
>> > 1) We can seed the smoke test suite with typehints tests, and add more 
>> > tests later if there is a need. We can identify them by the file path or 
>> > by special attributes in test files. Identifying them using 

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

Thank you for comment, Valentyn.

> 1) We can seed the smoke test suite with typehints tests, and add more tests 
> later if there is a need. We can identify them by the file path or by special 
> attributes in test files. Identifying them using filepath seems simpler and 
> independent of test runner.

Yes, making run_pylint.sh allow target test file paths as arguments is
good way if could.

> 3)  We should reduce the code duplication across  
> beam/sdks/python/test-suites/$runner/py3*. I think we could move the suite 
> definition into a common file like 
> beam/sdks/python/test-suites/$runner/build.gradle perhaps, and populate 
> individual suites (beam/sdks/python/test-suites/$runner/py38/build.gradle) 
> including the common file and/or logic from PythonNature [1].

Exactly. I'll check it out.

> 4) We have some tests that we run only under specific Python 3 versions, for 
> example: FlinkValidatesRunner test runs using Python 3.5: [2]
> HDFS Python 3 tests are running only with Python 3.7 [3]. Cross-language Py3 
> tests for Spark are running under Python 3.5[4]: , there may be more test 
> suites that selectively use particular versions.
> We need to correct such suites, so that we do not tie them  to a specific 
> Python version. I see several options here: such tests should run either for 
> all high-priority versions, or run only under the lowest version among the 
> high-priority versions.  We don't have to fix them all at the same time. In 
> general, we should try to make it as easy as possible to configure, whether a 
> suite runs across all  versions, all high-priority versions, or just one 
> version.

The way of high-priority/low-priority configuration would be useful for this.
And which versions to be tested may be related to 5).

> 5) If postcommit suites (that need to run against all versions) still 
> constitute too much load on the infrastructure, we may need to investigate 
> how to run these suites less frequently.

That's certainly true, beam_PostCommit_PythonXX and
beam_PostCommit_Python_Chicago_Taxi_(Dataflow|Flink) take about 1
hour.
Does anyone have knowledge about this?

2020年5月2日(土) 5:18 Valentyn Tymofieiev :
>
> Hi Yoshiki,
>
> Thanks a lot for your help with Python 3 support so far and most recently, 
> with your work on Python 3.8.
>
> Overall the proposal sounds good to me. I see several aspects here that we 
> need to address:
>
> 1) We can seed the smoke test suite with typehints tests, and add more tests 
> later if there is a need. We can identify them by the file path or by special 
> attributes in test files. Identifying them using filepath seems simpler and 
> independent of test runner.
>
> 2) Defining high priority/low priority versions in gradle.properties sounds 
> good to me.
>
> 3)  We should reduce the code duplication across  
> beam/sdks/python/test-suites/$runner/py3*. I think we could move the suite 
> definition into a common file like 
> beam/sdks/python/test-suites/$runner/build.gradle perhaps, and populate 
> individual suites (beam/sdks/python/test-suites/$runner/py38/build.gradle) 
> including the common file and/or logic from PythonNature [1].
>
> 4) We have some tests that we run only under specific Python 3 versions, for 
> example: FlinkValidatesRunner test runs using Python 3.5: [2]
> HDFS Python 3 tests are running only with Python 3.7 [3]. Cross-language Py3 
> tests for Spark are running under Python 3.5[4]: , there may be more test 
> suites that selectively use particular versions.
>
> We need to correct such suites, so that we do not tie them  to a specific 
> Python version. I see several options here: such tests should run either for 
> all high-priority versions, or run only under the lowest version among the 
> high-priority versions.  We don't have to fix them all at the same time. In 
> general, we should try to make it as easy as possible to configure, whether a 
> suite runs across all  versions, all high-priority versions, or just one 
> version.
>
> 5) If postcommit suites (that need to run against all versions) still 
> constitute too much load on the infrastructure, we may need to investigate 
> how to run these suites less frequently.
>
> [1] 
> https://github.com/apache/beam/blob/b78c7ed4836e44177a149155581cfa8188e8f748/sdks/python/test-suites/portable/py37/build.gradle#L19-L20
> [2] 
> https://github.com/apache/beam/blob/93181e792f648122d3b4a5080d683f21c6338132/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy#L34
> [3] 
> https://github.com/apache/beam/blob/93181e792f648122d3b4a5080d683f21c6338132/sdks/python/test-suites/direct/py37/build.gradle#L58
> [4] 
> https://github.com/apache/beam/blob/93181e792f648122d3b4a5080d683f21c6338132/.test-infra/jenkins/job_PostCommit

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

[Yoshiki Obata]

Hello everyone.

I'm working on Python 3.8 support[1] and now is the time for preparing
test infrastructure.
According to the discussion, I've considered how to prioritize tests.
My plan is as below. I'd like to get your thoughts on this.

- With all low-pri Python, apache_beam.typehints.*_test run in the
PreCommit test.
  New gradle task should be defined like "preCommitPy3*-minimum".
  If there are essential tests for all versions other than typehints,
please point out.

- With high-pri Python, the same tests as running in the current
PreCommit test run for testing extensively; "tox:py3*:preCommitPy3*",
"dataflow:py3*:preCommitIT" and "dataflow:py3*:preCommitIT_V2".

- Low-pri versions' whole PreCommit tests are moved to each PostCommit tests.

- High-pri and low-pri versions are defined in gralde.properties and
PreCommit/PostCommit task dependencies are built dynamically according
to them.
  It would be easy for switching priorities of Python versions.

[1] https://issues.apache.org/jira/browse/BEAM-8494

2020年4月4日(土) 7:51 Robert Bradshaw :
>
> https://pypistats.org/packages/apache-beam is an interesting data point.
>
> The good news: Python 3.x more than doubled to nearly 40% of downloads last 
> month. Interestingly, it looks like a good chunk of this increase was 3.5 
> (which is now the most popular 3.x version by this metric...)
>
> I agree with using Python EOL dates as a baseline, with the possibility of 
> case-by-case adjustments. Refactoring our tests to support 3.8 without 
> increasing the load should be our focus now.
>
>
> On Fri, Apr 3, 2020 at 3:41 PM Valentyn Tymofieiev  
> wrote:
>>
>> Some good news on  Python 3.x support: thanks to +David Song and +Yifan Zou 
>> we now have Python 3.8 on Jenkins, and can start working on adding Python 
>> 3.8 support to Beam (BEAM-8494).
>>
>>> One interesting variable that has not being mentioned is what versions of 
>>> python 3
>>> are available to users via their distribution channels (the linux
>>> distributions they use to develop/run the pipelines).
>>
>>
>> Good point. Looking at Ubuntu 16.04, which comes with Python 3.5.2, we can 
>> see that  the end-of-life for 16.04 is in 2024, end-of-support is April 2021 
>> [1]. Both of these dates are beyond the announced Python 3.5 EOL in 
>> September 2020 [2]. I think it would be difficult for Beam to keep Py3.5 
>> support until these EOL dates, and users of systems that stock old versions 
>> of Python have viable workarounds:
>> - install a newer version of Python interpreter via pyenv[3], from sources, 
>> or from alternative repositories.
>> - use a docker container that comes with a newer version of interpreter.
>> - use older versions of Beam.
>>
>> We didn't receive feedback from user@ on how long 3.x versions on the 
>> lower/higher end of the range should stay supported.  I would suggest for 
>> now that we plan to support all Python 3.x versions that were released and 
>> did not reach EOL. We can discuss exceptions to this rule on a case-by-case 
>> basis, evaluating any maintenance burden to continue support, or stop early.
>>
>> We should now focus on adjusting our Python test infrastructure to make it 
>> easy to split 3.5, 3.6, 3.7, 3.8  suites into high-priority and low-priority 
>> suites according to the Python version. Ideally, we should make it easy to 
>> change which versions are high/low priority without having to change all the 
>> individual test suites, and without losing test coverage signal.
>>
>> [1] https://wiki.ubuntu.com/Releases
>> [2] https://devguide.python.org/#status-of-python-branches
>> [3] https://github.com/pyenv/pyenv/blob/master/README.md
>>
>> On Fri, Feb 28, 2020 at 1:25 AM Ismaël Mejía  wrote:
>>>
>>> One interesting variable that has not being mentioned is what versions of 
>>> python
>>> 3 are available to users via their distribution channels (the linux
>>> distributions they use to develop/run the pipelines).
>>>
>>> - RHEL 8 users have python 3.6 available
>>> - RHEL 7 users have python 3.6 available
>>> - Debian 10/Ubuntu 18.04 users have python 3.7/3.6 available
>>> - Debian 9/Ubuntu 16.04 users have python 3.5 available
>>>
>>>
>>> We should consider this when we evaluate future support removals.
>>>
>>> Given  that the distros that support python 3.5 are ~4y old and since 
>>> python 3.5
>>> is also losing LTS support soon is probably ok to not support it in Beam
>>> anymore as Robert suggests.
>>>
>>>
>>> On Thu, Feb 27, 2020 at 3:57 AM Valentyn Tymofieiev  
>>> wrote:

 Thanks everyone for sharing your perspectives so far. It sounds like we 
 can mitigate the cost of test infrastructure by having:
 - a selection of (fast) tests that we will want to run against all Python 
 versions we support.
 - high priority Python versions, which we will test extensively.
 - infrequent postcommit test that exercise low-priority versions.
 We will need test infrastructure improvements to have the flexibility of 
 

Re: Jenkins jobs not running for my PR 10438

[Yoshiki Obata]

Hello Beam Committers,

I would appreciate if you could trigger precommit checks for these PRs;
https://github.com/apache/beam/pull/11493
https://github.com/apache/beam/pull/11494

Regards
yoshiki

2020年4月21日(火) 1:11 Luke Cwik :

> The precommits started and I provided the comments for the postcommits as
> you have requested but they have yet to start.
>
> On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar 
> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger the pre-commit checks on the PR:
>> https://github.com/apache/beam/pull/11210 along with the python
>> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>>
>> Thanks! Regards,
>>
>> *Shoaib Zafar*
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>>
>> 
>>
>>
>> On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía  wrote:
>>
>>> done
>>>
>>> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali <
>>> rehman.murad...@venturedive.com> wrote:
>>>
 Hello Beam Committers.

 Would you please trigger basic tests as well as validatesRunner test on
 this PR:

 
 https://github.com/apache/beam/pull/11350


 *Thanks & Regards*



 *Rehman Murad Ali*
 Software Engineer
 Mobile: +92 3452076766 <+92%20345%202076766>
 Skype: rehman.muradali


 On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay  wrote:

> Done.
>
> On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar <
> shoaib.za...@venturedive.com> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger the pre-commit checks on the PR:
>> https://github.com/apache/beam/pull/11210 along with the python
>> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>>
>> Thanks!
>>
>> *Shoaib Zafar*
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>>
>> 
>>
>>
>> On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía 
>> wrote:
>>
>>> done
>>>
>>> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
>>>  wrote:
>>> >
>>> > Hi Beam Committers!
>>> >
>>> > Thanks( Ismael )
>>> >
>>> > I appreciate if someone could trigger these tests on this PR
>>> https://github.com/apache/beam/pull/11154
>>> >
>>> > run dataflow validatesrunner
>>> > run flink validatesrunner
>>> > Run Java Flink PortableValidatesRunner Streaming
>>> >
>>> > Thanks
>>> >
>>> >
>>> >
>>> > Rehman Murad Ali
>>> > Software Engineer
>>> > Mobile: +92 3452076766 <+92%20345%202076766>
>>> > Skype: rehman.muradali
>>> >
>>> >
>>> >
>>> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía 
>>> wrote:
>>> >>
>>> >> done
>>> >>
>>> >> On Wed, Apr 1, 2020 at 9:15 AM Rehman Murad Ali
>>> >>  wrote:
>>> >> >
>>> >> > Hi Beam Committers!
>>> >> >
>>> >> > I appreciate if someone could trigger the tests on this PR
>>> https://github.com/apache/beam/pull/11154
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> >
>>> >> >
>>> >> > Rehman Murad Ali
>>> >> > Software Engineer
>>> >> > Mobile: +92 3452076766 <+92%20345%202076766>
>>> >> > Skype: rehman.muradali
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Mon, Mar 30, 2020 at 8:09 PM Kamil Wasilewski <
>>> kamil.wasilew...@polidea.com> wrote:
>>> >> >>
>>> >> >> Done.
>>> >> >>
>>> >> >> On Mon, Mar 30, 2020 at 4:58 PM Tomo Suzuki <
>>> suzt...@google.com> wrote:
>>> >> >>>
>>> >> >>> Hi Beam committers,
>>> >> >>> (Thanks Ahmet)
>>> >> >>>
>>> >> >>> Would you retrigger the following 5 jobs for
>>> https://github.com/apache/beam/pull/11156 ?
>>> >> >>> Run Dataflow ValidatesRunner
>>> >> >>> Run Java PreCommit
>>> >> >>> Run Portable_Python PreCommit
>>> >> >>> Run PythonFormatter PreCommit
>>> >> >>> Run Website PreCommit
>>> >> >>>
>>> >> >>>
>>> >> >>> On Thu, Mar 26, 2020 at 9:45 PM Ahmet Altay 
>>> wrote:
>>> >> 
>>> >>  Done.
>>> >> 
>>> >>  On Thu, Mar 26, 2020 at 2:37 PM Tomo Suzuki <
>>> suzt...@google.com> wrote:
>>> >> >
>>> >> > Hi Beam committers,
>>> >> > (Thanks Ahmet for previous attempt)
>>> >> >
>>> >> > Would you trigger
>>> >> > - The failed "Run Java Precommit" for
>>> https://github.com/apache/beam/pull/11208,
>>> >> > - and the precommit checks for
>>> https://github.com/apache/beam/pull/11156  with the following 6
>>> checks?
>>> >> > Run Java PostCommit
>>> >> > Run Java HadoopFormatIO Performance Test
>>> >> > Run BigQueryIO Streaming Performance Test Java
>>> >> > Run Dataflow ValidatesRunner
>>> >> > Run Spark ValidatesRunner
>>> >> > Run SQL 

Re: Jenkins jobs not running for my PR 10438

[Yoshiki Obata]

Hi Beam committers

It would be appreciated if anyone could trigger python precommit job to this PR:
https://github.com/apache/beam/pull/10141

Regards,
Yoshiki

Re: Is there good way to make Python SDK docs draft accessible?

[Yoshiki Obata]

Sorry for late reply.

I've checked release process and found following way would be good to
make the docs/scripts ready to be reviewed and merged.

1. Create PR for apache/beam-site about docs generated by scripts
(hereinafter called PR1)
PR1 is intended only to review docs, so it must not to be merged.
2. create PR for apache/beam about modification to docs-generating
scripts (hereinafter called PR2)
3. PR1 and PR2 are reviewed and confirmed to be OK, close PR1.
4. Merge PR2.

Please let me know if there is a problem.

2019年11月7日(木) 9:17 Valentyn Tymofieiev :


>
> Hi Yoshiki,
>
> Were you able to find the information you need to regenerate the 
> documentation?
>
> Thanks,
> Valentyn
>
> On Tue, Oct 29, 2019 at 8:01 AM Yoshiki Obata  wrote:
>>
>> Thank you for advising, Udi and Ahmet.
>> I'll take a look at the release process.
>>
>> 2019年10月29日(火) 3:47 Ahmet Altay :
>> >
>> > Thank you for doing this. It should be possible to run tox as Udi 
>> > suggested and create a PR for review purposes similar to the release 
>> > process (see: 
>> > https://beam.apache.org/contribute/release-guide/#build-the-pydoc-api-reference)
>> >
>> > /cc +Valentyn Tymofieiev -- This is likely a required item before retiring 
>> > python 2 support.
>> >
>> > Ahmet
>> >
>> > On Mon, Oct 28, 2019 at 11:21 AM Udi Meiri  wrote:
>> >>
>> >> I believe that generating pydoc for the website is still a manual process 
>> >> (unlike the rest of the website?).
>> >> The reviewer will need to manually generate the docs (checkout the PR, 
>> >> run tox -e docs).
>> >>
>> >> On Mon, Oct 28, 2019 at 10:55 AM Yoshiki Obata  
>> >> wrote:
>> >>>
>> >>> Hi all.
>> >>>
>> >>> I'm working on enabling to generate Python SDK docs with Python3 [1]
>> >>> I have modified scripts and now reviewing generated docs in someone’s
>> >>> eyes is needed.
>> >>>
>> >>> But there seems to be no existing way to upload generated docs to
>> >>> where anyone can access unlike website html which can be uploaded to
>> >>> GCS via Jenkins job.
>> >>> Would anyone know good way to make generated docs accessible for
>> >>> anyone for convenience of reviewing them?
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/BEAM-7847
>> >>>
>> >>>
>> >>> Best regards,
>> >>> Yoshiki
>> >>>
>> >>>
>> >>> --
>> >>> Yoshiki Obata
>> >>> mail: yoshiki.ob...@gmail.com
>> >>> gh: https://github.com/lazylynx

Re: Is there good way to make Python SDK docs draft accessible?

[Yoshiki Obata]

Thank you for advising, Udi and Ahmet.
I'll take a look at the release process.

2019年10月29日(火) 3:47 Ahmet Altay :
>
> Thank you for doing this. It should be possible to run tox as Udi suggested 
> and create a PR for review purposes similar to the release process (see: 
> https://beam.apache.org/contribute/release-guide/#build-the-pydoc-api-reference)
>
> /cc +Valentyn Tymofieiev -- This is likely a required item before retiring 
> python 2 support.
>
> Ahmet
>
> On Mon, Oct 28, 2019 at 11:21 AM Udi Meiri  wrote:
>>
>> I believe that generating pydoc for the website is still a manual process 
>> (unlike the rest of the website?).
>> The reviewer will need to manually generate the docs (checkout the PR, run 
>> tox -e docs).
>>
>> On Mon, Oct 28, 2019 at 10:55 AM Yoshiki Obata  
>> wrote:
>>>
>>> Hi all.
>>>
>>> I'm working on enabling to generate Python SDK docs with Python3 [1]
>>> I have modified scripts and now reviewing generated docs in someone’s
>>> eyes is needed.
>>>
>>> But there seems to be no existing way to upload generated docs to
>>> where anyone can access unlike website html which can be uploaded to
>>> GCS via Jenkins job.
>>> Would anyone know good way to make generated docs accessible for
>>> anyone for convenience of reviewing them?
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-7847
>>>
>>>
>>> Best regards,
>>> Yoshiki
>>>
>>>
>>> --
>>> Yoshiki Obata
>>> mail: yoshiki.ob...@gmail.com
>>> gh: https://github.com/lazylynx

Is there good way to make Python SDK docs draft accessible?

[Yoshiki Obata]

Hi all.

I'm working on enabling to generate Python SDK docs with Python3 [1]
I have modified scripts and now reviewing generated docs in someone’s
eyes is needed.

But there seems to be no existing way to upload generated docs to
where anyone can access unlike website html which can be uploaded to
GCS via Jenkins job.
Would anyone know good way to make generated docs accessible for
anyone for convenience of reviewing them?

[1] https://issues.apache.org/jira/browse/BEAM-7847


Best regards,
Yoshiki


--
Yoshiki Obata
mail: yoshiki.ob...@gmail.com
gh: https://github.com/lazylynx

contributor permission for Beam Jira

[Yoshiki Obata]

Hello everyone

This is Yoshiki Obata.
I created ticket BEAM-7137 and plan to fix it.
I'm glad someone would add me as contributer to Jira.

my Jira username is yoshiki.obata

Best regards,
Yoshiki


--
Yoshiki Obata
mail: yoshiki.ob...@gmail.com
gh: https://github.com/lazylynx