Re: Version Beam Website Documentation

It seems also be too complex for the Google Crawler as well. A lot of times I arrived on documentation on an older version of a product when I search (aka Google) for something. _/ _/ Alex Van Boxel On Fri, Dec 6, 2019 at 6:20 AM Kenneth Knowles wrote: > Since we are not making breaking

Re: Portable runner bundle scheduling (Streaming/Python/Flink)

PR for this is now open: https://github.com/apache/beam/pull/10313 Hey Max, Thanks for the feedback. --> On Sun, Nov 24, 2019 at 2:04 PM Maximilian Michels wrote: > Load-balancing the worker selection for bundle execution sounds like the > solution to uneven work distribution across the

Re: Version Beam Website Documentation

Since we are not making breaking changes (we hope) and we try to be careful about performance regressions, I think it is OK to simply encourage users to upgrade to the latest if they expect the narrative documentation to match their version. The versioned API docs are probably enough. We might

Re: [RELEASE] Tracking 2.18

Yeah, so I saw... On Thu, Dec 5, 2019 at 4:31 PM Udi Meiri wrote: > > Sorry Robert the release was already cut yesterday. > > > > On Thu, Dec 5, 2019 at 8:37 AM Ismaël Mejía wrote: >> >> Colm, I just merged your PR and cherry picked it into 2.18.0 >> https://github.com/apache/beam/pull/10296 >>

Re: Python staging file weirdness

Filed https://issues.apache.org/jira/browse/BEAM-8900 to address the inefficiency discussed here. Thanks everyone. On Thu, Dec 5, 2019 at 2:53 PM Valentyn Tymofieiev wrote: > Note that so far we have not been staging wheels, since SDK does not have > a knowledge of a target platform, but there

Re: Python interactive runner: test dependencies removed

Thanks Udi! On Thu, Dec 5, 2019 at 2:07 PM Udi Meiri wrote: > The pytest tasks are there for me (or someone else) to verify that they > can replace the nose ones. > If you make changes to tox environments, please make changes to the > corresponding -pytest env as well. > > Regarding extras, go

Re: Python staging file weirdness

Note that so far we have not been staging wheels, since SDK does not have a knowledge of a target platform, but there is https://issues.apache.org/jira/browse/BEAM-4032 to add this support. On Thu, Dec 5, 2019 at 2:35 PM Chad Dombrova wrote: > On Thu, Dec 5, 2019 at 12:36 PM Valentyn Tymofieiev

Re: Python staging file weirdness

On Thu, Dec 5, 2019 at 12:36 PM Valentyn Tymofieiev wrote: Ah nice, so then the workflow would be: download [missing] deps from pypi > into a long-lived cache directory, then download copy the same deps into > a short-lived temporary directory, using long-lived cache directory as > SoT, then

Re: Python interactive runner: test dependencies removed

The pytest tasks are there for me (or someone else) to verify that they can replace the nose ones. If you make changes to tox environments, please make changes to the corresponding -pytest env as well. Regarding extras, go ahead in adding "interactive" to the extras option (both py3x and

Re: Python interactive runner: test dependencies removed

Hi Udi, Are the temporary pytest tasks in use for pre-commit check or anything currently? I see there is still WIP for BEAM-3713 . There is only one task "pythonPreCommitPytest" depending on the pytest tasks using the pytest environment configs.

Re: Python staging file weirdness

Ah nice, so then the workflow would be: download [missing] deps from pypi into a long-lived cache directory, then download copy the same deps into a short-lived temporary directory, using long-lived cache directory as SoT, then stage files from a short-lived temporary directory and clean it up.

Re: Python staging file weirdness

Another way to copy only the deps you care about is to use `pip download` to do the copy. I believe you can provide the cache dir to `pip download --find-links` and it will read from that before reading from pypi (you may also need to set --wheel-dir to the cache dir as well), and thus it acts as

Re: Python staging file weirdness

Looked for a bit at pip download command. The alternative seems to parse the output of python -m pip download --dest . -r requirements.txt --exists-action i --no-binary :all: and see which files were downloaded and/or skipped since they were already present, and then stage only the files that

Re: Python staging file weirdness

I think reusing the same cache directory makes sense during downloading but why do we upload everything that is there? On Thu, Dec 5, 2019 at 9:24 AM Udi Meiri wrote: > Looking at the source, it seems that it should be using a > os.path.join(tempfile.gettempdir(), 'dataflow-requirements-cache')

Re: [RELEASE] Tracking 2.18

Colm, I just merged your PR and cherry picked it into 2.18.0 https://github.com/apache/beam/pull/10296 On Thu, Dec 5, 2019 at 10:54 AM jincheng sun wrote: > Thanks for the Tracking Udi! > > I have updated the status of some release blockers issues as follows: > > - BEAM-8733 closed > -

Precommits fire for wrong PRs

Hi all, I noticed that sometimes the precommit jobs are launched for unrelated PRs, eg. Python or website precommit runs for some Java and gradle changes. AFAIK, the 'responsibility regions' for the jobs are defined in Job DSL scripts and are regexes that file changed paths are checked against.

Re: [RELEASE] Tracking 2.18

Thanks for the Tracking Udi! I have updated the status of some release blockers issues as follows: - BEAM-8733 closed - BEAM-8620 reset the fix version to 2.19 - BEAM-8618 reset the fix version to 2.19 Best, Jincheng Colm O hEigeartaigh 于2019年12月5日周四 下午5:38写道： > Could we get this one in 2.18

Re: [RELEASE] Tracking 2.18

Could we get this one in 2.18 as well? https://issues.apache.org/jira/browse/BEAM-8861 Colm. On Wed, Dec 4, 2019 at 8:02 PM Udi Meiri wrote: > Following the release calendar, I plan on cutting the 2.18 release branch > today. > > There are currently 8 release blockers >