Re: [ANNOUNCE] New committer: Alan Myrvold

Congrats Alan! On Mon, Sep 30, 2019 at 11:12 AM Alan Myrvold wrote: > Thanks!! Looking forward to making more impact to Apache Beam > > On Mon, Sep 30, 2019 at 10:56 AM Mikhail Gryzykhin > wrote: > >> Congratulations! >> >> On Mon, Sep 30, 2019 at 9:47 AM David Cavazos >> wrote: >> >>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy wrote: > +1 > > wt., 1 paź 2019 o 11:29 Maximilian Michels napisał(a): > >> +1 >> >> On 30.09.19 23:03, Reza Rokni wrote: >> > +1 >> > >> > On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli > > > wrote: >> > >> > +1

Re: [Discuss] Beam mascot

Dumbo octopus anyone? https://youtu.be/DmqikqvLLLw?t=263 On Mon, Nov 11, 2019 at 2:06 PM Luke Cwik wrote: > The real answer, what cool schwag can we get based upon the mascot. > > On Mon, Nov 11, 2019 at 2:04 PM Kenneth Knowles wrote: > >> I'm with Luke on cuttlefish. We can have color

Re: [ANNOUNCE] New committer: Brian Hulette

Congrats Brian! On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang wrote: > Congrats Brian! > > On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu wrote: > >> Congrats, Brian! >> >> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira >> wrote: >> >>> Congratulations Brian! It's well deserved. >>> >>> On

Re: [discuss] Using a logger hierarchy in Python

+1, but can we use something less verbose and shift key heavy than _LOGGER like log or _log? Also please dedupe with these existing bugs: https://issues.apache.org/jira/browse/BEAM-3523 https://issues.apache.org/jira/browse/BEAM-1825 On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise wrote: >

Cython unit test suites running without Cythonized sources

I opened this bug today after commenting on Chad's type hints PR. https://issues.apache.org/jira/browse/BEAM-8572?filter=-1 I am 95% sure that our Precommit tests are using tarballs that are built without Cython (including the

Re: Python tests and setup

I've been looking at using pyenv in the past few days. I've also had a broken 3.6.8 install on my machine for a while now, but using pyenv I managed to install 3.6.9 with little difficulty: pyenv install 3.6.9 PYENV_VERSION=3.6.9 ./gradlew :sdks:python:test-suites:tox:py36:setupVirtualenv (You

Re: [PROPOSAL] Preparing for Beam 2.18 release

>> and/or release manager may decide to make some of these blockers if needed. >> I don't think the tool automatically makes the auto generated JIRAs >> release blockers. >> >> Thanks, >> Cham >> >> >>> [1]: >>> https://issues.apache

[RELEASE] Tracking 2.18

Following the release calendar, I plan on cutting the 2.18 release branch today. There are currently 8 release blockers . smime.p7s Description: S/MIME Cryptographic Signature

Re: Python staging file weirdness

#L161 > > [4] > https://github.com/apache/beam/blob/438055c95116f4e6e419e5faa9c42f7d329c421c/sdks/python/apache_beam/runners/portability/stager.py#L172 > > On Wed, Nov 27, 2019 at 11:55 AM Udi Meiri wrote: > >> I was investigating a Dataflow postcommit test failure (endpoints_pb2 >> missing

Python: pytest migration update

This PR (in review) migrates py27-gcp to using pytest. It reduces the testPy2Gcp task down to ~13m (from ~45m). This speedup will probably be lower once all 8 tasks are using

request for access: pypi and dockerhub

Hi, I'm following the release guide , and it says I need access to a couple of repos: - apache_beam pypi project, need maintainer or owner grant, user: udim - DockerHub Push Permission, need to be part of "maintainer team", user: udim Thanks!

Re: Cython unit test suites running without Cythonized sources

To follow up, since I'm trying to run cython-based tests using pytest: - tox does in fact correctly install apache-beam with cythonized modules in its virtualenv. - Since our tests are under apache_beam/, local sources shadow those in the installed apache_beam package. - The original issue I

Re: request for access: pypi and dockerhub

e added you as a maintainer in docker hub. >> I don't have privileges to add you in pypi. +Ahmet Altay >> can you? >> -P. >> >> On Mon, Dec 9, 2019 at 4:34 PM Udi Meiri wrote: >> >>> Hi, >>> >>> I'm following the release guide >>

Re: Cython unit test suites running without Cythonized sources

ing the wrong > module name to check for cythonization ("apache_beam.coders" which is a > package). > > -chad > > > On Tue, Dec 10, 2019 at 6:03 PM Udi Meiri wrote: > >> To follow up, since I'm trying to run cython-based tests using pytest: >> - tox does

Re: [RELEASE] Tracking 2.18

Also marked 3 Jiras from these cherrypicks as blockers . Current open blocker count: 7 <https://issues.apache.org/jira/projects/BEAM/versions/12346383>. On Thu, Dec 12, 2019 at 5:21 PM Udi Meiri wrote: > Just merged 6 PRs. :) > > On Thu, Dec 12, 2019 at 4:52 PM Udi Meiri wro

Re: [RELEASE] Tracking 2.18

Udi Meiri wrote: > Re: cherrypicks on top of the release-2.18.0 branch > The precommit tests are failing most likely due to some integration tests > (wordcount, etc.) that are expecting the new 2.18 worker on Dataflow. > I'm working on building an initial version of that worker so that

Re: [RELEASE] Tracking 2.18

Just merged 6 PRs. :) On Thu, Dec 12, 2019 at 4:52 PM Udi Meiri wrote: > Update: I'm accepting cherrypicks with failing tests if the corresponding > PR have passed them on master. > > I recall (without proof) that in the past, even with released worker > containers for the in-

Re: Python: pytest migration update

Update: 37m <https://scans.gradle.com/s/hqcvbxm2h6svg/timeline> precommit time with the latest PR <https://github.com/apache/beam/pull/10377> (in review). On Tue, Dec 10, 2019 at 11:21 AM Udi Meiri wrote: > > > On Mon, Dec 9, 2019 at 9:33 PM Kenneth Knowles wrote: >

Re: Root logger configuration

Pablo, does the issue affect debuggability of pipelines? On Mon, Dec 16, 2019 at 6:23 PM Chad Dombrova wrote: > > > On Mon, Dec 16, 2019 at 5:59 PM Pablo Estrada wrote: > >> +chad...@gmail.com is this consistent with behavior >> that you observed? >> > > I honestly can't recall, sorry. I

Re: Cython unit test suites running without Cythonized sources

The `changedir = {envsitepackagesdir}` setting is definitely something I haven't thought of. It solves the shadowing issue without needing to split tests and packages from one another. (though I still think it's unnecessary to include tests in the published package) IIUC, isolated_build=True and

Re: Python interactive runner: test dependencies removed

> "py36-pytest" in tox.ini >> <https://github.com/apache/beam/blob/master/sdks/python/tox.ini#L100> >> then? >> >> @Ahmet Altay fyi >> >> On Wed, Dec 4, 2019 at 5:22 PM Pablo Estrada wrote: >> >>> +Ning Kang +Sam Rohde fyi

Re: [RELEASE] Tracking 2.18

The remaining 4 open blockers all have recently merged cherrypicks (at least 1 blocker is waiting on verification since it's a release process issue). Will attempt an RC today. On Thu, Dec 12, 2019 at 5:33 PM Udi Meiri wrote: > Also marked 3 Jiras from these cherrypicks as blockers . > C

Re: Python: pytest migration update

onitoring and re-queuing timed-out jenkins jobs lately than I am > writing code. > > On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri wrote: > >> This PR <https://github.com/apache/beam/pull/10322> (in review) migrates >> py27-gcp to using pytest. >> It reduces the tes

Re: Python: pytest migration update

Valentyn, the speedup is due to parallelization. On Mon, Dec 9, 2019 at 6:12 PM Chad Dombrova wrote: > > On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri wrote: > >> I have given this some thought honestly don't know if splitting into >> separate jobs will help. >> -

Re: Python: pytest migration update

On Mon, Dec 9, 2019 at 9:33 PM Kenneth Knowles wrote: > > > On Mon, Dec 9, 2019 at 6:34 PM Udi Meiri wrote: > >> Valentyn, the speedup is due to parallelization. >> >> On Mon, Dec 9, 2019 at 6:12 PM Chad Dombrova wrote: >> >>> >>> On Mo

Re: Is there good way to make Python SDK docs draft accessible?

I believe that generating pydoc for the website is still a manual process (unlike the rest of the website?). The reviewer will need to manually generate the docs (checkout the PR, run tox -e docs). On Mon, Oct 28, 2019 at 10:55 AM Yoshiki Obata wrote: > Hi all. > > I'm working on enabling to

Re: Python Precommit duration pushing 2 hours

> > > On Tue, Oct 29, 2019, 6:46 AM Kenneth Knowles wrote: > >> > >> Noting for the benefit of the thread archive in case someone goes > digging and wonders if this affects other SDKs: the Java SDK memoizes > DoFnSignatures and generated DoFnInvoker classes. >

Re: Install Jenkins AnsiColor plugin

seed job > > Does anyone have any concerns with me trying this out now? > > -chad > > On Tue, Oct 22, 2019 at 11:42 AM Udi Meiri wrote: > >> Also note that changing the job DSL doesn't take effect until the "seed" >> job runs. (use the "run seed job&qu

Re: Install Jenkins AnsiColor plugin

Seems to be already installed: https://issues.apache.org/jira/browse/INFRA-16944 Do we just need to enable it somehow? This might work: https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.wrapper.WrapperContext.colorizeOutput BTW, our Jenkins is maintained by ASF's

Re: Install Jenkins AnsiColor plugin

ed to this. > > > > On Tue, Oct 22, 2019 at 10:45 AM Yifan Zou wrote: > >> Thanks, Udi! The ansicolor plugin was applied to ASF Jenkins universally. >> You might need to explicitly enable the coloroutput in your jenkins dsl. >> >> On Tue, Oct 22, 2019 a

Re: cython test instability

Correction: the error is not gone after removing the line. I get instead: error: [Errno 17] File exists: '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg' On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri wrote: > I managed to recreate one of the issues with this comm

Re: cython test instability

I managed to recreate one of the issues with this command: ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo "python setup.py -q nosetests --tests apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ; done This reliably gives me: OSError: [Errno 17] File exists:

Re: cython test instability

Basically, I believe what's happening is that a new Gradle task was added that uses setup.py but doesn't have the same dependency on some main setup.py task that all others depend on (list sdist). On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri wrote: > Correction: the error is not gone af

Re: cython test instability

hat only have source > distributions (i.e. no wheels). But why are the eggs being installed in > sdks/python/.eggs instead of into the virtualenv created by setupVirtualenv > gradle task or by tox? > > > On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri wrote: > >> Basically,

Re: cython test instability

;& pip install". On Tue, Nov 26, 2019 at 5:14 PM Chad Dombrova wrote: > Sorry wrong link: https://github.com/apache/beam/pull/9915 > > > > On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri wrote: > >> I looked at #9959 but it doesn't seem to modify setup.py? >> The a

Re: cython test instability

t-runner'] was present -- it's in a gradle traceback > now rather than the console log. I've also seen different packages listed > as the culprit (simplegeneric, timeloop). > > -chad > > > > On Tue, Nov 26, 2019 at 4:47 PM Udi Meiri wrote: > >> Chad, I believe the an

Python staging file weirdness

I was investigating a Dataflow postcommit test failure (endpoints_pb2 missing), and saw this in the staging directory: $ gsutil ls gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1126202146-314738.1574799706.314882

Python interactive runner: test dependencies removed

As part of a move to stop using the deprecated (and racey) setup.py keywords setup_requires and test_require, interactive runner dependencies have been removed from tests in https://github.com/apache/beam/pull/10227 If this breaks any tests, please let me know. smime.p7s Description: S/MIME

[PROPOSAL] Preparing for Beam 2.18 release

Hi all, The next (2.18) release branch cut is scheduled for Dec 4, according to the calendar . I would like to volunteer myself to do this release. The plan is to cut the branch on that date,

Re: [ANNOUNCE] New committer: Daniel Oliveira

Congrats Daniel! On Wed, Nov 20, 2019 at 12:42 PM Kyle Weaver wrote: > Congrats Dan! Keep up the good work :) > > On Wed, Nov 20, 2019 at 12:41 PM Cyrus Maden wrote: > >> Congratulations! This is great news. >> >> On Wed, Nov 20, 2019 at 3:24 PM Rui Wang wrote: >> >>> Congrats! >>> >>> >>>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[ ] Beaver [x] Hedgehog [x] Lemur [x] Owl [ ] Salmon [ ] Trout [x] Robot dinosaur [x] Firefly [ ] Cuttlefish [ ] Dumbo Octopus [ ] Angler fish On Wed, Nov 20, 2019 at 12:05 PM Eduardo Soldera < eduardo.sold...@arquivei.com.br> wrote: > [X ] Beaver > [] Hedgehog > [ ] Lemur > [ ] Owl > [ ] Salmon

Re: [Discuss] Beam Summit 2020 Dates & locations

+1 for better organization. I would have gone to ApacheCon LV had I known there was going to be a Beam summit there. On Tue, Nov 12, 2019 at 9:31 AM Alexey Romanenko wrote: > On 8 Nov 2019, at 11:32, Maximilian Michels wrote: > > > > The dates sounds good to me. I agree that the bay area has

Re: NOTICE: New Python PreCommit jobs

On Fri, Oct 4, 2019 at 10:35 AM Chad Dombrova wrote: > > I have a WiP PR to convert Beam to use pytest, but it's been stalled. >> > > What would it take to get it back on track? > Besides needing to convert ITs (removing save_main_session), which can be split out to a later PR, there's

Re: BEAM-8989 fix for 2.18.0 release

gt; > On Thu, Dec 19, 2019 at 10:18 AM Udi Meiri wrote: > >> The JIRA issue was assigned to me, but I have no background in the issue. >> Who would be the most suitable to take care of fixing, testing (Nemo >> quickstart), and cherrypicking? >> > smime.p7s Description: S/MIME Cryptographic Signature

Re: Unifying Build/contributing instructions

+1 for website focus On Thu, Dec 19, 2019 at 10:22 AM Elliotte Rusty Harold wrote: > That's two votes for > https://beam.apache.org/contribute/contribution-guide/ and a lot of > abstentions. I'll update the PR to move content to > https://beam.apache.org/contribute/contribution-guide/ > > On

[PROPOSAL] python precommit timeouts

Looking at this console log , it seems that some pytests got stuck (or slowed down considerably). I'd like to put a 10 minute default timeout on all unit tests, using the

Re: PostCommit_Py_VR_Dataflow timing out

Yes, there are objections since this would take up a Jenkins slot for longer. An alternative would be to set timeouts on individual tests. Debugging options: run the gradle tasks locally, try to pinpoint the culprit PR https://issues.apache.org/jira/browse/BEAM-8877 On Wed, Dec 18, 2019 at 1:25

Re: [PROPOSAL] python precommit timeouts

-test if needed? > > > > On Thu, Dec 19, 2019 at 5:52 PM Udi Meiri > <mailto:eh...@google.com>> wrote: > > > > Looking at this console log > > < > https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/timestamps/?time=HH:mm:s

Re: [PROPOSAL] python precommit timeouts

other than that, I think this is great. > > On Fri, Dec 20, 2019 at 9:39 AM Udi Meiri wrote: > >> https://issues.apache.org/jira/browse/BEAM-9009 >> >> On Fri, Dec 20, 2019 at 6:18 AM Maximilian Michels >> wrote: >> >>> +1 Good idea. We should als

Re: Sphinx Docs Command Error (:sdks:python:test-suites:tox:pycommon:docs)

I don't have those issues (running on Linux), but a possible workaround could be to remove the "-j 8" flags (2 locations) in generate_pydoc.sh. On Mon, Feb 10, 2020 at 11:06 AM Shoaib Zafar wrote: > Hello Beamers. > > Just curious does anyone having trouble running >

Re: [ANNOUNCE] New committer: Chad Dombrova

Congrats and welcome, Chad! On Mon, Feb 24, 2020 at 1:21 PM Pablo Estrada wrote: > Hi everyone, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Chad Dombrova > > Chad has contributed to the project in multiple ways, including > improvements to the testing

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

Welcome Kamil! On Fri, Feb 28, 2020 at 12:53 PM Mark Liu wrote: > Congrats, Kamil! > > On Fri, Feb 28, 2020 at 12:23 PM Ismaël Mejía wrote: > >> Congratulations Kamil! >> >> On Fri, Feb 28, 2020 at 7:09 PM Yichi Zhang wrote: >> >>> Congrats, Kamil! >>> >>> On Fri, Feb 28, 2020 at 9:53 AM

Re: Python Static Typing: Next Steps

Let's go forward with this and see. I volunteer to help as well. I believe that mypy via pre-commit hook will be faster than 10s since it only applies to modified files. On Mon, Mar 2, 2020 at 10:53 AM Robert Bradshaw wrote: > +1 > > We should enable this on jenkins, plus trivial instructions

Re: [DISCUSS] How many Python 3.x minor versions should Beam Python SDK aim to support concurrently?

I agree with having low-frequency tests for low-priority versions. Low-priority versions could be determined according to least usage. On Wed, Feb 26, 2020 at 4:06 PM Robert Bradshaw wrote: > On Wed, Feb 26, 2020 at 3:29 PM Kenneth Knowles wrote: > > > > Are these divergent enough that they

Re: Python Static Typing: Next Steps

Off-topic: Python lint via pre-commit should be much faster. (I wrote my own modified-file-only lint in the past) On Mon, Mar 2, 2020 at 2:08 PM Kyle Weaver wrote: > > Python lint takes 4-5mins to complete. I think if the mypy analysis is > really on the order of 10s, the additional time won't

Re: [DISCUSS] Autoformat python code with Black

;> first. >>> > >>> > Best, >>> > Kamil >>> > >>> > >>> > On Thu, Jan 23, 2020 at 10:37 AM Michał Walenia < >>> michal.wale...@polidea.com> wrote: >>> >> >>> >> Hi all, >>>

Re: [ANNOUNCE] New committer: Michał Walenia

Congratulations Michał! On Mon, Jan 27, 2020 at 3:49 PM Chamikara Jayalath wrote: > Congrats Michał! > > On Mon, Jan 27, 2020 at 2:59 PM Reza Rokni wrote: > >> Congratulations buddy! >> >> On Tue, 28 Jan 2020, 06:52 Valentyn Tymofieiev, >> wrote: >> >>> Congratulations, Michał! >>> >>> On

Re: Jenkins jobs not running for my PR 10438

done On Fri, Jan 31, 2020 at 9:07 AM Tomo Suzuki wrote: > HI Beam committers, > > Would you re-trigger the 2 failed checks in > https://github.com/apache/beam/pull/10714 ? > Run Java PreCommit > Run Java_Examples_Dataflow PreCommit > > > On Fri, Jan 31, 2020 at 7:51 AM Rehman Murad Ali < >

Re: [DISCUSSION] Improve release notes by adding a change list file

+1 to add this to the checklist On Mon, Feb 3, 2020 at 4:57 PM Robert Bradshaw wrote: > On Mon, Feb 3, 2020 at 4:49 PM Ahmet Altay wrote: > > > > On Mon, Feb 3, 2020 at 2:09 PM Robert Bradshaw > wrote: > >> > >> I would suggest we start with the simpler single file. If merge > >> conflicts

Re: [RELEASE VOTE RESULT] Release 2.19.0, release candidate #1

Thank you Boyuan! On Mon, Feb 3, 2020 at 3:40 PM Ahmet Altay wrote: > On Mon, Feb 3, 2020 at 1:22 PM Thomas Weise wrote: > >> Impressive, probably the fastest/smoothest Beam release so far. >> > > I agree! Thank you, Boyuan! > > >> >> On Mon, Feb 3, 2020 at 10:45 AM Boyuan Zhang wrote: >> >>>

Re: [DISCUSS] Autoformat python code with Black

as to just notice every PR that was affected by the >> change. >> >>>>>> And clearly document how to validate and autoformat the code. >> >>>>>> >> >>>>>> So the earlier the better, go go autoformat! >> >>

[RESULT] [VOTE] Release 2.18.0, release candidate #1

I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 4 of which are binding: * Ahmet Altay * Robert Bradshaw * Ismaël Mejía * Jean-Baptiste Onofré There are no disapproving votes. Thanks everyone! smime.p7s Description: S/MIME Cryptographic

Re: [ANNOUNCE] New committer: Hannah Jiang

Welcome and congrats Hannah! On Tue, Jan 28, 2020 at 4:52 PM Robin Qiu wrote: > Congratulations, Hannah! > > On Tue, Jan 28, 2020 at 4:50 PM Alan Myrvold wrote: > >> Congrats, Hannah >> >> On Tue, Jan 28, 2020 at 4:46 PM Connell O'Callaghan >> wrote: >> >>> Thank you for sharing Luke!!!

anyone working on updating com.google.cloud:google-cloud-spanner?

It's currently at 1.6.0, which a year old. https://github.com/googleapis/java-spanner/releases?after=1.9.0 I would appreciate any help with this. Tracking issues: https://issues.apache.org/jira/browse/BEAM-8758 https://issues.apache.org/jira/browse/BEAM-8682 smime.p7s Description: S/MIME

[ANNOUNCE] Beam 2.18.0 Released

. -- Udi Meiri, on behalf of The Apache Beam team

Re: Sphinx Docs Command Error (:sdks:python:test-suites:tox:pycommon:docs)

0 at 1:17 PM jincheng sun >> wrote: >> >>> I have verified that this issue could be reproduced in my local >>> environment (MacOS) and the solution suggested by Udi could work! >>> >>> Best, >>> Jincheng >>> >>> Udi Meiri 于2020年2月11日周

Re: jira search in chrome omnibox

create the > page Udi and I will add the Firefox section if you agree. > > > > > On Fri, Aug 31, 2018 at 2:31 AM Udi Meiri wrote: > >> Correction: this is the correct URL: >> https://issues.apache.org/jira/secure/QuickSearch.jspa?searchString=%s >> >> It u

Re: Labels on PR

Cool! On Mon, Feb 10, 2020 at 9:27 AM Robert Burke wrote: > +1 to autolabeling > > On Mon, Feb 10, 2020, 9:21 AM Luke Cwik wrote: > >> Nice >> >> On Mon, Feb 10, 2020 at 2:52 AM Alex Van Boxel wrote: >> >>> Ha, cool. I'll have a look at the autolabeler. The infra stuff is not >>> something

Re: [RELEASE] Tracking 2.18

RC1 is almost ready, but Nexus login is down due to LDAP issues with Apache. On Mon, Dec 16, 2019 at 9:53 AM Udi Meiri wrote: > The remaining 4 open blockers all have recently merged cherrypicks (at > least 1 blocker is waiting on verification since it's a release process &

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

SG +1 On Wed, Jan 15, 2020 at 12:59 PM Hannah Jiang wrote: > I have done some research about images released under apache namespace at > docker hub, and here is my proposal. > > Currently, we are using apachebeam as our namespace and each image has its > own repository. Version number is used

Re: [BEAM-9015] Adding pyXX-cloud instead of pyXX-gcp and pyXX-aws

blo Estrada wrote: > >> now back from the holidays, I intend to do this - one of these days. >> >> On Thu, Dec 26, 2019 at 12:51 PM Udi Meiri wrote: >> >>> +1 >>> >>> On Mon, Dec 23, 2019, 17:28 Robert Bradshaw wrote: >>> >>&g

Re: [VOTE] Release 2.18.0, release candidate #1

Dataflow containers have been updated. Test away. On Tue, Jan 14, 2020 at 6:37 PM Udi Meiri wrote: > Here my second take: > > Hi everyone, > Please review and vote on the release candidate #1 for the version 2.18.0, > as follows: > [ ] +1, Approve the release > [

[VOTE] Release 2.18.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #3 for the version 1.2.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: [DISCUSS] Python static type checkers

ype: 24,864 >>> mypy: 1,502,582 >>> >>> So to sum up this email in a sentence: running mypy in pre-commit checks >>> would be beneficial, but making pytype also pass would be a burden with >>> little benefit for the majority of users. >>> &

Re: [VOTE] Release 2.18.0, release candidate #1

few Batch and Streaming Python 3.7 > pipelines using Direct and Dataflow runners, and they all succeeded. > > On Mon, Jan 13, 2020 at 4:09 PM Udi Meiri wrote: > >> Hi everyone, >> Please review and vote on the release candidate #3 for the version 1.2.3, >> as follows: >&

Re: [VOTE] Release 2.18.0, release candidate #1

e blog/website. > > Udi, could you send an updated version of the voting text with TODOs, > template pieces removed? We can discuss changes to the template separately. > My vote is +1 pending an updated vote text. > > On Tue, Jan 14, 2020 at 4:47 PM Udi Meiri wrote: > &

Re: [VOTE] Release 2.18.0, release candidate #1

617819 [10] https://hub.docker.com/u/apachebeam On Tue, Jan 14, 2020 at 6:34 PM Udi Meiri wrote: > Please don't do any Dataflow-based verifications yet, because we'll have > to redo them once new Dataflow containers are built. > > On Tue, Jan 14, 2020 at 6:27 PM Ahmet Altay wrote: &g

Re: [DISCUSS] Autoformat python code with Black

+1 to autoformatting On Wed, Jan 22, 2020 at 9:57 AM Luke Cwik wrote: > +1 to autoformatters. Also the Beam Java SDK went through a one time pass > to apply the spotless formatting. > > On Tue, Jan 21, 2020 at 9:52 PM Ahmet Altay wrote: > >> +1 to autoformatters and yapf. It appears to be a

Re: [VOTE] Release 2.18.0, release candidate #1

> > I recall we had a similar thread before. Please, include the > > release managers in the PRs that are/will be merged into the > > release branch and tag JIRA issues with the all relevant > > releases that should be blocked on it. > &g

Re: [DISCUSS] Autoformat python code with Black

It sounds like there's a consensus for yapf. I volunteer to take this on On Wed, Jan 22, 2020, 10:31 Udi Meiri wrote: > +1 to autoformatting > > On Wed, Jan 22, 2020 at 9:57 AM Luke Cwik wrote: > >> +1 to autoformatters. Also the Beam Java SDK went through a one tim

Re: [DISCUSS] Autoformat python code with Black

Sorry, backing off on this due to time constraints. On Wed, Jan 22, 2020 at 3:39 PM Udi Meiri wrote: > It sounds like there's a consensus for yapf. I volunteer to take this on > > On Wed, Jan 22, 2020, 10:31 Udi Meiri wrote: > >> +1 to autoformatting >> >> On Wed

Re: [VOTE] Release 2.18.0, release candidate #1

faeda897e > > On Tue, Jan 21, 2020 at 9:49 AM Ahmet Altay wrote: > > > > All, could you help with validations and voting? > > > > On Wed, Jan 15, 2020 at 6:14 PM Ahmet Altay wrote: > >> > >> +1, validated the same things, they still work. Thank you. &

Request for new dockerhub repos

Hi, As part of the 2.18 release, we're adding 3 additional containers for Flink. I have write access but since I am not an owner I cannot create new repos. Could someone with access add these? flink1.7_job_server flink1.8_job_server flink1.9_job_server (go to https://hub.docker.com/repositories

release scripts as interactive notebooks?

What does the community think about converting our release scripts to be Jupyter notebooks using bash_kernel? Since these scripts frequently fail (especially for first time releasers), we often need to rerun parts manually. The

Re: Request for new dockerhub repos

Thank you the pushes were successful. On Fri, Jan 10, 2020 at 8:47 AM Hannah Jiang wrote: > Hi Udi > > The repositories are created. Were you added as a maintainer? If not, we > need your docker hub user ID. > > Thanks, > Hannah > > On Thu, Jan 9, 2020 at 5:48 P

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

Congrats Kasia! On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wrote: > Congrats Kasia! And thanks for sharing, Pablo. > > On Mon, Dec 23, 2019 at 4:16 PM Pablo Estrada wrote: > >> Hi everyone, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Kasia Kucharczyk >>

Re: [DISCUSS] Bump the version of GRPC from 1.21.0 to 1.22.0+ (May be the latest 1.26.0?)

Probably best to take one of these existing update bugs: https://issues.apache.org/jira/browse/BEAM-4938 There's no discussion on these bugs, so I'd go with 1.26.0 unless someone else has an objection. On Mon, Dec 23, 2019 at 10:04 PM jincheng sun wrote: > Hi folks, > > When submitting a Python

Re: [BEAM-9015] Adding pyXX-cloud instead of pyXX-gcp and pyXX-aws

+1 On Mon, Dec 23, 2019, 17:28 Robert Bradshaw wrote: > Makes sense to me. > > On Mon, Dec 23, 2019 at 3:33 PM Pablo Estrada wrote: > > > > Hi all, > > a couple of contributors [1][2] have been kind enough to add support for > s3 filesystem[3] for the Python SDK. Part of this involved adding a

[DISCUSS] Python static type checkers

Hi, We recently added mypy to the Jenkins Lint job for PRs (currently ignores errors). Mypy is a static type checker. There's a JIRA for adding another static type checker named pytype https://issues.apache.org/jira/browse/BEAM-9064 I wanted to ask the community their thoughts on this. (see JIRA

Re: Install Jenkins AnsiColor plugin

> seem to be some ansi plugin already installed but I couldn’t get it to > work. > > -chad > > > On Sun, Mar 8, 2020 at 10:52 AM Ismaël Mejía wrote: > >> Did this ever happen? If not what is blocking it? >> >> >> >> On Tue, Oct 22, 2019 at

Re: Website publish jobs fail recently

Hey, I was looking at this today but could not figure it out. The machines we run the publish jobs probably vary from our regular apache-beam-testing Jenkins ones. I tried researching all the reasons why this might be happening but came up empty. On Tue, Apr 14, 2020 at 10:19 AM Kyle Weaver

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

If this process is used in releases we would benefit from running it regularly to ensure it isn't broken and thus delay releases (and add work for the release manager). Does it make sense to put it in postcommit? On Wed, Apr 15, 2020 at 2:30 PM Kyle Weaver wrote: > Looks like the same error as

Re: Website publish jobs fail recently

elease. >> >> On Tue, Apr 14, 2020 at 9:01 PM Kenneth Knowles wrote: >> >>> Indeed, publish jobs have write access to things, which normal builds do >>> not. >>> >>> I suggest reaching out to bui...@apache.org >>> >>>

Re: Website publish jobs fail recently

enneth Knowles wrote: > >> Indeed, publish jobs have write access to things, which normal builds do >> not. >> >> I suggest reaching out to bui...@apache.org >> >> Kenn >> >> On Tue, Apr 14, 2020 at 5:58 PM Udi Meiri wrote: >> >>>

Re: Implementing type hints on multi-output PTransforms

PROPSAL] ..." to this mailing list describing what you want to change should be enough. A document could also work; I'm not aware of one that touches on this. Your PR would need to have an associated JIRA (feel free to take over any of the ones I've mentioned). > Best, > Joshua > > On

Re: [DISCUSS] Let's establish a guideline for using Python type annotations in Beam codebase

I agree with Robert to only put Any where it really can be any type. I'm not sure how much typing we should add. At minimum: external APIs and wherever mypy complains. Ideally I would like to have annotations everywhere, because this reduces uncertainty when modifying existing code. You are

Re: Hello Beam Community!

Welcome! On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang wrote: > Welcome! > > On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay wrote: > >> Welcome Brittany! >> >> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann >> wrote: >> >>> Hello Beam Community! >>> >>> My name is Brittany Hermann and I

Re: Are docker image tags shared within a jenkins worker?

The Python HDFS IT uses the jenkins BUILD_TAG to create unique names: PROJECT_NAME=$(echo hdfs_IT-${BUILD_TAG:-non-jenkins}) The BUILD_TAG is unique and easily traced back to the Jenkins job that made it. It might need some sanitizing though if it contains any invalid characters. On Tue, Mar 24,

Re: [BEAM-9322] Python SDK discussion on correct output tag names

On Thu, Mar 26, 2020 at 10:13 AM Luke Cwik wrote: > The issue seems to be that a PCollection can have a "tag" associated with > it and PTransform expansion can return an arbitrary nested dictionary/tuple > yet we need to figure out what the user wanted as the local name for the > PCollection

<    1   2   3   4   >