Re: gradle clean causes long-running python installs

s, etc? > > Kenn > > On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri wrote: > >> setup.py has requirements like setuptools, which are installed in the >> virtual environment. >> So even running the clean command requires the virtualenv to be set up. >> >>

Re: gradle clean causes long-running python installs

setup.py has requirements like setuptools, which are installed in the virtual environment. So even running the clean command requires the virtualenv to be set up. A possible fix could be to skip :beam-sdks-python:cleanPython if setupVirtualenv has not been run. (perhaps by checking for the

Re: Adding KMS support to generic filesystem interface

monday. I hope waiting a bit is ok with you if you want some feedback > from me. > > I am curious if you considered or are aware of this issue: > BEAM-3821 Support a pluggable key management system (KMS) > https://issues.apache.org/jira/browse/BEAM-3821 > > > On Fri, J

Confluence wiki edit access request

username: udim Thanks! smime.p7s Description: S/MIME Cryptographic Signature

TestDirectRunner for Java?

Hi, I want to use DirectRunner for a new IT I'm writing, since it's testing I/O code that's runner agnostic. The problem is that DirectRunner doesn't have a TestDataflowRunner analog, so features like OnSuccessMatcher aren't available. Any objections to adding a TestDirectRunner class?

Re: Confluence wiki edit access request

bump On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: > username: udim > > Thanks! > smime.p7s Description: S/MIME Cryptographic Signature

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

Some options: - You could wait to assert until after p.waitForFinish(). - You could PAssert using SerializableMatcher and allow any lastModifiedTime. On Tue, Jan 22, 2019 at 3:56 PM Alex Amato wrote: > +Jeff, Eugene, > > Hi Jeff and Eugene, > > I've noticed that Jeff's PR >

Re: FileIOTest.testMatchWatchForNewFiles flakey in java presubmit

for reference? >>> >>>- I guess to this you need to implement the SerializableMatcher >>>interface and use the matcher as an option in the pipeline options. >>> >>> >>> On Tue, Jan 22, 2019 at 4:28 PM Udi Meiri wrote: >>> >

Re: Confluence wiki edit access request

Thank you! On Wed, Jan 23, 2019 at 12:49 PM Ismaël Mejía wrote: > Done. > > On Tue, Jan 22, 2019 at 8:53 PM Udi Meiri wrote: > > > > bump > > > > On Fri, Jan 18, 2019 at 1:57 PM Udi Meiri wrote: > >> > >> username: udim > >>

Adding KMS support to generic filesystem interface

Hi, I'd like to add support for creating files using a cloud Key Management System. A KMS allows you to audit, create, rotate, and disable encryption keys. Both AWS and GCP have such a service.. I wanted to show the community what I've been working on and see if there are any comments or

Re: Issue with publishing maven artefacts locally

On Wed, Dec 12, 2018 at 11:00 AM Scott Wegner wrote: > Thanks for pointing this out Alexy. This seems like we unintentionally > broke something in PR#7197 [1] > > +Garrett Jones , who authored the change. > Garrett can you help investigate? > > I went to check to see if we have any existing

Java performance tests dashboard

Hi Lukasz, I was looking for statistics on I/O performance for writes of many files (~10k) on GCS. I found this dashboard and I have some questions. 1. The tests that are "local filesystem" seem to be running on Dataflow

excessive java precommit logging

Hi all, I'd like to reduce precommit log sizes on Jenkins. For example: https://builds.apache.org/job/beam_PreCommit_Java_Commit/3181/consoleFull is 79M, which makes Chrome sluggish to use on it (tab is constantly using a whole cpu core). I know this might be controversial, but I'd like to

Re: excessive java precommit logging

me >> 3. Download the consoleText output locally and use your favorite log >> viewer that can scale to large files. >> >> [1] https://gradle.com/s/ckhjrjdexpuzm >> >> On Wed, Dec 19, 2018 at 10:42 AM Udi Meiri wrote: >> >>> Hi all, >>> I'd

Re: [VOTE] Release 2.9.0, release candidate #1

For DirectRunner there are regressions in query 7 sql direct runner batch mode <https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424=732741424=411089194> (2x) and streaming mode (5x). On Thu, Dec 6, 2018 at 5:59 PM Udi Meiri wrote: > I see a regression for quer

Re: [VOTE] Release 2.9.0, release candidate #1

I see a regression for query 7 spark runner batch mode on about 2018-11-13. [image: image.png] On Thu, Dec 6, 2018 at 2:46 AM Chamikara Jayalath wrote: > Hi everyone, > > Please review and vote on

Re: Need help regarding memory leak issue

If you're working with Dataflow, it supports this flag: https://github.com/apache/beam/blob/75e9f645c7bec940b87b93f416823b020e4c5f69/sdks/python/apache_beam/options/pipeline_options.py#L602 which uses guppy for heap profiling. On Fri, Nov 16, 2018 at 3:08 PM Ruoyun Huang wrote: > Even tough the

Re: Enforce javadoc comments in public methods?

+1 On Mon, Jan 7, 2019 at 4:49 PM Daniel Oliveira wrote: > +1 > > I like this idea, especially with the line number requirement. The exact > number of lines is debatable, but you could go as low as 10 lines and that > would exclude any trivial setters and getters. Even better might be if it's >

Re: excessive java precommit logging

just want to quickly look for something. >> > >> > 2) curl /consoleText > log.txt >> > less log.txt >> > >> > Here we store the log to a file first, then use 'less' or 'grep' to >> search it. >> > >> > When in 'less', I use '/'

Re: Add code quality checks to pre-commits.

+1 for adding more code quality signals. Could we add them in an advisory-only mode at first? (a warning and not an error) I'm curious how the "technical debt" metric is determined. I'm not familiar with SonarQube. What languages does it support? On Thu, Jan 3, 2019 at 10:19 AM Mikhail

Re: workspace cleanups needed on jenkins master

On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía wrote: > Bringing this subject for awareness to dev@ > We are sadly part of this top. > Does somebody know what this data is? And if we can clean it periodically? > Can somebody with more sysadmin super powers take a look and act on this. > >

Re: workspace cleanups needed on jenkins master

Can you confirm? > > --Mikhail > > Have feedback <http://go/migryz-feedback>? > > > On Thu, Jan 3, 2019 at 1:59 PM Udi Meiri wrote: > >> >> >> On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía wrote: >> >>> Bringing this subject for aw

Re: Cross-language pipelines

Also debugability: collecting logs from each of these systems. On Tue, Jan 22, 2019 at 10:53 AM Chamikara Jayalath wrote: > Thanks Robert. > > On Tue, Jan 22, 2019 at 4:39 AM Robert Bradshaw > wrote: > >> Now that we have the FnAPI, I started playing around with support for >> cross-language

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

I don't use gradle commands for Python development either, because they are slow (no incremental testing). On Fri, Mar 29, 2019 at 9:16 AM Michael Luckey wrote: > > > On Fri, Mar 29, 2019 at 2:31 PM Robert Bradshaw > wrote: > >> On Fri, Mar 29, 2019 at 12:54 PM Michael Luckey >> wrote: >> >

Re: Build blocking on

It shouldn't stall. That's a bug. OTOH, I never use the `build` target. I'll try running that myself. On Mon, Mar 25, 2019, 07:24 Michael Luckey wrote: > Hi, > > trying to run './gradlew build' on vanilla setup, my build consistently > stalls during execution of python gcp tests, e.g. on both

Re: Build blocking on

epo/ cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be performed. https://gradle.com/s/x5zqbc5zwd3bg (Now I remember why I stopped using `build` :/) On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri wrote: > It shouldn't stall. That's a bug. > OTOH, I never use the `build` target. &

Re: Build blocking on

be missing some env settings for gap, dunno. Probably missed >> some docs. >> >> >> >> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey >> wrote: >> >>> Thanks Udi for trying that! >>> >>> In fact, the go dependency r

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congrats Mark! On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay wrote: > Congratulations, Mark!  > > On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson > wrote: > >> Congratulations Mark! >> >> >> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey >> wrote: >> >>> Nice! Congratulations, Mark. >>> >>> On

Re: Build blocking on

are likely to be different to what's in the lock file. >>> >>> This work will be tracked in >>> https://issues.apache.org/jira/browse/BEAM-5379 >>> GoGradle hasn't moved to support the new-go way of handling deps, so my >>> inclination is to simplify to si

Re: Build blocking on

ocks a final submission, rather than the work fortunately. > > On Tue, Mar 26, 2019, 1:08 PM Udi Meiri wrote: > >> "rm -r ~/.gradle/go/repo/" worked for me (there was more than one package >> with issues). >> My ~/.bashrc has >> export GOPATH=$HOME/go

Re: What quick command to catch common issues before pushing a python PR?

Talking about Python: I only know of "./gradlew lint", which include style and some py3 compliance checking. There is no auto-fix like spotlessApply AFAIK. As a side-note, I really dislike our python line continuation indent rule, since pycharm can't be configured to adhere to it and I find

Re: Virtualenv setup issues on new machine

I think gradle is complaining that the path can't be found. Is there more information if you run it with --info? On Thu, Feb 28, 2019, 14:35 Ankur Goenka wrote: > Hi Beamers, > > I am trying build python sdk from a fresh git checkout on a new linux > machine but the setupVirtualEnv task is

Re: precommit tests: please ignore Python_pytest

t; it might not be fully-functional yet. >> >> It would be nice if there was a way to enable those hooks only for a >> testing PR, such that they do not interfere with other PRs. >> >> Perhaps somebody has an idea how to do that? >> >> -Max >> >>

Re: Virtualenv setup issues on new machine

Weird. Is that a known bug? On Thu, Feb 28, 2019 at 3:19 PM Ankur Goenka wrote: > The issue seems to be with "." in the virtualenv path. > virtualenv works after moving from > "/usr/local/google/home/goenka/.local/bin" to "/usr/bin" > > On Thu, Feb

Re: [BEAM-6761] Pydoc is giving cryptic error messages, blocking my PR :(

I think it's referring to the big comment at the top of the sdks/python/apache_beam/testing/metric_result_matchers.py. The line numbers are relative to the beginning of the block. On Fri, Mar 1, 2019 at 2:21 PM Alex Amato wrote: > BEAM-6761

Re: gradle clean causes long-running python installs

n case of the latter, clean >> isn't called anyway, right? >> >> If the former, its probably worth to put more effort into python setup >> anyway, e.g. declaring proper input/outputs... If done correctly (if thats >> possible at all) clean will work out of the box anyw

Re: Added a Jira beginner's guide to the wiki.

My favorite way to navigate JIRA is using a Chrome search engine. You configure it like this: [image: Screenshot from 2019-02-27 17-11-26.png] (URL is: https://issues.apache.org/jira/secure/QuickSearch.jspa?searchString=%s) And search by writing in the location bar: "j BEAM-1234" will take you to

precommit tests: please ignore Python_pytest

Hi all, I'm testing running Python tests using pytest, and I've added a temporary Jenkins jobs that seems to be triggering for PRs, even though I set triggerPathPatterns to an empty list. (file ) Please

Re: Selectively running tests?

ore extensive for Java worker > changes. I noticed that my change runs Go, Python and Java precommits: > https://github.com/apache/beam/pull/8080 > > On Tue, Mar 19, 2019 at 10:27 AM Udi Meiri wrote: > >> Do you have an example PR Pablo? >> >> >> On Mo

Re: Python Datastore client upgrade plan

s a good plan to get us out of a tough situation with an >>> old dependency. >>> >>> On Tue, Oct 16, 2018 at 6:59 PM, Udi Meiri wrote: >>> >>>> Hi, >>>> Sadly upgrading googledatastore -> google-cloud-datastore is >>>>

Re: Selectively running tests?

Do you have an example PR Pablo? On Mon, Mar 18, 2019, 18:23 Alan Myrvold wrote: > The includedRegions was set up as part of > https://issues.apache.org/jira/browse/BEAM-4445 and there are additional > paths added from >

Re: Jenkins slowness

Precommits times for Python and Java have been slowly climbing: http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1=1546974237153=1549566237153=light On Thu, Feb 7, 2019 at 10:54 AM Udi Meiri wrote: > If anyone has done any investigation/is working on this please share. >

Jenkins slowness

If anyone has done any investigation/is working on this please share. I'm investigating Jenkins slowness. I've noticed it happening since yesterday: precommits taking 3 hours to start, phrase commands similarly taking as much time to register. My current theory is that we have a job that's are

Re: Jenkins slowness

There is also excessive python test logging tracked here: https://issues.apache.org/jira/browse/BEAM-6603 On Thu, Feb 7, 2019, 12:23 Udi Meiri I suggest disabling Jacoco and re-enabling the build cache until we can > migrate to Gradle 5. I imagine the migration to v5 is not a simple cha

Re: Jenkins slowness

ems to be lots of work done now which >> previously was cacheable. >> >> Not sure, whether this is the culprit- or part of it -, but I d suggest >> to upgrade to gradle 5 pretty fast. >> >> On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri wrote: >> >>> If anyone

Re: Signing off

Good luck Scott! On Fri, Feb 15, 2019 at 9:32 AM Alex Amato wrote: > Thanks's for your contributions Scott. We will miss you. > > On Fri, Feb 15, 2019 at 7:08 AM Etienne Chauchot > wrote: > >> Thank you for your contributions Scott ! Your new project seems very fun. >> Enjoy ! >> >> Etienne >>

Re: gradle clean causes long-running python installs

gt;>>>> Can we just limit clean to do cleanup?! >>>>> >>>>> >>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw >>>>> wrote: >>>>> >>>>>> Just some background, grpcio-tools is what's used to do the proto >>>

Re: Findbugs -> Spotbugs ?

+1 for spotbugs On Thu, Jan 31, 2019 at 5:09 AM Gleb Kanterov wrote: > Agree, spotbugs brings static checks that aren't covered in error-prone, > it's a good addition. There are few conflicts between error-prone and > spotbugs, for instance, the approach to enum switch exhaustiveness, but it >

Re: [PROPOSAL] Custom JVM initialization for Beam workers

Is this like the way Python SDK allows for a custom setup.py? example: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py On Fri, Apr 12, 2019 at 10:51 AM Lukasz Cwik wrote: > +1 on the use cases that Ahmet pointed out and the solution that

pickler.py issue with nested classes

I was looking at migrating unit tests to pytest and found this test which doesn't pass: https://gist.github.com/udim/a71fcb278b56a9a5b7962f4588e14efb (stack overflow) (requires installing python3.7 and "python3.7 -m pip install pytest".) The same command passes with python2.7 and python3.5. I

Re: pickler.py issue with nested classes

Not sure: my case is using a nested class and the error is a stack overflow (or infinite recursion detection is triggered). It is odd though that they have the same workaround. smime.p7s Description: S/MIME Cryptographic Signature

pytest migration progress

Hi, I'm making progress on the pytest migration here: https://github.com/apache/beam/pull/7949 The PR does not replace nose (yet) - that would require more work and a verification effort to make sure no test gets left behind. - Udi smime.p7s Description: S/MIME Cryptographic Signature

Re: python integration tests flake detection

est results in xunit format. > Would that work? > > [1] https://nose.readthedocs.io/en/latest/plugins/xunit.html > > On Tue, Jun 25, 2019 at 10:04 AM Udi Meiri wrote: > >> The current state of Python post-commit tests is pretty flaky. >> I was wondering if we had any sta

python integration tests flake detection

The current state of Python post-commit tests is pretty flaky. I was wondering if we had any stats for integration tests, to help identify which tests are causing the most failures. Jenkins keeps some history for tests (example

Re: [ANNOUNCE] New committer: Mikhail Gryzykhin

Congrats Mikhail! On Tue, Jun 25, 2019 at 2:32 AM Gleb Kanterov wrote: > Congratulations! > > On Tue, Jun 25, 2019 at 2:03 AM Connell O'Callaghan > wrote: > >> Thomas thank you for sharing this >> >> Congratulations on this Mikhail!!! >> >> On Mon, Jun 24, 2019 at 3:19 PM Kai Jiang wrote:

Re: GitHub checks not running

I think we reached an upper limit on the Jenkins queue length (the grey flat line): [image: graph.png] (https://builds.apache.org/label/beam/load-statistics?type=sec10) On Mon, Jun 17, 2019 at 9:27 AM Anton Kedin wrote: > They are getting triggered now. > > On Mon, Jun 17, 2019 at 9:10 AM

pickling typing types in Python 3.5+

It seems like pickling of typing types is broken in 3.5 and 3.6, fixed in 3.7: https://github.com/python/typing/issues/511 Here are my attempts: https://gist.github.com/udim/ec213305ca865390c391001e8778e91d My ideas: 1. I know that we override type object handling in pickler.py

Re: Quota: In use IP-adresses

We're running up against this limit: "Quota 'IN_USE_ADDRESSES' exceeded. Limit: 750.0 in region us-central1." On Fri, May 24, 2019 at 8:36 AM Valentyn Tymofieiev wrote: > I did this for a few other resources recently (CPU, Disk). If this keeps > being a problem we can lower test parallelism. >

Re: Quota: In use IP-adresses

I opened a support request to increase the quota. On Fri, May 24, 2019 at 9:59 AM Udi Meiri wrote: > We're running up against this limit: "Quota 'IN_USE_ADDRESSES' exceeded. > Limit: 750.0 in region us-central1." > > On Fri, May 24, 2019 at 8:36 AM Valentyn Tymofieie

Plans for Python type hints

Hi, I've written a document, with input from robertwb@, detailing the direction forward I want to take type hints in Python 3. The document contains background, a survey of existing type tools, and example usage. The summary of proposed changes is: 1. Update Beam's type hinting support to

Re: [DISCUSS] Portability representation of schemas

>From a Python type hints perspective, how do schemas fit? Type hints are currently used to determine which coder to use. It seems that given a schema field, it would be useful to be able to convert it to a coder (using URNs?), and to convert the coder into a typing type. This would allow for

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congrats Pablo! On Thu, May 16, 2019 at 9:27 AM Thomas Weise wrote: > Congratulations, Pablo! > > > On Thu, May 16, 2019 at 5:03 AM Katarzyna Kucharczyk < > ka.kucharc...@gmail.com> wrote: > >> Wow, great news!  Congratulations, Pablo! >> >> On Thu, May 16, 2019 at 1:28 PM Michał Walenia < >>

beam_PreCommit_Python_PVR_Flink_Commit most perma-red

FYI, I opened an issue here: https://issues.apache.org/jira/browse/BEAM-7378 Please triage if you know how these tests work. Thanks! smime.p7s Description: S/MIME Cryptographic Signature

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

failed and how to retry just them. On Fri, Apr 26, 2019 at 2:17 PM Alex Amato wrote: > @Udi Meiri , Is this true if the specific tests are > rerun? I don't think we should rerun all tests. > > On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev > wrote: > >> Preinstalling

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

2af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41, >> this was discussed >> [2] >> https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E >> discussed >> >> On Fri, Apr 26, 2019 a

Re: investigating python precommit wordcount_it failure

Correction: it's a postcommit failure On Thu, Apr 18, 2019 at 5:43 PM Udi Meiri wrote: > in https://issues.apache.org/jira/browse/BEAM-7111 > > If anyone has state please lmk > smime.p7s Description: S/MIME Cryptographic Signature

investigating python precommit wordcount_it failure

in https://issues.apache.org/jira/browse/BEAM-7111 If anyone has state please lmk smime.p7s Description: S/MIME Cryptographic Signature

Re: investigating python precommit wordcount_it failure

I believe these are separate issues. BEAM-7111 is about wordcount_it_test failing on direct runner in streaming mode On Thu, Apr 18, 2019 at 8:09 PM Valentyn Tymofieiev wrote: > I am working on a postcommit worcount it failure in BEAM-7063. > > On Thu, Apr 18, 2019 at 6:05 PM Udi Mei

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

Congrats everyone! On Thu, May 2, 2019 at 9:55 AM Ahmet Altay wrote: > Congratulations! > > On Thu, May 2, 2019 at 9:54 AM Yifan Zou wrote: > >> Congratulations! Well deserved! >> >> On Thu, May 2, 2019 at 9:37 AM Rui Wang wrote: >> >>> Congratulations! >>> >>> >>> -Rui >>> >>> On Thu, May 2,

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

That runs the risk of overloading our test machines when the build goes red. On Fri, Apr 26, 2019 at 11:29 AM Alex Amato wrote: > It would be ideal to not need manual steps. If known flakey tests can be > auto retried that would be a great improvement. > > On Fri, Apr 26, 2019 at 11:24 AM

apache-beam-jenkins-15 out of disk

Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648 Can someone investigate what's going on? smime.p7s Description: S/MIME Cryptographic Signature

Re: python integration tests flake detection

In lieu of doing a migration to pytest, which is a large effort, I'm trying to do the same using nose. Opened https://issues.apache.org/jira/browse/BEAM-7641 On Tue, Jun 25, 2019 at 4:01 PM Udi Meiri wrote: > I was thinking that our test infrastructure could use an upgrade to pytest. >

Re: apache-beam-jenkins-15 out of disk

t; 965.7MB 339.2MB 0 >>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_testlatest >>>>ab181c49d56e4 days ago 1.305GB >>>> 965.7MB 339.2MB 0 >>>> hdfs_it-jenkins-b

Re: Phrase triggering jobs problem

Still happening for me too. On Wed, Jul 10, 2019 at 10:40 AM Lukasz Cwik wrote: > This has happened in the past. Usually there is some issue where Jenkins > isn't notified of new PRs by Github or doesn't see the PR phrases and hence > Jenkins sits around idle. This is usually fixed after a few

Re: Phrase triggering jobs problem

I've opened a bug: https://issues.apache.org/jira/browse/BEAM-7723 If anyone is working on this please assign yourself On Wed, Jul 10, 2019 at 5:57 PM Udi Meiri wrote: > Thanks Kenn. > > On Wed, Jul 10, 2019 at 3:31 PM Kenneth Knowles wrote: > >> Just noticed this thread. In

Re: Phrase triggering jobs problem

Opened https://issues.apache.org/jira/browse/BEAM-7725 for migration off the old plugin onto the new (already deprecated I might add) plugin. Any takers? On Thu, Jul 11, 2019 at 10:53 AM Udi Meiri wrote: > Okay, phrase triggering is working again (they re-enabled the plugin). See > notes

python precommits failing at head

This is due to https://github.com/apache/beam/pull/8969 and https://github.com/apache/beam/pull/8934 being merged today. Fix is here: https://github.com/apache/beam/pull/9044 smime.p7s Description: S/MIME Cryptographic Signature

Re: [VOTE] Vendored dependencies release process

I left some comments. Being new to the Beam releasing process, my question might be trivial to someone actually performing the release. On Tue, Jul 2, 2019 at 4:49 PM Lukasz Cwik wrote: > Please vote based on the vendored dependencies release process as > discussed[1] and documented[2]. > >

Re: Dataflow IT failures on Python being investigated internally

It seems that at least some of the failures to start pipelines on DF were due to a CMEK misconfiguration. On Tue, Jul 2, 2019 at 6:45 PM Udi Meiri wrote: > The failures are of the type where the pipeline fails very quickly (10 > seconds) and there's a "Pipeline execution failed&qu

Re: Dataflow IT failures on Python being investigated internally

https://issues.apache.org/jira/browse/BEAM-7687 On Wed, Jul 3, 2019 at 11:57 AM Udi Meiri wrote: > It seems that at least some of the failures to start pipelines on DF were > due to a CMEK misconfiguration. > > On Tue, Jul 2, 2019 at 6:45 PM Udi Meiri wrote: > >> The fai

Dataflow IT failures on Python being investigated internally

The failures are of the type where the pipeline fails very quickly (10 seconds) and there's a "Pipeline execution failed" or "Workflow failed" error. smime.p7s Description: S/MIME Cryptographic Signature

Re: Stop using Perfkit Benchmarker tool in all tests?

The Python 3 incompatibility is reason enough to move off of Perfkit. (+1) On Mon, Jul 8, 2019 at 9:49 AM Mark Liu wrote: > Thanks for summarizing this discussion and post in dev list. I was closely > working on Python performance tests and those Perfkit problems are really > painful. So +1 to

Re: [DISCUSS] Contributor guidelines for iterating on PRs: when to squash commits.

I think there are already some guidelines here: https://beam.apache.org/contribute/committer-guide/#pull-request-review-objectives (maybe we could point to them from the PR template?) Yes, it is acceptable to ask for squash or if it's ok to squash to a single commit. On Mon, Jul 8, 2019 at 11:14

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congrats Yifan! On Mon, Apr 22, 2019 at 11:04 AM Valentyn Tymofieiev wrote: > Congratulations, Yifan! Thanks a lot for your continued contributions to > Beam. > > On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu wrote: > >> Congratulations Yifan! >> >> On Mon, Apr 22, 2019 at 10:17 AM Chamikara

Re: [ANNOUNCE] New committer: Robert Burke

Congrats Robert B.! On Tue, Jul 16, 2019 at 10:23 AM Ahmet Altay wrote: > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Robert > Burke. > > Robert has been contributing to Beam and actively involved in the > community for over a year. He has been actively

[PSA] Python: don't forget to close() your gRPC channels

https://github.com/grpc/grpc/issues/12531 I imagine this mostly affects GCP connectors, but perhaps grpcio is used elsewhere? (I haven't checked) Background is that I'm updating the minimum grpcio version to 1.12.1 ( https://github.com/grpc/grpc/releases/tag/v1.12.0). smime.p7s Description:

Python question about save_main_session

Hi, I'm trying to get pytest with the xdist plugin to run Beam tests. The issue is with save_main_session and a dependency of pytest-xdist called execnet, which triggers this error: *apache_beam/examples/complete/tfidf.py*:212: in run*output | 'write' >>

Re: Python question about save_main_session

re-writing the test to avoid save_main_session. > > On Fri, Aug 23, 2019 at 11:57 AM Udi Meiri wrote: > >> Hi, >> I'm trying to get pytest with the xdist plugin to run Beam tests. The >> issue is with save_main_session and a dependency of pytest-xdist called >>

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

Congrats! On Tue, Aug 27, 2019 at 9:50 AM Yichi Zhang wrote: > Congrats Valentyn! > > On Tue, Aug 27, 2019 at 7:55 AM Valentyn Tymofieiev > wrote: > >> Thank you everyone! >> >> On Tue, Aug 27, 2019 at 2:57 AM Alexey Romanenko < >> aromanenko@gmail.com> wrote: >> >>> Congrats, well

Re: [discuss] How we support our users on Slack / Mailing list / StackOverflow

I don't go on Slack, but I will be notified of mentions. It has the advantage of being an informal space. SO can feel just as intimidating as the mailing list IMO. Unlike the others, it doesn't lend itself very well to discussions (you can only post comments or answers). On Fri, Sep 6, 2019 at

Python postcommits broken: crossLanguagePythonJavaFlink

I opened a bug here: https://issues.apache.org/jira/browse/BEAM-7924 but I don't know who's the best person to take a look. Could someone assign this please? smime.p7s Description: S/MIME Cryptographic Signature

Re: [Update] Beam 2.15 Release Progress

https://github.com/apache/beam/pull/9240 has been merged On Wed, Aug 7, 2019 at 12:33 PM Anton Kedin wrote: > Perf regression is seemingly gone now. If this is caused by a PR we might > want to find out which one and cherry-pick it into the release. > > Regards, > Anton > > On Tue, Aug 6, 2019

Re: [ANNOUNCE] New committer: Kyle Weaver

Congrats Kyle! On Tue, Aug 6, 2019 at 2:00 PM Melissa Pashniak wrote: > Congratulations Kyle! > > On Tue, Aug 6, 2019 at 1:36 PM Yichi Zhang wrote: > >> Congrats Kyle! >> >> On Tue, Aug 6, 2019 at 1:29 PM Aizhamal Nurmamat kyzy < >> aizha...@google.com> wrote: >> >>> Thank you, Kyle! And

Re: Jira email notifications

Is your email set correctly? You can see it if you hit the edit button for profile details. [image: UaSgEcLBGeM.png] On Wed, Aug 7, 2019 at 5:16 PM sridhar inuog wrote: > Yes, I am already on the "Watchers" list > > On Wed, Aug 7, 2019 at 7:13 PM Pablo Estrada wrote: > >> Have you tried

Re: Allowing firewalled/offline builds of Beam

You can download it here: https://gradle.org/releases/ and run it instead of using the wrapper. Example: $ cd $ unzip Downloads/gradle-5.5.1-bin.zip $ cd ~/src/beam $ ~/gradle-5.5.1/bin/gradle lint On Thu, Aug 8, 2019 at 10:52 AM Chad Dombrova wrote: > This topic came up in another thread, so

precommits failing on git clean:

https://issues.apache.org/jira/browse/BEAM-7788 smime.p7s Description: S/MIME Cryptographic Signature

Re: precommits failing on git clean:

e it's pretty broad. > > https://github.com/apache/beam/pull/9096 > > On Fri, Jul 19, 2019, 11:53 AM Udi Meiri wrote: > >> https://issues.apache.org/jira/browse/BEAM-7788 >> > smime.p7s Description: S/MIME Cryptographic Signature

Re: On Auto-creating GCS buckets on behalf of users

ting flags (including many SDK-specific flags) would make it difficult > to implement. In addition, uniform argument names wouldn't necessarily > ensure uniform implementation. > >> > >> Kyle Weaver | Software Engineer | github.com/ibzib | > kcwea...@google.com > >> > >

Re: On Auto-creating GCS buckets on behalf of users

Java SDK creates one regional bucket per project and region combination . So it's not a

[REQUEST] Python Tests (pre/post-commits) Status

Hi, I've been trying to merge several Python PRs in the past weeks, but Jenkins pre- and post-commit jobs have been red all this time due to various reasons. I have a proposal in mind to help deal with this, but it can't happen without cooperation from a majority of committers. The request from

<    1   2   3   4   >