Re: Proposal: Fn API SDK harness container contract

This message landed in my SPAM folder, so bumping it up in case anyone missed it for the same reason. On Fri, Aug 11, 2017 at 11:45 AM, Henning Rohde wrote: > Hello everyone, > > First, I'm Henning and I have been working on the container model used by > Google

Proposal: introduce an option to pass SDK harness container image in Beam SDKs

Hi Beam Dev, Fn API overview [1] suggests using a docker container to encapsulate the language-specific SDK harness. I'd like to propose to add a pipeline option to Beam SDKs, to pass the container image to a runner. For example, in Python SDK this option can be called

Java pre/post commit test suite breakage

Hi Beam-Dev, It's been >5 days since the last successful run of a beam_PreCommit_Java_MavenInstall build[1] and >4 days since last successful run of beam_PreCommit_Java_MavenInstall[2]. Looking at build logs I see following problems. 1. After October 17, postcommit builds started to fail with

Re: [VOTE] Use Gradle for Apache Beam developmental processes

+1 I support the process change On Tue, Nov 28, 2017 at 9:56 AM, Kenneth Knowles wrote: > +1 (binding) > > On Tue, Nov 28, 2017 at 9:55 AM, Lukasz Cwik wrote: > >> This is a procedural vote for migrating to use Gradle for all our >> development related

Re: [VOTE] Release 2.2.0, release candidate #4

I have verified: SHA & MD5 signatures of Python artifacts in [2], and checked Python side of the validation checklist on Linux. There is one known issue in UserScore example for Dataflow runner. The issue has been fixed on master branch and does not require a cherry-pick at this point. A

Re: [VOTE] Release 2.2.0, release candidate #4

Apex, DirectRunner, Flink & Spark on Linux. > > The Gearpump runner is yet to have a quickstart listed on our website. > Adding the quickstart is already represented by this existing issue: > https://issues.apache.org/jira/browse/BEAM-2692 > > On Fri, Nov 17, 2017 at

Re: [VOTE] Release 2.2.0, release candidate #3

with release 2.1.0 and I did not see any > performance regression. > > Best > > Etienne > > > Le 09/11/2017 à 03:13, Valentyn Tymofieiev a écrit : > >> I looked at Python side of Dataflow & Direct runners on Linux. There are >> two findings: >> >

Re: [VOTE] Release 2.2.0, release candidate #2

Confirming Ismaël's finding - I also see this error and it did not see it on a candidate that was in the staging area yesterday. On Wed, Nov 8, 2017 at 9:07 AM, Ismaël Mejía wrote: > I tested the python version of the release I just created a new > virtualenv and run > >

Re: [VOTE] Release 2.2.0, release candidate #2

1:55 AM, Jean-Baptiste Onofré > > ><j...@nanthrax.net> > > >> wrote: > > >> > Let me take a look. Afair I didn't touch those files in last > > >release. > > >> > > > >> > I keep you posted. > > >> &

Re: [VOTE] Release 2.2.0, release candidate #3

I looked at Python side of Dataflow & Direct runners on Linux. There are two findings: 1. One of the mobile gaming examples did not pass for Dataflow runner, addressed in: https://github.com/apache/beam/pull/4102

Re: How to cope with Maven transient network issues?

Has this ever been brought up in Maven dev community? Perhaps they have some suggestions. It sounds like a reasonable feature request. On Fri, Dec 1, 2017 at 1:18 PM, Kenneth Knowles wrote: > I've repeatedly searched around for a way to just add proper retry to > maven or

Re: [VOTE] Release 2.2.0, release candidate #1

At this point it's safe to proceed cutting the RC2, I don't anticipate further changes to release branch to resolve the blocking issues. On Mon, Oct 30, 2017 at 10:35 PM, Jean-Baptiste Onofré wrote: > Great. Thanks for the update. > > Regards > JB > > On Oct 31, 2017, 01:04,

Re: Documenting Github PR jenkins trigger phrases

+1 to writing a Beam Jenkins spellbook. I have observed that Jenkins commands sometimes don't work for the first time, why could this be? Do end of lines at the end of command matter? On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud wrote: > It would be great to have the

Re: Missing dep in python sdk? "ImportError: No module named builtins"

Adding dependency on future in the setup.py fixes this for me, sending https://github.com/apache/beam/pull/5379. On Tue, May 15, 2018 at 3:58 PM, Pablo Estrada wrote: > I believe this is a consequence of the recent work to futurize Python > code[1]. I believe Robbe and

Re: Looking for contributors for Python 3 support

Thanks, Robbe! I also sent https://github.com/apache/beam-site/pull/441 to cover this information on Beam Site for better visibility. On Fri, May 11, 2018 at 6:36 AM, Robbe Sneyders wrote: > Hello everyone, > > We have started adding Python 3 support to Beam. It took a

Re: Survey: what is everyone working on that you want to share?

Hi Kenn, I sent https://github.com/apache/beam-site/pull/441 to cover efforts related to Python 3 support in Beam. Thanks, Valentyn On Tue, May 15, 2018 at 10:27 AM, David Morávek wrote: > Hi Kenn, > > Java 8 DSL > > JIRA: >

Re: [VOTE] Code Review Process

+1 On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay wrote: > +1 > > On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles wrote: > >> +1 >> >> On Fri, Jun 1, 2018 at 10:25 AM Thomas Groh wrote: >> >>> As we seem to largely have consensus in "Reducing Committer Load for >>> Code Reviews"[1], this is a

Re: [VOTE] Release 2.3.0, release candidate #2

Yes (thanks Kenn!): https://s.apache.org/beam-2.3.0-release-validation On Thu, Feb 8, 2018 at 10:14 AM, Eugene Kirpichov wrote: > Do we have a release validation spreadsheet for this one? > > On Thu, Feb 8, 2018 at 9:30 AM Ahmet Altay wrote: > >> +1 >>

Re: [VOTE] Release 2.3.0, release candidate #3

+1, Validated Python - Mobile game walkthrough, Python - Quickstart (Streaming Alpha). On Wed, Feb 14, 2018 at 10:42 AM, Alan Myrvold wrote: > +1 Validated java quickstarts for direct, dataflow, apex, flink, and spark. > > On Wed, Feb 14, 2018 at 9:21 AM, Lukasz Cwik

Re: Pushing daily/test containers for python

The GCR repository can be configured with public pull access, which I think will be required to use the container. On Thu, Dec 21, 2017 at 2:34 AM, David Sabater Dinter < david.saba...@gmail.com> wrote: > +1 > Hi, > It makes sense to use GCR (locality with GCP services and works like any > other

Re: "Maven JVM terminated unexpectedly with exit code 137"

I believe committers only have access to Jenkins workspaces, so you can try to open a ticket with INFRA and ask them to ssh and run particular commands on Jenkins machine. Related: https://issues.apache.org/jira/browse/BEAM-3057 On Wed, Dec 20, 2017 at 2:50 PM, Udi Meiri

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

Just wanted to highlight again to folks who are interested to help with qualifying the release: release validation checklist has 2.6.0 tab that shows what has been tested so far for this RC.

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #1

towards > (and thank those progressing) structured monitoring of performance across > releases. > > > > On Fri, Aug 3, 2018 at 9:32 AM, Valentyn Tymofieiev > wrote: > >> Just wanted to highlight again to folks who are interested to help with >> qualifying the re

Re: [VOTE] Apache Beam, version 2.6.0, release candidate #2

+1 (non-binding). I verified Python quick start on Direct & Dataflow runners on Windows, and a streaming wordcount pipeline on Linux. I noticed wheel files have ".rc2" suffix in their names, which was not the case for 2.5.0 but was also the case for 2.4.0. So I assume that's harmless. I also

Re: jira search in chrome omnibox

Thanks for sharing. I have also found useful following custom search query for PRs: https://github.com/apache/beam/pulls?q=is%3Apr%20%s Sample usage: type 'pr', space, type: 'author:tvalentyn'. You could also incorporate 'author:' into the query:

Re: Python 3: final step

This is awesome! Kudos to Robbe and Matthias who have been pushing this forward! On Wed, Sep 5, 2018 at 9:45 AM Charles Chen wrote: > This is great! Feel free to add me as a reviewer. > > On Wed, Sep 5, 2018 at 9:38 AM Andrew Pilloud wrote: > >> Cool! I know very little about Python 3, but

Re: New Post Commit Task fails in SetupVirtualEnv when running on Jenkins

Thanks for sharing. This may be caused by https://github.com/tox-dev/tox/issues/649. On Fri, Sep 7, 2018 at 4:48 PM Ankur Goenka wrote: > It seems that the issue was with the length of file name which was > >

Re: [VOTE] Release 2.4.0, release candidate #2

-1. Checked Python Quickstarts (Passed) and Python MobileGaming on DirectRunner. I observe an issue in BQ sink for hourly teams score example: https://issues.apache.org/jira/browse/BEAM-3824 On Fri, Mar 9, 2018 at 10:49 AM, Lukasz Cwik wrote: > I checked that word count

Re: slack @the-asf?

Do we need to ask ASF admin to periodically regenerate the invite link? On Sun, Mar 18, 2018 at 11:58 AM, Jean-Baptiste Onofré wrote: > Just back from flights. Thanks. > > Sorry for the delay, I was busy. > > Regards > JB > Le 18 mars 2018, à 19:57, Innocent Djiofack

Re: [VOTE] Release 2.4.0, release candidate #3

much luck either. Since I am not sure when these examples last worked and warnings shows the examples are using deprecated constructs, I suspect the issue is with examples, and we can address it separately from 2.4.0 release. On Mon, Mar 19, 2018 at 7:21 PM, Valentyn Tymofieiev <valen...@google.

Re: [VOTE] Release 2.4.0, release candidate #3

+1. Ran a Python Streaming wordcount pipeline on Direct and Dataflow runners and Batch mobile gaming examples on Dataflow runner. On Mon, Mar 19, 2018 at 6:02 PM, Alan Myrvold wrote: > +1 I ran the java quickstarts against 2.4.0 and they passed. > ./gradlew

Re: Python typing library is not provisional in Python 3.7

Hi Manu, We have added Python 3 environment to our tests see [1], and we are actively making changes to Beam code to make it Python 3-compatible. We are enabling tests module by module, although we have to disable some of the tests initially, when failures are likely introduced in other modules.

Re: Python 3: final step

ins ? It’s Python > 3.5.2 now. > > Thanks, > Manu Zhang > On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev , > wrote: > > I have put together a guide [1] to help get started with investigating > Python 3-related test failures that may be helpful for new folks join

Re: [PROPOSAL] Using Bazel and Docker for Python SDK development and tests

Thanks for starting the discussion. For Python 3, we specifically run tests only in Python 3.5, and as was shown in https://issues.apache.org/jira/browse/BEAM-5663, this does not provide sufficient coverage for Python 3 compatibility. To accelerate Py3 compatibility effort, we need to start

Re: Python 3: final step

nks Valentyn. Note some test failing issues are covered by “Finish > Python 3 porting for *** module”, e.g. > https://issues.apache.org/jira/browse/BEAM-5315. > > Manu > 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev ，写道： > > Hi Rakesh and Manu, > > Thanks to both of you for offer

Re: Python 3: final step

On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev wrote: > I agree there is some overlap between JIRAs that track individual failures > and module-level JIRAs. We originally wanted to do the conversion on a > module-by-module basis, however we learned that test failures in some > mod

Re: Python typing library is not provisional in Python 3.7

che/beam/pull/6397> module. Yes, I'd love to >> contribute more. >> The issue I raise here is specifically about Python 3.7, where the >> dependency on typing library would fail all the tests. >> Do you know which version of Python 3 is setup for our tests ? >> >> Man

Re: Python 3: final step

Hi Rakesh and Manu, Thanks to both of you for offering help (in different threads). It's great to see that more and more people get involved with helping to make Beam Python 3 compatible! There are a few PRs in flight, and several people in the community actively work on Python 3 support now. I

Re: Python PreCommit broken

The change https://github.com/apache/beam/pull/6424 was not deemed particularly risky, and it's purpose was adding more tests to precommit test suite. There was a green Precommit signal on Jenkins, and I believe Postcommit test suite (at the same time) wouldn't catch this. The reason the breakage

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1, releasing containers is a useful process that we need to build in Beam and it is required for FnApi users. Among other reasons, having officially-released Beam SDK harness container images will make it easier for users to do simple customizations to container images, as they will be able to

Re: :beam-sdks-python:docs fails with docs invocation failure

Hi, I just opened https://issues.apache.org/jira/browse/BEAM-6489, and plan to look into this. On Tue, Jan 22, 2019 at 3:13 PM Mikhail Gryzykhin wrote: > Hi everyone, > > I see python precommit tests fail with > > no such option: --process-dependency-links > > > Supposedly when invoking pip

Shall we use "tenacity" library to help deflake some of Python tests using retry logic?

I have been looking at a few test flakes in Python SDK recently, and some of them can benefit from a simple retry logic. See PR #7455 for an example[1]. I would not recommend retrying by default for all tests, or mechanically adding a retry to every test that we see flaking: some legitimate bugs

Re: Shall we use "tenacity" library to help deflake some of Python tests using retry logic?

tained library. >>> >>> We should use this sparingly, but it is helpful for algorithms that >>> have an intrinsic amount of randomness/noise (e.g. the sampling code) >>> to reduce a 1% chance of failure to a 1 in a million. >>> >>> On Fri, Jan 11

Re: Run Python3 tests in miniconda

Hi Manu, I saw your PR https://github.com/apache/beam/pull/6397 (thanks a lot!) - did you resolve the issue with setup? I have not tried Miniconda with Beam myself. Perhaps you could describe your setup in more detail, so that I (or other folks on the list) could try to reproduce the issue?

Re: Python36/37 not installed on Beam2 and Beam12?

I asked them yesterday on Slack, and commented on existing issue https://issues.apache.org/jira/browse/INFRA-17335, however didn't receive a response. We can try opening another infra ticket. Mark, perhaps you can quote/+1 my message from yesterday in thier slack channel :) ? On Wed, Mar 20, 2019

Re: Writing bytes to BigQuery with beam

an option ? I think when table is not available users have to > provide a schema to create a new table. > > >> >> >> On Wed, Mar 20, 2019 at 3:44 PM Valentyn Tymofieiev >> wrote: >> >>> Thanks Juta for detailed analysis. >>> >>&g

Re: Writing bytes to BigQuery with beam

be enabled by default and we should clearly advertise that bytes type > is not supported (or support it with extra information). Just my 2 cents. > > Thanks, > Cham > > >> >> On Mon, Mar 25, 2019 at 11:01 AM Chamikara Jayalath >> wrote: >> >>> &g

Re: Writing bytes to BigQuery with beam

We received feedback on https://issuetracker.google.com/issues/129006689 - BQ developers say that schema identification is done and they discourage to use schema autodetection in tables using BYTES. In light of this, I think may be fair to recommend Beam users to specify BQ schemas as well when

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations, Mark! Thanks for your contributions, in particular for your efforts to parallelize test execution for Python SDK and increase the speed of Python precommit checks. On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC

Re: Deprecating Avro for fastavro on Python 3

I would suggest to make fastavro a default option on Python 3, for the lack of alternative at the moment, but keep current default on Python 2. I would also keep both avro and avro-python3 dependencies and associated codepaths. This way, we will gradually increase the usage of fastavro, but keep

Re: Deprecating Avro for fastavro on Python 3

Thanks, Robbe and Frederik, for raising this. Over the course of making Beam Python 3 compatible this is at least the second time [1] we have to deal with an error in avro-python3 package. The release cadence of Apache Avro (1 release a year) is concerning to me [2]. Even if we have a new release

Re: Python sdist broken?

This most likely caused by https://github.com/apache/beam/pull/7675. I suggest we revert and roll-forward with a fix. We should also understand why this didn't surface in the precommit tests. On Thu, Mar 14, 2019 at 4:44 AM Michael Luckey wrote: > Hi, > > while trying to get build working on my

Re: hi from DevRel land

Hi Reza! Welcome to Beam. Very nice to have you onboard. Btw, the link seems broken. Thanks, Valentyn On Tue, Mar 12, 2019 at 6:04 AM Reza Ardeshir Rokni wrote: > Hi Folks, > > Just wanted to say hi to the good folks in the Beam community in my new > capacity as a Developer advocate for

Re: Python sdist broken?

>> wrote: >> >>> Thanks for that revert! >>> >>> I d really love to get those proposed changes of #7675 in, as this >>> '--keep-temp' also causes some troubles during build. >>> >>> Please let me know if/how I could support here. >&

Re: Writing bytes to BigQuery with beam

Thanks Juta for detailed analysis. I reached out to BigQuery team to improve documentation around treatment of Bytes and reported the issue that schema autodetection does not work for BYTES in GCP issue tracker

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

Congratulations, Boyuan! On Thu, Apr 11, 2019 at 4:54 PM Ankur Goenka wrote: > Congrats Boyuan! > > On Thu, Apr 11, 2019 at 4:52 PM Mark Liu wrote: > >> Congrats Boyuan! >> >> On Thu, Apr 11, 2019 at 9:53 AM Alexey Romanenko < >> aromanenko@gmail.com> wrote: >> >>> > since early 2018 >>> >

Re: pickler.py issue with nested classes

This looks very similar to https://github.com/uqfoundation/dill/issues/300, however we observed that bug on Python 3, and not on Python 2.7. On Tue, Apr 16, 2019 at 10:58 AM Udi Meiri wrote: > I was looking at migrating unit tests to pytest and found this test which > doesn't pass: >

What is preferred way to label Jira issues intended for new contributors?

I am seeing at least 4 labels in JIRA that can be well applicable for tagging issues for someone getting started on Beam: beginner, easyfix, newbie, starter. Are they materially different? Is it documented somewhere? If not, should we perhaps reduce this list? Thanks, Valentyn

Re: What is preferred way to label Jira issues intended for new contributors?

how would you maintain it? > > Kenn > > On Wed, Apr 17, 2019 at 2:42 PM Valentyn Tymofieiev > wrote: > >> I am seeing at least 4 labels in JIRA that can be well applicable for >> tagging issues for someone getting started on Beam: beginner, easyfix, >> newbie,

Insufficient CPU quota in apache-beam-testing causes test flakes

FYI, I have recently observed a large amount of test failures in Beam test suites where Dataflow Jobs failed due to a lack of CPU quota in apache-beam-testing project. We have been adding new suites for Python 3.x versions, which may have contributed to this. problem. I have not investigated yet

Re: Insufficient CPU quota in apache-beam-testing causes test flakes

e're working on the migrating the > rest of old Jenkins agents, and the old instances will be removed once > finish. That should relieve the pain of quota. > > Yifan > > On Tue, Apr 16, 2019 at 1:58 PM Valentyn Tymofieiev > wrote: > >> FYI, I have recently observed a large am

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

Congrats, Pablo! On Wed, May 15, 2019 at 10:41 AM Yifan Zou wrote: > Congratulations, Pablo! > > *From: *Maximilian Michels > *Date: *Wed, May 15, 2019 at 2:06 AM > *To: * > > Congrats Pablo! Thank you for your help to grow the Beam community! >> >> On 15.05.19 10:33, Tim Robertson wrote: >>

Re: Writing bytes to BigQuery with beam

Also, I filed https://issues.apache.org/jira/browse/BEAM-7346 to add more tests to Go SDK and verify the consistency of BQ IO behavior w.r.t. handling BYTES. On Thu, May 16, 2019 at 4:42 PM Valentyn Tymofieiev wrote: > > On Thu, May 16, 2019 at 1:12 PM Chamikara Jayalath &

Re: Writing bytes to BigQuery with beam

On Thu, May 16, 2019 at 1:12 PM Chamikara Jayalath wrote: > > > On Wed, May 15, 2019 at 12:26 PM Valentyn Tymofieiev > wrote: > >> I took a closer look at BigQuery IO implementation in Beam SDK and >> Dataflow runner while reviewing a few PRs to address BEAM-6

Re: Writing bytes to BigQuery with beam

Tue, Mar 26, 2019 at 11:27 AM Pablo Estrada wrote: > Sure, we can make users explicitly ask for schema autodetection, instead > of it being the default when no schema is provided. I think that's > reasonable. > > > On Mon, Mar 25, 2019, 7:19 PM Valentyn Tymofieiev > wrote:

Re: Writing bytes to BigQuery with beam

, it is desirable to have a consistent UX across SDK, especially given that we are working on adding support for cross-language pipelines ( https://beam.apache.org/roadmap/connectors-multi-sdk/). On Wed, May 15, 2019 at 12:26 PM Valentyn Tymofieiev wrote: > I took a closer look at BigQuery IO implementat

Re: [VOTE] Release 2.13.0, release candidate #1

ull/8668 >>>>> in to 2.13.0 since it fixes a logging setup issue on Dataflow (BEAM-7406). >>>>> >>>>> On Tue, May 28, 2019 at 10:22 AM Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>

Re: [VOTE] Release 2.13.0, release candidate #1

Thanks, Juta Staes, for reporting this issue. On Tue, May 28, 2019, 9:19 AM Valentyn Tymofieiev wrote: > -1. > I would like us to fix > https://issues.apache.org/jira/browse/BEAM-7439 for 2.13.0. It is a > regression that happened in 2.12.0, but was not caught by existing tests

Re: [VOTE] Release 2.13.0, release candidate #1

-1. I would like us to fix https://issues.apache.org/jira/browse/BEAM-7439 for 2.13.0. It is a regression that happened in 2.12.0, but was not caught by existing tests. Thanks, Valentyn On Wed, May 22, 2019, 4:30 PM Ankur Goenka wrote: > Hi everyone, > > Please review and vote on the release

Re: [PROPOSAL] Standardize Gradle structure in Python SDK

Hey Mark & others, We've been following the structure proposed in this thread to extend test coverage for Beam Python SDK on Python 3.5, 3.6, 3.7 interpreters, see [1]. This structure allowed us to add 3.x suites without slowing down the pre/postcommit execution time. We can actually see a drop

Re: [VOTE] Release 2.13.0, release candidate #2

Thanks, Ankur, for driving the release. Do we have a draft of user-friendly summary of release notes with high-level changes somewhere? If so, please tag me on a document or a PR, or post the link in this thread. Thank you! On Mon, Jun 3, 2019 at 5:38 PM Ankur Goenka wrote: > +1 > Thanks for

Re: [VOTE] Release 2.13.0, release candidate #1

help me out with one inbuilt function > which is there in apache spark with name as Monotonically increasing id is > there any smilar kind in apache beam if so please revert it with more in > detail thanks in advance > > On Tue, May 28, 2019 at 9:49 PM Valentyn Tymofieiev > wrot

Re: Plan for dropping python 2 support

I like the update Ismaël referenced [1], I think we should prepare a similar update for Beam users. I would propose the following: - Designate last LTS release that we will have in 2019 to be the last LTS release with Python 2 support. - Add a Beam-specific deprecation warning on Python 2 starting

Re: Plan for dropping python 2 support

g/jira/browse/BEAM-1251 [2] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245=detail On Tue, Jun 18, 2019 at 12:03 AM Valentyn Tymofieiev wrote: > I like the update Ismaël referenced [1], I think we should prepare a > similar update for Beam users. I would propose the foll

Re: Blogpost Beam Summit 2019

Hi Juta, Thanks for sharing! You can also consider sending it to user mailing list. Note that Datastore IO now supports Python 3: https://lists.apache.org/thread.html/0a1fdb9b6b42b08a82eebf3b5b7898893ca236b5d3bb5c4751664034@%3Cuser.beam.apache.org%3E . Thanks, Valentyn On Tue, Jun 25, 2019 at

Re: Plan for dropping python 2 support

eleases in 2019 where we can mark Python 2 support as deprecated and remove it with the first release in 2020. > > On Tue, Jun 18, 2019 at 12:08 AM Valentyn Tymofieiev > wrote: > >> To give a better understanding where we are w.r.t. Python 3, I'd like to >> give a quick ove

Jenkins not triggering test runs? Be careful with merges.

Is something happening with Jenkins infra? On a recent PR of mine precommit tests have not been triggered for at least 10 min, and are still not triggered. Another PR was merged before tests on a new revision of PR started to run, while passing test signal actually referred to an earlier

Jira component for HDFS issues with Python SDK

Hi, Could someone please help with addition io-python-hadoop or similar component to Jira? Also, there is a small discrepancy in naming py vs python between: io-python-gcp and sdk-py-core - consider unifying them. Thank you!

Re: Jira component for HDFS issues with Python SDK

tps://jira.apache.org/jira/projects/BEAM?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page >>> >>> On Thu, May 23, 2019 at 9:35 AM Chamikara Jayalath >>> wrote: >>> >>>> >>>> >>>> On Thu, May 23, 201

Re: Quota: In use IP-adresses

I did this for a few other resources recently (CPU, Disk). If this keeps being a problem we can lower test parallelism. On Thu, May 23, 2019, 3:48 PM Mikhail Gryzykhin wrote: > Hello everybody, > > Some of our jobs fail with 1/0 in use IP-addresses quota exception. > > Seems that we spin-up too

Re: :beam-sdks-java-io-hadoop-input-format:test is extremely flaky

BeamFnControlServiceTest is being worked on in https://issues.apache.org/jira/browse/BEAM-5709. On Mon, Apr 29, 2019 at 2:01 PM Reuven Lax wrote: > yeah, that testClientConnecting test is also extremely flaky. > > On Mon, Apr 29, 2019 at 6:50 AM Jean-Baptiste Onofré > wrote: > >> Agree, +1 >>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

f the specific tests are >> rerun? I don't think we should rerun all tests. >> >> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev >> wrote: >> >>> Preinstalling dependencies may affect the dependency resolution, and we >>> may end up testing a d

Tip: Search through Beam mailing lists using a custom search engine.

Custom search URLs: Dev: https://lists.apache.org/list.html?dev@beam.apache.org:lte=99M:%s User: https://lists.apache.org/list.html?u...@beam.apache.org:lte=99M:%s How to add a custom search engine in Google Chrome: https://support.google.com/chrome/answer/95426

Re: Beam at Google Summer of Code 2019

Congrats & good luck Tanay & Pablo! *From: *Connell O'Callaghan *Date: *Mon, May 6, 2019 at 4:15 PM *To: * *Cc: * Well done Tanay - good luck with this project!!! > > +1 Pablo - thank you for this mentorship!!! > > On Mon, May 6, 2019 at 1:11 PM Chamikara Jayalath > wrote: > >> Congrats

Re: Better naming for runner specific options

I think using RunnerOptions was an idea at some point, but in Python, we ended up parsing options from the runner api without populating RunnerOptions, and RunnerOptions was eventually removed [1]. If we decide to rename options, a path forward may be to have runners recognize both old and new

Re: investigating python precommit wordcount_it failure

I am working on a postcommit worcount it failure in BEAM-7063. On Thu, Apr 18, 2019 at 6:05 PM Udi Meiri wrote: > Correction: it's a postcommit failure > > On Thu, Apr 18, 2019 at 5:43 PM Udi Meiri wrote: > >> in https://issues.apache.org/jira/browse/BEAM-7111 >> >> If anyone has state please

Re: [ANNOUNCE] New committer announcement: Udi Meiri

Congrats, Udi! *From: *Thomas Weise *Date: *Mon, May 6, 2019 at 7:50 AM *To: * Congrats! > > > On Mon, May 6, 2019 at 2:25 AM Łukasz Gajowy wrote: > >> Congrats! :) >> >> pon., 6 maj 2019 o 10:45 Alexey Romanenko >> napisał(a): >> >>> Congratulations, Udi! Thanks for your work! >>> >>> On 4

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

No, just re-run the tests. For the record, there are PyPi error-rates in https://status.python.org/#system-metrics. On Fri, Apr 26, 2019 at 11:15 AM Alex Amato wrote: > Okay but this occurred on jenkins. So does the machine need an update? > > On Fri, Apr 26, 2019 at 10:43 AM

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

ies to > try to fix this sort of problem. > > Perhaps a mixture of reusing a virtualenv, and having retries when > creating it? > > On Fri, Apr 26, 2019 at 11:15 AM Alex Amato wrote: > >> Okay but this occurred on jenkins. So does the machine need an update? >> >>

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

I think you hit a pypi flake. pip install futures>=2.2.0 works fine for me. On Fri, Apr 26, 2019 at 9:41 AM Alex Amato wrote: > Would be nice to fix this as it can slow down PRs. I am not sure if this one > is fixed on retry yet or not. > > > >

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Preinstalling dependencies may affect the dependency resolution, and we may end up testing a different configuration than a user would have after installing beam into a clean environment. I do think pip uses cache, unless one specifies "--no-cache-dir". By default the cache is ~/.cache/pip.

Re: python precommits failing at head

No, we did not reduce the timeout recently. Looking at console logs, nothing happened for an hour or so, *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :) 06:57:50 * *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out (after 120 minutes). Marking the build

Re: [DISCUSS] Contributor guidelines for iterating on PRs: when to squash commits.

tter-guide/#pull-request-review-objectives >> (maybe >> we could point to them from the PR template?) >> Yes, it is acceptable to ask for squash or if it's ok to squash to a >> single commit. >> >> On Mon, Jul 8, 2019 at 11:14 AM Valentyn Tymofieiev >> wro

[DISCUSS] Contributor guidelines for iterating on PRs: when to squash commits.

I have observed a pattern where authors force-push their changes during every review iteration, so that a pull request always contains one commit. This creates the following problems: 1. It is hard to see what has changed between review iterations. 2. Sometimes authors make changes in parts of

Re: [DISCUSS] Contributor guidelines for iterating on PRs: when to squash commits.

h or if it's ok to squash to a > single commit. > > On Mon, Jul 8, 2019 at 11:14 AM Valentyn Tymofieiev > wrote: > >> I have observed a pattern where authors force-push their changes during >> every review iteration, so that a pull request always contains one com

Re: PostCommit tests broken

This is actually a quite frequent error right now. https://issues.apache.org/jira/browse/BEAM-6202 is opened to make Dataflow runner more robust when polling status of a running Dataflow job. On Wed, Mar 6, 2019 at 8:18 PM Valentyn Tymofieiev wrote: > Hi Juta, > > I al

Re: [VOTE] Release 2.12.0, release candidate #4

I think we should also leverage/invest in the automation for RC validation. We have some validation scripts, but last time I looked at them they worked only partially and had several usability issues. On Tue, Apr 23, 2019 at 3:24 PM Ahmet Altay wrote: > > > On Tue, Apr 23, 2019 at 3:21 PM

Re: [ANNOUNCE] New committer announcement: Yifan Zou

Congratulations, Yifan! Thanks a lot for your continued contributions to Beam. On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu wrote: > Congratulations Yifan! > > On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath > wrote: > >> Congrats Yifan! >> >> On Mon, Apr 22, 2019 at 10:02 AM Maximilian

Re: pickling typing types in Python 3.5+

I have checked that cloudpickle (an alternative to dill) is able to pickle and unpickle typing types on Python 3.5, 3.6, which seems to be a recent change, see: https://github.com/cloudpipe/cloudpickle/issues/63#issuecomment-501624383. I am evaluating cloudpickle as a potential avenue to address

Re: Update on Python Gradle test configuration unification

Thank you, Mark. Everyone, if you have recently added, or are planning to add, a Python SDK test suite, please make sure to replicate this suite to run under at least one Python 3 minor version to make sure use cases work on Python 3. If you pick only one Python 3 minor version (to save resources

  1   2   3   4   >