Re: Azure(ADLS) compatibility on Beam with Spark runner

Hi JB, I'm working on adding HDFS support to the Python runner. We're planning on using libhdfs3, which doesn't seem to support anything other than HDFS. On Mon, Nov 27, 2017 at 12:44 PM Lukasz Cwik wrote: > Out of curiosity, does using the DirectRunner with ADL work

Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')

than >> the ones we happened to choose. (Imagine the pain of two or more of >> our dependencies pinned all their dependencies...) >> >> On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <eh...@google.com> wrote: >> > +1 to pinning to exact versions, to be sure

HDFS Support for Python SDK

Hi, I've done some research into implementing HDFS support for Python SDK and I'd like your input. This work is regarding BEAM-3099 . This doc lists several options for implementing HDFS support and attempts to weigh the differences.

Pubsub on directrunner: direct_runner.py and transform_evaluator.py

Hi, I'm having trouble understanding why there's an extra level of indirection when doing pubsub reads via directrunner vs writes. For reads, we have these translations: beam_pubsub.ReadFromPubSub -> direct_runner._DirectReadFromPubSub -> transform_evaluator._PubSubReadEvaluator For writes, this

Documentation for Beam on Windows

Hi all, I was looking yesterday for a quickstart guide on how to use Beam on Windows but saw that those guides are exclusively for Linux users. What documentation is available for people wanting to use Beam on Windows machines? Thanks! smime.p7s Description: S/MIME Cryptographic Signature

Proposal: keeping precommit times fast

HI, I have a proposal to improve contributor experience by keeping precommit times low. I'm looking to get community consensus and approval about: 1. How long should precommits take. 2 hours @95th percentile over the past 4 weeks is the current proposal. 2. The process for dealing with slowness.

Re: I'm back and ready to help grow our community!

Welcome back and congrats again! On Thu, May 17, 2018 at 2:23 PM Dmitry Demeshchuk wrote: > While this may be a bit off topic, I still want to say this. > > Congratulations on your graduation, Gris! > > On Thu, May 17, 2018 at 2:19 PM, Griselda Cuevas

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

Another bug: reading from PubSub with_attributes=True is broken on Python with Dataflow. https://issues.apache.org/jira/browse/BEAM-4536 JB, I'm making a PR that removes this keyword and I'd like to propose it as a cherrypick to 2.5.0. (feature should be fixed in the next release) On Mon, Jun

Re: Proposal: keeping precommit times fast

> just the single previous build, so if any previous jobs (up to the cache > size limit) built/tested artifacts unchanged by the current PR, the results > would live in the cache. > > I would look at (a) and (b) only if this isn't already good enough. > > On Wed, Jun 6, 2018

Re: [DISCUSS] Use Confluence wiki for non-user-facing stuff

(a) Yes. (b) I'm interested in putting documentation for contributors there. (test triage guide, precommit and postcommit guidelines, processes, etc.) It'd be faster than having to go through the motions of a github pull request and a review process. (c) Anything that goes to a wide audience, such

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

Cherrypick created: https://github.com/apache/beam/pull/5607 Tests still running. On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri wrote: > -1: Would like to cherry pick a fix for > https://issues.apache.org/jira/browse/BEAM-4536 > > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Ono

Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1

+1 to ignoring flaky test. FYI there's a fourth cherrypick: https://github.com/apache/beam/pull/5624 On Wed, Jun 13, 2018 at 3:45 PM Pablo Estrada wrote: > Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky > test. As JB is the release manager, I'l let him make the call on

Re: Precommits broken?

+1 for separate jobs if it gets us faster to pre-commit filtering On Thu, Jun 14, 2018 at 11:22 AM Kenneth Knowles wrote: > I like Andrew's solution. Just totally separate jobs for automatic and > manual. > > Kenn > > On Thu, Jun 14, 2018 at 9:56 AM Lukasz Cwik wrote: > >> That seems like a

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

+1 On Mon, Jun 18, 2018 at 6:16 PM Boyuan Zhang wrote: > Follow up with pervious email( sorry for the inconvenience). > > Hey JB, > > If you haven't built python wheels yet, then you can use files in the > previous email to move forward. If you have done all of them, then please > ignore

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

-1: Would like to cherry pick a fix for https://issues.apache.org/jira/browse/BEAM-4536 On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré wrote: > Any update about your vote and fix ? > > Thanks ! > Regards > JB > > On 12/06/2018 04:02, Udi Meiri wrote: > > Anoth

Re: Reducing Committer Load for Code Reviews

I thought this was the norm already? I have been the sole reviewer a few PRs by committers and I'm only a contributor. +1 On Wed, May 30, 2018 at 2:13 PM Kenneth Knowles wrote: > ++1 > > This is good reasoning. If you trust someone with the committer > responsibilities [1] you should trust

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

+1 On Fri, Jun 1, 2018 at 4:27 PM Lukasz Cwik wrote: > +1 > > On Fri, Jun 1, 2018 at 2:53 PM Thomas Weise wrote: > >> +1 >> >> On Fri, Jun 1, 2018 at 2:17 PM, Robert Bradshaw >> wrote: >> >>> +1 >>> >>> On Fri, Jun 1, 2018 at 1:43 PM Andrew Pilloud >>> wrote: >>> +1 On Fri,

Re: Proposal: keeping precommit times fast

e as well...) >>> >>> >>>>> I also wouldn't be opposed to a quicker immediate signal, plus more >>> extensive tests before actually merging. It's also nice to not have to >>> wait >>> an hour to see that you have a lint error; quick stuff

Re: Proposal: keeping precommit times fast

only run Python precommit tests. On Tue, Jun 5, 2018 at 7:24 PM Udi Meiri wrote: > I've been having a separate discussion on the proposal doc, which is ready > for another round of reviews. > Change summary: > - Changed fast requirement to be < 30 minutes and simplify the check as

Filesystems.copy and .rename behavior

Hi, I've been working on HDFS code for the Python SDK and I've noticed some behaviors which are surprising. I wanted to know if these behaviors are known and intended. 1. When renaming files during finalize_write, rename errors are ignored

Re: Filesystems.copy and .rename behavior

wish it didn't give an error if the stuff I'm copying > doesn't exist" - the option exists only because we couldn't come up with > another way to implement idempotent rename on GCS. > > What's your idea of how a safe retryable GCS rename() could work? > > On Wed, Jan 31, 2018

Re: Filesystems.copy and .rename behavior

run didn't run on different output? > Overwriting seems more accurate than ignoring. Does handling this error at > sink level distinguish between the two (another run vs second attempt)? > > > On Wed, Jan 31, 2018 at 12:32 PM, Udi Meiri <eh...@google.com> wrote: > >> Yeah

Re: Filesystems.copy and .rename behavior

hat any step in Beam can be executed multiple times, including the rename >> step. If the rename step gets run twice, the second run should succeed >> vacuously. >> >> >> On Tue, Jan 30, 2018 at 6:19 PM, Udi Meiri <eh...@google.com> wrote: >> >>>

Re: Filesystems.copy and .rename behavior

would overwrite output >> if it exists >> -- reshuffle -- >> 2) delete, which would not fail if input is missing. >> > > Something like this is needed only in streaming, right? > > Raghu. > > >> That way first everything is copied (possibly via multiple at

Re: Removing documentation for old Beam versions

, 2018 at 11:51 AM Daniel Oliveira wrote: > The older docs should be recorded in the commit history of the website > repository, right? If they're not currently used in the website and they're > in the commit history then I don't see a reason to save them. > > On Tue, Jul 31, 2018

Re: Removing documentation for old Beam versions

using some of these old releases. It >>>> is unclear from Beam website which releases are still supported or not. It >>>> probably makes sense to drop documentation for releases < 2.0. (I would >>>> suggest keeping docs for 2.0). For the future I can work on upda

Re: Removing documentation for old Beam versions

[image: pr-520.png] (trying that image again) On Thu, Aug 2, 2018 at 7:00 PM Udi Meiri wrote: > Alright, created https://github.com/apache/beam-site/pull/520 > [image: pr-520.png] > Reduces staging upload from 500M down to 270M, and halves the number of > files from ~22k to 11k. &g

Re: Removing documentation for old Beam versions

gt;> time they clone repo, which is not that good of an idea to do. >> >> Regards, >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> >> >> On Thu, Aug 2, 2018 at 6:42 PM Udi Meiri wrote: >> >>> Pablo, the docs a

Re: CODEOWNERS for apache/beam repo

concretely need. > > Regards > JB > Le 24 juil. 2018, à 04:17, Udi Meiri a écrit: >> >> I was recently told about Prow >> <https://github.com/kubernetes/test-infra/tree/master/prow>, which >> automates testing and merging for Kubernetes and other project

Re: CODEOWNERS for apache/beam repo

Summary doc for CODEOWNERS, Mention-bot, Prow: https://docs.google.com/document/d/1S8spggJsxDNYZ7aNwZN6VhLhNW372SVRezjblt-7lNQ/edit?usp=sharing This doc will get updated as we gain experience with Mention-bot and Prow. On Wed, Jul 25, 2018 at 5:15 PM Udi Meiri wrote: > So I configured P

Removing documentation for old Beam versions

Hi all, I'm writing a PR for apache/beam-site and beam_PreCommit_Website_Stage is timing out after 100 minutes, because it's trying to deletes 22k files and then copy 22k files (warning large file ). It seems that we

Re: CODEOWNERS for apache/beam repo

t read access (for requesting reviews). On Fri, Jul 27, 2018, 09:37 Udi Meiri wrote: > Summary doc for CODEOWNERS, Mention-bot, Prow: > https://docs.google.com/document/d/1S8spggJsxDNYZ7aNwZN6VhLhNW372SVRezjblt-7lNQ/edit?usp=sharing > This doc will get updated as we gain experience w

Re: [DISCUSSION] Tracking & Visualizing various metrics of the Beam community

These tables look very cool! I certainly don't object to using tools made by another organization. The only issue might be compatibility with our process. >From my understanding of the Kubernetes review process, they use Github's tagging feature to specify PR statuses such as LGTM and approval.

Re: gradlew :rat broken

Found an elegant solution: https://github.com/apache/beam/pull/6236 On Wed, Aug 15, 2018 at 4:25 PM Udi Meiri wrote: > nosetests.xml doesn't seem to get cleaned by :clean. Same goes for the > containers sub-directory. > > On Wed, Aug 15, 2018 at 4:23 PM Yifan Zou wrote: &g

Re: gradlew :rat broken

ainer >> >> Your files are container/vendor/, and sdks/python/nosetests.xml >> (which is generated when running nosetests). >> >> On Wed, Aug 15, 2018 at 4:11 PM Udi Meiri wrote: >> >>> Hi, >>> Whenever I run ../../gradlew :rat fro

gradlew :rat broken

Hi, Whenever I run ../../gradlew :rat from the sdks/python directory I get errors about unapproved licenses. Example: https://scans.gradle.com/s/gdpcgbexwyrpm This didn't use to be the case a few weeks back. Any ideas what could be causing this? smime.p7s Description: S/MIME Cryptographic

Re: [PROPOSAL] Prepare Beam 2.7.0 release

+1 On Mon, Aug 20, 2018 at 3:33 PM Boyuan Zhang wrote: > +1 > Thanks for volunteering, Charles! > > On Mon, Aug 20, 2018 at 3:22 PM Rafael Fernandez > wrote: > >> +1, thanks for volunteering, Charles! >> >> On Mon, Aug 20, 2018 at 12:09 PM Charles Chen wrote: >> >>> Thank you Andrew for

Re: Removing documentation for old Beam versions

eveloper will have to pull all the versions of website every > time they clone repo, which is not that good of an idea to do. > >> > >> Regards, > >> --Mikhail > >> > >> Have feedback? > >> > >> > >> On Thu, Aug 2, 2018 at 6:42 PM Udi Meiri w

Re: BEAM-5180 for 2.7.0 ?

+Ankur Goenka (Kenneth is out of office) On Fri, Aug 24, 2018 at 3:20 AM Tim Robertson wrote: > Thanks Jozef for bringing this to dev@ and your work in reporting Jiras > and offering fixes. > > I propose we consider BEAM-5180, BEAM-2277 blockers on 2.7.0. They break > word count and file IO

Re: Design Proposal: Beam-Site Automation Reliability

FYI, we are about to add a new branch to apache/beam, named 'asf-site', which will contain generated website sources. On Thu, Jun 7, 2018 at 10:18 AM Jason Kuster wrote: > Sounds good; I'm really excited about these changes Scott. Thanks for > taking this on! > > On Tue, Jun 5, 2018 at 4:00 PM

jira search in chrome omnibox

In case you want to quickly look up JIRA tickets, e.g., typing 'j', space, 'BEAM-4696'. Search URL: https://issues.apache.org/jira/QuickSearch.jspa?searchString=%s smime.p7s Description: S/MIME Cryptographic Signature

Re: CODEOWNERS for apache/beam repo

t;> JB >> >>>>> Le 16 juil. 2018, à 18:59, Holden Karau < holden.ka...@gmail.com> >> a écrit: >> >>>>>> >> >>>>>> Would folks be OK with me asking infra to turn on blame based >> suggestions for Beam and tr

Re: CODEOWNERS for apache/beam repo

. On Tue, Jul 17, 2018 at 11:04 AM Udi Meiri wrote: > +1 to generating the file. > I'll go ahead and file a PR to remove CODEOWNERS > > On Tue, Jul 17, 2018 at 9:28 AM Holden Karau wrote: > >> So it doesn’t support doing that right now, although if we find it’s a &g

Re: builds.apache.org refused connections since last night

My mistake. I thought you needed to appear as "member" on apache/beam-site as well. On Fri, Aug 31, 2018 at 5:06 PM Thomas Weise wrote: > All committers should be able to merge the site changes and wasn't that > the case till the Jenkins outage? > > > On Fri, Aug 31,

Re: builds.apache.org refused connections since last night

I believe the bot only listens to members. On Fri, Aug 31, 2018 at 2:56 PM Thomas Weise wrote: > Any idea why the beam-site merge bot doesn't work? > > The PRs are showing a "Merging is blocked" check that I don't remember > seeing before. > > > On Fri, Aug 31, 2018 at 2:06 AM Maximilian

Re: jira search in chrome omnibox

the query: > https://github.com/apache/beam/pulls?q=is%3Apr%20author%3A > > On Tue, Aug 28, 2018 at 4:26 PM Daniel Oliveira > wrote: > >> This seems pretty useful. Thanks Udi! >> >> On Mon, Aug 27, 2018 at 3:54 PM Udi Meiri wrote: >> >>> In case yo

CODEOWNERS for apache/beam repo

Hi everyone, I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS mechanism. Initial version is here: *https://github.com/apache/beam/pull/5909/files * I need help from the community in determining owners for each component. Feel

Re: CODEOWNERS for apache/beam repo

some > areas. > > On Tue, Jul 10, 2018 at 10:23 AM, Jean-Baptiste Onofré > wrote: > >> +1 >> >> I added my name on some components ;) >> >> Thanks ! >> Regards >> JB >> >> On 10/07/2018 02:06, Udi Meiri wrote: >> > Hi eve

Re: CODEOWNERS for apache/beam repo

example we can >> run monthly job to see if owner was active during this period. >> >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> >> >> On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri wrote: >> >>> Thanks all! >

Re: CODEOWNERS for apache/beam repo

> > Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit : > > Hi everyone, > > I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS > mechanism. > Initial version is here: *https://github.com/apache/beam/pull/5909/files > <https://github.com/apach

Re: CODEOWNERS for apache/beam repo

d a lot of other people to it, and I had to remove them. > >> This seems like a big inconvenience to me, is there a way to disable > this? > >> > >> Thanks. > >> > >> On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri >> <mailto:eh...@google.com

Re: CODEOWNERS for apache/beam repo

change the list of reviewers. I propose renaming the file to disable the auto-reviewer assignment feature. In its place I'll add a script that suggests reviewers. On Fri, Jul 13, 2018 at 9:09 AM Udi Meiri wrote: > Hi Etienne, > > Yes you could be as precise as you want. The paths I listed

Re: CODEOWNERS for apache/beam repo

ture appears to be >> effectively unusable for Beam but I'd hope that Github might pay attention >> and fix things if we submit feedback, with us being one of the most active >> Apache projects - did anyone do this yet / planning to? >> >> On Fri, Jul 13, 2018 at 10

Re: Jenkins slowness

extra slowness. > > > On Fri, Mar 9, 2018 at 10:18 AM Udi Meiri <eh...@google.com> wrote: > >> Hi, >> >> Does anybody know why Jenkins hosts take so long to run? For example, >> beam1 was running beam_PostCommit_Python_Verify and I saw this time for >> runn

Jenkins slowness

Hi, Does anybody know why Jenkins hosts take so long to run? For example, beam1 was running beam_PostCommit_Python_Verify and I saw this time for running "tox -e py27": Ran 1535 tests in 403.860s on my workstation I got: Ran 1535 tests in 160.242s Is there any way to troubleshoot this? Each

Python incremental testing

I cooked up a quick Bazel proof of concept to run Python tests. Bazel's advantage over Nosetests is that it caches results and does dependency tracking on tests to know when to rerun them. (I am not trying to replace Gradle.) The config is almost working (121 out of 123 tests files work, 2 have

Python postcommit and precommit

Hi, I noticed that Python precommit runs using this command: mvn clean install -pl sdks/python -am -amd while postcommit invocation is simply a bash script: bash sdks/python/run_postcommit.sh Both run unit tests via Tox, however since the runtime environment setup is configured in different

Jenkins wait times

Hi, I've been seeing increased wait times on Jenkins. It's frustrating to wait 8h for a build, or 4h for it just to schedule. Data point:

Pubsub API feedback

Hi, I wanted to get feedback about the upcoming Python Pubsub API. It is currently experimental and only supports reading and writing UTF-8 strings. My current proposal only concerns reading from Pubsub. Classes: - PubsubMessage: encapsulates Pubsub message payload and attributes. PTransforms: -

debugging python jenkins precommit

Hi, I'm trying to debug a jenkins precommit error for PR #4877. (Side rant: It's taking a long time to run precommits (between 24m and 3h), and I don't have access to the jenkins VM to debug things locally.) Partial log: py27-cython create:

Re: Jenkins wait times

> > On Thu, Mar 22, 2018 at 10:14 AM Udi Meiri <eh...@google.com> wrote: > >> Hi, >> I've been seeing increased wait times on Jenkins. It's frustrating to >> wait 8h >> <https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/3265/> >> for a

Re: [DISCUSS] Automation for Java code formatting

Python already has lint checks: :beam-sdks-python:lint There are autoformatting tools, which we don't use AFAIK: https://github.com/myint/autoflake https://github.com/hhatto/autopep8 On Thu, Jun 28, 2018 at 12:31 AM Daniel Kulp wrote: > > > On Jun 28, 2018, at 6:01 AM, Kenneth Knowles wrote:

Python SDK: pytest vs nose

Hi, I'm currently leaning towards migrating us to pytest, since it has features like --last-failed, is actively maintained, and seems more polished in general. (Pytest is supposed to be able to run nose tests, but not all our tests work with it out of the box.) Does anyone have a preference for

Re: Python SDK: pytest vs nose

https://issues.apache.org/jira/browse/BEAM-3713 On Thu, Jun 28, 2018 at 3:35 PM Udi Meiri wrote: > Hi, > I'm currently leaning towards migrating us to pytest, since it has > features like --last-failed, is actively maintained, and seems more > polished in general. > > (

Re: python post-commit failures

More details in https://issues.apache.org/jira/browse/BEAM-5442 On Fri, Oct 5, 2018 at 10:26 AM Udi Meiri wrote: > I'm seeing these errors at least in one test: > "Python sdk harness failed: > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/

python post-commit failures

I'm seeing post-commit failures in :beam-sdks-python:postCommitITTests: https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/console https://builds.apache.org/job/beam_PostCommit_Python_Verify/6182/console smime.p7s Description: S/MIME Cryptographic Signature

Re: python post-commit failures

it > (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) -> > Bigquery table not found > test_game_stats_it > (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) -> > Bigquery table not found > streaming related tests -> Assertion errors >

Re: python post-commit failures

to > https://beam.apache.org/contribute/testing/#pre-commit or is there a > better place? > > Thanks > > > > > On Fri, Oct 5, 2018 at 10:38 AM Udi Meiri wrote: > >> More details in https://issues.apache.org/jira/browse/BEAM-5442 >> >> On Fri, Oct 5,

Re: [DISCUSS] - Separate JIRA notifications to a new mailing list

+1 to split JIRA notifications On Thu, Oct 11, 2018 at 9:13 AM Kenneth Knowles wrote: > > On Thu, Oct 11, 2018 at 9:10 AM Mikhail Gryzykhin < > gryzykhin.mikh...@gmail.com> wrote: > >> +1. >> Should we separate Jenkins notifications as well? >> > > I'm worried this question will get buried in

Re: [DISCUSS] Gradle for the build ?

I agree with the points made that our Gradle configuration is too complicated (learning Groovy and BeamModulePlugin come to mind). But I mainly write for Python SDK, and I've done experiments in the past with Bazel builds, which was a much better fit for Python as an incremental and parallel

post-commit failure emails

Hi, https://github.com/apache/beam/pull/6635 is an attempt to notify commit authors if their commit is suspected to have broken post-commit test. I'd would like to get some feedback about this feature. Is it accurate? Is it spammy? Thanks! smime.p7s Description: S/MIME Cryptographic Signature

Re: post-commit failure emails

would be a problem > if this doesn't get turned on until at least one run has passed on each job. > > Andrew > > On Thu, Oct 11, 2018 at 2:44 PM Udi Meiri wrote: > >> Hi, >> https://github.com/apache/beam/pull/6635 is an attempt to notify commit >> authors if th

[PROPOSAL] Using Bazel and Docker for Python SDK development and tests

Hi, In light of increasing Python pre-commit times due to the added Python 3 tests, I thought it might be time to re-evaluate the tools used for Python tests and development, and propose an alternative. Currently, we use nosetests, tox, and virtualenv for testing. The proposal is to use Bazel,

Re: [PROPOSAL] Using Bazel and Docker for Python SDK development and tests

On Wed, Oct 17, 2018 at 1:38 AM Robert Bradshaw wrote: > On Tue, Oct 16, 2018 at 12:48 AM Udi Meiri wrote: > >> Hi, >> >> In light of increasing Python pre-commit times due to the added Python 3 >> tests, >> I thought it might be time to re-eval

Python Datastore client upgrade plan

Hi, Sadly upgrading googledatastore -> google-cloud-datastore is non-trivial ( https://issues.apache.org/jira/browse/BEAM-4543). I wrote a doc to summarize the plan: https://docs.google.com/document/d/1sL9p7NE5Z0p-5SB5uwpxWrddj_UCESKSrsvDTWNKqb4/edit?usp=sharing Contents pasted below: Beam Python

Re: [PROPOSAL] Using Bazel and Docker for Python SDK development and tests

work for Python as well. >> > > +1, that's a great idea. Together with --parallel--safe-build should be > sufficient. > > We could separately look into whether it's worth adding annotations > (positive or negative) to mark tests which have low value to be run in all > the

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

+1 for explicit --runner_option=param=val,... It's hard to tell otherwise where an option is going to, On Mon, Oct 15, 2018 at 8:04 AM Robert Bradshaw wrote: > On Mon, Oct 15, 2018 at 3:58 PM Maximilian Michels wrote: > > > > I agree that the current approach breaks the pipeline options

Re: error with DirectRunner

This looks like a FnApiRunner bug. When I override use_fnapi_runner = False in direct_runner.py the pipeline works. It seems like either the side-input to _copy_number or the Flatten operation is the culprit. On Mon, Oct 29, 2018 at 2:37 PM Allie Chen wrote: > Hi, > > I have a project that

Re: error with DirectRunner

> >> Thanks Udi. I agree, since it works fine removing either the side input >> or the last flatten and combine operation. >> >> On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri wrote: >> >>> This looks like a FnApiRunner bug. >>> When I override u

Re: error with DirectRunner

+Robert Bradshaw I would be happy to debug and fix this, but I'd need more guidance on where to look. On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri wrote: > Created https://issues.apache.org/jira/browse/BEAM-5927 > > On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik wrote: > >> Udi,

BEAM-6018: memory leak in thread pool instantiation

HI, I've identified a memory leak when GcsUtil.java instantiates a ThreadPoolExecutor (https://issues.apache.org/jira/browse/BEAM-6018). The code uses the getExitingExecutorService

Re: BEAM-6018: memory leak in thread pool instantiation

ing the executor within > the GcsUtil as a static instance instead of creating one each time. Could > have been missed during code review / slow code changes over time. GcsUtil > is not well "loved". > > On Thu, Nov 8, 2018 at 11:00 AM Udi Meiri wrote: > >> HI,

Re: Please do not merge Python PRs

All clear, Python tests are reporting errors correctly again. On Wed, Nov 14, 2018 at 5:57 PM Udi Meiri wrote: > https://github.com/apache/beam/pull/7048 is the rollback PR > > On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > >> Thank you Udi. Could you send a rollback P

Please do not merge Python PRs

It seems that Gradle is not getting the correct exit status from test runs. Possible culprit: https://github.com/apache/beam/pull/6903 smime.p7s Description: S/MIME Cryptographic Signature

Re: Please do not merge Python PRs

https://github.com/apache/beam/pull/7048 is the rollback PR On Wed, Nov 14, 2018 at 5:28 PM Ahmet Altay wrote: > Thank you Udi. Could you send a rollback PR? > > I believe this is https://issues.apache.org/jira/browse/BEAM-6048 > > On Wed, Nov 14, 2018 at 5:16 PM, U

Re: Please do not merge Python PRs

Recreated locally: https://gradle.com/s/psqgcywnc3h2m On Wed, Nov 14, 2018 at 5:16 PM Udi Meiri wrote: > It seems that Gradle is not getting the correct exit status from test runs. > Possible culprit: https://github.com/apache/beam/pull/6903 > smime.p7s Description: S/MIME Cryp

Re: BEAM-6018: memory leak in thread pool instantiation

/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java#L133-L138 On Thu, Nov 8, 2018 at 11:41 PM Dan Halperin wrote: > >> On Thu, Nov 8, 2018 at 2:12 PM Udi Meiri wrote: >> >>> Both options risk delaying worker shutdown if the executor's shutdown() >>&g

Re: [VOTE] Mark 2.7.0 branch as a long term support (LTS) branch

+1 On Fri, Nov 9, 2018 at 8:31 AM Maximilian Michels wrote: > +1 > > On 09.11.18 09:38, Robert Bradshaw wrote: > > +1 approve. > > On Fri, Nov 9, 2018 at 2:47 AM Ahmet Altay wrote: > >> > >> Hi all, > >> > >> Please review the following statement: > >> > >> "2.7.0 branch will be marked as the

Re: Spotless and lint precommit

+1 and parallelize the 3 lint tasks On Tue, Nov 13, 2018 at 10:43 AM Thomas Weise wrote: > +1 > > > On Tue, Nov 13, 2018 at 9:06 AM Ruoyun Huang wrote: > >> +1 >> >> On Tue, Nov 13, 2018 at 8:29 AM Maximilian Michels >> wrote: >> >>> +1 >>> >>> On 13.11.18 14:22, Robert Bradshaw wrote: >>> >

Re: Python PostCommit failures

Thanks Yifan! On Thu, Oct 4, 2018 at 11:45 AM Yifan Zou wrote: > The integration test has been fixed by #6567 > . The Python PostCommit Verify > is back to normal. > Thanks. > > - Yifan > > On Thu, Oct 4, 2018 at 12:22 AM Yifan Zou wrote: > >> We are

Re: python post-commit failures

t;> wordcount_streaming integration tests as part of pre-commit tests. >> >> Can look into that. >> >> Thanks, >> Max >> >> On 05.10.18 23:08, Ahmet Altay wrote: >> > >> > >> > On Fri, Oct 5, 2018 at 1:51 PM, Udi Meiri > > <ma

Python SDK: .options deprecation

Hey, does anybody know why the pipeline.options property was deprecated? I found this bug: https://issues.apache.org/jira/browse/BEAM-2124 but there's no explanation. smime.p7s Description: S/MIME Cryptographic Signature

Re: Python SDK: .options deprecation

t;pipeline.options() >> >> On Fri, Sep 21, 2018 at 2:11 PM Udi Meiri wrote: >> >>> Hey, does anybody know why the pipeline.options property was deprecated? >>> I found this bug: https://issues.apache.org/jira/browse/BEAM-2124 >>> but there's no explanation. >>> >> > smime.p7s Description: S/MIME Cryptographic Signature

Re: Removing documentation for old Beam versions

e/BEAM-5459 >> >> On Fri, Aug 24, 2018 at 11:48 AM Thomas Weise wrote: >> >>> Hi Udi, >>> >>> Good to know you will continue this work. >>> >>> Let me know if you want to try the buildbot route (which does not >>> require generat

Re: Removing documentation for old Beam versions

> > Scott had filed https://issues.apache.org/jira/browse/BEAM-5459 - > it would eliminate the need to place generated docs into git repos. > > On Mon, Sep 24, 2018 at 11:06 AM Udi Meiri wrote: > >> I believe that beam.apache.org is populated from the asf-site branch of >>

Re: Removing documentation for old Beam versions

>>>> it incentivizes putting the generated sources branch on the same repo. >>>>>> I've >>>>>> documented a few of the reasons in the Appendix of the design doc [1]: >>>>>> - It's easier to maintain a single repository; easily ap

Python License code

Hi, I'm reviewing a PR that has code licensed under Python License. It is under category A so it's okay to include. The question is: where do we put the license notice? Is it sufficient to place the code in a separate module with the license

Re: Python License code

gt; In addition to the page you linked, have you followed the link to the > LEGAL Jira space ( > https://www.apache.org/legal/resolved.html#asking-questions)? You may > find your answer there, or could ask. > > Kenn > > On Wed, Sep 26, 2018 at 11:25 AM Udi Meiri wrote: > >

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

Congrats! On Thu, Sep 20, 2018 at 10:09 AM Raghu Angadi wrote: > Congrats Kenn! > > On Wed, Sep 19, 2018 at 12:54 PM Davor Bonaci wrote: > >> Hi everyone -- >> It is with great pleasure that I announce that at today's meeting of the >> Foundation's Board of Directors, the Board has appointed

Re: TestDirectRunner for Java?

that was introduced, versus waiting for termination > and running the assertion more directly. Can you enlighten me? > > Kenn > > On Tue, Jan 15, 2019 at 4:15 PM Udi Meiri wrote: > >> Hi, >> I want to use DirectRunner for a new IT I'm writing, since it's testin

  1   2   3   4   >