Re: Hello

Thanks for adding me to the group. On Wed, Nov 1, 2017 at 12:04 PM, Lukasz Cwik <lc...@google.com.invalid> wrote: > Welcome, you have been added. > > On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka <goe...@google.com.invalid> > wrote: > > > Hi Guys,

[Discussion] [Doc] Proposal to multithread workers in Beam Python SDK

Hi all, I am working on enhancing the multithreading in python SDK to improve performance. The document summarize the problem and proposed solution with some backgroud of the code structure. Please provide feedback on the solution. Thanks, Ankur​ Link to Jira:

Hello

Hi Guys, I recently joined Google and will be contributing to Apache Beam project. Please add me to the apache jira. My jira id is angoenka Looking forward to work with you guys! Thanks, Ankur

Re: Launching a Portable Pipeline

Updated link <https://docs.google.com/document/d/1xOaEEJrMmiSHprd-WiYABegfT129qqF-idUBINjxz8s/edit> to the document as the previous link was not working for some people. On Fri, May 11, 2018 at 7:56 PM Ankur Goenka <goe...@google.com> wrote: > Hi, > > Recent effort on porta

Fwd: Launching a Portable Pipeline

Hi, Recent effort on portability has introduced JobService and ArtifactService to the beam stack along with SDK. This has open up a few questions around how we start a pipeline in a portable setup (with JobService). I am trying to document our approach to launching a portable pipeline and take

Re: Documenting Github PR jenkins trigger phrases

In my experience affect of white space in commit is inconsistent for certain commands they do matter while for others they don't. On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev wrote: > +1 to writing a Beam Jenkins spellbook. > I have observed that Jenkins commands

Re: Documenting Github PR jenkins trigger phrases

These actions are documented in groovy scripts like https://github.com/apache/beam/blob/b7bfca9b196699a096786506777a49237e0b3776/.test-infra/jenkins/job_beam_PostCommit_Python_ValidatesContainer_Dataflow.groovy#L36 We can create a tool which can just go over these scripts and print the comments

Re: Launching a Portable Pipeline

/cancel Pipelines is the first thing > I can think of abusing. > On Tue, May 22, 2018 at 9:40 PM Ankur Goenka <goe...@google.com> wrote: > > > Thank you guys for the input. > > > Here is the summary. > > > Responsibility of Beam on Job Management > > >

Re: Launching a Portable Pipeline

Yes, JobService can be implemented by a runner and can be bade available using an endpoint. The component reuse is more of a code reuse. On Wed, May 23, 2018 at 3:14 PM Reuven Lax <re...@google.com> wrote: > > > On Wed, May 23, 2018 at 3:09 PM Ankur Goenka <goe...@google.com&g

Portable Artifact Staging

Hi, Artifact Staging is still an evolving topic in Beam Portability. I have started a document to go over different approaches for artifact staging. Please review the document and provide your

Re: Launching a Portable Pipeline

> Thanks Ankur, I think there's consensus, so it's probably ready to share :) > > On Fri, May 18, 2018 at 3:00 PM Ankur Goenka <goe...@google.com> wrote: > >> Thanks for all the input. >> I have summarized the discussions at the bottom of the document ( here >>

Re: Launching a Portable Pipeline

> > +Aljoscha Krettek <aljos...@data-artisans.com> +Thomas Weise > <t...@apache.org> +Henning Rohde <hero...@google.com> > > On Mon, May 14, 2018 at 12:34 PM Ankur Goenka <goe...@google.com> wrote: > >> Updated link >> <https:

Re: [FYI] New Apache Beam Swag Store!

Awesome! On Fri, Jun 8, 2018 at 4:24 PM Pablo Estrada wrote: > Nice : D > > On Fri, Jun 8, 2018, 3:43 PM Raghu Angadi wrote: > >> Woo-hoo! This is terrific. >> >> If we are increasing color choices I would like black or charcoal... Beam >> logo would really pop on a dark background. >> >> On

Re: Java compiler OOMs on Jenkins/Gradle

I am facing OOM while locally building the project using Gradle. Here is the scan https://scans.gradle.com/s/t3n42rw5666us The issue is happening from :rat task. Is this issue related? On Tue, May 1, 2018 at 4:40 PM Scott Wegner wrote: > Sorry about the instability. We need to get the Gradle

Re: [VOTE] Code Review Process

+1 On Fri, Jun 1, 2018 at 11:28 AM Charles Chen wrote: > +1 > > On Fri, Jun 1, 2018 at 11:20 AM Valentyn Tymofieiev > wrote: > >> +1 >> >> On Fri, Jun 1, 2018 at 10:40 AM, Ahmet Altay wrote: >> >>> +1 >>> >>> On Fri, Jun 1, 2018 at 10:37 AM, Kenneth Knowles wrote: >>> +1 On

Re: [DISCUSS] Automation for Java code formatting

+1 Intellij can help but still formatting is an additional thing to keep in mind. Enabling auto formatting at gradle level will remove this additional thing to keep in mind. On Tue, Jun 26, 2018 at 9:49 PM Eugene Kirpichov wrote: > +1! > > In some cases the temptation to format code manually

Re: Pubsub to Beam SQL

I like the idea of exposing source timestamp in TBLPROPERTIES which is closely tied to source (KafkaIO, KinesisIO, MqttIO, AmqpIO, unbounded FileIO, PubSubIO). Exposing timestamp as a top level keyword will break the symmetry between streaming and batch pipelines. TBLPROPERTIES gives us

Re: Community Examples Repository

I like he initiative but I feel that fragmenting the codebase will make it harder to discover examples. Having examples in a separate repo makes it easier to forget that examples should get the same love as the rest of the codebase. The other challenge is the tooling and integration which is

Re: How to best test fn_api_layers (operation.py, bundle_processor.py)

Thanks for getting attention to this topic. We relied on ValidatesRunner tests and encapsulating tests which is not optimum. For the PR, approach 4 seems to OK I think, approach 1 should be our long term plan. This will require quite a bit of refactoring so we can take it as a tech debt or we can

Re: Process JobBundleFactory for portable runner

Thanks for starting the discussion. I will be happy to help. I agree, we should have pluggable SDKHarness environment Factory. We can register multiple Environment factory using service registry and use the PipelineOption to pick the right one on per job basis. There are a couple of things which

Discussion: Scheduling across runner and SDKHarness in Portability framework

Hi, tl;dr Dead Lock in task execution caused by limited task parallelism on SDKHarness. *Setup:* - Job type: *Beam Portable Python Batch* Job on Flink standalone cluster. - Only a single job is scheduled on the cluster. - Everything is running on a single machine with single Flink task

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

; > On Mon, Aug 20, 2018 at 1:47 PM Ankur Goenka wrote: > >> I would prefer to to keep it dynamic as it can be changed by the >> infrastructure or the pipeline author. >> Like in case of Python, number of concurrent bundle can be changed by >> setting pipeline option wor

Re: Bootstrapping Beam's Job Server

Option 4) We are also thinking about adding process based SDKHarness. This will avoid docker in docker scenario. Process based SDKHarness also has other applications and might be desirable in some of the production use cases. On Mon, Aug 20, 2018 at 11:49 AM Henning Rohde wrote: > Option 3)

Re: Should we mention TF Transform in Beam site?

+1 Adding to Thomas's suggestion, we can also add the future plans to keep people excited about what's in store. On Mon, Aug 20, 2018 at 11:09 AM Thomas Weise wrote: > +1 > > It would also be helpful to mention important current restrictions wrt > availability on runners, etc. > > On Mon, Aug

Re: BEAM-5180 for 2.7.0 ?

Replies on the Jira and PR. For now we should go ahead with rollback to unblock 2.7 On Fri, Aug 24, 2018 at 10:21 AM Udi Meiri wrote: > +Ankur Goenka (Kenneth is out of office) > > On Fri, Aug 24, 2018 at 3:20 AM Tim Robertson > wrote: > >> Thanks Jozef for bringing this

Re: Tensor Flow support on Flink Runner

Hi Rakesh, You can use DirectRunner to run Tensorflow on Beam. Please refer https://www.tensorflow.org/tfx/transform/get_started to learn more about it. We are working on making Tensorflow available on Flink Runner. You can build Beam from Master and run the Tensorflow model on it. Here are

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

o say that I'm not great at anything more then a single bundle at a > time but utilizing this information by a runner should be optional. > > > > On Fri, Aug 17, 2018 at 1:53 PM Ankur Goenka wrote: > >> To recap the discussion it seems that we have come-up with following >&

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

t; >>>> The problem would be resolved if we scheduled the tasks in topological >>>> order. Doing that is not easy because they run in separate Flink >>>> operators and the SDK Harness would have to have insight into the >>>> execution grap

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

ther it should ask the SDK to do more work or less but its > not an explicit don't do more then X bundles in parallel. > > On Fri, Aug 17, 2018 at 5:55 PM Ankur Goenka wrote: > >> Makes sense. Having exposed upper bound on concurrency with optimum >> concurrency can give a good b

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

that number). > > A dynamic computation would use the same limits from the SDK, but take > into account its own resource usage (incl. the usage by running bundles). > > On Fri, Aug 17, 2018 at 6:20 PM Ankur Goenka wrote: > >> I am thinking upper bound to be more on the lines

Re: Vendoring / Shading Protobuf and gRPC

time. >>> >>> On Tue, Jul 17, 2018 at 7:01 PM Thomas Weise wrote: >>> >>>> Thanks, the classpath order matters indeed. >>>> >>>> Still not able to run RemoteExecutionTest, but I was able to get the >>>> Flink portable te

Re: Live coding & reviewing adventures

+1 For reference here is a sample job https://github.com/axelmagn/model-analysis/blob/axelmagn-hacks/examples/chicago_taxi/preprocess_flink.sh Also 1 quick heads up, output file will be created in docker container if you use local file system. On Tue, Jul 17, 2018 at 2:27 PM Holden Karau

Re: Vendoring / Shading Protobuf and gRPC

For reference: I was able to make intellij work with the master by doing following steps 1. Remove module :beam:vendor-sdks-java-extensions-protobuf from intellij. 2. Adding :beam-model-fn-execution/build/libs/beam-model-fn-execution-2.6.0-SNAPSHOT.jar and

Re: Vendoring / Shading Protobuf and gRPC

since other dependencies are missing. > > > On Tue, Jul 17, 2018 at 4:02 PM Ankur Goenka wrote: > >> For reference: >> I was able to make intellij work with the master by doing following steps >> >>1. Remove module :beam:vendor-sdks-java-extensions-protobuf from >

Re: Process JobBundleFactory for portable runner

execute that one >> > > >> > standalone. We should make sure this is >> > > also an >> > > >> option. >> > > >> > >> > &g

Enhancing Environment Proto to support Docker and Process Environments.

Hi, We recently added the ProceessEnvironment which uses forked process instead of Docker container to run SDKHarness. But the current Environment proto does not have a well defined structure to represent a process. Current Proto: message Environment { // (Required) The URL of a container

New Post Commit Task fails in SetupVirtualEnv when running on Jenkins

Hi, I added a new Post Commit task "beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle" to Jenkins in https://github.com/apache/beam/pull/6322 but the jenkins job is failing with logs below. However the gradle task works fine when i execute it on my local machine using ./gradlew

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome Dan! Thank you for the contribution. On Thu, Sep 6, 2018 at 3:13 PM Ismaël Mejía wrote: > Dal Tran this is really nice, this is a much awaited contribution, welcome! > On Thu, Sep 6, 2018 at 9:18 PM Ahmet Altay wrote: > > > > Welcome! Thank you! > > > > On Thu, Sep 6, 2018 at 11:40 AM,

Re: Discussion: Scheduling across runner and SDKHarness in Portability framework

gt; However, it still seems to me to be a semantically problematic idea to >>> send non-executable bundles to the SDK harness. I understand it's useful >>> for Flink, but is that really the best path forward? >>> >>> >>> >>> On Mon, Aug 20, 2018 at 5:44 PM An

Re: New Post Commit Task fails in SetupVirtualEnv when running on Jenkins

. On Thu, Sep 6, 2018 at 6:37 PM Ankur Goenka wrote: > Hi, > > I added a new Post Commit task > "beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle" to Jenkins in > https://github.com/apache/beam/pull/6322 but the jenkins job is failing > with logs below. >

Re: Live coding & reviewing adventures

Thanks Holden for doing this. Looking forward to attend the live session. Suggestion: It will super useful to do a live session for beam setup for users and another for contributors. On Fri, Jul 13, 2018 at 12:33 PM Innocent Djiofack wrote: > Thanks I think this will be super useful. I will

Re: Beam Summit community feedback

Thanks Max for sharing. On Fri, Oct 5, 2018 at 7:05 AM Maximilian Michels wrote: > Hi, > > What do you think about collecting some of the feedback from the > community at Beam Summit last week? Here's what I've come across: > > > * The Kubernetes / Docker Story > > Multiple users reported that

Re: [DISCUSS] - Separate JIRA notifications to a new mailing list

+1 On Thu, Oct 11, 2018 at 12:14 PM Suneel Marthi wrote: > +1 > > Sent from my iPhone > > On Oct 11, 2018, at 8:03 PM, Łukasz Gajowy > wrote: > > This is a good idea. +1 > > Łukasz > > > czw., 11 paź 2018, 18:01 użytkownik Udi Meiri napisał: > >> +1 to split JIRA notifications >> >> On Thu,

Re: Docker missing on Beam15

apache.org/jira/browse/BEAM-5467 >>> >>> Any help to further investigate or fix would be appreciated! >>> >>> Thanks, >>> Thomas >>> >>> >>> >>> On Fri, Oct 19, 2018 at 4:51 PM Yifan Zou wrote: >>> >>>&g

Re: Docker missing on Beam15

rm -rf > /var/lib/apt/lists/**11:02:12* ---> Running in 5e9b67be03f9*11:02:12* grpc: > the connection is unavailable > > > - Yifan > > > > On Fri, Oct 19, 2018 at 12:45 PM Ankur Goenka wrote: > >> Hi, >> >> Flink Validates Runner test cases are failing on Bea

Re: a new contributor

Welcome Heejong! On Fri, Oct 19, 2018 at 12:27 PM Rui Wang wrote: > Welcome! > > -Rui > > On Fri, Oct 19, 2018 at 11:55 AM Robin Qiu wrote: > >> Welcome, Heejong! >> >> On Fri, Oct 19, 2018 at 11:55 AM Ahmet Altay wrote: >> >>> Welcome! >>> >>> On Fri, Oct 19, 2018 at 11:48 AM, Heejong Lee

Docker missing on Beam15

Hi, Flink Validates Runner test cases are failing on Beam 15 because docker is not installed. Failing tasks https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/buildTimeTrend Can we install docker on all the machines as the Portable Validates Runner tests need it. Thanks, Ankur

Re: New Edit button on beam.apache.org pages

Great addition to the website  On Wed, Oct 24, 2018 at 2:51 PM Ruoyun Huang wrote: > Looks awesome! > > On Wed, Oct 24, 2018 at 2:24 PM Alan Myrvold wrote: > >> To make small documentation changes easier, there is now an Edit button >> at the top right of the pages on https://beam.apache.org.

Re: Never get spotless errors with this one weird trick

Thanks for sharing the trick. On Thu, Nov 1, 2018 at 9:30 AM Kenneth Knowles wrote: > Hi all, > > Scott just separated the spotless check from the Java unit test precommit > job, so you get faster feedback on spotless errors. > > I wondered if there was a good place to just always reformat,

Re: Stackoverflow Questions

+1 for the daily/weekly digest to user@ On Mon, Nov 5, 2018 at 10:52 AM Maximilian Michels wrote: > Great idea! I'd prefer a daily/weekly digest if possible. > > On 05.11.18 19:44, Tim Robertson wrote: > > Thanks for raising this Anton > > > > It would be very easy to forward new SO

Re: Python profiling

All containers are destroyed by default on termination so to analyze profiling data for portable runners, either disable container cleanup (using --retainDockerContainers=true) or use remote distributed file system path. On Mon, Nov 5, 2018 at 1:05 AM Robert Bradshaw wrote: > Any portable

Re: How to use "PortableRunner" in Python SDK?

Hi, The Portable Runner requires a job server uri to work with. The current default job server docker image is broken because of docker inside docker issue. Please refer to https://beam.apache.org/roadmap/portability/#python-on-flink for how to run a wordcount using Portable Flink Runner.

Add cleanup flag to DockerPayload

Hi, In portable flink runner, SDK Harness docker containers are created dynamically and are not garbage collected. SDK Harness container pull the staging artifact, generate logs and tmp files which is stored as an additional layer on top of image. These dead container layers accumulates over time

Re: Splitting the repo

Hi, I think the subtext here is that development is hard in general. I agree to it. And a major cause of it is diversity of languages, complexity of the project and legacy code. To alleviate language related issues, we are trying to have modular code which we already have to a certain extent. On

Re: Log output from Dataflow tests

Hi Max, I don't have edit privileges for the project so can't modify user. On Wed, Oct 10, 2018 at 9:02 AM Maximilian Michels wrote: > Thank you Scott! Ismael also sent me the logs and I could fix the error. > > It seems we have granted read-only access to project members in the > past. I just

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

Congrats Kenn! On Wed, Sep 19, 2018 at 1:35 PM Amit Sela wrote: > Well deserved! Congrats Kenn. > > On Wed, Sep 19, 2018 at 4:25 PM Kai Jiang wrote: > >> Congrats, Kenn! >> ᐧ >> >> On Wed, Sep 19, 2018 at 1:23 PM Alan Myrvold wrote: >> >>> Congrats, Kenn. >>> >>> On Wed, Sep 19, 2018 at 1:08

Cleanup Jenkins old jobs

Hi, Jenkins UI has accumulated a lot of old jobs over time which are not in use any more. Shall we clean old jobs (Jobs which did not run in last 7 days) from the jenkins UI for a cleaner view of valid jobs? This is a low risk cleanup as Seed Job will recreate valid jobs if one gets removed.

Re: Modular IO presentation at Apachecon

Thanks for sharing. Great slides and looking for the recorded session. Do we have a central location where we link all the beam presentations for discoverability? On Wed, Sep 26, 2018 at 9:35 PM Thomas Weise wrote: > Thanks for sharing. I'm looking forward to see the recording of the talk >

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

I agree with the cleanliness of the Commit history. "Fixup!", "Address comments", "Address even more comments" type of comments does not convey meaningful information and are not very useful. Its a good idea to squash them. However, I think its ok to keep separate commits for different logical

Re: Python Flink tests failing on Jenkins

The problem can be because of long task name. I have seen this happen is post commits and hence shortened the name to " beam_PostCommit_Python_VR_Flink" (Mind the VR instead of validates runner). Created a new PR to address this https://github.com/apache/beam/pull/7539 On Wed, Jan 16, 2019 at

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

- Do we also need additional code changes for snapshots and releases to > default to these specific containers? There could be a version based > mechanism to resolve the correct container to use. > > On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: > >> Hi All, >

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

h Python 3 minor version that >> Beam is compatible with. >> > > Probably worth a separate discussion. I would favor first releasing a > python 3 compatible version before figuring out how we would target > multiple python 3 versions. > > >> >> On Wed, Jan

Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

Hi All, As portability/FnApi is taking shape and are compatible with ULR and Flink. I wanted to discuss the release plan release of SDKHarness Docker images. Of-course users can create their own images but it will be useful to have a default image available out of box. Pre build image are a must

Re: Beam JobService Problem

Thanks Sam for bringing this to the list. As preparation_ids are not reusable, having preparation_id and job_id same makes sense to me for Flink. Another option is to have a subscription for all states/messages on the JobServer. This will be similar to "docker". As the container id is created

Re: How to use "PortableRunner" in Python SDK?

;). > > > > If this issue is non-trivial to solve, I would vote for removing > > > > this default behavior as part of the consolidation. > > > > > > > > On Thu, Nov 8, 2018 at 2:58 AM Maximilian Michels < > m...@apache.org > &g

Re: How to use "PortableRunner" in Python SDK?

> > On Thu, Nov 8, 2018 at 2:58 AM Maximilian Michels > > > mailto:m...@apache.org> > > > > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote: > > > > > > >

Re: [RFC] I made a new tabbed Beam view in Jenkins

This is really cool. Another request for cleaning the old build history from the list :) On Thu, Dec 20, 2018 at 2:22 AM Łukasz Gajowy wrote: > Looks great. +1 for making this a top level view. > > >> After that, maybe we could clean the categories so they fit into the tabs >> more easily with

Re: beam9 failing most of the python tests

t;, line 1423, in install_python raise e OSError: [Errno 11] Resource temporarily unavailable Running virtualenv with interpreter /usr/bin/python2 On Mon, Dec 3, 2018 at 1:12 PM Ankur Goenka wrote: > Hi, > > I see that beam9 is failing significantly more number of python related

beam9 failing most of the python tests

Hi, I see that beam9 is failing significantly more number of python related builds [1]. This also results in more failure of beam_PreCommit_Portable_Python_Commit [2] on beam9. Can someone with access to beam9 take a look? Thanks, Ankur [1] https://builds.apache.org/computer/beam9/builds [2]

Re: What is Jenkins job "Portable_Python" in PreCommit?

We added new precommit which tests a wordcount pipeline on portable flink runner. In recent time we missed catching some obvious issues which broke potability and could have been caught by this test. The current test is fairly light weight and executes in ~5min which seems to be reasonable for a

Re: Questions on [MD5] hash code of staged files

Hi Ruoyun, We moved from MD5 to SHA256 hashing which caused this problem. The java and python code was updated in PR https://github.com/apache/beam/pull/6583 though GO code was not updates. Go caches the generated code which caused tests to pass. Though I am not sure why we did not break

Re: JB's back

Good to have you back! On Wed, Nov 21, 2018, 10:06 AM Thomas Weise Welcome back, JB! > > Also good to know that the Spark runner will get some attention. > > On Wed, Nov 21, 2018 at 8:37 AM K Fred wrote: > >> Welcome back! >> >> On Thu, Nov 22, 2018, 00:29 Lukasz Cwik > >>> Good to see you

Re: Enforce javadoc comments in public methods?

I think it makes sense. Having an annotation to suppress this check for a method/class instead of adding trivial comment would be useful. On Mon, Jan 7, 2019 at 9:53 AM Ruoyun Huang wrote: > Yeah. Agree there is no reason to enforce anything for trivial methods > like setter/getter. > > What I

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 On Sun, Sep 16, 2018 at 3:20 AM Maximilian Michels wrote: > +1 (binding) > > On 15.09.18 20:07, Reuven Lax wrote: > > +1 > > > > On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > > wrote: > > > > +1 > > > > -Rui > > > > On Sat, Sep 15, 2018 at 12:32 AM Robert

Re: Migrating Beam SQL to Calcite's code generation

Thats Awesome! Thanks Team! On Tue, Sep 18, 2018 at 10:58 AM Jean-Baptiste Onofré wrote: > Awesome, > > thanks Andrew !! > > Regards > JB > > On 17/09/2018 23:27, Andrew Pilloud wrote: > > I've adapted Calcite's EnumerableCalc code generation to generate the > > BeamCalc DoFn. The primary

Re: Jenkins seed job fails

It seems that the cron is not deleted if the job is deleted. I ran the seed job manually to test the new job which would have created the cron. I am not aware of jenkins internal but it will be great if we can clean up the cron on job deletion. On Thu, Jan 24, 2019 at 7:40 AM Thomas Weise

Re: Our jenkins beam1 server is down

Beam 1 seems to be down again https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou wrote: > The beam1 and 14 are back and building. > > On Thu, Jan

Re: [DISCUSS] Backwards compatibility of @Experimental features

I think a release version with Experimental flag makes sense. In addition, I think many of our user start to rely on experimental features because they are not even aware that these features are experimental and its really hard to find the experimental features used without giving a good look at

Re: [ANNOUNCE] New committer announcement: Mark Liu

Congratulations Mark! On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster wrote: > Wonderful, congrats Mark! > > On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold wrote: > >> congratulations, Mark!!! >> >> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang wrote: >> >>> Congratulations Mark! >>> >>> On Mon,

Re: Virtualenv setup issues on new machine

gt; Is there more information if you run it with --info? > > On Thu, Feb 28, 2019, 14:35 Ankur Goenka wrote: > >> Hi Beamers, >> >> I am trying build python sdk from a fresh git checkout on a new linux >> machine but the setupVirtualEnv task is failing with the error

Virtualenv setup issues on new machine

Hi Beamers, I am trying build python sdk from a fresh git checkout on a new linux machine but the setupVirtualEnv task is failing with the error below. The complete build scan is at https://scans.gradle.com/s/h3jwzeg5aralk/failure?openFailures=WzBd=WzQsM10#top=0 >From the error it seems that

Re: Virtualenv setup issues on new machine

Not sure, dog seems to be the only meaningful difference in 2 scenarios. On Thu, Feb 28, 2019 at 8:04 PM Udi Meiri wrote: > Weird. Is that a known bug? > > On Thu, Feb 28, 2019 at 3:19 PM Ankur Goenka wrote: > >> The issue seems to be with "." in the virtualenv pat

beam9 bad worker

Beam9 is failing all the scheduled jobs. Can we reboot the machine?

Re: [ANNOUNCE] New committer announcement: Michael Luckey

Congratulations Michael! On Wed, Feb 27, 2019 at 2:25 PM Thomas Weise wrote: > Congrats Michael! > > > On Wed, Feb 27, 2019 at 12:41 PM Gleb Kanterov wrote: > >> Congratulations and welcome! >> >> On Wed, Feb 27, 2019 at 8:57 PM Connell O'Callaghan >> wrote: >> >>> Excellent thank you for

Re: Python PVR Reference post-commit tests failing

On that note, we should also think about adding PVR for python reference runners. Jira: https://issues.apache.org/jira/browse/BEAM-6837 On Thu, Mar 14, 2019 at 12:57 PM Kenneth Knowles wrote: > How about this compromise: > > 1. disable the test since clearly no one is relying on the

Re: Hazelcast Jet Runner

Hi Can, Like GreedyPipelineFuser, we have added many more components which makes building a Portable Runner easy. Here is a link [1] to slides which explains at a very high level what is needed to add a new portable runner. Still adding a portable runner will be more complex than adding a native

Re: Another new contributor!

Welcome Brian! On Fri, Feb 1, 2019 at 6:49 AM Mikhail Gryzykhin < gryzykhin.mikh...@gmail.com> wrote: > Welcome to the community! > > On Thu, Jan 31, 2019, 16:50 Alex Amato wrote: > >> Great to start working with you Brian, welcome. >> >> On Thu, Jan 31, 2019 at 4:23 PM Brian Hulette >> wrote:

Re: Beam Python streaming pipeline on Flink Runner

Hi Matthias, Unfortunately, unbounded reads including pubsub are not yet supported for portable runners. Thanks, Ankur On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens wrote: > Hi everyone, > > Last few days I have been trying to run a streaming pipeline (code on > Github

Re: New contributor: Michał Walenia

Welcome Michał! long press "l" on mac to type "ł' :) On Wed, Jan 30, 2019 at 7:57 PM Maximilian Michels wrote: > Welcome Michał! > > I do have to find out how to type ł without copy/pasting it every time ;) > > On 30.01.19 15:22, Łukasz Gajowy wrote: > > Hi all, > > > > a new fellow joined

Re: [EXT] Re: [DOC] Portable Spark Runner

Thanks for sharing. This looks great! On Mon, Apr 15, 2019 at 2:54 PM Kenneth Knowles wrote: > Great. Thanks for sharing! > > On Mon, Apr 15, 2019 at 2:38 PM Lei Xu wrote: > >> This is super nice! Really look forward to use this. >> >> On Mon, Apr 15, 2019 at 2:34 PM Thomas Weise wrote: >>

Re: [review?] WordCount in Kotlin

Thanks Pablo and Harshit. Just a quick reminder, please squash the "fixup" sort of commits in the PR based on the prior discussion on the mailing list https://lists.apache.org/thread.html/6d922820d6fc352479f88e5c8737f2c8893ddb706a1e578b50d28948@%3Cdev.beam.apache.org%3E On Fri, Apr 12, 2019 at

Re: [review?] WordCount in Kotlin

Best > -P. > > On Fri, Apr 12, 2019, 12:22 PM Ankur Goenka wrote: > >> Thanks Pablo and Harshit. >> >> Just a quick reminder, please squash the "fixup" sort of commits in the >> PR based on the prior discussion on t

Re: Enable security for data channels in portability

this secret to authenticate the worker (by using gRPC's interceptor for >>>>this customized auth) >>>>3. By having the 2 steps above, we *no longer need transport layer >>>>security *(SSL/TLS). So we abandon our initial plan to enable >>>&

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

ramework. >> > >>> >> > >>> Mark >> > >>> >> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold >> wrote: >> > >>>> >> > >>>> +1 This would be great. gcr.io seems like a

Re: [VOTE] Release 2.13.0, release candidate #2

this branch. > > Thanks, > Thomas > > > On Tue, Jun 4, 2019 at 1:59 AM Maximilian Michels wrote: > >> The summary is not correct. Binding votes (in order): >> >> Ahmet Altay >> Robert Bradshaw >> Maximilian Michels >> Jean-Baptiste Onofré &g

Re: [VOTE] Release 2.13.0, release candidate #2

; I would actually suggest that the following search needs to be triaged to > zero before cutting an RC: > https://github.com/apache/beam/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+base%3Arelease-2.13.0 > . > > On Tue, Jun 4, 2019 at 11:17 AM Ankur Goenka wrote: > >> Sorry, I mis

Re: [VOTE] Release 2.13.0, release candidate #2

t we ask all cherry pick requests to > have a JIRA filed against that release and marked as blockers? > > On Tue, Jun 4, 2019 at 1:05 PM Ankur Goenka wrote: > >> That makes sense. >> I would also like to add that the corresponding PR should be added to an >> open blocki

Re: [VOTE] Release 2.13.0, release candidate #1

run without errors on the 2.13.0 Spark runner ( > https://github.com/tensorflow/tfx/pull/84). > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > | +1650203 > > > On Tue, May 28, 2019 at 11:53 AM Ankur Goenka wrote: > >> Thanks for the validatio

Re: [VOTE] Release 2.13.0, release candidate #1

Hi All, In the meanwhile Please validate RC1 to catch anyother issues. Thanks, Ankur On Tue, May 28, 2019 at 3:37 PM Ankur Goenka wrote: > Valentyn, Can you please send the cherry pick PR for > https://issues.apache.org/jira/browse/BEAM-7439 > > On Tue, May 28, 2019 at 3:04 PM

  1   2   >