Gradle Tips and tricks

After some discussion on slack it is clear that we need to document some of the gradle replacements of our common maven commands. We started a shared doc yesterday to share some of those and other gradle tips and tricks. I invite everyone who can help to add their favorite gradle 'incantations' the

Re: Gradle Tips and tricks

Agree, it's what I mentioned this morning on Slack. I think it would be great to have a summary of the current state on the dev mailing list. At the end of the day, the contribution guide should be updated (we have a Jira about that afair). Regards JB On 04/06/2018 09:03 AM, Ismaël Mejía wrote:

Re: Gradle Tips and tricks

By the way, I'm doing a new pass on the current state of the gradle build. Regards JB On 04/06/2018 09:03 AM, Ismaël Mejía wrote: > After some discussion on slack it is clear that we need to document > some of the gradle replacements of our common maven commands. We > started a shared doc yesterd

[PROPOSAL] Preparing 2.5.0 release next week

Hi guys, Apache Beam 2.4.0 has been released on March 20th. According to our cycle of release (roughly 6 weeks), we should think about 2.5.0. I'm volunteer to tackle this release. I'm proposing the following items: 1. We start the Jira triage now, up to Tuesday 2. I would like to cut the rele

Re: [PROPOSAL] Preparing 2.5.0 release next week

+1 to get 2.5 out asap (fixes blockers so always good to let it be upgraded) +1000 to split beam releases (and even repos) by concerns Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Gith

Re: [PROPOSAL] Python 3 support

Hi all, I don't seem to have the permissions to create a Kanban board or even assign tasks to myself. Who could help me with this? I've updated the coders package pull request [1] and added the applied strategy to the proposal document [2]. It would be great to get some feedback on this, so we ca

Re: Gradle questions on Eclipse and End to End tests

> On Apr 5, 2018, at 5:31 PM, Daniel Oliveira wrote: > > So I'm working on updating the Beam Contributor's Guide to swap references to > Maven with Gradle. I'm wondering if anyone can help with two trouble spots > I'm having: > > 1. Has anyone set up Eclipse to work with Gradle for beam deve

Re: Gradle questions on Eclipse and End to End tests

> > > 2. I'm having trouble finding relevant information for this section: E2E > Testing Framework > . > Does anyone know what the progress is on E2E tests in Gradle? > > This section seems to relate to both things like WordCountIT

building on top of filesystem, can beam help?

Hi guys, I have a use case where I'd like to be able to expose to a user some file system navigation and enable him to visualize the file system (as in beam sense) Technically it is a matter of being able to use glob pattern to browse the file system using match(specs). What is important in that

About the Gauge metric API

Hello all, As I was working on adding support for Gauges in Dataflow, some noted that Gauge is a fairly unusual kind of metric for a distributed environment, since many workers will report different values and stomp on each other's all the time. We also looked at Flink and Dropwizard Gauge metrics

Re: About the Gauge metric API

+1 on the proposal to support a "String" gauge. To expand a bit, the current API doesn't make it clear that the gauge value is based on local state. If a runner chooses to parallelize a DoFn across many workers, each worker will have its own local Gauge metric and its updates will overwrite other

Re: Gradle questions on Eclipse and End to End tests

Gradle has plugins for creating the necessary project files for various IDEs. I haven't tried it yet, but I would recommend starting there: * https://docs.gradle.org/current/userguide/eclipse_plugin.html * https://docs.gradle.org/current/userguide/idea_plugin.html On Fri, Apr 6, 2018 at 8:15 AM Ł

Re: About the Gauge metric API

Some metrics backend label the value, for instance with the worker that sent it. Then the aggregation is latest per label. This makes it useful for holding values such as "memory usage" that need to hold current value. On Fri, Apr 6, 2018, 9:00 AM Scott Wegner wrote: > +1 on the proposal to supp

Re: About the Gauge metric API

See for instance how gauge metrics are handled in Prometheus, Datadog and Stackdriver monitoring. Gauges are perfect for use in distributed systems, they just need to be properly labeled. Perhaps we should apply a default tag or allow users to specify one. On Fri, Apr 6, 2018, 9:14 AM Ben Chambers

Re: About the Gauge metric API

Hi Ben : D Sure, that's reasonable. And perhaps I started the discussion in the wrong direction. I'm not questioning the utility of Gauge metrics. What I'm saying is that Beam only supports integers,, but Gauges are aggregated by dropping old values depending on their update times; so it might be

Gradle Status [April 6]

I wanted to start a thread to summarize the current state of Gradle migration. We've made lots of good progress so far this week. Here's the status from what I can tell-- please add or correct anything I missed: * Release artifacts can be built and published for Snapshot and officlal releases [1]

Re: Gradle Status [April 6]

Hi Scott, is it right that 2 doesn't handle the hierachy anymore and that it doesn't handle profiles for runners as it is currently with maven? Romain Manni-Bucau @rmannibucau | Blog | Old Blog

Re: About the Gauge metric API

> > I would be in favor of replacing the existing Gauge.set(long) API with the > String version and removing the old one. This would be a breaking change. > However this is a relatively new API and is still marked @Experimental. > Keeping the old API would retain the potential confusion. It's bette

Re: About the Gauge metric API

A String API makes it clear(er) that the values will not be aggregated in any way across workers. I don't think retaining both APIs (except for possibly some short migration period) worthwhile. On another note, I still find the distributed gague API to be a bit odd in general. On Fri, Apr 6, 2018

Re: About the Gauge metric API

I am not opposed to removing other data types, though they are extra convenience for user. In Scott's example above, if the metric is a counter, what are the guarantees provided? E.g. would it match the global count using GBK? If yes, then gauges (especially per-key gauges) can be very useful too

Re: Gradle Status [April 6]

Romain, are you talking about the profiles that exist as part of the archetype examples? If so, then those still exist and haven't been changed. If not, can you provide a link to the profile in a pom file to be clearer? On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau wrote: > Hi Scott, > > i

Re: About the Gauge metric API

Just naively - the use cases that Gauge addresses seem relevant, and the information seems feasible to gather and present. The bit that doesn't seem to make sense is aggregating gauges by clobbering each other. So I think that's just +1 Ben? On Fri, Apr 6, 2018 at 10:26 AM Raghu Angadi wrote: >

Re: About the Gauge metric API

A gauge API is only useful if there's a correlation to a distributed worker, because "clobber" is not a useful aggregation method. There's no useful correlation of that signal to anything actionable. Ben's already noted this point that metrics backends do this, but it seems that if gauge is to be a

Re: Gradle Status [April 6]

I'm working on finding a solution for launching the Nexmark suite with each runner. This doesn't have to be done via Gradle, but we anyhow need built artifacts that don't require user classpath intervention. It looks to me like the examples are also missing this - they have separate configuration

Re: About the Gauge metric API

Gauges are incredibly useful for exposing the current state of the system. For instance, number of elements in a queue, current memory usage, number of RPCs in flight, etc. As mentioned above, these concepts exist in numerous systems for monitoring distributed environments, including Stackdriver Mo

Re: Gradle Status [April 6]

Le 6 avr. 2018 20:09, "Lukasz Cwik" a écrit : Romain, are you talking about the profiles that exist as part of the archetype examples? Was more thinking to this kind of profiles https://github.com/apache/beam/blob/master/sdks/java/nexmark/pom.xml (which should hit all IO at some point to ensure

Re: Gradle Tips and tricks

I think Scott just sent a summary. I agree, even when all coding work is done, the migration isn't done until the contribution guide (and any other documentation) is updated. On Fri, Apr 6, 2018 at 1:07 AM Jean-Baptiste Onofré wrote: > Agree, it's what I mentioned this morning on Slack. > > I t

Re: Gradle Status [April 6]

Why building a zip per runner which its stack and just pointing out on that zip and let beam lazy load the runner: --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the fromSystemProperties() if it gets merged a day ;)) Le 6 avr. 2018 20:21, "Kenneth Knowles" a écrit : > I'm w

Re: building on top of filesystem, can beam help?

Personally, this is a case where I think forking might be a better option, even though I'm not generally a fan of duplicating code. In past projects, depending on internal modules of other projects never seems lead to good outcomes. FileSystem exists for Beam today, and Beam might make changes to

Re: About the Gauge metric API

Nobody wants to get rid of Gauges. I see that we have three separate themes being discussed here, and I think it's useful to point them out and address them independently: 1. Whether Gauges should change to hold string values. 2. If Gauges are to support string values, whether Gauges should also c

Re: About the Gauge metric API

Thanks for unraveling those themes, Pablo! 1. Seems reasonable in light of behaviors metrics backends support. 2. Those same backends support histogramming of data, so having integer types is very useful. 3. I believe that is the case, for the reasons I mentioned earlier, Gauges should only clobbe

Re: About the Gauge metric API

In terms of natural language, I don't think "gauge" makes sense strings. A gauge measures a quantity. A string is not a quantity. So I like a separate API, like Robert says. Backends can go ahead and implement leaf String and Gauge collectors with the same data structure if they like. In implement

Re: About the Gauge metric API

Generally strong +1 to everything Bill said. I would suggest though that the per-worker segmentation might be specified using some more general tagging/labeling API. For instance, all of the following seem like reasonable uses to support: 1. Gauge that is tagged with worker to get per-worker segme

Re: About the Gauge metric API

On Fri, Apr 6, 2018 at 12:23 PM Ben Chambers wrote: > Generally strong +1 to everything Bill said. I would suggest though that > the per-worker segmentation might be specified using some more general > tagging/labeling API. For instance, all of the following seem like > reasonable uses to support

Re: building on top of filesystem, can beam help?

Issues forking are: 1. I have to drop beam FileIO (in all its flavors) which means not taking any benefit from beam in that area which is 50% of beam gain (the other being the portability) 2. I have to maintain a bridge for all filesystem impl - being said it still misses some info ATM 3. It is st

Re: building on top of filesystem, can beam help?

So is this project of yours also built on top of Beam, or is it unrelated? On Fri, Apr 6, 2018 at 1:12 PM Romain Manni-Bucau wrote: > Issues forking are: > > 1. I have to drop beam FileIO (in all its flavors) which means not taking > any benefit from beam in that area which is 50% of beam gain (

Re: Gradle Tips and tricks

The goal of this document is about extending the contribution guide with some of the cases that we use from time to time during development for quick or specific validations. This comes from a previous discussion where Kenneth mentioned that he had also a list of 'incantations' for maven (like i di

Re: building on top of filesystem, can beam help?

Partially. Will run with beam in half of the cases or without in the remaining 50% (and in this case the dependencies+api are currently blocking). My constraint is that to activate any feature i must be able to cover both cases. Le 6 avr. 2018 22:14, "Reuven Lax" a écrit : > So is this projec

Re: building on top of filesystem, can beam help?

In the other thread, we suggested writing a Beam FileSystem impl that wraps VFS. Is that a path forward here? Then you can build on top of VFS instead, and simply layer VfsFilesystem on top of it when running on Beam. On Fri, Apr 6, 2018 at 1:23 PM Romain Manni-Bucau wrote: > Partially. Will run

Re: building on top of filesystem, can beam help?

I did a PR for that but it just beings connectivity to beam. To solve any issue the opposite is the only valid option. Le 6 avr. 2018 22:31, "Reuven Lax" a écrit : In the other thread, we suggested writing a Beam FileSystem impl that wraps VFS. Is that a path forward here? Then you can build on

Re: [PROPOSAL] Python 3 support

I tried to create a shared kanban board but I failed. I think I am lacking some permission to create a shared filter. Could someone help with creating this? The filter I planned to use was "project = BEAM AND (parent = BEAM-2784 OR parent = BEAM-1251) ORDER BY Rank ASC" Ahmet On Fri, Apr 6, 2018

Re: [PROPOSAL] Python 3 support

I had a similar problem. On Fri, Apr 6, 2018, 6:23 PM Ahmet Altay wrote: > I tried to create a shared kanban board but I failed. I think I am lacking > some permission to create a shared filter. Could someone help with creating > this? > > The filter I planned to use was "project = BEAM AND (par

Re: Gradle Status [April 6]

Here's an end-of-day update on migration work: * Snapshot unsigned dailies and signed release builds are working (!!). PR/5048 [1] merges changes from Luke's branch * python precommit failing... will investigate python precommit Monday * All Precommits are gradle only * All Postcommits except pe