Re: Gradle Status [April 6]

[Reuven Lax]

So if I understand correctly, we've migrated all precommit, most
postcommits, and we have a working release process using Gradle. There are
a few bugs left, but at this pace it sounds like we're close to fully
migrated.

I know that multiple people put it long hours last getting this done last
week (just look at the Slack messages!). This is awesome progress, and a
hearty thank you to everyone who put in their time.

Reuven

On Fri, Apr 6, 2018 at 7:52 PM Scott Wegner  wrote:

> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are working (!!).
> PR/5048 [1] merges changes from Luke's branch
>   * python precommit failing... will investigate python precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and Java_JDK_Versions_Test  use
> gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> wrote:
>
>> Why building a zip per runner which its stack and just pointing out on
>> that zip and let beam lazy load the runner:
>>
>> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
>> fromSystemProperties() if it gets merged a day ;))
>>
>> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>>
>>> I'm working on finding a solution for launching the Nexmark suite with
>>> each runner. This doesn't have to be done via Gradle, but we anyhow need
>>> built artifacts that don't require user classpath intervention.
>>>
>>> It looks to me like the examples are also missing this - they have
>>> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
>>> compared to a free-form launching of a main() program with a runner profile.
>>>
>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>>>
 Romain, are you talking about the profiles that exist as part of the
 archetype examples?

 If so, then those still exist and haven't been changed. If not, can you
 provide a link to the profile in a pom file to be clearer?

 On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it
> doesn't handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and
>> officlal releases [1]
>> * Gradle-generated releases have been validated with the the Apache
>> Beam archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed 
>> into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework
>> [5]
>> * Daniel is working on updating documentation to refer to Gradle
>> instead of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>> releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far!
>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>> issues assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] 

Re: About the Gauge metric API

[Reuven Lax]

Interesting point. Some runners might implement metrics via a control
plane, in which case per-key stuff is problematic for large numbers of
keys. However other runners may decide to implement metrics inside the
graph itself (e.g. by generating a CombinePerKey), in which case per-key
aggregation is less scary. Limits will be runner dependent.

On Fri, Apr 6, 2018 at 12:51 PM Robert Bradshaw  wrote:

> On Fri, Apr 6, 2018 at 12:23 PM Ben Chambers  wrote:
>
>> Generally strong +1 to everything Bill said. I would suggest though that
>> the per-worker segmentation might be specified using some more general
>> tagging/labeling API. For instance, all of the following seem like
>> reasonable uses to support:
>>
>> 1. Gauge that is tagged with worker to get per-worker segmentation (such
>> as queue size, memory usage, etc.)
>> 2. Gauge that is tagged with the "key" being processed. Would be useful
>> for things like how much data is buffered, where are watermark holds for
>> each key, etc. If processing is partitioned by key, this is strictly more
>> granular than per-worker.
>> 3. Counter tagged with the "key" being processed. Would be useful for
>> time spent processing each key, etc.
>>
>
> Per-key stuff gets really dangerous, as then the counter (control) plane
> has O(#keys) items to keep track of. That is unless it is paired with some
> kind of a lossy top/histogram aggregation.
>
> However, I think Bill hits the nail on the head that there is an implicit
> (ill-defined, in the model at least) segmentation desired here, with
> different aggregations happening within vs. across segments. (Also, FWIW,
> clobber is not the only aggregation that makes sense at the lowest level.)
> Default counters use the same aggregation across both levels, giving useful
> and well-defined semantics regardless of bundling and work distribution
> (assuming associative aggregation of course), but perhaps the
> counter/metrics APIs could be augmented to be able to explicitly request
> the level and differing aggregation with respect to this segmentation.
>
>
>> On Fri, Apr 6, 2018 at 11:39 AM Bill Neubauer  wrote:
>>
>>> Thanks for unraveling those themes, Pablo!
>>>
>>> 1. Seems reasonable in light of behaviors metrics backends support.
>>> 2. Those same backends support histogramming of data, so having integer
>>> types is very useful.
>>> 3. I believe that is the case, for the reasons I mentioned earlier,
>>> Gauges should only clobber previously values reported by the same entity.
>>> Two workers with the same gauge should not be overwriting each other's
>>> values, only their own. This implies per-worker segmentation.
>>>
>>>
>>> On Fri, Apr 6, 2018 at 11:35 AM Pablo Estrada 
>>> wrote:
>>>
 Nobody wants to get rid of Gauges. I see that we have three separate
 themes being discussed here, and I think it's useful to point them out and
 address them independently:

 1. Whether Gauges should change to hold string values.
 2. If Gauges are to support string values, whether Gauges should also
 continue to have an int API.
 3. Whether Beam should support some sort of label / tag / worker-id for
 aggregation of Gauges (maybe other metrics?)

 -P.

 On Fri, Apr 6, 2018 at 11:21 AM Ben Chambers 
 wrote:

> Gauges are incredibly useful for exposing the current state of the
> system. For instance, number of elements in a queue, current memory usage,
> number of RPCs in flight, etc. As mentioned above, these concepts exist in
> numerous systems for monitoring distributed environments, including
> Stackdriver Monitoring. The key to making them work is the addition of
> labels or tags, which as an aside are also useful for *all* metric types,
> not just Gauges.
>
> If Beam gets rid of Gauges, how would we go about exporting "current"
> values like memory usage, RPCs in flight, etc.?
>
> -- Ben
>
> On Fri, Apr 6, 2018 at 11:13 AM Kenneth Knowles 
> wrote:
>
>> Just naively - the use cases that Gauge addresses seem relevant, and
>> the information seems feasible to gather and present. The bit that 
>> doesn't
>> seem to make sense is aggregating gauges by clobbering each other. So I
>> think that's just +1 Ben?
>>
>> On Fri, Apr 6, 2018 at 10:26 AM Raghu Angadi 
>> wrote:
>>
>>> I am not opposed to removing other data types, though they are extra
>>> convenience for user.
>>>
>>> In Scott's example above, if the metric is a counter, what are the
>>> guarantees provided? E.g. would it match the global count using GBK? If
>>> yes, then gauges (especially per-key gauges) can be very useful too 
>>> (e.g.
>>> backlog for each Kafka partition/split).
>>>
>>> On Fri, Apr 6, 2018 at 10:01 AM Robert Bradshaw 

Re: About the Gauge metric API

[Ben Chambers]

Great summary overall. A few small things to add, along with a note that
more examples of the intended is for each metric/aggregation may be helpful.

It is worth looking at what existing metric systems provide. Specifically,
two things to consider:

1. Is scoping implict / automatic or explicit. In some systems a metric can
be declared as having one or more labels, and reporting it requires
providing values for those labels. This allows user defined labelling, and
also simplifies scoping to just ensuring the right values are available for
use as labels.

2. There is only one runner that supports logical metrics, and no standard
metrics backend support those. Perhaps metrics in Beam should focus on the
standardly available set? Logical makes the most sense for system metrics
(such as elements in a collection) and don't need to use the standard
mechanisms.

This would simultaneously simplify the metrics system and bring it more in
line across runners and metric backends.



On Sat, Apr 7, 2018, 11:37 AM Andrea Foegler  wrote:

> Hi folks,
>
> A lot of great / interesting use cases have come up on this thread!
>
> I would like to advocate for this discussion to include scoping and
> aggregation in a way that is consistent across types of metrics: Gauge,
> Cumulative, etc. In my head, Gauge = "instantaneous reading" and Cumulative
> = "incrementally changing".  There might be some implication on meaningful
> aggregations there, but feels like the specification and handling of scope
> and aggregations could be shared.
>
> The existing Cumulative Metric implementation may provide a few challenges
> to this, but also a good starting point.  Below I've summarized my
> interpretation of Cumulative Metric behavior and some considerations around
> flexibility.
>
> Cumulative Metrics
> * Programming model: Implicitly scoped to reporting Transform;
> * Reporting: Implicitly scoped to Transform and Bundle;
> * Monitoring: Explicitly scoped to Transform
>
> Re: accumulation
> The Bundle scoping at runtime is only strictly required for handling
> "Bundle failed" semantics.  But it also conveniently handles the
> distributed nature of the computation. Regardless of the "logical" nature
> of these metrics, I don't know that external monitoring systems support a
> notion of implicit aggregation across reporting entities.  I believe they
> would require a per-reporting-entity id/scope of some sort.  Bundle id
> works.  A worker identifier would work too.  Then the monitoring system
> would allow aggregations over those per-bundle or per-worker metrics.  But
> the reported metric values must have been received segmented by reporter.
> **Important note: the cardinality of Bundle is likely far to large to
> handle as a scope explicitly.  As Robert mentioned, the same is true for
> "key".  This could be an argument for making certain scopes only available
> to Distributions or at reporting time for internal aggregations.
>
> Re: scoping / identification
> There's an interesting consideration moving between the scoping of the
> programming language and the scoping of the intended metric.  By implicitly
> scoping Cumulative metrics to a ParDo, the scope of the metric and the
> scope of the variable are in sync.  By scoping to anything else, there's an
> implication that 2 different Metric variables parameterized by the same
> name/ID actually refer to the same Metric. So if a user wanted an
> rpc_error_count metric that accumulated RPC errors between the 3 different
> ParDos that make RPC calls, Cumulative Metrics can't currently support
> that.  If we plan to support something like that for Gauge, consistency
> would nice.
>
> Aside:  The current usage of Gauge in the SDK looks to be for a Globally
> observable value where the user would like to see the value exported, but
> there's no clear owner.  So every entity reports and the newest value
> wins.  Just mentioning so that that use case is considered.
>
> Not a proposal, just a high level organization of metric parameters that
> might be useful for discussion. (I'm not sure how to account for local
> aggregation here.  Kind of seems like any arbitrary code block could be
> supported there.)
> Metric Name: 
> Metric Type: Gauge, Cumulative, Distribution...
> Scope: Transform, Global, Worker... (set of?)
> Data Type: Integer, Double, String...
> Aggregation: Sum, Latest, Max...
> Metric ID:   + (s)
>
> Some potentially interesting questions:
> 1. How should the user specify the parameters of the metric?  What
> parameter combos are supported?
> 2. Should the SDK support different Metric variables contributing to the
> same Metric value? If so, how does the SDK communicate that?
> 3. Should runtime-only scopes be exposed to the user?  Useful for
> Distributions?
> 4. What should be exported with the metric - say to an external monitoring
> system?
>
> Cheers
> Andrea
>
>
> On Fri, Apr 6, 2018 at 12:51 PM Robert Bradshaw 
> wrote:
>
>> On 

Build failed in Jenkins: beam_Release_NightlySnapshot #738

[Apache Jenkins Server]

See 


Changes:

[wcn] Add float support for the SDK.

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

[wcn] Add diagnostics for sinks.

[wcn] Add hooks to the Go SDK.

[lcwik] [BEAM-3250] Add JUnit test reporting to Jenkins.

[altay] Migrate JUnit test archiving to Python Gradle build

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 2.79 MB...]
2018-04-07T08:52:33.452 [INFO] Excluding 
org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2 from 
the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding io.netty:netty-all:jar:4.1.17.Final 
from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding io.netty:netty:jar:3.9.9.Final from 
the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
io.dropwizard.metrics:metrics-jvm:jar:3.1.5 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
io.dropwizard.metrics:metrics-json:jar:3.1.5 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
io.dropwizard.metrics:metrics-graphite:jar:3.1.5 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding org.apache.ivy:ivy:jar:2.4.0 from the 
shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding net.razorvine:pyrolite:jar:4.13 from 
the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding net.sf.py4j:py4j:jar:0.10.6 from the 
shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
org.apache.spark:spark-tags_2.11:jar:2.3.0 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
org.apache.commons:commons-crypto:jar:1.0.0 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
2018-04-07T08:52:33.452 [INFO] Excluding 
org.apache.spark:spark-streaming_2.11:jar:2.3.0 from the shaded jar.
2018-04-07T08:52:35.675 [INFO] Replacing original artifact with shaded artifact.
2018-04-07T08:52:35.783 [INFO] 
2018-04-07T08:52:35.783 [INFO] --- maven-assembly-plugin:3.1.0:single 
(source-release-assembly) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:35.787 [INFO] Skipping the assembly in this project because 
it's not the Execution Root
2018-04-07T08:52:35.895 [INFO] 
2018-04-07T08:52:35.895 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.005 [INFO] 
2018-04-07T08:52:36.005 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.113 [INFO] 
2018-04-07T08:52:36.113 [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar 
(attach-javadocs) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.118 [INFO] Not executing Javadoc as the project is not a 
Java classpath-capable package
2018-04-07T08:52:36.228 [INFO] 
2018-04-07T08:52:36.228 [INFO] --- 
reproducible-build-maven-plugin:0.4:strip-jar (default) @ 
beam-sdks-java-javadoc ---
2018-04-07T08:52:36.229 [INFO] Stripping 

2018-04-07T08:52:36.691 [INFO] 
2018-04-07T08:52:36.691 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.693 [INFO] Skipping plugin execution
2018-04-07T08:52:36.800 [INFO] 
2018-04-07T08:52:36.800 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.802 [INFO] Installing 

 to 

2018-04-07T08:52:36.827 [INFO] Installing 

 to 

2018-04-07T08:52:36.976 [INFO] 
2018-04-07T08:52:36.977 [INFO] --- maven-deploy-plugin:2.8.2:deploy 
(default-deploy) @ beam-sdks-java-javadoc ---
2018-04-07T08:52:36.980 [INFO] Downloading from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-javadoc/2.5.0-SNAPSHOT/maven-metadata.xml
2018-04-07T08:52:37.233 [INFO] Downloaded from apache.snapshots.https: