Re: Documenting Metrics API?

There is an open task for this on JIRA: https://issues.apache.org/jira/browse/BEAM-1974 On Tue, May 15, 2018 at 10:56 AM Etienne Chauchot wrote: > Hi Pablo, > I don't know if it is what you seek but we have at least this doc that is > user facing but it is a bit old: > > https://docs.google.com

Re: [ANNOUCEMENT] New Foundation members!

Congrats! On Sat, Mar 31, 2018 at 2:30 AM Ahmet Altay wrote: > Congratulations to all of you! > > > On Fri, Mar 30, 2018, 4:29 PM Pablo Estrada wrote: > >> Congratulations y'all! Very cool. >> Best >> -P. >> >> On Fri, Mar 30, 2018 at 4:09 PM Davor Bonaci wrote: >> >>> Now that this is public.

Re: [INFO] Spark runner updated to Spark 2.2.1

Nice! On Mon, Dec 18, 2017 at 12:51 PM Jean-Baptiste Onofré wrote: > Hi all, > > We are pleased to announce that Spark 2.x support in Spark runner has been > merged this morning. It supports Spark 2.2.1. > > In the same PR, we did update to Scala 2.11, including Flink artifacts > update to > 2.1

Re: Hi

Added to contributors list, welcome aboard! On Tue, Oct 3, 2017 at 12:45 PM Dennis Jung wrote: > Hello, > Thanks Aviem! I'll start with that. > > JIRA ID : djkooks > > BR, > Dennis > > 2017-10-03 18:23 GMT+09:00 Aviem Zur : > > > Hi Dennis, > > >

Re: Hi

Hi Dennis, You can take a look at the "Contribute to Beam" page on the website, and most importantly the Contribution Guide https://beam.apache.org/contribute/ You can find open "starter" tasks on JIRA using the labels "starter" or "newbie" like so: https://issues.apache.org/jira/issues/?jql=proj

Re: Hi

Srinivas you have been added. Welcome aboard! On Tue, Oct 3, 2017 at 10:27 AM Srinivas Reddy wrote: > Hi, > > I am Srinivas, Sr Data Engineer at Kogentix. I would like to contribute to > beam project. > > Could you add me to contributor list. My JIRA username is : mrsrinivas > > > On 3 October 2

Re: Hi

Welcome Yoni On Tue, Oct 3, 2017 at 3:03 AM Lukasz Cwik wrote: > You have been added. Welcome. > > On Mon, Oct 2, 2017 at 2:15 AM, Yonatan Seneor wrote: > > > Hi > > My username on Apache JIRA is: yseneor > > Thanks > > Yoni Seneor > > > > > > ‫בתאריך יום ב׳, 2 באוק׳ 2017 ב-11:26 מאת ‪Yonatan

Re: Contributor introduction

Added, welcome Uri! On Mon, Oct 2, 2017 at 3:32 PM Jean-Baptiste Onofré wrote: > Hi Uri, > > what's your Jira ID ? > > Thanks, > Regards > JB > > On 10/02/2017 02:31 PM, Uri Silberstein wrote: > > Hi all, > > > > My name is Uri Silberstein and I am part of a PayPal team that works with > > Beam.

Re: new guy

Welcome JB #2! Glad to have you on board. On Tue, Aug 29, 2017 at 5:38 PM Joey Baruch wrote: > my jira username is joeyfezster > > thanks > > On Tue, Aug 29, 2017 at 4:12 PM Jean-Baptiste Onofré > wrote: > > > Welcome ! > > > > What's your apache id ? > > > > Regards > > JB > > > > On 08/29/20

Re: kafka docs

Hi Joey. This would be great. Also, KafkaIO requires a specific dependency to be added (beam-sdks-java-io-kafka), we should probably put that as a maven snippet in the README as well. Feel free to create a PR with this README on GitHub. Regarding the long series of links you need to click in the

Re: Policy for stale PRs

Makes sense to close after a long time of inactivity and no response, and as Kenn mentioned they can always re-open. On Wed, Aug 16, 2017 at 12:20 AM Jean-Baptiste Onofré wrote: > If we consider the author, it makes sense. > > Regards > JB > > On Aug 15, 2017, 01:29, at 01:29, Ted Yu wrote: > >

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

Congrats! On Mon, Aug 14, 2017 at 6:43 PM Tyler Akidau wrote: > Congrats and thanks all around! > > On Sat, Aug 12, 2017 at 12:09 AM Aljoscha Krettek > wrote: > > > Congrats, everyone! It's well deserved. > > > > Best, > > Aljoscha > > > > > On 12. Aug 2017, at 08:06, Pei HE wrote: > > > > > >

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

We also have two tests failing in Spark runner as detailed by the following two tickets: https://issues.apache.org/jira/browse/BEAM-2670 https://issues.apache.org/jira/browse/BEAM-2671 On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré wrote: > Hi all, > > due to https://issues.apache.org/jir

Re: [VOTE] Release 2.1.0, release candidate #2

a \ > >> --settings ../settings.xml \ > >> -Pdirect-runner \ > >> -D exec.mainClass=org.apache.beam.examples.WordCount \ > >> -D exec.args="--inputFile=pom.xml --output=counts" > >> > >> > >> mvn compile exec:j

Re: [VOTE] Release 2.1.0, release candidate #2

- Distributions are available here: > > https://dist.apache.org/repos/dist/dev/beam/2.1.0/ > > > > - Artifacts are on the staging repository: > > https://repository.apache.org/content/repositories/orgapachebeam-1019/ > > > > Regards > > JB > > > >

Re: [VOTE] Release 2.1.0, release candidate #2

Have the jars for RC2 been uploaded somewhere? On Wed, Jul 19, 2017 at 10:19 AM Jean-Baptiste Onofré wrote: > So, I guess you are voting +1 on RC2, correct (just for the tracking) ? > > Thanks, > Regards > JB > > On 07/19/2017 08:00 AM, Ahmet Altay wrote: > > Thank you JB. > > > > I validated py

Re: [DISCUSS] Bridge beam metrics to underlying runners to support metrics reporters?

Hi Cody, Some of the runners have their own metrics sink, for example Spark runner uses Spark's metrics sink which you can configure to send the metrics to backends such as Graphite. There have been ideas floating around for a Beam metrics sink extension which will allow users to send Beam metric

Re: [DISCUSS] Apache Beam 2.1.0 release next week ?

+1 There are important bug fixes that need to be released. On Thu, Jun 22, 2017 at 11:42 AM Etienne Chauchot wrote: > +1 on Ismaël words, but not a blocking point indeed, maybe more a nice > to have. > > > Le 22/06/2017 à 06:59, Ismaël Mejía a écrit : > > Thahks JB for keeping the time based rel

Re: low availability in the coming 4 weeks

Congratulations! On Fri, May 26, 2017 at 9:21 AM Kenneth Knowles wrote: > Congrats! > > On Thu, May 25, 2017 at 2:00 PM, Raghu Angadi > wrote: > > > Congrats Mingmin. All the best! > > > > On Wed, May 24, 2017 at 8:33 PM, Mingmin Xu wrote: > > > > > Hello everyone, > > > > > > I'll take 4 week

Re: First stable release completed!

Awesome! Now let's make Beam the standard in data processing. On Thu, May 18, 2017 at 5:05 AM Jason Kuster wrote: > Fantastic work everyone! I'm really excited to see what we've accomplished, > and the future for Beam looks bright. > > On Wed, May 17, 2017 at 2:00 PM, Mark Liu > wrote: > > > Co

Re: Website homepage visual refresh

Cool! On Wed, May 17, 2017 at 12:33 PM Mark Liu wrote: > This is awesome! thanks Jeremy. > > On Tue, May 16, 2017 at 10:49 AM, Sourabh Bajaj < > sourabhba...@google.com.invalid> wrote: > > > +1 this is great. > > > > On Tue, May 16, 2017 at 10:18 AM Jesse Anderson < > je...@bigdatainstitute.io >

Re: Process for getting the first stable release out

+1. A document similar to the one we had for the Hackathon could serve us here again. A section for acceptance criteria compiled by the community and a matrix of tests per runner to be filled for each RC version could help us synchronize and get there. On Fri, May 5, 2017 at 10:42 PM Dan Halperin

Re: [INFO] Build is broken on the archetypes

Looks like this is due to a bug in generate-sources.sh Until we fix that bug you can fix your local directory by running the following: rm -rf sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src rm -rf sdks/java/maven-archetypes/examples-java8/src/main/resources/archetyp

Re: Congratulations Davor!

Congrats Davor! :) On Thu, May 4, 2017 at 10:42 AM Jean-Baptiste Onofré wrote: > Congrats ! Well deserved ;) > > Regards > JB > > On 05/04/2017 09:30 AM, Jason Kuster wrote: > > Hi all, > > > > The ASF has just published a blog post[1] welcoming new members of the > > Apache Software Foundation,

Re: [DISCUSSION] Encouraging more contributions

@ Sourabh Bajaj The mentoring on starter tickets is an interesting Idea. How would it technically work?. A new contributor assigns a starter ticket to themselves. What happens from there? On Tue, Apr 25, 2017 at 12:01 PM Ismaël Mejía wrote: > I think it is important to clarify that the develop

Re: An Update on Jenkins

Thanks for the update, Jason! On Wed, Apr 26, 2017 at 6:51 AM Jason Kuster wrote: > Hey folks, > > There have been a couple of different issues over the last couple of days > related to some necessary updates Infra has been working on. We've tracked > down the last couple of issues, and the late

Re: Community hackathon

No problem, Sean. Invite sent. On Tue, Apr 25, 2017 at 6:14 PM Sean Story wrote: > I'd also love to be added to the slack channel > > > Thanks, > > Sean Story > > > > On Apr 25, 2017, at 12:54 AM, Davor Bonaci wrote: > > > > Thanks everyone for the enthusiasm! > > > > Let's go with this Wednesd

Re: Hanging Jenkins builds.

t; /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_ > >> MavenInstall/sdks/java/core/src/main/java/org/apache/beam/ > >> sdk/transforms/CombineFns.java:147: > >> warning - Tag @link:can't find composeKeyed() in > >> org.apache.beam.sdk.transforms.Com

[DISCUSSION] Encouraging more contributions

Hi all, I wanted to start a discussion about actions we can take to encourage more contributions to the project. A few points I've been thinking of: 1. Have people unassign themselves from issues they're not actively working on. 2. Have the community engage more in triage, improving tickets desc

Hanging Jenkins builds.

Hi all, Please be aware that Beam builds (precommit + postcommit validations) are hanging since a few hours ago. This seems to be a problem in builds of other projects as well (for example, Kafka). I've opened an INFRA ticket: https://issues.apache.org/jira/browse/INFRA-13949

Re: [DISCUSSION] PAssert success/failure count validation for all runners

; > transforms rather than an absolute number of successes. > >>> > > >>> > Kenn > >>> > > >>> > > >>> >>> On 7. Apr 2017, at 12:42, Kenneth Knowles > >>> >> wrote: > >>> &

Re: Pipeline termination in the unified Beam model

+1 To help integrate this we can start by adding `ValidatesRunner` tests with a new category and run it only with runners which adhere to the rules mentioned, and eventually in all runners. On Fri, Mar 3, 2017 at 12:46 AM Amit Sela wrote: > +1 on Eugene's words - this shows how batch is concept

Re: Renaming SideOutput

+1 On Wed, Apr 12, 2017 at 6:06 AM JingsongLee wrote: > strong +1 > best, > JingsongLee--From:Tang > Jijun(上海_技术部_数据平台_唐觊隽) Time:2017 Apr 12 (Wed) > 10:39To:dev@beam.apache.org Subject:答复: Renaming > SideOutput > +1 more clearer > >

[PROPOSAL] Standard IO Metrics

Hi all, We are currently in the process of introducing IO metrics to Beam. Questions have been raised as to what the metrics names should be, and if they should be standard across different IOs. I've written this up as a proposal found here: https://s.apache.org/standard-io-metrics As usual, th

Re: Combine.Global

mCombineFn())).setCoder(CustomTuple.coder); > > > The InputT is not the same as OutputT, so the input coder can't be used. > > On 2017-04-07 08:58 (-0500), Aviem Zur wrote: > > Have you set the coder for your input PCollection? The one on which you > > perform the Co

[DISCUSSION] PAssert success/failure count validation for all runners

Currently, PAssert assertions may not happen and tests will pass while silently hiding issues. Up until now, several runners have implemented an assertion that the number of expected successful assertions have actually happened, and that no failed assertions have happened. (runners which check thi

Re: Combine.Global

Have you set the coder for your input PCollection? The one on which you perform the Combine? On Fri, Apr 7, 2017 at 4:24 PM Paul Gerver wrote: > Hello All, > > I'm trying to test out a Combine.Globally transform which takes in a small > custom class (CustomA) and outputs a secondary custom class

Re: [DISCUSSION] Consistent use of loggers

t use cases in the future... > > On Mon, Apr 3, 2017 at 8:14 PM, Jean-Baptiste Onofré > wrote: > > > Fair enough. +1 especially for the documentation. > > > > Regards > > JB > > > > > > On 04/03/2017 08:48 PM, Aviem Zur wrote: > > > >

Re: [DISCUSSION] Consistent use of loggers

I think we can exclude log4j from the Flink dependencies. It’s > somewhat annoying that they are there in the first place. > > The Flink doc has this to say about the topic: > https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/logging.html > > On 3. Apr 2017,

Re: [DISCUSSION] Consistent use of loggers

> Good point. > > > > > > I think, in our dependencies set, we should just depend to > slf4j-api > > and > > > let the > > > user provides the binding he wants (slf4j-log4j12, slf4j-simple, > > whatever). > >

Re: [DISCUSSION] Consistent use of loggers

set, we should just depend to slf4j-api and > let the > user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever). > > We define a binding only with test scope in our modules. > > Regards > JB > > On 03/22/2017 04:58 AM, Aviem Zur wrote: > > Hi all,

[DISCUSSION] Consistent use of loggers

Hi all, There have been a few reports lately (On JIRA [1] and on Slack) from users regarding inconsistent loggers used across Beam's modules. While we use SLF4J, different modules use a different logger behind it (JUL, log4j, etc) So when people add a log4j.properties file to their classpath for

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

commits (22,122 > > ++ > > > / > > > > > > 18,407 --) since February 2016 [1]. In recent months, he’s been > > > driving > > > > > the > > > > > > Splittable DoFn effort [4]. A true expert on IO subsystem, Eugene

Default shading configuration and opting out

Hi all, https://github.com/apache/beam/pull/2096 introduced a common shading configuration for all of the modules in the project. The reason for this is that modules which are dependent on Guava may leak this dependency to the user and this could conflict with the version of Guava they require. A

Add GitHub topics to Beam repository

About a month ago GitHub introduced topics, which let GitHub users query for repositories by topics (domains that the repos deal with). We can leverage these to increase Beam's exposure on GitHub. Example topics we could add: big-data, google-cloud-dataflow, spark, flink, apex, gearpump We can als

Re: Interest in a (virtual) contributor meeting?

+1 On Wed, Feb 22, 2017 at 5:45 AM Jesse Anderson wrote: > Sounds good. > > On Tue, Feb 21, 2017, 7:19 PM Davor Bonaci wrote: > > > In the early days of the project, we have held a few meetings for the > > initial community to get to know each other. Since then, the community > has > > grown a

Re: Better developer instructions for using Maven?

#x27;t skip > > checkstyle, findbugs, etc, for that, we have a fastinstall property). > > > > Maybe it would make more sense to use a specific property like > > -DfastBuild=true. > > > > WDYT ? > > > > Regards > > JB > > > > > > On

Re: Better developer instructions for using Maven?

I've created a PR which disables slow verifications if '-DskipTests' was specified, otherwise runs them. I think this satisfies all the considerations mentioned in this thread. PTAL: https://github.com/apache/beam/pull/2048 Ticket: https://issues.apache.org/jira/browse/BEAM-1513 On Thu, Feb 16, 2

Re: Metrics for Beam IOs.

Is there a way to leverage runners' existing metrics sinks? As stated by Amit & Stas, Spark runner uses Spark's metrics sink to report Beam's aggregators and metrics. Other runners may also have a similar capability, I'm not sure. This could remove the need for a plugin, and dealing with push/pull.

Re: Metrics for Beam IOs.

Hi Ismaël, You've raised some great points. Please see my comments inline. On Tue, Feb 14, 2017 at 3:37 PM Ismaël Mejía wrote: > ​Hello, > > The new metrics API allows us to integrate some basic metrics into the Beam > IOs. I have been following some discussions about this on JIRAs/PRs, and I >

Re: Projects for Google Summer of Code 2017

Kenn that scholarly documents project sounds awesome. On Fri, Feb 3, 2017 at 11:48 PM Kenneth Knowles wrote: > In fact, I have just learned that our deadline to file project _is_ > February 9th. Having good ideas is part of the ASF's application process. > > Here's a TL;DR of the instructions: >

Re: Better developer instructions for using Maven?

t;. > > > > I've listed the fraction of commits I think will break one of the > following > > if that property is not tested: > > > > * compiling (100%) > > * tests (100%) > > * checkstyle (90%) > > * javadoc (30%) > > * findbu

Re: Better developer instructions for using Maven?

Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457 On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré wrote: > Yeah. Agree. Time extend is not huge and it's worth to add it in verify > phase. > > Regards > JB > > On Feb 10, 2017, 10:13, at 10:13,

Re: Better developer instructions for using Maven?

l when I'm developing. > > No problem to change that. > > Regards > JB > > On Feb 10, 2017, 07:51, at 07:51, Aviem Zur wrote: > >Can we consider adding rat-plugin and findbugs to the default verify > >phase? > >Currently they only run when the `release` p

Re: Better developer instructions for using Maven?

;>> > > > >>> Right now we are living with the combination of the bad aspects - > > > >default > > > >>> is not thorough but not actually fast and a thorough check is > > > >>> undocumented. > > > >>> > > >

Re: Issue with Coder documentation regarding context

ava/org/apache/beam/sdk/coders/KvCoder.java#L89 > > On Thu, Feb 9, 2017 at 10:12 AM, Aviem Zur wrote: > > > Hi, > > > > I think improvements can be made to the documentation of `encode` and > > `decode` methods in `Coder`. > > > > A coder may be used to enc

Issue with Coder documentation regarding context

Hi, I think improvements can be made to the documentation of `encode` and `decode` methods in `Coder`. A coder may be used to encode/decode several objects using a single stream, you cannot assume that the stream the coder encodes to/decodes from only contains bytes representing a single object.

Re: PTransform style guide PR

Very well written. Examples for every concept make it very easily relatable and understandable. On Tue, Jan 31, 2017 at 3:52 AM Eugene Kirpichov wrote: > I don't think I'll have capacity to review every PR that brings particular > Beam transforms in accordance with the style guide - but I'm happ

Re: TextIO binary file

hink requires a rather less savvy user) > > It just seems like a really exotic set of circumstances to me. If the > problem is that people don't realize that the easiest way to serialize > their data is make it a POJO and use AvroIO, we can solve by improving > documentation

Re: TextIO binary file

ly. I also like Dan's suggestion to use AvroIO > to serialize byte[] arrays and you can do whatever you want with them (e.g. > use another serialization library, say, Kryo, or Java serialization, etc.) > > On Sun, Feb 5, 2017 at 11:37 AM Aviem Zur wrote: > > > I agree that the

Re: TextIO binary file

12:04 PM, Aviem Zur wrote: > > +1 on what Stas said. > > I think there is value in not having the user write a custom IO for a > > protocol they use which is not covered by Beam IOs. Plus having them deal > > with not only the encoding but also the IO part is not ideal. > &

Re: TextIO binary file

to write objects to files using > Beam coders, as opposed to just using e.g. AvroIO? > > Coders (should) make no promise as to what their wire format is, so such > files could be read back only by other Beam pipelines using the same IO. > > On Tue, Jan 31, 2017 at 2:48 AM Aviem Zu

Re: TextIO binary file

Looks like Eugene addressed this in the following ticket: https://issues.apache.org/jira/browse/BEAM-1354 Just added a bullet regarding updating the javadoc. On Tue, Jan 31, 2017 at 12:47 PM Aviem Zur wrote: > So If I understand the general agreement is that TextIO should not support > an

Re: TextIO binary file

> > > The proper way to address your use case is to write a custom > > FileBasedSource. > > On Mon, Jan 30, 2017 at 2:52 AM Aviem Zur wrote: > > > > > The Javadoc of TextIO states: > > > > > > * By default, {@link TextIO.Read} returns a {@link PC

Re: TextIO binary file

an-Baptiste Onofré wrote: > Hi Aviem, > > TextIO is not designed to write/read binary file: it's pure Text, so > String. > > Regards > JB > > On 01/30/2017 09:24 AM, Aviem Zur wrote: > > Hi, > > > > While trying to use TextIO to write/read a bin

TextIO binary file

Hi, While trying to use TextIO to write/read a binary file rather than String lines from a textual file I ran into an issue - the delimiter TextIO uses seems to be hardcoded '\n'. See `findSeparatorBounds` - https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sd

Re: Committed vs. attempted metrics results

le let's do > so. Preferably in a seperate PR or commit. > > B) Yes, direct runner should match whatever semantics we choose. > > On Thu, Jan 26, 2017, 5:30 AM Aviem Zur wrote: > > > Ben - yes, there is still some ambiguity regarding the querying of the > > metric

Pipeline graph reflection

Hi all, While working on implementing metrics support in the Spark Runner a need arose for composing a unique identifier of a transform, to differentiate it from other transforms with the same name. With the help of @bjchambers I understood that something similar to this exists in the Dataflow Ru

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Congrats! On Fri, Jan 27, 2017, 06:25 Thomas Weise wrote: > Congrats! > > > On Thu, Jan 26, 2017 at 7:49 PM, María García Herrero < > mari...@google.com.invalid> wrote: > > > Congratulations and thank you for your contributions thus far! > > > > On Thu, Jan 26, 2017 at 6:00 PM, Robert Bradshaw <

Re: Committed vs. attempted metrics results

methods? Runners could add them as necessary > > > based > > > > > on the levels of querying the support. > > > > > > > > > > The other desire was to make the accuracy clear. One implementation > > > path > > > > > was report

Committed vs. attempted metrics results

Hi all, While working on the implementation of metrics API in Spark runner the question of committed vs. attempted results has come up, sparking (no pun intended) an interesting conversation. (See API: MetricResult

Re: Better developer instructions for using Maven?

I agree with Dan and Lukasz. Developers should not be expected to know beforehand which specific profiles to run. The phase specified in the PR instructions (`verify`) should run all the relevant verifications and be the "slower" build, while a preceding lifecycle, such as `test`, should run the "f