Re: Documenting Metrics API?

There is an open task for this on JIRA: https://issues.apache.org/jira/browse/BEAM-1974 On Tue, May 15, 2018 at 10:56 AM Etienne Chauchot wrote: > Hi Pablo, > I don't know if it is what you seek but we have at least this doc that is > user facing but it is a bit old: > >

Re: [ANNOUCEMENT] New Foundation members!

Congrats! On Sat, Mar 31, 2018 at 2:30 AM Ahmet Altay wrote: > Congratulations to all of you! > > > On Fri, Mar 30, 2018, 4:29 PM Pablo Estrada wrote: > >> Congratulations y'all! Very cool. >> Best >> -P. >> >> On Fri, Mar 30, 2018 at 4:09 PM Davor Bonaci

Re: [INFO] Spark runner updated to Spark 2.2.1

Nice! On Mon, Dec 18, 2017 at 12:51 PM Jean-Baptiste Onofré wrote: > Hi all, > > We are pleased to announce that Spark 2.x support in Spark runner has been > merged this morning. It supports Spark 2.2.1. > > In the same PR, we did update to Scala 2.11, including Flink

Re: Hi

Added to contributors list, welcome aboard! On Tue, Oct 3, 2017 at 12:45 PM Dennis Jung <inylov...@gmail.com> wrote: > Hello, > Thanks Aviem! I'll start with that. > > JIRA ID : djkooks > > BR, > Dennis > > 2017-10-03 18:23 GMT+09:00 Aviem Zur <aviem...@gmai

Re: Hi

Hi Dennis, You can take a look at the "Contribute to Beam" page on the website, and most importantly the Contribution Guide https://beam.apache.org/contribute/ You can find open "starter" tasks on JIRA using the labels "starter" or "newbie" like so:

Re: Hi

Srinivas you have been added. Welcome aboard! On Tue, Oct 3, 2017 at 10:27 AM Srinivas Reddy wrote: > Hi, > > I am Srinivas, Sr Data Engineer at Kogentix. I would like to contribute to > beam project. > > Could you add me to contributor list. My JIRA username is :

Re: Hi

Welcome Yoni On Tue, Oct 3, 2017 at 3:03 AM Lukasz Cwik wrote: > You have been added. Welcome. > > On Mon, Oct 2, 2017 at 2:15 AM, Yonatan Seneor wrote: > > > Hi > > My username on Apache JIRA is: yseneor > > Thanks > > Yoni Seneor > > > > > >

Re: Contributor introduction

Added, welcome Uri! On Mon, Oct 2, 2017 at 3:32 PM Jean-Baptiste Onofré wrote: > Hi Uri, > > what's your Jira ID ? > > Thanks, > Regards > JB > > On 10/02/2017 02:31 PM, Uri Silberstein wrote: > > Hi all, > > > > My name is Uri Silberstein and I am part of a PayPal team that

Re: new guy

Welcome JB #2! Glad to have you on board. On Tue, Aug 29, 2017 at 5:38 PM Joey Baruch wrote: > my jira username is joeyfezster > > thanks > > On Tue, Aug 29, 2017 at 4:12 PM Jean-Baptiste Onofré > wrote: > > > Welcome ! > > > > What's your apache id ?

Re: kafka docs

Hi Joey. This would be great. Also, KafkaIO requires a specific dependency to be added (beam-sdks-java-io-kafka), we should probably put that as a maven snippet in the README as well. Feel free to create a PR with this README on GitHub. Regarding the long series of links you need to click in the

Re: Policy for stale PRs

Makes sense to close after a long time of inactivity and no response, and as Kenn mentioned they can always re-open. On Wed, Aug 16, 2017 at 12:20 AM Jean-Baptiste Onofré wrote: > If we consider the author, it makes sense. > > Regards > JB > > On Aug 15, 2017, 01:29, at

Re: [ANNOUNCEMENT] New committers, August 2017 edition!

Congrats! On Mon, Aug 14, 2017 at 6:43 PM Tyler Akidau wrote: > Congrats and thanks all around! > > On Sat, Aug 12, 2017 at 12:09 AM Aljoscha Krettek > wrote: > > > Congrats, everyone! It's well deserved. > > > > Best, > > Aljoscha > > > > > On

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

We also have two tests failing in Spark runner as detailed by the following two tickets: https://issues.apache.org/jira/browse/BEAM-2670 https://issues.apache.org/jira/browse/BEAM-2671 On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré wrote: > Hi all, > > due to

Re: [VOTE] Release 2.1.0, release candidate #2

gt; >> > >> mvn compile exec:java \ > >> --settings ../settings.xml \ > >> -Pdirect-runner \ > >> -D exec.mainClass=org.apache.beam.examples.WordCount \ > >> -D exec.args="--inputFile=pom.xml --output=counts" > >&g

Re: [VOTE] Release 2.1.0, release candidate #2

t; > > as mentioned in the first e-mail: > > > > - Distributions are available here: > > https://dist.apache.org/repos/dist/dev/beam/2.1.0/ > > > > - Artifacts are on the staging repository: > > https://repository.apache.org/content/repositories/orgapachebeam

Re: [VOTE] Release 2.1.0, release candidate #2

Have the jars for RC2 been uploaded somewhere? On Wed, Jul 19, 2017 at 10:19 AM Jean-Baptiste Onofré wrote: > So, I guess you are voting +1 on RC2, correct (just for the tracking) ? > > Thanks, > Regards > JB > > On 07/19/2017 08:00 AM, Ahmet Altay wrote: > > Thank you JB. >

Re: [DISCUSS] Bridge beam metrics to underlying runners to support metrics reporters?

Hi Cody, Some of the runners have their own metrics sink, for example Spark runner uses Spark's metrics sink which you can configure to send the metrics to backends such as Graphite. There have been ideas floating around for a Beam metrics sink extension which will allow users to send Beam

Re: low availability in the coming 4 weeks

Congratulations! On Fri, May 26, 2017 at 9:21 AM Kenneth Knowles wrote: > Congrats! > > On Thu, May 25, 2017 at 2:00 PM, Raghu Angadi > wrote: > > > Congrats Mingmin. All the best! > > > > On Wed, May 24, 2017 at 8:33 PM, Mingmin Xu

Re: First stable release completed!

Awesome! Now let's make Beam the standard in data processing. On Thu, May 18, 2017 at 5:05 AM Jason Kuster wrote: > Fantastic work everyone! I'm really excited to see what we've accomplished, > and the future for Beam looks bright. > > On Wed, May 17, 2017 at

Re: Website homepage visual refresh

Cool! On Wed, May 17, 2017 at 12:33 PM Mark Liu wrote: > This is awesome! thanks Jeremy. > > On Tue, May 16, 2017 at 10:49 AM, Sourabh Bajaj < > sourabhba...@google.com.invalid> wrote: > > > +1 this is great. > > > > On Tue, May 16, 2017 at 10:18 AM Jesse Anderson <

Re: Process for getting the first stable release out

+1. A document similar to the one we had for the Hackathon could serve us here again. A section for acceptance criteria compiled by the community and a matrix of tests per runner to be filled for each RC version could help us synchronize and get there. On Fri, May 5, 2017 at 10:42 PM Dan

Re: [INFO] Build is broken on the archetypes

Looks like this is due to a bug in generate-sources.sh Until we fix that bug you can fix your local directory by running the following: rm -rf sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/src rm -rf

Re: Congratulations Davor!

Congrats Davor! :) On Thu, May 4, 2017 at 10:42 AM Jean-Baptiste Onofré wrote: > Congrats ! Well deserved ;) > > Regards > JB > > On 05/04/2017 09:30 AM, Jason Kuster wrote: > > Hi all, > > > > The ASF has just published a blog post[1] welcoming new members of the > > Apache

Re: An Update on Jenkins

Thanks for the update, Jason! On Wed, Apr 26, 2017 at 6:51 AM Jason Kuster wrote: > Hey folks, > > There have been a couple of different issues over the last couple of days > related to some necessary updates Infra has been working on. We've tracked > down the

Re: Community hackathon

No problem, Sean. Invite sent. On Tue, Apr 25, 2017 at 6:14 PM Sean Story wrote: > I'd also love to be added to the slack channel > > > Thanks, > > Sean Story > > > > On Apr 25, 2017, at 12:54 AM, Davor Bonaci wrote: > > > > Thanks everyone

Re: Hanging Jenkins builds.

^ > >> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_ > >> MavenInstall/sdks/java/core/src/main/java/org/apache/beam/ > >> sdk/transforms/CombineFns.java:147: > >> warning - Tag @link:can't find composeKeyed() in > >> org.apach

Hanging Jenkins builds.

Hi all, Please be aware that Beam builds (precommit + postcommit validations) are hanging since a few hours ago. This seems to be a problem in builds of other projects as well (for example, Kafka). I've opened an INFRA ticket: https://issues.apache.org/jira/browse/INFRA-13949

Re: [DISCUSSION] PAssert success/failure count validation for all runners

; > > >>> >>> On 7. Apr 2017, at 12:42, Kenneth Knowles <k...@google.com.INVALID> > >>> >> wrote: > >>> >>> > >>> >>> We also have a design that improves the signal even without > metrics, > >>> so > >>>

Re: Pipeline termination in the unified Beam model

+1 To help integrate this we can start by adding `ValidatesRunner` tests with a new category and run it only with runners which adhere to the rules mentioned, and eventually in all runners. On Fri, Mar 3, 2017 at 12:46 AM Amit Sela wrote: > +1 on Eugene's words - this

Re: Renaming SideOutput

+1 On Wed, Apr 12, 2017 at 6:06 AM JingsongLee wrote: > strong +1 > best, > JingsongLee--From:Tang > Jijun(上海_技术部_数据平台_唐觊隽) Time:2017 Apr 12 (Wed) > 10:39To:dev@beam.apache.org

[PROPOSAL] Standard IO Metrics

Hi all, We are currently in the process of introducing IO metrics to Beam. Questions have been raised as to what the metrics names should be, and if they should be standard across different IOs. I've written this up as a proposal found here: https://s.apache.org/standard-io-metrics As usual,

Re: Combine.Global

obally(new > CustomCombineFn())).setCoder(CustomTuple.coder); > > > The InputT is not the same as OutputT, so the input coder can't be used. > > On 2017-04-07 08:58 (-0500), Aviem Zur <aviem...@gmail.com> wrote: > > Have you set the coder for your input PCollection? The one on wh

[DISCUSSION] PAssert success/failure count validation for all runners

Currently, PAssert assertions may not happen and tests will pass while silently hiding issues. Up until now, several runners have implemented an assertion that the number of expected successful assertions have actually happened, and that no failed assertions have happened. (runners which check

Re: Combine.Global

Have you set the coder for your input PCollection? The one on which you perform the Combine? On Fri, Apr 7, 2017 at 4:24 PM Paul Gerver wrote: > Hello All, > > I'm trying to test out a Combine.Globally transform which takes in a small > custom class (CustomA) and outputs a

Re: [DISCUSSION] Consistent use of loggers

future... > > On Mon, Apr 3, 2017 at 8:14 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Fair enough. +1 especially for the documentation. > > > > Regards > > JB > > > > > > On 04/03/2017 08:48 PM, Aviem Zur wrote: > > &

Re: [DISCUSSION] Consistent use of loggers

> On 3. Apr 2017, at 17:56, Aviem Zur <aviem...@gmail.com> wrote: > > > >> * java.util.logging could be a good choice for the Direct Runner > > Yes, this will be great for users (Instead of having no logging when > using > > direct runner). > > >

Re: [DISCUSSION] Consistent use of loggers

Python SDK needs to expands its logging capabilities. Filed [1] > for this. > > Ahmet > > [1] https://issues.apache.org/jira/browse/BEAM-1825 > > > > > > On 3/22/17, 5:46 AM, "Aviem Zur" <aviem...@gmail.com> wrote: > > > > +1 to what JB said. &

Re: [DISCUSSION] Consistent use of loggers

> I think, in our dependencies set, we should just depend to slf4j-api and > let the > user provides the binding he wants (slf4j-log4j12, slf4j-simple, whatever). > > We define a binding only with test scope in our modules. > > Regards > JB > > On 03/22/2017 04:58 AM, Aviem Zur

Re: [ANNOUNCEMENT] New committers, March 2017 edition!

> > > driving > > > > > the > > > > > > Splittable DoFn effort [4]. A true expert on IO subsystem, Eugene > > has > > > > > > reviewed nearly every IO contributed to Beam. Finally, Eugene > > > > contributed > > > > >

Default shading configuration and opting out

Hi all, https://github.com/apache/beam/pull/2096 introduced a common shading configuration for all of the modules in the project. The reason for this is that modules which are dependent on Guava may leak this dependency to the user and this could conflict with the version of Guava they require.

Add GitHub topics to Beam repository

About a month ago GitHub introduced topics, which let GitHub users query for repositories by topics (domains that the repos deal with). We can leverage these to increase Beam's exposure on GitHub. Example topics we could add: big-data, google-cloud-dataflow, spark, flink, apex, gearpump We can

Re: Interest in a (virtual) contributor meeting?

+1 On Wed, Feb 22, 2017 at 5:45 AM Jesse Anderson wrote: > Sounds good. > > On Tue, Feb 21, 2017, 7:19 PM Davor Bonaci wrote: > > > In the early days of the project, we have held a few meetings for the > > initial community to get to know each other.

Re: Metrics for Beam IOs.

Is there a way to leverage runners' existing metrics sinks? As stated by Amit & Stas, Spark runner uses Spark's metrics sink to report Beam's aggregators and metrics. Other runners may also have a similar capability, I'm not sure. This could remove the need for a plugin, and dealing with

Re: Metrics for Beam IOs.

Hi Ismaël, You've raised some great points. Please see my comments inline. On Tue, Feb 14, 2017 at 3:37 PM Ismaël Mejía wrote: > ​Hello, > > The new metrics API allows us to integrate some basic metrics into the Beam > IOs. I have been following some discussions about this

Re: Projects for Google Summer of Code 2017

Kenn that scholarly documents project sounds awesome. On Fri, Feb 3, 2017 at 11:48 PM Kenneth Knowles wrote: > In fact, I have just learned that our deadline to file project _is_ > February 9th. Having good ideas is part of the ASF's application process. > > Here's a

Re: Better developer instructions for using Maven?

Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457 On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Yeah. Agree. Time extend is not huge and it's worth to add it in verify > phase. > > Regards > JB > > On Feb 10, 2017,

Issue with Coder documentation regarding context

Hi, I think improvements can be made to the documentation of `encode` and `decode` methods in `Coder`. A coder may be used to encode/decode several objects using a single stream, you cannot assume that the stream the coder encodes to/decodes from only contains bytes representing a single object.

Re: PTransform style guide PR

Very well written. Examples for every concept make it very easily relatable and understandable. On Tue, Jan 31, 2017 at 3:52 AM Eugene Kirpichov wrote: > I don't think I'll have capacity to review every PR that brings particular > Beam transforms in accordance with

Re: TextIO binary file

like that got answered properly. I also like Dan's suggestion to use AvroIO > to serialize byte[] arrays and you can do whatever you want with them (e.g. > use another serialization library, say, Kryo, or Java serialization, etc.) > > On Sun, Feb 5, 2017 at 11:37 AM Aviem Zur <av

Re: TextIO binary file

<rober...@google.com.invalid> wrote: > On Tue, Jan 31, 2017 at 12:04 PM, Aviem Zur <aviem...@gmail.com> wrote: > > +1 on what Stas said. > > I think there is value in not having the user write a custom IO for a > > protocol they use which is not covered by Beam IOs.

Re: TextIO binary file

ing the same IO. > > On Tue, Jan 31, 2017 at 2:48 AM Aviem Zur <aviem...@gmail.com> wrote: > > > So If I understand the general agreement is that TextIO should not > support > > anything but lines from files as strings. > > I'll go ahead and file a

Re: TextIO binary file

of text as String, and > not > > have a withCoder parameter at all. > > > > The proper way to address your use case is to write a custom > > FileBasedSource. > > On Mon, Jan 30, 2017 at 2:52 AM Aviem Zur <aviem...@gmail.com> wrote: > > > &g

Re: TextIO binary file

wrote: > Hi Aviem, > > TextIO is not designed to write/read binary file: it's pure Text, so > String. > > Regards > JB > > On 01/30/2017 09:24 AM, Aviem Zur wrote: > > Hi, > > > > While trying to use TextIO to write/read a binary file rather than Strin

TextIO binary file

Hi, While trying to use TextIO to write/read a binary file rather than String lines from a textual file I ran into an issue - the delimiter TextIO uses seems to be hardcoded '\n'. See `findSeparatorBounds` -

Pipeline graph reflection

Hi all, While working on implementing metrics support in the Spark Runner a need arose for composing a unique identifier of a transform, to differentiate it from other transforms with the same name. With the help of @bjchambers I understood that something similar to this exists in the Dataflow

Re: [ANNOUNCEMENT] New committers, January 2017 edition!

Congrats! On Fri, Jan 27, 2017, 06:25 Thomas Weise wrote: > Congrats! > > > On Thu, Jan 26, 2017 at 7:49 PM, María García Herrero < > mari...@google.com.invalid> wrote: > > > Congratulations and thank you for your contributions thus far! > > > > On Thu, Jan 26, 2017 at 6:00 PM,

Re: Committed vs. attempted metrics results

stion might be does the > > > pipeline > > > > > result even need query methods? Runners could add them as necessary > > > based > > > > > on the levels of querying the support. > > > > > > > > > > The other desire was to make the a