Re: [KUDOS] Contributed runner: Apache Apex!

[Bhupendra Mishra]

Congratulations guys. Good job.

Sent from my iPhone

> On 17-Oct-2016, at 10:21 PM, Kenneth Knowles  wrote:
> 
> Hi all,
> 
> I would to, once again, call attention to a great addition to Beam: a
> runner for Apache Apex.
> 
> After lots of review and much thoughtful revision, pull request #540 has
> been merged to the apex-runner feature branch today. Please do take a look,
> and help us put the finishing touches on it to get it ready for the master
> branch.
> 
> And please also congratulate and thank Thomas Weise for this large
> endeavor, Vlad Rosov who helped get the integration tests working, and
> Guarav Gupta who contributed review comments.
> 
> Kenn

Re: Exploring Performance Testing

[amir bahmanyari]

Hi Jason,I have been busy bench-marking Flink Cluster (Spark next) under Beam.I 
can share my experience. Can you list items of interest to know so I can answer 
them to the best of my knowledge.Cheers

  From: Jason Kuster 
 To: dev@beam.incubator.apache.org 
 Sent: Monday, October 17, 2016 5:06 PM
 Subject: Exploring Performance Testing
   
Hey all,

Now that we've covered some of the initial ground with regard to
correctness testing, I'm going to be starting work on performance testing
and benchmarking. I wanted to reach out and see what people's experiences
have been with performance testing and benchmarking
frameworks, particularly in other Apache projects. Anyone have any
experience or thoughts?

Best,

Jason

-- 
---
Jason Kuster
Apache Beam (Incubating) / Google Cloud Dataflow


   

Exploring Performance Testing

[Jason Kuster]

Hey all,

Now that we've covered some of the initial ground with regard to
correctness testing, I'm going to be starting work on performance testing
and benchmarking. I wanted to reach out and see what people's experiences
have been with performance testing and benchmarking
frameworks, particularly in other Apache projects. Anyone have any
experience or thoughts?

Best,

Jason

-- 
---
Jason Kuster
Apache Beam (Incubating) / Google Cloud Dataflow

Re: Documentation for IDE setup

[Jesse Anderson]

That was the compilation error I got with Eclipse too. Thanks for
sorting it out.

On 10/17/16, Daniel Kulp  wrote:
> Just a follow up based on some discoveries while trying to rebase my branch
> on master this morning.
>
> Eclipse JDT outputs methods/fields into class files in a different order
> than Oracle compiler.   That’s perfectly acceptable from a “binary
> compatibility” standpoint, but it has a side effect of causing potential
> problems with AutoValue.   If the AutoValue class tries to get it’s values
> from an interface, the methods on the interface will come in a different
> order than with Oracle and the resulting constructor/fields/etc… will be
> different.   Based on some experiments and back and forth with Dan H., I
> believe the best fix is to explicitly define the properties on the AutoValue
> class as if it didn’t pull those via the interface.   Thus, the APT
> processing gets the attributes in the order intended and generates the right
> code. The alternative would be to use the Builder pattern instead of the
> constructor, but that requires more code to be written than just defining
> the attributes in the right order.   However, if you are already defining a
> Builder, that might be the best option.
>
> Anyway, something to be aware of when using the AutoValue things.   Once we
> get the branch merged, travis should automatically pick this up.
>
> Dan
>
>
>
>
>> On Oct 14, 2016, at 11:37 AM, Daniel Kulp  wrote:
>>
>>
>>> On Oct 14, 2016, at 10:06 AM, Jesse Anderson 
>>> wrote:
>>>
>>> Last week I imported Beam with IntelliJ and everything worked.
>>>
>>> That said, I tried to import the Eclipse project and that doesn't
>>> compile
>>> anymore. I didn't have time to figure out what happened though.
>>>
>>
>> I have a pull request https://github.com/apache/incubator-beam/pull/1094
>> that fixes the compile issues.  It has two LGTM’s, just needs someone to
>> merge it.
>>
>> With eclipse, you need to have all the needed m2e connectors.   Some of
>> them (find bugs, check style) can be auto-detected and installed when beam
>> is first imported.   The apt one doesn’t.   You need to go to the eclipse
>> marketplace, install it, then configure it in the Eclipse properties to
>> turn on the “experimental” m2e-apt processing.   Once you do that, a
>> refresh of the maven projects should result in it building/compiling.
>>
>> Running tests is another matter.   Since eclipse compiles everything in a
>> module in one pass (instead of two like maven), one of the apt processors
>> doesn’t know where to output files and always dumps the files in /classes
>> instead of /test-classes.   Thus, any test that relies on a runner will
>> likely fail as it results in the “test” versions of various services from
>> core being picked up.  A simple:
>>
>> rm sdks/java/core/target/classes/META-INF/services/*
>>
>> From the command line will fix that.   That should also be documented on
>> the IDE page until someone can figure out how to work around it.
>>
>> Dan
>>
>>
>>
>>> On Fri, Oct 14, 2016 at 1:21 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Hi Christian,

 IntelliJ doesn't need any special config (maybe the code style can be
 documented or imported).

 Anyway, agree to add such on website in the contribute directory. I
 think it could be part of the contribution-guide as it's first setup
 step.

 Regards
 JB

 On 10/14/2016 10:17 AM, Christian Schneider wrote:
> Hello all,
>
> I am new to the beam community and currently start making myself
> familiar with the code.  I quickly found the contribution guide and
> was
> able to clone the code and build beam using maven.
>
> The first obstacle I faced was getting the code build in eclipse. I
> naively imported as existing maven projects but got lots of compile
> errors. After talking to Dan Kulp we found that this is due to the apt
> annotation processing for auto value types. Dan explained me how I
> need
> to setup eclipse to make it work.
>
> I still got 5 compile errors (Some bound mismatch at Read.bounded, and
> one ambiguous method empty). These errors seem to be present for
> everyone using eclipse and Dan works on it. So I think this is not a
> permanent problem.
>
> To make it easier for new people I would like to write a documentation
> about the IDE setup. I can cover the eclipse part but I think intellij
> should also be described.
>
> I already started with it and placed it in /contribute/ide-setup. Does
> that make sense?
>
> I currently did not link to it from anywhere. I think it should be
> linked in the contribute/index and in the Contribute menu.
>
> Christian
>

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - 

Re: Documentation for IDE setup

[Daniel Kulp]

Just a follow up based on some discoveries while trying to rebase my branch on 
master this morning.   

Eclipse JDT outputs methods/fields into class files in a different order than 
Oracle compiler.   That’s perfectly acceptable from a “binary compatibility” 
standpoint, but it has a side effect of causing potential problems with 
AutoValue.   If the AutoValue class tries to get it’s values from an interface, 
the methods on the interface will come in a different order than with Oracle 
and the resulting constructor/fields/etc… will be different.   Based on some 
experiments and back and forth with Dan H., I believe the best fix is to 
explicitly define the properties on the AutoValue class as if it didn’t pull 
those via the interface.   Thus, the APT processing gets the attributes in the 
order intended and generates the right code. The alternative would be to 
use the Builder pattern instead of the constructor, but that requires more code 
to be written than just defining the attributes in the right order.   However, 
if you are already defining a Builder, that might be the best option.

Anyway, something to be aware of when using the AutoValue things.   Once we get 
the branch merged, travis should automatically pick this up.

Dan




> On Oct 14, 2016, at 11:37 AM, Daniel Kulp  wrote:
> 
> 
>> On Oct 14, 2016, at 10:06 AM, Jesse Anderson  wrote:
>> 
>> Last week I imported Beam with IntelliJ and everything worked.
>> 
>> That said, I tried to import the Eclipse project and that doesn't compile
>> anymore. I didn't have time to figure out what happened though.
>> 
> 
> I have a pull request https://github.com/apache/incubator-beam/pull/1094 that 
> fixes the compile issues.  It has two LGTM’s, just needs someone to merge it. 
> 
> With eclipse, you need to have all the needed m2e connectors.   Some of them 
> (find bugs, check style) can be auto-detected and installed when beam is 
> first imported.   The apt one doesn’t.   You need to go to the eclipse 
> marketplace, install it, then configure it in the Eclipse properties to turn 
> on the “experimental” m2e-apt processing.   Once you do that, a refresh of 
> the maven projects should result in it building/compiling.
> 
> Running tests is another matter.   Since eclipse compiles everything in a 
> module in one pass (instead of two like maven), one of the apt processors 
> doesn’t know where to output files and always dumps the files in /classes 
> instead of /test-classes.   Thus, any test that relies on a runner will 
> likely fail as it results in the “test” versions of various services from 
> core being picked up.  A simple:
> 
> rm sdks/java/core/target/classes/META-INF/services/*
> 
> From the command line will fix that.   That should also be documented on the 
> IDE page until someone can figure out how to work around it.
> 
> Dan
> 
> 
> 
>> On Fri, Oct 14, 2016 at 1:21 AM Jean-Baptiste Onofré 
>> wrote:
>> 
>>> Hi Christian,
>>> 
>>> IntelliJ doesn't need any special config (maybe the code style can be
>>> documented or imported).
>>> 
>>> Anyway, agree to add such on website in the contribute directory. I
>>> think it could be part of the contribution-guide as it's first setup step.
>>> 
>>> Regards
>>> JB
>>> 
>>> On 10/14/2016 10:17 AM, Christian Schneider wrote:
 Hello all,
 
 I am new to the beam community and currently start making myself
 familiar with the code.  I quickly found the contribution guide and was
 able to clone the code and build beam using maven.
 
 The first obstacle I faced was getting the code build in eclipse. I
 naively imported as existing maven projects but got lots of compile
 errors. After talking to Dan Kulp we found that this is due to the apt
 annotation processing for auto value types. Dan explained me how I need
 to setup eclipse to make it work.
 
 I still got 5 compile errors (Some bound mismatch at Read.bounded, and
 one ambiguous method empty). These errors seem to be present for
 everyone using eclipse and Dan works on it. So I think this is not a
 permanent problem.
 
 To make it easier for new people I would like to write a documentation
 about the IDE setup. I can cover the eclipse part but I think intellij
 should also be described.
 
 I already started with it and placed it in /contribute/ide-setup. Does
 that make sense?
 
 I currently did not link to it from anywhere. I think it should be
 linked in the contribute/index and in the Contribute menu.
 
 Christian
 
>>> 
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>> 
> 
> -- 
> Daniel Kulp
> dk...@apache.org - http://dankulp.com/blog
> Talend Community Coder - http://coders.talend.com
> 

-- 
Daniel Kulp
dk...@apache.org - http://dankulp.com/blog
Talend Community Coder - 

Re: [KUDOS] Contributed runner: Apache Apex!

[Sergio Fernández]

awesome!

On Oct 17, 2016 18:52, "Kenneth Knowles"  wrote:

> Hi all,
>
> I would to, once again, call attention to a great addition to Beam: a
> runner for Apache Apex.
>
> After lots of review and much thoughtful revision, pull request #540 has
> been merged to the apex-runner feature branch today. Please do take a look,
> and help us put the finishing touches on it to get it ready for the master
> branch.
>
> And please also congratulate and thank Thomas Weise for this large
> endeavor, Vlad Rosov who helped get the integration tests working, and
> Guarav Gupta who contributed review comments.
>
> Kenn
>

Re: Introduction

[Jesse Anderson]

Neelesh,

I saw you talked about the Hadoop MapReduce runner support too. I'd love to
see that happen. When Tyler and I spoke at Strata NYC, I was surprised how
many people were there with only MR code.

This would definitely ease the testing burden if they can port to Beam and
run on MR before going to another runner.

Thanks,

Jesse

On Mon, Oct 17, 2016 at 11:28 AM Amit Sela  wrote:

> Done.
>
> Feel free to take a pick at the Spark runner since you have Spark
> experience and that's great!
>
> Most open issues are usually automatically assigned to me, but ping me (dev
> list/Slack) if you want to work on something and not sure what's the status
> there.
>
> Thanks,
> Amit
>
>
> On Mon, Oct 17, 2016 at 9:14 PM Neelesh Salian 
> wrote:
>
> > Hello folks,
> >
> > I am Neelesh Salian; I recently joined the Beam community and I wanted to
> > take this opportunity to formally introduce myself.
> >
> > I have been working with the Hadoop and Spark ecosystems over the past
> two
> > years and started working on Flink over the past few weeks as well.
> >
> >
> > If someone in the community could please add me to the list of
> contributors
> > to help assign JIRAs to myself to work on, that would be super helpful.
> >
> >
> > Excited to start working and help build the community. :)
> > Thank you.
> >
> > --
> > Neelesh Srinivas Salian
> > Engineer
> >
>

Re: [KUDOS] Contributed runner: Apache Apex!

[Jesse Anderson]

Awesome!

On Mon, Oct 17, 2016 at 10:41 AM Thomas Weise  wrote:

> Thanks to Kenn for helping with the review and many questions!
>
> The focus till here has been on making the runner functional. I will start
> creating JIRAs for follow-up work.
>
> Looking forward to the next steps to make it a top-level runner and input
> from the community on the same.
>
> Thanks!
> Thomas
>
>
> On Mon, Oct 17, 2016 at 10:35 AM, Amit Sela  wrote:
>
> > Congrats and thanks to everyone who was involved in this effort!
> >
> > On Mon, Oct 17, 2016 at 8:07 PM Neelesh Salian 
> > wrote:
> >
> > > Awesome. Great work.
> > >
> > > On Mon, Oct 17, 2016 at 10:03 AM, Aljoscha Krettek <
> aljos...@apache.org>
> > > wrote:
> > >
> > > > Congrats! :-)
> > > >
> > > > On Mon, 17 Oct 2016 at 18:55 Kenneth Knowles  >
> > > > wrote:
> > > >
> > > > > *I would like to :-)
> > > > >
> > > > > On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles 
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would to, once again, call attention to a great addition to
> > Beam: a
> > > > > > runner for Apache Apex.
> > > > > >
> > > > > > After lots of review and much thoughtful revision, pull request
> > #540
> > > > has
> > > > > > been merged to the apex-runner feature branch today. Please do
> > take a
> > > > > look,
> > > > > > and help us put the finishing touches on it to get it ready for
> the
> > > > > master
> > > > > > branch.
> > > > > >
> > > > > > And please also congratulate and thank Thomas Weise for this
> large
> > > > > > endeavor, Vlad Rosov who helped get the integration tests
> working,
> > > and
> > > > > > Guarav Gupta who contributed review comments.
> > > > > >
> > > > > > Kenn
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Neelesh Srinivas Salian
> > > Customer Operations Engineer
> > >
> >
>

Re: Introduction

[Amit Sela]

Done.

Feel free to take a pick at the Spark runner since you have Spark
experience and that's great!

Most open issues are usually automatically assigned to me, but ping me (dev
list/Slack) if you want to work on something and not sure what's the status
there.

Thanks,
Amit


On Mon, Oct 17, 2016 at 9:14 PM Neelesh Salian  wrote:

> Hello folks,
>
> I am Neelesh Salian; I recently joined the Beam community and I wanted to
> take this opportunity to formally introduce myself.
>
> I have been working with the Hadoop and Spark ecosystems over the past two
> years and started working on Flink over the past few weeks as well.
>
>
> If someone in the community could please add me to the list of contributors
> to help assign JIRAs to myself to work on, that would be super helpful.
>
>
> Excited to start working and help build the community. :)
> Thank you.
>
> --
> Neelesh Srinivas Salian
> Engineer
>

Introduction

[Neelesh Salian]

Hello folks,

I am Neelesh Salian; I recently joined the Beam community and I wanted to
take this opportunity to formally introduce myself.

I have been working with the Hadoop and Spark ecosystems over the past two
years and started working on Flink over the past few weeks as well.


If someone in the community could please add me to the list of contributors
to help assign JIRAs to myself to work on, that would be super helpful.


Excited to start working and help build the community. :)
Thank you.

-- 
Neelesh Srinivas Salian
Engineer

Re: [KUDOS] Contributed runner: Apache Apex!

[Thomas Weise]

Thanks to Kenn for helping with the review and many questions!

The focus till here has been on making the runner functional. I will start
creating JIRAs for follow-up work.

Looking forward to the next steps to make it a top-level runner and input
from the community on the same.

Thanks!
Thomas


On Mon, Oct 17, 2016 at 10:35 AM, Amit Sela  wrote:

> Congrats and thanks to everyone who was involved in this effort!
>
> On Mon, Oct 17, 2016 at 8:07 PM Neelesh Salian 
> wrote:
>
> > Awesome. Great work.
> >
> > On Mon, Oct 17, 2016 at 10:03 AM, Aljoscha Krettek 
> > wrote:
> >
> > > Congrats! :-)
> > >
> > > On Mon, 17 Oct 2016 at 18:55 Kenneth Knowles 
> > > wrote:
> > >
> > > > *I would like to :-)
> > > >
> > > > On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would to, once again, call attention to a great addition to
> Beam: a
> > > > > runner for Apache Apex.
> > > > >
> > > > > After lots of review and much thoughtful revision, pull request
> #540
> > > has
> > > > > been merged to the apex-runner feature branch today. Please do
> take a
> > > > look,
> > > > > and help us put the finishing touches on it to get it ready for the
> > > > master
> > > > > branch.
> > > > >
> > > > > And please also congratulate and thank Thomas Weise for this large
> > > > > endeavor, Vlad Rosov who helped get the integration tests working,
> > and
> > > > > Guarav Gupta who contributed review comments.
> > > > >
> > > > > Kenn
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Neelesh Srinivas Salian
> > Customer Operations Engineer
> >
>

Re: [KUDOS] Contributed runner: Apache Apex!

[Amit Sela]

Congrats and thanks to everyone who was involved in this effort!

On Mon, Oct 17, 2016 at 8:07 PM Neelesh Salian  wrote:

> Awesome. Great work.
>
> On Mon, Oct 17, 2016 at 10:03 AM, Aljoscha Krettek 
> wrote:
>
> > Congrats! :-)
> >
> > On Mon, 17 Oct 2016 at 18:55 Kenneth Knowles 
> > wrote:
> >
> > > *I would like to :-)
> > >
> > > On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would to, once again, call attention to a great addition to Beam: a
> > > > runner for Apache Apex.
> > > >
> > > > After lots of review and much thoughtful revision, pull request #540
> > has
> > > > been merged to the apex-runner feature branch today. Please do take a
> > > look,
> > > > and help us put the finishing touches on it to get it ready for the
> > > master
> > > > branch.
> > > >
> > > > And please also congratulate and thank Thomas Weise for this large
> > > > endeavor, Vlad Rosov who helped get the integration tests working,
> and
> > > > Guarav Gupta who contributed review comments.
> > > >
> > > > Kenn
> > > >
> > >
> >
>
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>

Re: [KUDOS] Contributed runner: Apache Apex!

[Neelesh Salian]

Awesome. Great work.

On Mon, Oct 17, 2016 at 10:03 AM, Aljoscha Krettek 
wrote:

> Congrats! :-)
>
> On Mon, 17 Oct 2016 at 18:55 Kenneth Knowles 
> wrote:
>
> > *I would like to :-)
> >
> > On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles  wrote:
> >
> > > Hi all,
> > >
> > > I would to, once again, call attention to a great addition to Beam: a
> > > runner for Apache Apex.
> > >
> > > After lots of review and much thoughtful revision, pull request #540
> has
> > > been merged to the apex-runner feature branch today. Please do take a
> > look,
> > > and help us put the finishing touches on it to get it ready for the
> > master
> > > branch.
> > >
> > > And please also congratulate and thank Thomas Weise for this large
> > > endeavor, Vlad Rosov who helped get the integration tests working, and
> > > Guarav Gupta who contributed review comments.
> > >
> > > Kenn
> > >
> >
>



-- 
Neelesh Srinivas Salian
Customer Operations Engineer

Re: [KUDOS] Contributed runner: Apache Apex!

[Aljoscha Krettek]

Congrats! :-)

On Mon, 17 Oct 2016 at 18:55 Kenneth Knowles  wrote:

> *I would like to :-)
>
> On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles  wrote:
>
> > Hi all,
> >
> > I would to, once again, call attention to a great addition to Beam: a
> > runner for Apache Apex.
> >
> > After lots of review and much thoughtful revision, pull request #540 has
> > been merged to the apex-runner feature branch today. Please do take a
> look,
> > and help us put the finishing touches on it to get it ready for the
> master
> > branch.
> >
> > And please also congratulate and thank Thomas Weise for this large
> > endeavor, Vlad Rosov who helped get the integration tests working, and
> > Guarav Gupta who contributed review comments.
> >
> > Kenn
> >
>

Re: [KUDOS] Contributed runner: Apache Apex!

[Kenneth Knowles]

*I would like to :-)

On Mon, Oct 17, 2016 at 9:51 AM Kenneth Knowles  wrote:

> Hi all,
>
> I would to, once again, call attention to a great addition to Beam: a
> runner for Apache Apex.
>
> After lots of review and much thoughtful revision, pull request #540 has
> been merged to the apex-runner feature branch today. Please do take a look,
> and help us put the finishing touches on it to get it ready for the master
> branch.
>
> And please also congratulate and thank Thomas Weise for this large
> endeavor, Vlad Rosov who helped get the integration tests working, and
> Guarav Gupta who contributed review comments.
>
> Kenn
>

[KUDOS] Contributed runner: Apache Apex!

[Kenneth Knowles]

Hi all,

I would to, once again, call attention to a great addition to Beam: a
runner for Apache Apex.

After lots of review and much thoughtful revision, pull request #540 has
been merged to the apex-runner feature branch today. Please do take a look,
and help us put the finishing touches on it to get it ready for the master
branch.

And please also congratulate and thank Thomas Weise for this large
endeavor, Vlad Rosov who helped get the integration tests working, and
Guarav Gupta who contributed review comments.

Kenn