Re: Gradle / Mvn diff

[Jean-Baptiste Onofré]

Hi Romain,

I think we are pretty close: agree to add some explicit tasks & projects names.

We can add additional tasks like skipAudit, for instance.

As reminder, gradle tasks provides the list of tasks and gradle projects
provides the list of projects/modules.

Regards
JB

On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote:
> Hmm, I have to admit docs dont have my favor cause they are easily outdated 
> and
> hard to search but you hit a good point. Starting by renaming properly the 
> tasks
> and maybe writing what is done in build files - since it is code and even "api
> for dev", it requires as much comments than the main api - can be better to 
> start.
> 
> Also a big switch flag to bypass checkstyle/findbugs/... can be good while in
> dev since these phases cost a looot for nothing while you validates your code 
> in
> runners modules for instance.
> 
> Le 23 janv. 2018 07:15, "Kenneth Knowles"  > a écrit :
> 
> On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau 
>  > wrote:
> 
> @Kenneth: why not dropping the doc for a script with comments in the
> project? A "RUNME.sh" ;).
> 
> 
> That's cool, too, but also any shell one liner can be a gradle one liner 
> or
> mvn two/three liner :-). it is just trading one command that you cannot
> guess easily for a different one that you still can't guess easily.
> 
> For example, are the SparkRunner ValidatesRunner tests in the SparkRunner 
> or
> the core SDK or a third module that integrates the two? And why would you
> know that the example ITs are called "sparkRunnerPreCommit"? It doesn't 
> even
> make sense really to have "precommit" or "postcommit" except as aliases to
> make it easy to repro Jenkins' behavior - they have no other intrinsic 
> meaning.
> 
> So I was proposing a mapping from "full sentence + description" to one 
> liner
> to help people navigate the targets that we set up. Some web page or doc
> that people can just quickly scan to find out to do common things, easier
> than groovy or XML.
> 
> Kenn
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Gradle / Mvn diff

[Romain Manni-Bucau]

Hmm, I have to admit docs dont have my favor cause they are easily outdated
and hard to search but you hit a good point. Starting by renaming properly
the tasks and maybe writing what is done in build files - since it is code
and even "api for dev", it requires as much comments than the main api -
can be better to start.

Also a big switch flag to bypass checkstyle/findbugs/... can be good while
in dev since these phases cost a looot for nothing while you validates your
code in runners modules for instance.

Le 23 janv. 2018 07:15, "Kenneth Knowles"  a écrit :

On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau 
wrote:

> @Kenneth: why not dropping the doc for a script with comments in the
> project? A "RUNME.sh" ;).
>

That's cool, too, but also any shell one liner can be a gradle one liner or
mvn two/three liner :-). it is just trading one command that you cannot
guess easily for a different one that you still can't guess easily.

For example, are the SparkRunner ValidatesRunner tests in the SparkRunner
or the core SDK or a third module that integrates the two? And why would
you know that the example ITs are called "sparkRunnerPreCommit"? It doesn't
even make sense really to have "precommit" or "postcommit" except as
aliases to make it easy to repro Jenkins' behavior - they have no other
intrinsic meaning.

So I was proposing a mapping from "full sentence + description" to one
liner to help people navigate the targets that we set up. Some web page or
doc that people can just quickly scan to find out to do common things,
easier than groovy or XML.

Kenn

Re: Gradle / Mvn diff

[Kenneth Knowles]

On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau 
wrote:

> @Kenneth: why not dropping the doc for a script with comments in the
> project? A "RUNME.sh" ;).
>

That's cool, too, but also any shell one liner can be a gradle one liner or
mvn two/three liner :-). it is just trading one command that you cannot
guess easily for a different one that you still can't guess easily.

For example, are the SparkRunner ValidatesRunner tests in the SparkRunner
or the core SDK or a third module that integrates the two? And why would
you know that the example ITs are called "sparkRunnerPreCommit"? It doesn't
even make sense really to have "precommit" or "postcommit" except as
aliases to make it easy to repro Jenkins' behavior - they have no other
intrinsic meaning.

So I was proposing a mapping from "full sentence + description" to one
liner to help people navigate the targets that we set up. Some web page or
doc that people can just quickly scan to find out to do common things,
easier than groovy or XML.

Kenn

Re: [DISCUSS] State of the project: Feature roadmap for 2018

[Jean-Baptiste Onofré]

Hi Ben,

about the "technical roadmap", we have a thread about "Beam 3.x roadmap".

It already provides ideas for points 3 & 4.

Regards
JB

On 01/22/2018 09:15 PM, Ben Chambers wrote:
> Thanks Davor for starting the state of the project discussions [1].
> 
> 
> In this fork of the state of the project discussion, I’d like to start the
> discussion of the feature roadmap for 2018 (and beyond).
> 
> 
> To kick off the discussion, I think the features could be divided into several
> areas, as follows:
> 
>  1.
> 
> Enabling Contributions: How do we make it easier to add new features to 
> the
> supported runners? Can we provide a common intermediate layer below the
> existing functionality that features are translated to so that runners 
> only
> need to support the intermediate layer and new features only need to 
> target
> it? What other ways can we make it easier to contribute to the development
> of Beam?
> 
>  2.
> 
> Realizing Portability: What gaps are there in the promise of portability?
> For example in [1] we discussed the fact that users must write per-runner
> code to push system metrics from runners to their monitoring platform. 
> This
> limits their ability to actually change runners. Credential management for
> different environments also falls into this category.
> 
>  3.
> 
> Large Features: What major features (like Beam SQL, Beam Python, etc.) 
> would
> increase the Beam user base in 2018?
> 
>  4.
> 
> Improvements: What small changes could make Beam more appealing to users?
> Are there API improvements we could make or common mistakes we could 
> detect
> and/or prevent?
> 
> 
> Thanks in advance for participating in the discussion. I believe that 2018 
> could
> be a great year for Beam, providing easier, more complete runner portability 
> and
> features that make Beam easier to use for everyone.
> 
> 
> Ben
> 
> 
> [1]
> https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f473094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E
> 
> [2]
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Eclipse support

[zlgonzalez]

Thanks Ismael. I'll try and take a stab at it when I've read more about the 
eclipse compiler.
In the meantime, do you have any pointers? Is it just about finding the right 
Eclipse JDT compiler options?
Thanks,Ron


Sent via the Samsung Galaxy S7 active, an AT 4G LTE smartphone
 Original message From: Ismaël Mejía  Date: 
1/22/18  1:29 AM  (GMT-08:00) To: dev@beam.apache.org Subject: Re: Eclipse 
support 
Hello,

Thanks for bringing this info, I tried to compile with the eclipse
compiler and I can confirm that it does not wok, Eclipse's JDT is more
annoying about generics so it could be related to this.

Filled https://issues.apache.org/jira/browse/BEAM-3508 to track it.
Feel free to contribute a fix if you feel like it.

Ismaël


On Sat, Jan 20, 2018 at 10:20 PM, Ron Gonzalez  wrote:
> Hello again,
>   Sorry to keep asking about this, but I can't seem to get Eclipse working
> for this project.
>
>   I did mvn eclipse:clean eclipse:eclipse command and I've reduced the
> problem now down to the Java generics issues related to autovalue types. I
> had to run mvn generate-sources generate-test-sources in each of the
> sub-modules to make a lot of the errors in Eclipse go away since after
> running mvn -DskipTests clean install, the target/generated-sources and
> target/generated-test-sources are empty for some reason.
>
>   Interestingly enough, if I run mvn -Peclipse-jdt -DskipTests clean install
> from the command line, I am able to reproduce the same errors that I see in
> my Eclipse installation. I am using Eclipse Neon.
>
>
>   One such error is:
>
> @Experimental(Kind.FILESYSTEM)
> @Deprecated
> public  TypedWrite to(
> DynamicAvroDestinations
> dynamicDestinations) {
>   return toBuilder()
>   .setDynamicDestinations((DynamicAvroDestinations)
> dynamicDestinations)
>   .build();
> }
>
>  gives the error of:
>
> Description Resource Path Location Type
> Type mismatch: cannot convert from
> AvroIO.TypedWrite to
> AvroIO.TypedWrite AvroIO.java
> /beam-sdks-java-core/src/main/java/org/apache/beam/sdk/io line 1007 Java
> Problem
>
>  I have 48 errors left and most of these remaining errors are of this kind
> (related to Java generics of an auto-valued type). I enabled the Maven
> Annotation setting, per the Eclipse tips in the website.
>  I double-checked the compiling JDK in Eclipse, and it's using my system
> JDK, which I assume is being used by the command line compilation, which is
> successful.
>  Does anybody have Eclipse completely compiling for them without errors? Any
> tips?
>
> Thanks,
> Ron
>
> On Wednesday, January 17, 2018, 10:34:48 AM PST, Ted Yu
>  wrote:
>
>
> Have you tried running 'mvn eclipse:eclipse' and importing from the root of
> workspace ?
>
> On Wed, Jan 17, 2018 at 10:32 AM, Ron Gonzalez  wrote:
>
> Hi,
>   I've been trying this for a couple of days now, but I can't seem to get a
> clean Eclipse import.
>   I refreshed to latest master, got a clean mvn -DskipTests clean install,
> ran through the Eclipse setup steps for m2e-apt installation.
>   I'm getting errors like below. Do you have any tips to get this going?
>
> Thanks,
> Ron
>
> Description Resource Path Location Type
> ACCUMULATING cannot be resolved to a variable WindowingStrategyTranslation.
> java /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 56 Java Problem
> AFTER_ALL cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 34 Java Problem
> AFTER_ALL cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 241 Java Problem
> AFTER_ANY cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 37 Java Problem
> AFTER_ANY cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 243 Java Problem
> AFTER_EACH cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 59 Java Problem
> AFTER_EACH cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 245 Java Problem
> AFTER_END_OF_WINDOW cannot be resolved to a variable
> TriggerStateMachines.java /beam-runners-core-java/src/
> main/java/org/apache/beam/ runners/core/triggers line 40 Java Problem
> AFTER_END_OF_WINDOW cannot be resolved to a variable TriggerTranslation.java
> 

Re: TestPipeline drops options with JsonIgnore annotation

[Paul Gerver]

I am hitting this issue on several integration tests, but these tests are only 
using options set by the `beamTestPipelineOptions` system property and not 
going through convertToArgs from my side.

I apologize for the email switch but the mail list isn’t letting log in through 
Google+.

On 2018/01/22 21:14:15, Lukasz Cwik  wrote:
> Are your talking about integration tests that use a main like WordCountIT?>
>
> If so, then https://github.com/apache/beam/pull/4346 was an attempt to get>
> around this limitation but I suggested that we get rid of convertToArgs>
> completely so there is no serialization round trip before the args are>
> passed to the TestPipeline.>
> If you have some ideas in this space, feel free to contribute a PR.>
>
> On Fri, Jan 19, 2018 at 4:54 PM, Paul Gerver  wrote:>
>
> > Hello,>
> >>
> > With Beam 2.2 it looks like the TestPipeline now serializes options before>
> > giving them to the parent Pipeline to run (in order to utilize runtime>
> > options). I have some options that were marked with the `JsonIgnore`>
> > annotation which now seem to be getting dropped for my runner.>
> >>
> > Is there something I'm missing which would allow me to skip this>
> > serialization piece in the TestPipeline? If not, this seems like a side>
> > effect of 2.2.>
> >>
> > Let me know!>
> >>
> > Thanks,>
> > -->
> > *Paul Gerver*>
> >>
>
Sent from Mail for Windows 10

Re: Gradle / Mvn diff

[Kenneth Knowles]

This thread does bring up a nice idea: We should add a little FAQ /
Cookbook somewhere for common tasks, like running just an IT with a
particular runtime config or running the unit tests of a deep module.

On Mon, Jan 22, 2018 at 2:58 PM, Lukasz Cwik  wrote:

>
>
>
>
> On Mon, Jan 22, 2018 at 2:40 PM, Romain Manni-Bucau  > wrote:
>
>>
>>
>> Le 22 janv. 2018 21:46, "Lukasz Cwik"  a écrit :
>>
>> 1. Are you trying to have version overrides in a module depend on the
>> parent's version and not in one global place?,
>> Doesn't this lead to compatibility issues if you don't live with a single
>> version of a dependency across the entire repo (unless that dependency is
>> shaded away of course).
>>
>>
>> Ismael can detail this point more than me but this is sadly already the
>> case. We were looking to override part of the tree due to incompatibilities
>> between spark and bigquery driver.
>>
>>
>> 2. How is what you describe different from './gradlew
>> :runners:spark:build'
>>
>>
>> I want to run only spark in the wordcount module for instance. Not a
>> runner module but a runner execution in a multi runners module.
>>
>
> I see. I think your referring to './gradlew :examples:java:
> sparkRunnerPreCommit'
>
>
>>
>>
>> 3. Can be overridden on command line or per user properties file but I
>> would rather have our users execute as close to what we test in Jenkins so
>> that the differences between a dev and CI workflow is minimal.
>>
>>
>> Hmm, are you sure it is the case everywhere? For me default should be
>> usable and CI use overrides but both options are valid. Just a very
>> different experience compared to maven.
>>
>
> '--no-parallel' and '--max-workers' on the command line,
> 'org.gradle.parallel' and 'org.gradle.workers.max' within
> 'gradle.properties' in gradle user home
>
>
>>
>> 4. configuration on demand is enabled by default and it should only
>> configure the projects that are needed so I'm not sure what you are asking
>> for here.
>>
>>
>> Running a submodule only is slower than maven so when working on a module
>> - during dev - it is quite costly, in particular debugging through the
>> build.
>>
>
> Is there a better alternative in Maven then continuously installing
> modules anytime you make a cross module change which is error prone or
> build all the modules with '-am' all the time which is slow?
>
>
>>
>>
>> On Mon, Jan 22, 2018 at 12:20 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> As mentionned in another thread, Im sending this mail to report some
>>> differences between maven and gradle setups - percieved as regressions from
>>> this side of the fence:
>>>
>>> 1. Parent versions are not usable in children as variables - btw why not
>>> putting them in gradle.properties as ut is often done? (Not blocking)
>>> 2. Multi executions are not all runnable once by one. Typical example is
>>> surefire executions are selectable using surefire:test@id with maven
>>> but the for loop in gradle is never parameterized so no way to run only
>>> spark runner test suite for instance. (Almost blocking to work efficiently
>>> but easy to fix)
>>> 3. Concurrency is hardcoded and way too high for most computers leading
>>> to freezing the computer and preventing the user to do anything (tested on
>>> an i7 with 32G of RAM and a SSD). (Blocking but easy to fix i guess if we
>>> use the rule of thumb to keep concurrency off by default)
>>> 4. Setup is slow when not using the daemon since it browses the whole
>>> project so a lazy setup can be beneficial when working on submodules (not
>>> that blocking until you rely on build.gradle setup)
>>>
>>>
>>>
>>
>>
>

Re: Gradle / Mvn diff

[Lukasz Cwik]

On Mon, Jan 22, 2018 at 2:40 PM, Romain Manni-Bucau 
wrote:

>
>
> Le 22 janv. 2018 21:46, "Lukasz Cwik"  a écrit :
>
> 1. Are you trying to have version overrides in a module depend on the
> parent's version and not in one global place?,
> Doesn't this lead to compatibility issues if you don't live with a single
> version of a dependency across the entire repo (unless that dependency is
> shaded away of course).
>
>
> Ismael can detail this point more than me but this is sadly already the
> case. We were looking to override part of the tree due to incompatibilities
> between spark and bigquery driver.
>
>
> 2. How is what you describe different from './gradlew :runners:spark:build'
>
>
> I want to run only spark in the wordcount module for instance. Not a
> runner module but a runner execution in a multi runners module.
>

I see. I think your referring to './gradlew
:examples:java:sparkRunnerPreCommit'


>
>
> 3. Can be overridden on command line or per user properties file but I
> would rather have our users execute as close to what we test in Jenkins so
> that the differences between a dev and CI workflow is minimal.
>
>
> Hmm, are you sure it is the case everywhere? For me default should be
> usable and CI use overrides but both options are valid. Just a very
> different experience compared to maven.
>

'--no-parallel' and '--max-workers' on the command line,
'org.gradle.parallel' and 'org.gradle.workers.max' within
'gradle.properties' in gradle user home


>
> 4. configuration on demand is enabled by default and it should only
> configure the projects that are needed so I'm not sure what you are asking
> for here.
>
>
> Running a submodule only is slower than maven so when working on a module
> - during dev - it is quite costly, in particular debugging through the
> build.
>

Is there a better alternative in Maven then continuously installing modules
anytime you make a cross module change which is error prone or build all
the modules with '-am' all the time which is slow?


>
>
> On Mon, Jan 22, 2018 at 12:20 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Hi
>>
>> As mentionned in another thread, Im sending this mail to report some
>> differences between maven and gradle setups - percieved as regressions from
>> this side of the fence:
>>
>> 1. Parent versions are not usable in children as variables - btw why not
>> putting them in gradle.properties as ut is often done? (Not blocking)
>> 2. Multi executions are not all runnable once by one. Typical example is
>> surefire executions are selectable using surefire:test@id with maven but
>> the for loop in gradle is never parameterized so no way to run only spark
>> runner test suite for instance. (Almost blocking to work efficiently but
>> easy to fix)
>> 3. Concurrency is hardcoded and way too high for most computers leading
>> to freezing the computer and preventing the user to do anything (tested on
>> an i7 with 32G of RAM and a SSD). (Blocking but easy to fix i guess if we
>> use the rule of thumb to keep concurrency off by default)
>> 4. Setup is slow when not using the daemon since it browses the whole
>> project so a lazy setup can be beneficial when working on submodules (not
>> that blocking until you rely on build.gradle setup)
>>
>>
>>
>
>

Re: Gradle / Mvn diff

[Romain Manni-Bucau]

Le 22 janv. 2018 21:46, "Lukasz Cwik"  a écrit :

1. Are you trying to have version overrides in a module depend on the
parent's version and not in one global place?,
Doesn't this lead to compatibility issues if you don't live with a single
version of a dependency across the entire repo (unless that dependency is
shaded away of course).


Ismael can detail this point more than me but this is sadly already the
case. We were looking to override part of the tree due to incompatibilities
between spark and bigquery driver.


2. How is what you describe different from './gradlew :runners:spark:build'


I want to run only spark in the wordcount module for instance. Not a runner
module but a runner execution in a multi runners module.


3. Can be overridden on command line or per user properties file but I
would rather have our users execute as close to what we test in Jenkins so
that the differences between a dev and CI workflow is minimal.


Hmm, are you sure it is the case everywhere? For me default should be
usable and CI use overrides but both options are valid. Just a very
different experience compared to maven.


4. configuration on demand is enabled by default and it should only
configure the projects that are needed so I'm not sure what you are asking
for here.


Running a submodule only is slower than maven so when working on a module -
during dev - it is quite costly, in particular debugging through the build.


On Mon, Jan 22, 2018 at 12:20 PM, Romain Manni-Bucau 
wrote:

> Hi
>
> As mentionned in another thread, Im sending this mail to report some
> differences between maven and gradle setups - percieved as regressions from
> this side of the fence:
>
> 1. Parent versions are not usable in children as variables - btw why not
> putting them in gradle.properties as ut is often done? (Not blocking)
> 2. Multi executions are not all runnable once by one. Typical example is
> surefire executions are selectable using surefire:test@id with maven but
> the for loop in gradle is never parameterized so no way to run only spark
> runner test suite for instance. (Almost blocking to work efficiently but
> easy to fix)
> 3. Concurrency is hardcoded and way too high for most computers leading to
> freezing the computer and preventing the user to do anything (tested on an
> i7 with 32G of RAM and a SSD). (Blocking but easy to fix i guess if we use
> the rule of thumb to keep concurrency off by default)
> 4. Setup is slow when not using the daemon since it browses the whole
> project so a lazy setup can be beneficial when working on submodules (not
> that blocking until you rely on build.gradle setup)
>
>
>

Re: TestPipeline drops options with JsonIgnore annotation

[Lukasz Cwik]

Are your talking about integration tests that use a main like WordCountIT?

If so, then https://github.com/apache/beam/pull/4346 was an attempt to get
around this limitation but I suggested that we get rid of convertToArgs
completely so there is no serialization round trip before the args are
passed to the TestPipeline.
If you have some ideas in this space, feel free to contribute a PR.

On Fri, Jan 19, 2018 at 4:54 PM, Paul Gerver  wrote:

> Hello,
>
> With Beam 2.2 it looks like the TestPipeline now serializes options before
> giving them to the parent Pipeline to run (in order to utilize runtime
> options). I have some options that were marked with the `JsonIgnore`
> annotation which now seem to be getting dropped for my runner.
>
> Is there something I'm missing which would allow me to skip this
> serialization piece in the TestPipeline? If not, this seems like a side
> effect of 2.2.
>
> Let me know!
>
> Thanks,
> --
> *Paul Gerver*
>

Re: Gradle / Mvn diff

[Lukasz Cwik]

1. Are you trying to have version overrides in a module depend on the
parent's version and not in one global place?,
Doesn't this lead to compatibility issues if you don't live with a single
version of a dependency across the entire repo (unless that dependency is
shaded away of course).

2. How is what you describe different from './gradlew :runners:spark:build'

3. Can be overridden on command line or per user properties file but I
would rather have our users execute as close to what we test in Jenkins so
that the differences between a dev and CI workflow is minimal.

4. configuration on demand is enabled by default and it should only
configure the projects that are needed so I'm not sure what you are asking
for here.

On Mon, Jan 22, 2018 at 12:20 PM, Romain Manni-Bucau 
wrote:

> Hi
>
> As mentionned in another thread, Im sending this mail to report some
> differences between maven and gradle setups - percieved as regressions from
> this side of the fence:
>
> 1. Parent versions are not usable in children as variables - btw why not
> putting them in gradle.properties as ut is often done? (Not blocking)
> 2. Multi executions are not all runnable once by one. Typical example is
> surefire executions are selectable using surefire:test@id with maven but
> the for loop in gradle is never parameterized so no way to run only spark
> runner test suite for instance. (Almost blocking to work efficiently but
> easy to fix)
> 3. Concurrency is hardcoded and way too high for most computers leading to
> freezing the computer and preventing the user to do anything (tested on an
> i7 with 32G of RAM and a SSD). (Blocking but easy to fix i guess if we use
> the rule of thumb to keep concurrency off by default)
> 4. Setup is slow when not using the daemon since it browses the whole
> project so a lazy setup can be beneficial when working on submodules (not
> that blocking until you rely on build.gradle setup)
>
>
>

Gradle / Mvn diff

[Romain Manni-Bucau]

Hi

As mentionned in another thread, Im sending this mail to report some
differences between maven and gradle setups - percieved as regressions from
this side of the fence:

1. Parent versions are not usable in children as variables - btw why not
putting them in gradle.properties as ut is often done? (Not blocking)
2. Multi executions are not all runnable once by one. Typical example is
surefire executions are selectable using surefire:test@id with maven but
the for loop in gradle is never parameterized so no way to run only spark
runner test suite for instance. (Almost blocking to work efficiently but
easy to fix)
3. Concurrency is hardcoded and way too high for most computers leading to
freezing the computer and preventing the user to do anything (tested on an
i7 with 32G of RAM and a SSD). (Blocking but easy to fix i guess if we use
the rule of thumb to keep concurrency off by default)
4. Setup is slow when not using the daemon since it browses the whole
project so a lazy setup can be beneficial when working on submodules (not
that blocking until you rely on build.gradle setup)

[DISCUSS] State of the project: Feature roadmap for 2018

[Ben Chambers]

Thanks Davor for starting the state of the project discussions [1].

In this fork of the state of the project discussion, I’d like to start the
discussion of the feature roadmap for 2018 (and beyond).

To kick off the discussion, I think the features could be divided into
several areas, as follows:

   1.

   Enabling Contributions: How do we make it easier to add new features to
   the supported runners? Can we provide a common intermediate layer below the
   existing functionality that features are translated to so that runners only
   need to support the intermediate layer and new features only need to target
   it? What other ways can we make it easier to contribute to the development
   of Beam?
   2.

   Realizing Portability: What gaps are there in the promise of
   portability? For example in [1] we discussed the fact that users must write
   per-runner code to push system metrics from runners to their monitoring
   platform. This limits their ability to actually change runners. Credential
   management for different environments also falls into this category.
   3.

   Large Features: What major features (like Beam SQL, Beam Python, etc.)
   would increase the Beam user base in 2018?
   4.

   Improvements: What small changes could make Beam more appealing to
   users? Are there API improvements we could make or common mistakes we could
   detect and/or prevent?


Thanks in advance for participating in the discussion. I believe that 2018
could be a great year for Beam, providing easier, more complete runner
portability and features that make Beam easier to use for everyone.

Ben

[1]
https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f473094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E
[2]
https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E

Re: Can Window PTransform drop tuples that violate allowed lateness?

[Shen Li]

Hi Kenn,

Thanks for the explanation.

> So now elements are droppable if they belong to an expired window.

Say I have two consecutive window transforms with FixedWindows WindowFn
(just an example, most likely won't appear in real pipeline). The first
windowFn says the element belongs to an expired window. But according to
the second windowFn, the element's window is not yet expired. In this case,
can the first Window transform drop the element?

Best,
Shen

On Mon, Jan 22, 2018 at 2:07 PM, Kenneth Knowles  wrote:

> Hi Shen,
>
> This is a documentation issue. The Beam model switched from dropping
> individual elements to expiring windows. So now elements are droppable if
> they belong to an expired window. This works a little better with the
> purpose of windowing and allowed lateness: to say when an aggregation is
> "complete". Any element that manages to make it to an aggregation before
> the accumulator is expired is allowed to be included now and only after the
> whole window expires we drop any further incoming elements for that window.
>
> Kenn
>
> On Mon, Jan 22, 2018 at 10:52 AM, Shen Li  wrote:
>
>> Hi,
>>
>> The Window#withAllowedLateness(Duration) doc says "Any elements that are
>> later than this as decided by the system-maintained watermark will be
>> dropped". Can the runner safely discard a tuple that violates the allowed
>> lateness in the Window operator? Or does it have to drop it in the
>> downstream GBK operator just in case that there could be another Window
>> transform in between overriding the allowed lateness (or other
>> configurations)?
>>
>> Thanks,
>> Shen
>>
>
>

Re: Can Window PTransform drop tuples that violate allowed lateness?

[Kenneth Knowles]

Hi Shen,

This is a documentation issue. The Beam model switched from dropping
individual elements to expiring windows. So now elements are droppable if
they belong to an expired window. This works a little better with the
purpose of windowing and allowed lateness: to say when an aggregation is
"complete". Any element that manages to make it to an aggregation before
the accumulator is expired is allowed to be included now and only after the
whole window expires we drop any further incoming elements for that window.

Kenn

On Mon, Jan 22, 2018 at 10:52 AM, Shen Li  wrote:

> Hi,
>
> The Window#withAllowedLateness(Duration) doc says "Any elements that are
> later than this as decided by the system-maintained watermark will be
> dropped". Can the runner safely discard a tuple that violates the allowed
> lateness in the Window operator? Or does it have to drop it in the
> downstream GBK operator just in case that there could be another Window
> transform in between overriding the allowed lateness (or other
> configurations)?
>
> Thanks,
> Shen
>

Can Window PTransform drop tuples that violate allowed lateness?

[Shen Li]

Hi,

The Window#withAllowedLateness(Duration) doc says "Any elements that are
later than this as decided by the system-maintained watermark will be
dropped". Can the runner safely discard a tuple that violates the allowed
lateness in the Window operator? Or does it have to drop it in the
downstream GBK operator just in case that there could be another Window
transform in between overriding the allowed lateness (or other
configurations)?

Thanks,
Shen

Re: Switching to Java 8

[Jean-Baptiste Onofré]

+1 ;)

Thanks for the heads up Eugene.

I'm working now on the examples merging.

Regards
JB

On 01/22/2018 07:36 PM, Eugene Kirpichov wrote:
> FYI: JB's PR https://github.com/apache/beam/pull/4424 has been merged, and now
> it's a free-for-all in terms of upgrading code to Java8!
> 
> (A bunch of other items on https://issues.apache.org/jira/browse/BEAM-3426 
> still
> remain)
> 
> On Tue, Jan 16, 2018 at 12:28 PM Eugene Kirpichov  > wrote:
> 
> Thanks. I think the one changing build is higher priority because it 
> enables
> people to start modernizing the code (e.g. FileIO) and it'd be good to do
> that before 2.3 cut. I wasn't able to find the PR you mentioned
> in https://github.com/apache/beam/pulls , which one is it?
> 
> On Mon, Jan 15, 2018 at 10:41 PM Jean-Baptiste Onofré  > wrote:
> 
> Hi
> 
> I created the PR about build during the weekend. I'm working on the
> examples merge PR and also polishing the first one. I will add you as
> reviewer.
> 
> Regards
> JB
> Le 16 janv. 2018, à 07:35, Eugene Kirpichov  > a écrit:
> 
> Hi JB - any updates here?
> 
> 
> On Tue, Jan 9, 2018, 2:51 AM Jean-Baptiste Onofré < 
> j...@nanthrax.net
> > wrote:
> 
> Actually, it's part of the build and I will "expand" the java
> version in the
> enforcer.
> 
> Regards
> JB
> 
> On 01/09/2018 11:46 AM, Etienne Chauchot wrote:
> > Hi,
> >
> > +1 as well, excellent news !
> >
> > I would add also: remove (AFAIK in some IOs) the enforcer
> configuration (like
> > [1]) that were put when java 8 was needed in a java 7 build.
> >
> > [1]
> >
> > 
> >     [1.8,)
> > 
> >
> >
> > Etienne
> >
> >
> > Le 08/01/2018 à 14:02, Jean-Baptiste Onofré a écrit :
> >> I created https://issues.apache.org/jira/browse/BEAM-3426 
> as
> umbrella Jira and
> >> created the sub-tasks related to build and examples.
> >>
> >> Feel free to add the relevant sub-tasks there.
> >>
> >> Regards
> >> JB
> >>
> >> On 01/08/2018 11:33 AM, Ismaël Mejía wrote:
> >>> Excellent news ! Probably a good idea to fill JIRAs to all
> of those. I
> >>> would add:
> >>>
> >>> - Remove the references in the website to Java 7
> >>> - Remove Java 7 and any related task from the CI
> >>> - Update the docker dev build images (I will take this one
> since
> >>> reproducible build is my pet project)
> >>> - Upgrade the IOs who were still in older versions because
> of client
> >>> compatibility. I remember SolfIO was one case but probably
> there are
> >>> others.
> >>>
> >>>
> >>> On Mon, Jan 8, 2018 at 7:49 AM, Jean-Baptiste Onofré <
> j...@nanthrax.net > wrote:
>  Yes, that's the plan: build first, example "merge" after.
> 
>  Regards
>  JB
> 
>  On 01/08/2018 07:43 AM, Eugene Kirpichov wrote:
> >
> > Sounds great, thanks! Probably best done as 2 separate
> steps, because
> > after updating the build scripts, everything else can
> begin in parallel?
> >
> > On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré <
> j...@nanthrax.net 
> > mailto:j...@nanthrax.net>>> 
> wrote:
> >
> >  Hi Eugene,
> >
> >  I'm taking the build update: Maven/Gradle with
> enforcer + merge of the
> > examples
> >  all together.
> >
> >  Regards
> >  JB
> >
> >  On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
> >   > The vote on user@ about switching to 

Re: Switching to Java 8

[Eugene Kirpichov]

FYI: JB's PR https://github.com/apache/beam/pull/4424 has been merged, and
now it's a free-for-all in terms of upgrading code to Java8!

(A bunch of other items on
https://issues.apache.org/jira/browse/BEAM-3426 still
remain)

On Tue, Jan 16, 2018 at 12:28 PM Eugene Kirpichov 
wrote:

> Thanks. I think the one changing build is higher priority because it
> enables people to start modernizing the code (e.g. FileIO) and it'd be good
> to do that before 2.3 cut. I wasn't able to find the PR you mentioned in
> https://github.com/apache/beam/pulls , which one is it?
>
> On Mon, Jan 15, 2018 at 10:41 PM Jean-Baptiste Onofré 
> wrote:
>
>> Hi
>>
>> I created the PR about build during the weekend. I'm working on the
>> examples merge PR and also polishing the first one. I will add you as
>> reviewer.
>>
>> Regards
>> JB
>> Le 16 janv. 2018, à 07:35, Eugene Kirpichov  a
>> écrit:
>>>
>>> Hi JB - any updates here?
>>>
>>> On Tue, Jan 9, 2018, 2:51 AM Jean-Baptiste Onofré < j...@nanthrax.net>
>>> wrote:
>>>
 Actually, it's part of the build and I will "expand" the java version
 in the
 enforcer.

 Regards
 JB

 On 01/09/2018 11:46 AM, Etienne Chauchot wrote:
 > Hi,
 >
 > +1 as well, excellent news !
 >
 > I would add also: remove (AFAIK in some IOs) the enforcer
 configuration (like
 > [1]) that were put when java 8 was needed in a java 7 build.
 >
 > [1]
 >
 > 
 > [1.8,)
 > 
 >
 >
 > Etienne
 >
 >
 > Le 08/01/2018 à 14:02, Jean-Baptiste Onofré a écrit :
 >> I created https://issues.apache.org/jira/browse/BEAM-3426 as
 umbrella Jira and
 >> created the sub-tasks related to build and examples.
 >>
 >> Feel free to add the relevant sub-tasks there.
 >>
 >> Regards
 >> JB
 >>
 >> On 01/08/2018 11:33 AM, Ismaël Mejía wrote:
 >>> Excellent news ! Probably a good idea to fill JIRAs to all of
 those. I
 >>> would add:
 >>>
 >>> - Remove the references in the website to Java 7
 >>> - Remove Java 7 and any related task from the CI
 >>> - Update the docker dev build images (I will take this one since
 >>> reproducible build is my pet project)
 >>> - Upgrade the IOs who were still in older versions because of
 client
 >>> compatibility. I remember SolfIO was one case but probably there
 are
 >>> others.
 >>>
 >>>
 >>> On Mon, Jan 8, 2018 at 7:49 AM, Jean-Baptiste Onofré <
 j...@nanthrax.net> wrote:
  Yes, that's the plan: build first, example "merge" after.
 
  Regards
  JB
 
  On 01/08/2018 07:43 AM, Eugene Kirpichov wrote:
 >
 > Sounds great, thanks! Probably best done as 2 separate steps,
 because
 > after updating the build scripts, everything else can begin in
 parallel?
 >
 > On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré <
 j...@nanthrax.net
 > > wrote:
 >
 >  Hi Eugene,
 >
 >  I'm taking the build update: Maven/Gradle with enforcer +
 merge of the
 > examples
 >  all together.
 >
 >  Regards
 >  JB
 >
 >  On 01/08/2018 07:34 AM, Eugene Kirpichov wrote:
 >   > The vote on user@ about switching to Java 8 has
 concluded,
 > affirmatively.
 >   >
 >   > What needs to be done to complete the switch? I can see
 at least
 > the
 >  following:
 >   > - Change maven and gradle scripts to use 1.8 source and
 target
 > version
 >   > - Fix resulting compilation/test errors (Java8 has
 slightly
 > different type
 >   > checking, more minor issues may arise)
 >   > - Remove all special-casing of java8 in build scripts
 >   > - Merge all modules like "java8 examples" and "java8
 tests" into
 > respective
 >   > non-"java8" modules
 >   > - Organize an effort to modernize code to use Java 8
 constructs
 > where
 >   > appropriate. Especially important to modernize examples.
 To a large
 >  extent this
 >   > can probably be automated with an IDE.
 >   >
 >   > Anything else?
 >   >
 >
 >  --
 >  Jean-Baptiste Onofré
 >  jbono...@apache.org 
 >  http://blog.nanthrax.net
 >  Talend - http://www.talend.com
 >
 
  --
  Jean-Baptiste Onofré
  jbono...@apache.org
  http://blog.nanthrax.net
  Talend - http://www.talend.com
 >>
 >

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend 

Re: [DISCUSS] State of the project

[Etienne Chauchot]

Thanks Davor for bringing this discussion up!

I particularly like that you listed the different areas of improvement 
and proposed to assign people based on their tastes.


I wanted to add a point about consistency across runners, but through 
the dev point of view: I've been working on a trans-runner feature 
lately (metrics push agnostic of the runners) for which I compared the 
behavior of the runners and wired up this feature into the flink and 
spark runners themselves. I must admit that I had a hard time figuring 
out how to wire it up in the different runners and that it was 
completely different between the runners. Also, their use (or non-use) 
of runner-core facilities vary. Even in the architecture of the tests: 
some, like spark, own their validates runner tests in the runner module 
and others runners run validates runner tests that are owned by sdk-core 
module. I also noticed some differences in the way to do streaming test: 
for some runners to trigger streaming mode it is needed to use an 
equivalent of direct runner's TestStream in the pipeline but for others 
putting streaming=true in pipelineOptions is enough.


=> long story short, IMHO I think that It could be interesting to 
enhance the runner API to contain more than run(). This could have the 
benefit to increase the coherence between runners. Besides we would need 
to find the correct balance between too many methods in the runner api 
that would reduce the flexibility of the runner implementations and too 
few methods that would reduce the coherence between the runners.


=>In addition, to enhance the coherence (dev point of view) between the 
runners, having all the runners run the exact same validates runner 
tests in both batch and streaming modes would be awesome!


Another thing: big +1 to have a programmatic way of defining the 
capability matrix as Romain suggested.


Also agree on Ismaël's point about too flexible concepts across runners 
(termination, bundling, ...)


Also big +1 to what Jessee wrote. I was myself in the past in the user 
architect position, and I can confirm that all the points that he 
mentioned are accurate.


Best,

Etienne


Le 16/01/2018 à 17:39, Ismaël Mejía a écrit :

Thanks Davor for opening this discussion and HUGE +1 to do this every
year or in cycles. I will fork this thread into a new one for the
Culture / Project management part issues as suggested.

About the diversity of users across runners subject I think that this
requires more attention to unification and implies at least work in
different areas:

* Automatized validation and consistent semantics among runners

Users should be confident that moving their code from one runner to
the other just works and the only way to ensure this is by having a
runner to pass ValidatesRunner/TCK tests and with this 'graduate' such
support as Romain suggested. The capatility-matrix is really nice but
it is not a programmatic way to do this. And also usually individual
features do work, but feature combinations produce issues so we need
to have a more exact semantics to avoid these.

Some parts of Beam's semantics are loose (e.g. bundle partiiioning,
pipeline termination, etc.), this I suppose has been a design decision
to allow flexibility in the runners implementation but it becomes
inconvenient when users move among runners and have different results.
I am not sure if the current tradeoff is worth the usability sacrifice
for the end user.

* Make user experience across runners a priority

Today all runners not only behave in different ways but the way users
publish and package their applications differ. Of course this is not a
trivial problem because deployment normally is a end user problem, but
we can improve in this area, e.g. guaranteeing a consistent deployment
mechanism across runners, and making IO integration easier for example
when using multiple IOs and switching runners it is easy to run into
conflicts, we should try to minimize this for the end-users.

* Simplify operational tasks among runners

We need to add a minimum degree of consistent observability across
runners. Of course Beam has metrics to do this, but it is not enough,
an end-user that starts on one runner and moves to another has to deal
with a totally different set of logs and operational issues. We can
try to improve this too, of course without trying to cover the full
spectrum but at least bringing some minimum set of observability. I
hope that the current work on portability will bring some improvements
in this area. But this is crucial for users that probably pass more
time running (and dealing) with issues in their jobs than writing
them.

We need to have some integration tests that simulate common user
scenarios and some distribution use cases, e.g. Probably the most
common data store used for streaming is Kafka (at least in Open
Source). We should have an IT that tests some common issues that can
arrive when you use kafka, what happens if a kafka broker goes down,

Re: Eclipse support

[Ismaël Mejía]

Hello,

Thanks for bringing this info, I tried to compile with the eclipse
compiler and I can confirm that it does not wok, Eclipse's JDT is more
annoying about generics so it could be related to this.

Filled https://issues.apache.org/jira/browse/BEAM-3508 to track it.
Feel free to contribute a fix if you feel like it.

Ismaël


On Sat, Jan 20, 2018 at 10:20 PM, Ron Gonzalez  wrote:
> Hello again,
>   Sorry to keep asking about this, but I can't seem to get Eclipse working
> for this project.
>
>   I did mvn eclipse:clean eclipse:eclipse command and I've reduced the
> problem now down to the Java generics issues related to autovalue types. I
> had to run mvn generate-sources generate-test-sources in each of the
> sub-modules to make a lot of the errors in Eclipse go away since after
> running mvn -DskipTests clean install, the target/generated-sources and
> target/generated-test-sources are empty for some reason.
>
>   Interestingly enough, if I run mvn -Peclipse-jdt -DskipTests clean install
> from the command line, I am able to reproduce the same errors that I see in
> my Eclipse installation. I am using Eclipse Neon.
>
>
>   One such error is:
>
> @Experimental(Kind.FILESYSTEM)
> @Deprecated
> public  TypedWrite to(
> DynamicAvroDestinations
> dynamicDestinations) {
>   return toBuilder()
>   .setDynamicDestinations((DynamicAvroDestinations)
> dynamicDestinations)
>   .build();
> }
>
>  gives the error of:
>
> Description Resource Path Location Type
> Type mismatch: cannot convert from
> AvroIO.TypedWrite to
> AvroIO.TypedWrite AvroIO.java
> /beam-sdks-java-core/src/main/java/org/apache/beam/sdk/io line 1007 Java
> Problem
>
>  I have 48 errors left and most of these remaining errors are of this kind
> (related to Java generics of an auto-valued type). I enabled the Maven
> Annotation setting, per the Eclipse tips in the website.
>  I double-checked the compiling JDK in Eclipse, and it's using my system
> JDK, which I assume is being used by the command line compilation, which is
> successful.
>  Does anybody have Eclipse completely compiling for them without errors? Any
> tips?
>
> Thanks,
> Ron
>
> On Wednesday, January 17, 2018, 10:34:48 AM PST, Ted Yu
>  wrote:
>
>
> Have you tried running 'mvn eclipse:eclipse' and importing from the root of
> workspace ?
>
> On Wed, Jan 17, 2018 at 10:32 AM, Ron Gonzalez  wrote:
>
> Hi,
>   I've been trying this for a couple of days now, but I can't seem to get a
> clean Eclipse import.
>   I refreshed to latest master, got a clean mvn -DskipTests clean install,
> ran through the Eclipse setup steps for m2e-apt installation.
>   I'm getting errors like below. Do you have any tips to get this going?
>
> Thanks,
> Ron
>
> Description Resource Path Location Type
> ACCUMULATING cannot be resolved to a variable WindowingStrategyTranslation.
> java /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 56 Java Problem
> AFTER_ALL cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 34 Java Problem
> AFTER_ALL cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 241 Java Problem
> AFTER_ANY cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 37 Java Problem
> AFTER_ANY cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 243 Java Problem
> AFTER_EACH cannot be resolved to a variable TriggerStateMachines.java
> /beam-runners-core-java/src/ main/java/org/apache/beam/
> runners/core/triggers line 59 Java Problem
> AFTER_EACH cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 245 Java Problem
> AFTER_END_OF_WINDOW cannot be resolved to a variable
> TriggerStateMachines.java /beam-runners-core-java/src/
> main/java/org/apache/beam/ runners/core/triggers line 40 Java Problem
> AFTER_END_OF_WINDOW cannot be resolved to a variable TriggerTranslation.java
> /beam-runners-core- construction-java/src/main/
> java/org/apache/beam/runners/ core/construction line 248 Java Problem
> AFTER_PROCESSING_TIME cannot be resolved to a variable
> TriggerStateMachines.java /beam-runners-core-java/src/
> main/java/org/apache/beam/ runners/core/triggers line 62 Java Problem
> AFTER_PROCESSING_TIME cannot be resolved to a variable
> TriggerTranslation.java /beam-runners-core- construction-java/src/main/
> 

Re: Fwd: Google Summer of Code 2018 is coming

[Jean-Baptiste Onofré]

Hi Davor,

you were faster than me: I was just preparing an e-mail about GSoC ;)

+1

Regards
JB

On 01/22/2018 08:37 AM, Davor Bonaci wrote:
> This is a great way to grow the community. If you are interested in mentoring
> students contributing to Beam, please apply!
> 
> -- Forwarded message --
> From: *Ulrich Stärk* >
> Date: Sun, Jan 21, 2018 at 1:22 PM
> Subject: Google Summer of Code 2018 is coming
> To: ment...@community.apache.org 
> 
> 
> Hello PMCs (incubator Mentors, please forward this email to your podlings),
> 
> Google Summer of Code [1] is a program sponsored by Google allowing students 
> to
> spend their summer
> working on open source software. Students will receive stipends for developing
> open source software
> full-time for three months. Projects will provide mentoring and project ideas,
> and in return have
> the chance to get new code developed and - most importantly - to identify and
> bring in new committers.
> 
> The ASF will apply as a participating organization meaning individual projects
> don't have to apply
> separately.
> 
> If you want to participate with your project we ask you to do the following
> things as soon as
> possible but please no later than 2017-01-30:
> 
> 1. understand what it means to be a mentor [2].
> 
> 2. record your project ideas.
> 
> Just create issues in JIRA, label them with gsoc2018, and they will show up at
> [3]. Please be as
> specific as possible when describing your idea. Include the programming
> language, the tools and
> skills required, but try not to scare potential students away. They are 
> supposed
> to learn what's
> required before the program starts.
> 
> Use labels, e.g. for the programming language (java, c, c++, erlang, python,
> brainfuck, ...) or
> technology area (cloud, xml, web, foo, bar, ...) and record them at [5].
> 
> Please use the COMDEV JIRA project for recording your ideas if your project
> doesn't use JIRA (e.g.
> httpd, ooo). Contact d...@community.apache.org 
> 
> if you need assistance.
> 
> [4] contains some additional information (will be updated for 2017 shortly).
> 
> 3. subscribe to ment...@community.apache.org
> ; restricted to potential mentors, meant 
> to
> be used as a
> private list - general discussions on the public d...@community.apache.org
>  list as much as possible
> please). Use a recognized address when subscribing (@apache.org
>  or one of your alias addresses on
> record).
> 
> Note that the ASF isn't accepted as a participating organization yet,
> nevertheless you *have to*
> start recording your ideas now or we will not get accepted.
> 
> Over the years we were able to complete hundreds of projects successfully. 
> Some
> of our prior
> students are active contributors now! Let's make this year a success again!
> 
> Cheers,
> 
> Uli
> 
> [1] https://summerofcode.withgoogle.com/ 
> 
> [2] http://community.apache.org/guide-to-being-a-mentor.html
> 
> [3] http://s.apache.org/gsoc2018ideas 
> [4] http://community.apache.org/gsoc.html 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com