Jenkins build is back to normal : beam_PostCommit_MavenVerify #368

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request: Revert "Add wildcard to checkstyle or...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/305


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Revert "Add wildcard to checkstyle ordering"

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4e5cfc4cf -> 07c60a965


Revert "Add wildcard to checkstyle ordering"

This reverts commit 7562a936527a4594cb76fcd1df77579e3b7f809d.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1f80c096
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1f80c096
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1f80c096

Branch: refs/heads/master
Commit: 1f80c096e9387fcc95f039b09b29197b5dda9e87
Parents: 4e5cfc4
Author: Davor Bonaci 
Authored: Sun May 8 13:45:58 2016 -0700
Committer: Davor Bonaci 
Committed: Sun May 8 13:45:58 2016 -0700

--
 sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1f80c096/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml 
b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index f92a0a1..2a4f832 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -97,7 +97,7 @@ page at http://checkstyle.sourceforge.net/config.html -->
   
 
   
-  
+  
   
   
   

[2/2] incubator-beam git commit: This closes #305

[davor]

This closes #305


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/07c60a96
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/07c60a96
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/07c60a96

Branch: refs/heads/master
Commit: 07c60a965b971d8a1135e70db615424fe8dc0d59
Parents: 4e5cfc4 1f80c09
Author: Davor Bonaci 
Authored: Sun May 8 13:47:50 2016 -0700
Committer: Davor Bonaci 
Committed: Sun May 8 13:47:50 2016 -0700

--
 sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request: Revert "Add wildcard to checkstyle or...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam/pull/305

Revert "Add wildcard to checkstyle ordering"

This reverts commit 7562a936527a4594cb76fcd1df77579e3b7f809d.

Breaks build. Will merge immediately. Cc: @sammcveety, @dhalperi.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam rollback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/305.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #305


commit 1f80c096e9387fcc95f039b09b29197b5dda9e87
Author: Davor Bonaci 
Date:   2016-05-08T20:45:58Z

Revert "Add wildcard to checkstyle ordering"

This reverts commit 7562a936527a4594cb76fcd1df77579e3b7f809d.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-77) Reorganize Directory structure

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-77.

Resolution: Fixed

> Reorganize Directory structure
> --
>
> Key: BEAM-77
> URL: https://issues.apache.org/jira/browse/BEAM-77
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Frances Perry
>Assignee: Davor Bonaci
>
> Now that we've done the initial Dataflow code drop, we will restructure 
> directories to provide space for additional SDKs and Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-77) Reorganize Directory structure

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-77:
-
Assignee: Davor Bonaci  (was: Jean-Baptiste Onofré)

> Reorganize Directory structure
> --
>
> Key: BEAM-77
> URL: https://issues.apache.org/jira/browse/BEAM-77
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Frances Perry
>Assignee: Davor Bonaci
>
> Now that we've done the initial Dataflow code drop, we will restructure 
> directories to provide space for additional SDKs and Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-154) Provide Maven BOM

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-154.
-
Resolution: Fixed

> Provide Maven BOM
> -
>
> Key: BEAM-154
> URL: https://issues.apache.org/jira/browse/BEAM-154
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> When using the Java SDK (for instance to develop IO), the developer has to 
> add dependencies in his pom.xml (like junit, hamcrest, slf4j, ...).
> To simplify the way to define the dependencies, each Beam SDK could provide a 
> Maven BoM (Bill of Material) describing these dependencies. Then the 
> developer could simply define this BoM as pom.xml dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-256) Add lifecycle event verifiers for Beam pipelines.

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-256:
--
Assignee: Jason Kuster  (was: Davor Bonaci)

> Add lifecycle event verifiers for Beam pipelines.
> -
>
> Key: BEAM-256
> URL: https://issues.apache.org/jira/browse/BEAM-256
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-254) Add maven-rat-plugin execution

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275710#comment-15275710
 ] 

Davor Bonaci commented on BEAM-254:
---

We should double-check that the checkstyle plugin cannot be configured to do 
the same thing. If so, it would be better to maintain one system, as opposed to 
two.

> Add maven-rat-plugin execution
> --
>
> Key: BEAM-254
> URL: https://issues.apache.org/jira/browse/BEAM-254
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> In order to check the license headers and validate we are "Apache compliant", 
> I will add the maven-rat-plugin execution in the main parent POM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-264) Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and Construction of Such Structures from Scalars, Other Structures and Queries

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275707#comment-15275707
 ] 

Davor Bonaci commented on BEAM-264:
---

Can you develop your idea a little bit please?

I'd suggest working through a scenario, trying to express it in the Beam model, 
and then trying to identify areas for improvement. The things you mention can 
be expressed in the Beam model already. There are certainly areas for 
improvement, however. To be able to assess this JIRA issue, we'll need more 
data as to what actually do you propose.

> Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and 
> Construction of Such Structures from Scalars, Other Structures and Queries
> -
>
> Key: BEAM-264
> URL: https://issues.apache.org/jira/browse/BEAM-264
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Not all timeseries data is scalar values hence support other types of values 
> like vectors, matrices. Also ability to construct these constructs. E.g. if I 
> want a time series vector of asset prices as input to a pipeline. Also I will 
> have to choose what stocks are in the vector and also how to handle if the 
> choosing is time varying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-264) Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and Construction of Such Structures from Scalars, Other Structures and Queries

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-264:
--
Assignee: Davor Bonaci

> Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and 
> Construction of Such Structures from Scalars, Other Structures and Queries
> -
>
> Key: BEAM-264
> URL: https://issues.apache.org/jira/browse/BEAM-264
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Not all timeseries data is scalar values hence support other types of values 
> like vectors, matrices. Also ability to construct these constructs. E.g. if I 
> want a time series vector of asset prices as input to a pipeline. Also I will 
> have to choose what stocks are in the vector and also how to handle if the 
> choosing is time varying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-264) Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and Construction of Such Structures from Scalars, Other Structures and Queries

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-264:
--
Component/s: sdk-ideas

> Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and 
> Construction of Such Structures from Scalars, Other Structures and Queries
> -
>
> Key: BEAM-264
> URL: https://issues.apache.org/jira/browse/BEAM-264
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-ideas
>Reporter: Suminda Dharmasena
>
> Not all timeseries data is scalar values hence support other types of values 
> like vectors, matrices. Also ability to construct these constructs. E.g. if I 
> want a time series vector of asset prices as input to a pipeline. Also I will 
> have to choose what stocks are in the vector and also how to handle if the 
> choosing is time varying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275699#comment-15275699
 ] 

Davor Bonaci commented on BEAM-262:
---

Oh, actually, it is quite the opposite ;)

Some runners/backends, including those mentioned, don't have all the features 
that are needed to run Beam pipelines at scale. You should probably refer to 
the Capability Matrix on our website.

In general, there's some mismatch with the Beam project vision and 
architecture. We'd love to have this discussion, but this topic is much more 
complex to have it in a JIRA issue. I'd invite you to join one of the future 
Beam (virtual) meetings, and discuss there.

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275697#comment-15275697
 ] 

Davor Bonaci commented on BEAM-265:
---

Fair point -- in a (very) limited scope, user-defined functions can be 
language-independent.

The real question is what are the tradeoffs: limiting scope of user-defined 
functions is a big downside. On the other hand, language-independence at that 
level is not providing us appropriate value. Once we clarify the Runner / Fn 
APIs, there'll be a set of core SDKs that will have to provide 
language-specific "Fn harness". This achieves total independence between 
runners and SDKs, without sacrifices in scope.

In this sense, you are challenging top-level project vision / architecture. It 
is a very complex and long discussion -- it would be better to have this part 
of the discussion on the dev@ mailing list. Even better, during one of the Beam 
(virtual) meetings.

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-262) Native Runners | Direct Compiler

[Suminda Dharmasena (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275695#comment-15275695
 ] 

Suminda Dharmasena commented on BEAM-262:
-

For what Beam does you do not need the fully fledged overhead of Apache Flink 
or Apache Spark. If you look at languages like CAL Actor Language what is 
needed is much more rudimentary to run dataflow pipelines. So what is needed is 
what you have to have to run the DF pipelines and not all that is present in 
the underlying frameworks.

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Suminda Dharmasena (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275691#comment-15275691
 ] 

Suminda Dharmasena edited comment on BEAM-265 at 5/8/16 7:54 PM:
-

User defined functions can be language independent if defined in a DSL of its 
own or a supported subset of the implementation language. As you go on the 
subset can be expanded to support more of the language like libraries.


was (Author: sirinath):
User defined functions can be language independent if defined in a DSL of its 
own or a supported subset of the implementation language.

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-262:
--
Assignee: Davor Bonaci

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-263) Generate Processing Infrastructure Code and Code for Runners in High Level Languages

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-263:
--
Assignee: Davor Bonaci

> Generate Processing Infrastructure Code and Code for Runners in High Level 
> Languages
> 
>
> Key: BEAM-263
> URL: https://issues.apache.org/jira/browse/BEAM-263
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> In addition you can auto generate processing infrastructure code in languages 
> like C/C++, Java, Scala, C#, F#, Rust, Julia, Swift, D, Nim, etc.
> Auto generated code will be the runner itself and should not need any 
> additional infrastructure to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-263) Generate Processing Infrastructure Code and Code for Runners in High Level Languages

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275690#comment-15275690
 ] 

Davor Bonaci commented on BEAM-263:
---

I'd say this is beyond the scope of the Apache Beam project at present time.

To be able to run data-processing pipelines at scale, such a runner would have 
to encompass the whole functionality of a data-processing backend. We don't aim 
to do that -- instead, we rely on existing backends and services, like Google 
Cloud Dataflow, Apache Flink or Apache Spark. At present, we don't plan to 
increase scope to try to create our own, internal backend.

Make sense?

> Generate Processing Infrastructure Code and Code for Runners in High Level 
> Languages
> 
>
> Key: BEAM-263
> URL: https://issues.apache.org/jira/browse/BEAM-263
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>
> In addition you can auto generate processing infrastructure code in languages 
> like C/C++, Java, Scala, C#, F#, Rust, Julia, Swift, D, Nim, etc.
> Auto generated code will be the runner itself and should not need any 
> additional infrastructure to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-262:
--
Component/s: runner-ideas

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Suminda Dharmasena (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275691#comment-15275691
 ] 

Suminda Dharmasena commented on BEAM-265:
-

User defined functions can be language independent if defined in a DSL of its 
own or a supported subset of the implementation language.

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275692#comment-15275692
 ] 

Davor Bonaci commented on BEAM-262:
---

I'd say this is beyond the scope of the Apache Beam project at present time.

To be able to run data-processing pipelines at scale, such a runner would have 
to encompass the whole functionality of a data-processing backend. We don't aim 
to do that – instead, we rely on existing backends and services, like Google 
Cloud Dataflow, Apache Flink or Apache Spark. At present, we don't plan to 
increase scope to try to create our own, internal backend.

Make sense?

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-266) Visual Pipeline Designers / Editors

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275688#comment-15275688
 ] 

Davor Bonaci commented on BEAM-266:
---

I agree; there are many ideas in this space. This is a very interesting problem 
space, but a really big piece of work.

> Visual Pipeline Designers / Editors
> ---
>
> Key: BEAM-266
> URL: https://issues.apache.org/jira/browse/BEAM-266
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Suminda Dharmasena
>
> This concept can be used by business people with little or no programming 
> ability if you have a Visual Editors. If the editor is an components, the 
> editor can be reused in integration with other products.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_MavenVerify #367

[Apache Jenkins Server]

See 

Changes:

[davor] Add wildcard to checkstyle ordering

--
[...truncated 3431 lines...]
[WARNING] grpc-protobuf-0.12.0.jar, grpc-all-0.12.0.jar define 4 overlapping 
classes: 
[WARNING]   - io.grpc.protobuf.ProtoUtils$2
[WARNING]   - io.grpc.protobuf.ProtoUtils
[WARNING]   - io.grpc.protobuf.ProtoUtils$1
[WARNING]   - io.grpc.protobuf.ProtoInputStream
[WARNING] grpc-okhttp-0.12.0.jar, grpc-all-0.12.0.jar define 76 overlapping 
classes: 
[WARNING]   - io.grpc.okhttp.OkHttpSettingsUtil
[WARNING]   - io.grpc.okhttp.AsyncFrameWriter$12
[WARNING]   - io.grpc.okhttp.NegotiationType
[WARNING]   - io.grpc.okhttp.OkHttpTlsUpgrader
[WARNING]   - io.grpc.okhttp.Utils
[WARNING]   - io.grpc.okhttp.AsyncFrameWriter$WriteRunnable
[WARNING]   - io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator
[WARNING]   - io.grpc.okhttp.OkHttpChannelBuilder$2
[WARNING]   - io.grpc.okhttp.internal.framed.Huffman$Node
[WARNING]   - io.grpc.okhttp.AsyncFrameWriter$7
[WARNING]   - 66 more...
[WARNING] grpc-auth-0.12.0.jar, grpc-all-0.12.0.jar define 2 overlapping 
classes: 
[WARNING]   - io.grpc.auth.ClientAuthInterceptor$1
[WARNING]   - io.grpc.auth.ClientAuthInterceptor
[WARNING] java-sdk-all-0.1.0-incubating-SNAPSHOT.jar, 
google-cloud-dataflow-java-runner-0.1.0-incubating-SNAPSHOT.jar define 1717 
overlapping classes: 
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.TreeRangeSet$ComplementRangesByLowerBound$2
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.WellBehavedMap$EntrySet$1$1
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.util.concurrent.CycleDetectingLockFactory$Policies$1
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Maps$6
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.primitives.UnsignedBytes$LexicographicalComparatorHolder$UnsafeComparator$1
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Range$1
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.Collections2$OrderedPermutationCollection
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.base.Splitter$2
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.base.Equivalence$Identity
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Lists$1
[WARNING]   - 1707 more...
[WARNING] grpc-core-0.12.0.jar, grpc-all-0.12.0.jar define 248 overlapping 
classes: 
[WARNING]   - io.grpc.ClientInterceptors$1
[WARNING]   - io.grpc.internal.AbstractReferenceCounted
[WARNING]   - io.grpc.inprocess.InProcessTransport
[WARNING]   - io.grpc.internal.ServerImpl
[WARNING]   - io.grpc.internal.MessageFramer$OutputStreamAdapter
[WARNING]   - io.grpc.internal.ReadableBuffers$BufferInputStream
[WARNING]   - io.grpc.internal.SerializingExecutor$TaskRunner
[WARNING]   - io.grpc.internal.SharedResourceHolder$ScheduledExecutorFactory
[WARNING]   - io.grpc.internal.ContextRunnable
[WARNING]   - io.grpc.NameResolver
[WARNING]   - 238 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Attaching shaded artifact.
[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (runnable-on-service-tests) @ 
spark-runner ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @ spark-runner ---
[INFO] Starting audit...
[ERROR] 
:31:
 Import kafka.serializer.Decoder appears after other imports that it should 
precede [ImportOrder]
[ERROR] 
:35:
 Import scala.Tuple2 appears after other imports that it should precede 
[ImportOrder]
[ERROR] 
:37:
 Import scala.Tuple2 appears after other imports that it should precede 
[ImportOrder]
[ERROR] 
:87:
 Import scala.Tuple2 appears after other imports that it should precede 
[ImportOrder]
[ERROR] 

Build failed in Jenkins: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #367

[Apache Jenkins Server]

See 


--
[...truncated 219 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

[jira] [Updated] (BEAM-269) Create BigDecimal Coder

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-269:
--
Assignee: Jesse Anderson  (was: Davor Bonaci)

> Create BigDecimal Coder
> ---
>
> Key: BEAM-269
> URL: https://issues.apache.org/jira/browse/BEAM-269
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
>
> There isn't a coder for BigDecimal. This class is especially important for 
> financial companies to represent money.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275685#comment-15275685
 ] 

Davor Bonaci commented on BEAM-265:
---

There are many things conflated here ;)

Are you referring to data-dependent pipeline structure? If so, we fully agree. 
This is on the wider roadmap, but not being worked on currently.

In terms of language-independence, Runner API and Fn API work currently in 
progress achieves that goal. (There's never full language-independence -- 
user-defined functions are always language-specific.)

Do you agree, or am I missing something?

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-265:
--
Assignee: Davor Bonaci

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-265:
--
Component/s: beam-model

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-216) Create Storm Runner

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-216.
-

> Create Storm Runner 
> 
>
> Key: BEAM-216
> URL: https://issues.apache.org/jira/browse/BEAM-216
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Sriharsha Chintalapani
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-260) Know the getSideInputWindow upper bound so can gc side input state

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-260:
--
Assignee: Mark Shields  (was: Frances Perry)

> Know the getSideInputWindow upper bound so can gc side input state
> --
>
> Key: BEAM-260
> URL: https://issues.apache.org/jira/browse/BEAM-260
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> We currently have no static knowledge about the getSideInputWindow function, 
> and runners are thus forced to hold on to all side input state / elements in 
> case a future element reaches back into an earlier side input element.
> Maybe we need an upper bound on lag from current to result of 
> getSideInputWindow so we can have a progressing gc horizon as we do for  GKB 
> window state. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #292

[davor]

This closes #292


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4e5cfc4c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4e5cfc4c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4e5cfc4c

Branch: refs/heads/master
Commit: 4e5cfc4cf0ece304932aeca7699001702e72efea
Parents: 0f3b053 7562a93
Author: Davor Bonaci 
Authored: Sun May 8 12:31:58 2016 -0700
Committer: Davor Bonaci 
Committed: Sun May 8 12:31:58 2016 -0700

--
 sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Add wildcard to checkstyle ordering

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0f3b05335 -> 4e5cfc4cf


Add wildcard to checkstyle ordering

Add wildcard to checkstyle, to handle unexpected package prefixes.  These 
should still be ordered before the sun and java packages.

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7562a936
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7562a936
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7562a936

Branch: refs/heads/master
Commit: 7562a936527a4594cb76fcd1df77579e3b7f809d
Parents: 0f3b053
Author: sammcveety 
Authored: Thu May 5 12:15:24 2016 -0700
Committer: Davor Bonaci 
Committed: Sun May 8 12:31:48 2016 -0700

--
 sdks/java/build-tools/src/main/resources/beam/checkstyle.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7562a936/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml 
b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index 2a4f832..f92a0a1 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -97,7 +97,7 @@ page at http://checkstyle.sourceforge.net/config.html -->
   
 
   
-  
+  
   
   
   

[jira] [Commented] (BEAM-77) Reorganize Directory structure

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275489#comment-15275489
 ] 

Jean-Baptiste Onofré commented on BEAM-77:
--

I think we can consider this Jira as fixed, right ?

> Reorganize Directory structure
> --
>
> Key: BEAM-77
> URL: https://issues.apache.org/jira/browse/BEAM-77
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Frances Perry
>Assignee: Jean-Baptiste Onofré
>
> Now that we've done the initial Dataflow code drop, we will restructure 
> directories to provide space for additional SDKs and Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-267) Enable Chekstyle check in Spark runner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-267.
---
Resolution: Fixed

> Enable Chekstyle check in Spark runner
> --
>
> Key: BEAM-267
> URL: https://issues.apache.org/jira/browse/BEAM-267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-267) Enable Chekstyle check in Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275486#comment-15275486
 ] 

ASF GitHub Bot commented on BEAM-267:
-

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/298


> Enable Chekstyle check in Spark runner
> --
>
> Key: BEAM-267
> URL: https://issues.apache.org/jira/browse/BEAM-267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-267] Enable checkstyle in Spark...

[jbonofre]

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-267] Enable checkstyle in Spark runner

[jbonofre]

[BEAM-267] Enable checkstyle in Spark runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0f3b0533
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0f3b0533
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0f3b0533

Branch: refs/heads/master
Commit: 0f3b053356f3321d08d4e2ee457a037df778bee4
Parents: ff825b0
Author: Jean-Baptiste Onofré 
Authored: Fri May 6 17:47:46 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Sun May 8 07:59:26 2016 +0200

--
 runners/spark/pom.xml   |   2 -
 .../runners/spark/SparkPipelineOptions.java |   3 +
 .../beam/runners/spark/SparkPipelineRunner.java |  13 +-
 .../runners/spark/SparkRunnerRegistrar.java |   5 +-
 .../spark/aggregators/AggAccumParam.java|   4 +
 .../spark/aggregators/NamedAggregators.java |  64 +++--
 .../beam/runners/spark/coders/CoderHelpers.java |  13 +-
 .../runners/spark/coders/NullWritableCoder.java |  10 +-
 .../runners/spark/coders/WritableCoder.java |  19 +-
 .../apache/beam/runners/spark/io/ConsoleIO.java |   7 +
 .../beam/runners/spark/io/CreateStream.java |   6 +-
 .../apache/beam/runners/spark/io/KafkaIO.java   |  18 +-
 .../beam/runners/spark/io/hadoop/HadoopIO.java  |  27 +-
 .../spark/io/hadoop/ShardNameBuilder.java   |   7 +-
 .../io/hadoop/ShardNameTemplateHelper.java  |   7 +-
 .../io/hadoop/TemplatedAvroKeyOutputFormat.java |   9 +-
 .../TemplatedSequenceFileOutputFormat.java  |   7 +-
 .../io/hadoop/TemplatedTextOutputFormat.java|   7 +-
 .../runners/spark/translation/DoFnFunction.java |  38 +--
 .../spark/translation/EvaluationContext.java|  26 +-
 .../spark/translation/MultiDoFnFunction.java|  42 +--
 .../spark/translation/SparkContextFactory.java  |   3 +
 .../translation/SparkPipelineEvaluator.java |  13 +-
 .../translation/SparkPipelineTranslator.java|   3 +-
 .../spark/translation/SparkProcessContext.java  |  72 ++---
 .../spark/translation/SparkRuntimeContext.java  |  51 ++--
 .../spark/translation/TransformEvaluator.java   |  11 +-
 .../spark/translation/TransformTranslator.java  | 271 +++
 .../streaming/StreamingEvaluationContext.java   |  19 +-
 .../streaming/StreamingTransformTranslator.java |  85 +++---
 .../StreamingWindowPipelineDetector.java|   6 +-
 .../runners/spark/util/BroadcastHelper.java |  12 +-
 .../beam/runners/spark/util/ByteArray.java  |   7 +-
 .../util/SinglePrimitiveOutputPTransform.java   |   3 +
 .../beam/runners/spark/EmptyInputTest.java  |   6 +
 .../beam/runners/spark/SimpleWordCountTest.java |  21 +-
 .../runners/spark/SparkRunnerRegistrarTest.java |  12 +-
 .../beam/runners/spark/io/AvroPipelineTest.java |   6 +-
 .../beam/runners/spark/io/NumShardsTest.java|   3 +
 .../io/hadoop/HadoopFileFormatPipelineTest.java |  21 +-
 .../spark/io/hadoop/ShardNameBuilderTest.java   |   3 +
 .../spark/translation/CombineGloballyTest.java  |  11 +-
 .../spark/translation/CombinePerKeyTest.java|   3 +
 .../spark/translation/DoFnOutputTest.java   |   3 +
 .../translation/MultiOutputWordCountTest.java   |   6 +
 .../spark/translation/SerializationTest.java|  16 +-
 .../spark/translation/SideEffectsTest.java  |   6 +-
 .../translation/SparkPipelineOptionsTest.java   |   3 +
 .../translation/WindowedWordCountTest.java  |   3 +
 .../streaming/FlattenStreamingTest.java |   2 +-
 .../streaming/KafkaStreamingTest.java   |   2 +-
 .../streaming/SimpleStreamingWordCountTest.java |   5 +-
 .../streaming/utils/EmbeddedKafkaCluster.java   |   4 +
 .../streaming/utils/PAssertStreaming.java   |   3 +-
 54 files changed, 634 insertions(+), 395 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0f3b0533/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index e673246..5daf1e1 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -211,12 +211,10 @@
 org.apache.maven.plugins
 maven-jar-plugin
   
-  
   
 org.apache.maven.plugins
 maven-source-plugin

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0f3b0533/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index bdf832b..091382e 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++