[jira] [Commented] (BEAM-80) Support combiner lifting for (Keyed)CombineWithContext

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203109#comment-15203109
 ] 

Davor Bonaci commented on BEAM-80:
--

Pei, is this complete?

> Support combiner lifting for (Keyed)CombineWithContext
> --
>
> Key: BEAM-80
> URL: https://issues.apache.org/jira/browse/BEAM-80
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Pei He
>Assignee: Pei He
>
> This is a missing feature of combine with context.
> Combiner lifting is currently disabled for (Keyed)CombineWithContext with a 
> passing through ParDo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-96) Support composing combine functions

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203110#comment-15203110
 ] 

Davor Bonaci commented on BEAM-96:
--

Pei, is this done?

> Support composing combine functions
> ---
>
> Key: BEAM-96
> URL: https://issues.apache.org/jira/browse/BEAM-96
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>
> The proposal of composed combine functions is following:
> pc.apply(
> Combine.perKey(
>  CombineFns.composeKeyed()
> .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
> .with(identityFn, new MeanFn(), meanLatencyTag)));
> Example code:
>* PCollection> latencies = ...;
>*
>* TupleTag maxLatencyTag = new TupleTag();
>* TupleTag meanLatencyTag = new TupleTag();
>*
>* SimpleFunction identityFn =
>* new SimpleFunction() {
>*   @Override
>*   public Integer apply(Integer input) {
>*   return input;
>*   }};
>* PCollection> maxAndMean = latencies.apply(
>* Combine.perKey(
>* CombineFns.composeKeyed()
>*.with(identityFn, new MaxIntegerFn(), maxLatencyTag)
>*.with(identityFn, new MeanFn(), meanLatencyTag)));
>*
>* PCollection finalResultCollection = maxAndMean
>* .apply(ParDo.of(
>* new DoFn, T>() {
>*   @Override
>*   public void processElement(ProcessContext c) throws Exception {
>* KV e = c.element();
>* Integer maxLatency = e.getValue().get(maxLatencyTag);
>* Double meanLatency = e.getValue().get(meanLatencyTag);
>*  Do Something 
>* c.output(...some T...);
>*   }
>* }));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-122) GlobalWindow and allowedLateness can cause inconsistent timer interpretation

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-122:
--
Assignee: Mark Shields  (was: Frances Perry)

> GlobalWindow and allowedLateness can cause inconsistent timer interpretation 
> -
>
> Key: BEAM-122
> URL: https://issues.apache.org/jira/browse/BEAM-122
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> In ReduceFnRunner we have code such as
>window.getMaxTimestamp().plus(windowingStrategy.getAllowedLateness())
> If window is global then maxTimestamp will be 
> BoundedWindow.TIMESTAMP_MAX_VALUE.
> Meanwhile, timestamps beyond BoundedWindow.TIMESTAMP_MAX_VALUE will be 
> clipped in most runners.
> This could cause the time of an expected timer (eg for garbage collection) to 
> not match the actual time of a fired timer.
> We should either make non-zero allowedLateness on the Global window illegal 
> (probably obnoxious) or explicitly clip it to zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-134) Investigate use of AutoValue

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-134:
-

 Summary: Investigate use of AutoValue
 Key: BEAM-134
 URL: https://issues.apache.org/jira/browse/BEAM-134
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Scott Wegner
Assignee: Davor Bonaci
Priority: Minor


The initial PR for [BEAM-118] added a dependency on AutoValue to auto-implement 
equality semantics for a new POJO. We decided to remove the dependency because 
the cost of adding the dependency for this feature may not be worth it for the 
value.

However, we could use AutoValue for all of our POJO's, it might be worth it. 
The proposal here is to follow-up with an investigation on whether we would 
gain significant value to porting our code to use AutoValue instead of 
hand-written POJO's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-126) KafkaWindowedWordCountExample fails with JobName invalid

[William McCarthy (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201682#comment-15201682
 ] 

William McCarthy commented on BEAM-126:
---

Sweet. Thank you!

> KafkaWindowedWordCountExample fails with JobName invalid
> 
>
> Key: BEAM-126
> URL: https://issues.apache.org/jira/browse/BEAM-126
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: William McCarthy
>Assignee: Maximilian Michels
>
> I get the following when I try to run the KafkaWindowedWordCountExample.
> I'm able to fix it by changing line 106 of that file to:
> options.setJobName("kafkawindowword" + options.getWindowSize() + 
> "seconds");
> flink run -c 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample
>  target/beam-1.0-SNAPSHOT.jar 
> test cl-mdgy:2181 cl-pu4p:9092,cl-y06o:9093 mygroup
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
>   at 
> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> Caused by: java.lang.RuntimeException: Failed to construct instance from 
> factory method FlinkPipelineRunner#fromOptions(interface 
> com.google.cloud.dataflow.sdk.options.PipelineOptions)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>   at 
> com.google.cloud.dataflow.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:57)
>   at com.google.cloud.dataflow.sdk.Pipeline.create(Pipeline.java:134)
>   at 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample.main(KafkaWindowedWordCountExample.java:114)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
>   ... 6 more
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>   ... 15 more
> Caused by: java.lang.IllegalArgumentException: JobName invalid; the name must 
> consist of only the characters [-a-z0-9], starting with a letter and ending 
> with a letter or number
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.fromOptions(FlinkPipelineRunner.java:92)
>   ... 20 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-116) Change runners artifactId to runners-parent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201319#comment-15201319
 ] 

ASF GitHub Bot commented on BEAM-116:
-

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/59

[BEAM-116] change runners artifactId to runners-parent



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-116

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit 447c8af23b3fa727bba5aa97093699745d6e6f5b
Author: Maximilian Michels 
Date:   2016-03-16T11:03:07Z

[BEAM-116] change runners artifactId to runners-parent




> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/8] incubator-beam git commit: Add casts to fix mvn verify errors

[kenn]

Add casts to fix mvn verify errors


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f06e2a97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f06e2a97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f06e2a97

Branch: refs/heads/master
Commit: f06e2a977b8e8bee456d3323ab664da3b97f0aa5
Parents: 659f0b8
Author: bchambers 
Authored: Wed Mar 16 12:44:20 2016 -0700
Committer: bchambers 
Committed: Thu Mar 17 11:12:36 2016 -0700

--
 .../dataflow/sdk/runners/DataflowPipelineRunner.java   | 13 +++--
 .../sdk/runners/inprocess/FlattenEvaluatorFactory.java |  6 --
 .../runners/inprocess/GroupByKeyEvaluatorFactory.java  |  7 ---
 .../runners/inprocess/ParDoMultiEvaluatorFactory.java  |  5 -
 .../runners/inprocess/ParDoSingleEvaluatorFactory.java |  5 -
 .../sdk/runners/inprocess/ViewEvaluatorFactory.java|  7 ---
 .../cloud/dataflow/sdk/transforms/DoFnReflector.java   |  7 ---
 .../google/cloud/dataflow/sdk/util/DoFnRunners.java|  4 +++-
 8 files changed, 34 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f06e2a97/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
index 29f9b85..5fbbcac 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
@@ -866,12 +866,13 @@ public class DataflowPipelineRunner extends 
PipelineRunner
   input.getWindowingStrategy().getWindowFn().windowCoder();
 
   @SuppressWarnings({"rawtypes", "unchecked"})
-  PCollectionView view = PCollectionViews.singletonView(
-  input.getPipeline(),
-  (WindowingStrategy) input.getWindowingStrategy(),
-  hasDefault,
-  defaultValue,
-  defaultValueCoder);
+  PCollectionView view =
+  (PCollectionView) PCollectionViews.singletonView(
+  input.getPipeline(),
+  (WindowingStrategy) input.getWindowingStrategy(),
+  hasDefault,
+  defaultValue,
+  defaultValueCoder);
 
   IsmRecordCoder ismCoder =
   coderForSingleton(windowCoder, defaultValueCoder);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f06e2a97/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
index bde1df4..ce315be 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
@@ -30,13 +30,15 @@ import com.google.cloud.dataflow.sdk.values.PCollectionList;
  * {@link PTransform}.
  */
 class FlattenEvaluatorFactory implements TransformEvaluatorFactory {
-  @SuppressWarnings({"unchecked", "rawtypes"})
   @Override
   public  TransformEvaluator forApplication(
   AppliedPTransform application,
   CommittedBundle inputBundle,
   InProcessEvaluationContext evaluationContext) {
-return createInMemoryEvaluator((AppliedPTransform) application, 
inputBundle, evaluationContext);
+@SuppressWarnings({"cast", "unchecked", "rawtypes"})
+TransformEvaluator evaluator = (TransformEvaluator) 
createInMemoryEvaluator(
+(AppliedPTransform) application, inputBundle, evaluationContext);
+return evaluator;
   }
 
   private  TransformEvaluator createInMemoryEvaluator(

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f06e2a97/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
index ec63be8..dec78d6 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java

Jenkins build became unstable: beam_MavenVerify #14

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-125) PubSubIO.Read should be implemented by an UnboundedSource

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-125.
--
Resolution: Duplicate

I went ahead and assigned that one to you.

Feel free to change the title/desc if you feel it's not well suited?

> PubSubIO.Read should be implemented by an UnboundedSource
> -
>
> Key: BEAM-125
> URL: https://issues.apache.org/jira/browse/BEAM-125
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> So any runner can stream messages from pub/sub, we need an UnboundendSource 
> implementation of PubSubIO.Read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-133) Test flakiness in the Spark runner

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-133:
--
Component/s: (was: runner-spark)
 project-management

> Test flakiness in the Spark runner
> --
>
> Key: BEAM-133
> URL: https://issues.apache.org/jira/browse/BEAM-133
> Project: Beam
>  Issue Type: Bug
>  Components: project-management
>Reporter: Davor Bonaci
>Assignee: Davor Bonaci
>
> Jenkins shows some flakiness in the Spark runner in the context of an 
> unrelated pre-commit test.
> {code}
> Results :
> Tests in error: 
>   AvroPipelineTest.testGeneric:75 » Runtime java.io.IOException: Could 
> not creat...
>   NumShardsTest.testText:77 » Runtime java.io.IOException: Could not 
> create File...
>   HadoopFileFormatPipelineTest.testSequenceFile:83 » Runtime 
> java.io.IOException...
>   
> TransformTranslatorTest.testTextIOReadAndWriteTransforms:76->runPipeline:96 » 
> Runtime
>   KafkaStreamingTest.testRun:121 » Runtime java.io.IOException: failure 
> to login...
> Tests run: 27, Failures: 0, Errors: 5, Skipped: 0
> [ERROR] There are test failures.
> {code}
> https://builds.apache.org/job/beam_PreCommit/98/console
> Amit, does this sounds like a test code issue or the infrastructure issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-133) Test flakiness in the Spark runner

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202292#comment-15202292
 ] 

Davor Bonaci commented on BEAM-133:
---

Marking resolved for now.

> Test flakiness in the Spark runner
> --
>
> Key: BEAM-133
> URL: https://issues.apache.org/jira/browse/BEAM-133
> Project: Beam
>  Issue Type: Bug
>  Components: project-management
>Reporter: Davor Bonaci
>Assignee: Davor Bonaci
>
> Jenkins shows some flakiness in the Spark runner in the context of an 
> unrelated pre-commit test.
> {code}
> Results :
> Tests in error: 
>   AvroPipelineTest.testGeneric:75 » Runtime java.io.IOException: Could 
> not creat...
>   NumShardsTest.testText:77 » Runtime java.io.IOException: Could not 
> create File...
>   HadoopFileFormatPipelineTest.testSequenceFile:83 » Runtime 
> java.io.IOException...
>   
> TransformTranslatorTest.testTextIOReadAndWriteTransforms:76->runPipeline:96 » 
> Runtime
>   KafkaStreamingTest.testRun:121 » Runtime java.io.IOException: failure 
> to login...
> Tests run: 27, Failures: 0, Errors: 5, Skipped: 0
> [ERROR] There are test failures.
> {code}
> https://builds.apache.org/job/beam_PreCommit/98/console
> Amit, does this sounds like a test code issue or the infrastructure issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-116) Change runners artifactId to runners-parent

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned BEAM-116:
---

Assignee: Maximilian Michels

> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #41

[lcwik]

This closes #41


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f7aaee2e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f7aaee2e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f7aaee2e

Branch: refs/heads/master
Commit: f7aaee2ea759c0687588e62e5603ad50332d18ea
Parents: a461e00 91de072
Author: Luke Cwik 
Authored: Sat Mar 19 15:12:26 2016 -0700
Committer: Luke Cwik 
Committed: Sat Mar 19 15:12:26 2016 -0700

--
 .../dataflow/sdk/options/PipelineOptions.java   |  3 +-
 .../sdk/options/PipelineOptionsFactory.java | 41 ++---
 .../PipelineOptionsFactoryJava8Test.java| 90 
 3 files changed, 122 insertions(+), 12 deletions(-)
--

Jenkins build is back to stable : beam_MavenVerify » Spark Beam Runner #23

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request: Filter Synthetic Methods in PipelineO...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/41


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-118) Add DisplayData to SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199983#comment-15199983
 ] 

ASF GitHub Bot commented on BEAM-118:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/56

[BEAM-118] Add DipslayData primitives to SDK

This introduces the new SDK primitives necessary for the Static Display 
Metadata feature.

Context:
* https://issues.apache.org/jira/browse/BEAM-117
* 
https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam displaydata-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56


commit de1bce41d366cc97a4b4bdf569e597d3836e123c
Author: Scott Wegner 
Date:   2016-03-17T17:22:42Z

Add DipslayData primitives to SDK




> Add DisplayData to SDK
> --
>
> Key: BEAM-118
> URL: https://issues.apache.org/jira/browse/BEAM-118
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build became unstable: beam_MavenVerify #14

[Amit Sela]

Looks like the cached MapR Hadoop dependencies issue is back..
I didn't push anything lately, and the only commit was unrelated.

On Thu, Mar 17, 2016 at 4:54 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
>

[GitHub] incubator-beam pull request: [BEAM-116] change runners artifactId ...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/59

[BEAM-116] change runners artifactId to runners-parent



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-116

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit 447c8af23b3fa727bba5aa97093699745d6e6f5b
Author: Maximilian Michels 
Date:   2016-03-16T11:03:07Z

[BEAM-116] change runners artifactId to runners-parent




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-96] Add composed `CombineFn` builders in `CombineFns`

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ac63fd6d4 -> c30326007


[BEAM-96] Add composed `CombineFn` builders in `CombineFns`

* `compose()` or `composeKeyed()` are used to start composition
* `with()` is used to add an input-transformation, a `CombineFn`
  and an output `TupleTag`.
* A non-`CombineFn` initial builder is used to ensure that every
  composition includes at least one item
* Duplicate output tags are not allowed in the same composition


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/23b43780
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/23b43780
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/23b43780

Branch: refs/heads/master
Commit: 23b437802546f32a167b38f8d0bc7a566abde224
Parents: ac63fd6
Author: Pei He 
Authored: Fri Mar 4 13:54:34 2016 -0800
Committer: bchambers 
Committed: Thu Mar 17 13:54:40 2016 -0700

--
 .../dataflow/sdk/transforms/CombineFns.java | 1100 ++
 .../dataflow/sdk/transforms/CombineFnsTest.java |  413 +++
 2 files changed, 1513 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/23b43780/sdk/src/main/java/com/google/cloud/dataflow/sdk/transforms/CombineFns.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/transforms/CombineFns.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/transforms/CombineFns.java
new file mode 100644
index 000..656c010
--- /dev/null
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/transforms/CombineFns.java
@@ -0,0 +1,1100 @@
+/*
+ * Copyright (C) 2016 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+package com.google.cloud.dataflow.sdk.transforms;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.cloud.dataflow.sdk.coders.CannotProvideCoderException;
+import com.google.cloud.dataflow.sdk.coders.Coder;
+import com.google.cloud.dataflow.sdk.coders.CoderException;
+import com.google.cloud.dataflow.sdk.coders.CoderRegistry;
+import com.google.cloud.dataflow.sdk.coders.StandardCoder;
+import com.google.cloud.dataflow.sdk.transforms.Combine.CombineFn;
+import com.google.cloud.dataflow.sdk.transforms.Combine.KeyedCombineFn;
+import com.google.cloud.dataflow.sdk.transforms.CombineFnBase.GlobalCombineFn;
+import com.google.cloud.dataflow.sdk.transforms.CombineFnBase.PerKeyCombineFn;
+import 
com.google.cloud.dataflow.sdk.transforms.CombineWithContext.CombineFnWithContext;
+import com.google.cloud.dataflow.sdk.transforms.CombineWithContext.Context;
+import 
com.google.cloud.dataflow.sdk.transforms.CombineWithContext.KeyedCombineFnWithContext;
+import com.google.cloud.dataflow.sdk.util.PropertyNames;
+import com.google.cloud.dataflow.sdk.values.TupleTag;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+/**
+ * Static utility methods that create combine function instances.
+ */
+public class CombineFns {
+
+  /**
+   * Returns a {@link ComposeKeyedCombineFnBuilder} to construct a composed
+   * {@link PerKeyCombineFn}.
+   *
+   * The same {@link TupleTag} cannot be used in a composition multiple 
times.
+   *
+   * Example:
+   * { @code
+   * PCollection> latencies = ...;
+   *
+   * TupleTag maxLatencyTag = new TupleTag();
+   * TupleTag meanLatencyTag = new TupleTag();
+   *
+   * SimpleFunction identityFn =
+   * new SimpleFunction() {
+   *   @Override
+   *   public Integer apply(Integer input) {
+   *   return input;
+   *   }};
+   * PCollection> maxAndMean = latencies.apply(
+   * Combine.perKey(
+   * CombineFns.composeKeyed()
+   *

[jira] [Updated] (BEAM-133) Test flakiness in the Spark runner

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-133:
--
Assignee: Davor Bonaci  (was: Amit Sela)

> Test flakiness in the Spark runner
> --
>
> Key: BEAM-133
> URL: https://issues.apache.org/jira/browse/BEAM-133
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Davor Bonaci
>Assignee: Davor Bonaci
>
> Jenkins shows some flakiness in the Spark runner in the context of an 
> unrelated pre-commit test.
> {code}
> Results :
> Tests in error: 
>   AvroPipelineTest.testGeneric:75 » Runtime java.io.IOException: Could 
> not creat...
>   NumShardsTest.testText:77 » Runtime java.io.IOException: Could not 
> create File...
>   HadoopFileFormatPipelineTest.testSequenceFile:83 » Runtime 
> java.io.IOException...
>   
> TransformTranslatorTest.testTextIOReadAndWriteTransforms:76->runPipeline:96 » 
> Runtime
>   KafkaStreamingTest.testRun:121 » Runtime java.io.IOException: failure 
> to login...
> Tests run: 27, Failures: 0, Errors: 5, Skipped: 0
> [ERROR] There are test failures.
> {code}
> https://builds.apache.org/job/beam_PreCommit/98/console
> Amit, does this sounds like a test code issue or the infrastructure issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-104) Capability matrix for Beam website

[Tyler Akidau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Akidau resolved BEAM-104.
---
Resolution: Fixed

Pull requests:
https://github.com/apache/incubator-beam-site/pull/2
https://github.com/apache/incubator-beam-site/pull/3

> Capability matrix for Beam website
> --
>
> Key: BEAM-104
> URL: https://issues.apache.org/jira/browse/BEAM-104
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Tyler Akidau
>Assignee: Frances Perry
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Given the variety of runners we're going to support, we'd like to have a 
> clear and relatively comprehensive capability matrix published on the website 
> and maintained over time. We're currently planning on having multiple levels 
> of granularity, from simple yes/no to detailed paragraph-level explanations, 
> to allow for various display options.
> Our proposal is:
> 1. Have runner implementors fill out an initial set of capabilities in this 
> spreadsheet Frances cooked up: 
> https://docs.google.com/spreadsheets/d/1OM077lZBARrtUi6g0X0O0PHaIbFKCD6v0djRefQRE1I/edit#gid=939018601
> If there are additional capabilities you'd like to propose, please do so.
> 2. Once filled out, I will move those data into YAML form in the website 
> source and provide a basic rendering of it on the website. We'll also have an 
> initial blog post announcing it. The YAML will become the source of truth 
> that gets maintained over time, with version control thanks to being in the 
> repo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build became unstable: beam_MavenVerify #14

[Andreas Veithen]

It looks like nowadays, when enabling "Use private Maven repository", you
also need to select a strategy. If you want to shield the build against
pollution of the local Maven repository by other builds (which I highly
recommend, speaking from experience), you should use "Local to the
workspace". That's currently not the case for that build: it's set to
"Default (~/.m2/respository)", which doesn't sound very private...

Andreas

On Thu, Mar 17, 2016 at 2:57 PM, Amit Sela  wrote:

> Looks like the cached MapR Hadoop dependencies issue is back..
> I didn't push anything lately, and the only commit was unrelated.
>
> On Thu, Mar 17, 2016 at 4:54 PM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
> > See 
> >
> >
>

[GitHub] incubator-beam pull request: Minor Javadoc fixes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/49


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: [Beam 109] Combine.PerKey ignores gro...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/63

[Beam 109] Combine.PerKey ignores grouping also by windows



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-109

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/63.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #63


commit 2c090ede5bbede511a378ca64d3d280ad26f425a
Author: Sela 
Date:   2016-03-19T10:25:02Z

[BEAM-109] Better testing for FixedWindows and SlidingWindows

commit 442a14944216792a7cd1884c43aed5d147ed1fd5
Author: Sela 
Date:   2016-03-19T20:40:48Z

[BEAM-109] lower counts is unordered so better to compare entire result and 
not just iterator head

commit 0c2442100622df6ff16036caa7700780413d3f50
Author: Sela 
Date:   2016-03-19T20:41:31Z

[BEAM-109] fix support for FixedWindows and SlidingWindows in batch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam-site git commit: Capability matrix page + blog post: - Content as discussed in this thread: http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/%3CCAB8MnHW6mE3

[takidau]

Capability matrix page + blog post:
- Content as discussed in this thread:

http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/%3CCAB8MnHW6mE3GXvemDStCvn_1zMxqXj0ZWLJBgO9hNuHed9ue%2Bw%40mail.gmail.com%3E
  and as iterated upon by relevant committers in this doc:

https://docs.google.com/spreadsheets/d/1OM077lZBARrtUi6g0X0O0PHaIbFKCD6v0djRefQRE1I/edit
- Enumerates current capabilities per-runner in _data/capability-matrix.yml.
  This file should be kept up to date over time as runners evolve, new
  runners are added, etc.
- Creates new page with live capability matrix, which will be updated over
  time via changes to the aforementioned YAML file.
- Create new blog post with summary snapshot of the current matrix.
- Update authors support in blog post templates to handle multiple authors.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/931d7f51
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/931d7f51
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/931d7f51

Branch: refs/heads/asf-site
Commit: 931d7f51002341201bb3534db81da46e54c97716
Parents: a8ebbad
Author: Tyler Akidau 
Authored: Thu Mar 17 14:40:19 2016 -0700
Committer: Tyler Akidau 
Committed: Thu Mar 17 14:52:09 2016 -0700

--
 _data/authors.yml   |   8 +
 _data/capability-matrix.yml | 561 +
 _includes/authors-list.md   |   1 +
 _includes/capability-matrix-common.md   |   7 +
 _includes/capability-matrix-row-blog.md |   1 +
 _includes/capability-matrix-row-full.md |   1 +
 _includes/capability-matrix-row-summary.md  |   1 +
 _includes/capability-matrix.md  |  28 +
 _includes/header.html   |  13 +-
 _layouts/post.html  |   4 +-
 _pages/blog.md  |   5 +-
 _pages/capability-matrix.md |  41 +
 _posts/2016-02-22-beam-has-a-logo.markdown  |   3 +-
 _posts/2016-02-22-beam-has-a-logo0.markdown |   3 +-
 _posts/2016-03-17-compatability-matrix.md   | 596 ++
 _sass/capability-matrix.scss| 127 +++
 .../python/sdk/2016/02/25/beam-has-a-logo0.html |   8 +-
 .../website/2016/02/22/beam-has-a-logo.html |  10 +-
 content/blog/index.html | 795 +-
 content/feed.xml| 807 ++-
 content/getting_started/index.html  |   1 +
 content/index.html  |   3 +
 content/issue_tracking/index.html   |   1 +
 content/mailing_lists/index.html|   1 +
 content/privacy_policy/index.html   |   1 +
 content/source_repository/index.html|  10 +-
 content/styles/site.css | 107 +++
 content/team/index.html |   1 +
 styles/site.scss|   1 +
 29 files changed, 3118 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/931d7f51/_data/authors.yml
--
diff --git a/_data/authors.yml b/_data/authors.yml
index f76b24f..b71ad8a 100644
--- a/_data/authors.yml
+++ b/_data/authors.yml
@@ -1,4 +1,12 @@
+fjp:
+name: Frances Perry
+email: fran...@apache.org
+twitter: francesjperry
 jamesmalone:
 name: James Malone
 email: jamesmal...@apache.org
 twitter: chimerasaurus
+takidau:
+name: Tyler Akidau
+email: taki...@apache.org
+twitter: takidau

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/931d7f51/_data/capability-matrix.yml
--
diff --git a/_data/capability-matrix.yml b/_data/capability-matrix.yml
new file mode 100644
index 000..785854a
--- /dev/null
+++ b/_data/capability-matrix.yml
@@ -0,0 +1,561 @@
+columns:
+  - class: model
+name: Beam Model
+  - class: dataflow
+name: Google Cloud Dataflow
+  - class: flink
+name: Apache Flink
+  - class: spark
+name: Apache Spark
+
+categories:
+  - description: What is being computed?
+anchor: what
+color-b: 'ca1'
+color-y: 'ec3'
+color-p: 'fe5'
+color-n: 'ddd'
+rows:
+  - name: ParDo
+values:
+  - class: model
+l1: 'Yes'
+l2: element-wise processing
+l3: Element-wise transformation parameterized by a chunk of user 
code. Elements are processed in bundles, with initialization and termination 
hooks. Bundle size is chosen by the runner and cannot be controlled by user 
code. 

Jenkins build became unstable: beam_MavenVerify » Spark Beam Runner #22

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_MavenVerify #15

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-116) Change runners artifactId to runners-parent

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197806#comment-15197806
 ] 

Kenneth Knowles commented on BEAM-116:
--

+1. I reverted it to "runners" but I am convinced by what you say here. Do you 
perhaps know a way to share maven configuration other than parent 
relationships, so there is no artifact implied?

> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>Reporter: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-136) PipelineOptionsFactory should detect all runners

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-136:


 Summary: PipelineOptionsFactory should detect all runners
 Key: BEAM-136
 URL: https://issues.apache.org/jira/browse/BEAM-136
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Daniel Halperin
Priority: Minor


Even with flink-runner jar on classpath, we didn't find it.

See linked thread here: 
http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/%3CCAA8k_FLekCURJKYO8qab%3DR%3DG-sRrTmYc7_NQXcrY1kHyATAtBw%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-126) KafkaWindowedWordCountExample fails with JobName invalid

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels closed BEAM-126.
---
Resolution: Fixed

Thank you for reporting! Fixed with 81d5ff5a561ebcf323caea5bdc4363353e5e60dd.

> KafkaWindowedWordCountExample fails with JobName invalid
> 
>
> Key: BEAM-126
> URL: https://issues.apache.org/jira/browse/BEAM-126
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: William McCarthy
>Assignee: Maximilian Michels
>
> I get the following when I try to run the KafkaWindowedWordCountExample.
> I'm able to fix it by changing line 106 of that file to:
> options.setJobName("kafkawindowword" + options.getWindowSize() + 
> "seconds");
> flink run -c 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample
>  target/beam-1.0-SNAPSHOT.jar 
> test cl-mdgy:2181 cl-pu4p:9092,cl-y06o:9093 mygroup
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
>   at 
> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> Caused by: java.lang.RuntimeException: Failed to construct instance from 
> factory method FlinkPipelineRunner#fromOptions(interface 
> com.google.cloud.dataflow.sdk.options.PipelineOptions)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>   at 
> com.google.cloud.dataflow.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:57)
>   at com.google.cloud.dataflow.sdk.Pipeline.create(Pipeline.java:134)
>   at 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample.main(KafkaWindowedWordCountExample.java:114)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
>   ... 6 more
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>   ... 15 more
> Caused by: java.lang.IllegalArgumentException: JobName invalid; the name must 
> consist of only the characters [-a-z0-9], starting with a letter and ending 
> with a letter or number
>   at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineRunner.fromOptions(FlinkPipelineRunner.java:92)
>   ... 20 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-116) Change runners artifactId to runners-parent

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201804#comment-15201804
 ] 

Jean-Baptiste Onofré commented on BEAM-116:
---

FYI, the PR that I did about the "renaming" already change the parent pom there 
(afair).

> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam-site git commit: This closes #3

[davor]

This closes #3


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/2f7336f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/2f7336f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/2f7336f0

Branch: refs/heads/asf-site
Commit: 2f7336f0e93255ff83e100a99fd433df9752c12b
Parents: b04b002 04e1fbb
Author: Davor Bonaci 
Authored: Thu Mar 17 16:39:07 2016 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 17 16:39:07 2016 -0700

--
 _posts/2016-03-17-capability-matrix.md  |  596 +++
 _posts/2016-03-17-compatability-matrix.md   |  596 ---
 .../2016/03/17/capability-matrix.html   |  896 ++
 content/blog/index.html |2 +-
 content/capability-matrix/index.html| 1652 ++
 content/feed.xml|   10 +-
 content/index.html  |2 +-
 7 files changed, 3151 insertions(+), 603 deletions(-)
--

[jira] [Closed] (BEAM-11) Integrate Spark runner with Beam

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-11?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-11.
-
Resolution: Fixed

Closed by PR-42

> Integrate Spark runner with Beam
> 
>
> Key: BEAM-11
> URL: https://issues.apache.org/jira/browse/BEAM-11
> Project: Beam
>  Issue Type: Task
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Refactor and integrate the Spark runner code against Google's contributed 
> version of Dataflow - Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-120) Integrate DisplayData into DirectPipelineRunner

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-120:
-

 Summary: Integrate DisplayData into DirectPipelineRunner
 Key: BEAM-120
 URL: https://issues.apache.org/jira/browse/BEAM-120
 Project: Beam
  Issue Type: Sub-task
Reporter: Scott Wegner






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [flink] improvements to the Kafka Example

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0f137169e -> ef1e32dee


[flink] improvements to the Kafka Example

- use timestamp extractor after ingestion
- fix coder runtime exception
- correct logging


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ef1e32de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ef1e32de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ef1e32de

Branch: refs/heads/master
Commit: ef1e32deefb9886584556c7125e87b2873c63ebf
Parents: 0f13716
Author: Maximilian Michels 
Authored: Thu Mar 17 14:49:09 2016 +0100
Committer: Maximilian Michels 
Committed: Thu Mar 17 14:53:41 2016 +0100

--
 .../examples/streaming/KafkaWindowedWordCountExample.java | 2 +-
 .../flink/translation/FlinkStreamingTransformTranslators.java | 3 ++-
 .../wrappers/streaming/io/UnboundedFlinkSource.java   | 7 +++
 runners/flink/runner/src/main/resources/log4j.properties  | 2 +-
 4 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ef1e32de/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
index 3942d0d..8fca1d4 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
@@ -103,7 +103,7 @@ public class KafkaWindowedWordCountExample {
   public static void main(String[] args) {
 PipelineOptionsFactory.register(KafkaStreamingWordCountOptions.class);
 KafkaStreamingWordCountOptions options = 
PipelineOptionsFactory.fromArgs(args).as(KafkaStreamingWordCountOptions.class);
-options.setJobName("KafkaExample");
+options.setJobName("KafkaExample - WindowSize: " + options.getWindowSize() 
+ " seconds");
 options.setStreaming(true);
 options.setCheckpointingInterval(1000L);
 options.setNumberOfExecutionRetries(5);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ef1e32de/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index bdefeaf..2b9b0ee 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
@@ -44,6 +44,7 @@ import org.apache.flink.api.common.functions.FlatMapFunction;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.core.fs.FileSystem;
 import org.apache.flink.streaming.api.datastream.*;
+import org.apache.flink.streaming.api.functions.IngestionTimeExtractor;
 import org.apache.flink.util.Collector;
 import org.joda.time.Instant;
 import org.slf4j.Logger;
@@ -179,7 +180,7 @@ public class FlinkStreamingTransformTranslators {
   public void flatMap(String s, Collector 
collector) throws Exception {
 collector.collect(WindowedValue.of(s, Instant.now(), 
GlobalWindow.INSTANCE, PaneInfo.NO_FIRING));
   }
-});
+}).assignTimestampsAndWatermarks(new IngestionTimeExtractor());
   } else {
 source = context.getExecutionEnvironment()
 .addSource(new 
UnboundedSourceWrapper<>(context.getPipelineOptions(), transform));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ef1e32de/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
index 05a8956..82984cb 100644
--- 

[1/2] incubator-beam git commit: Close Readers in InProcess Read Evaluators

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d39346823 -> 5b5c0e28f


Close Readers in InProcess Read Evaluators

The readers were formerly left open, which prevents release of any
resources that should be released.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fad6da89
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fad6da89
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fad6da89

Branch: refs/heads/master
Commit: fad6da89079791952a937aed257f0d2db1467053
Parents: d393468
Author: Thomas Groh 
Authored: Tue Mar 15 11:50:38 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Mar 17 21:01:18 2016 -0700

--
 .../inprocess/BoundedReadEvaluatorFactory.java  |  49 --
 .../UnboundedReadEvaluatorFactory.java  |  53 +++---
 .../BoundedReadEvaluatorFactoryTest.java| 136 ++-
 .../UnboundedReadEvaluatorFactoryTest.java  | 168 +++
 4 files changed, 366 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fad6da89/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
index 2a164c3..eaea3ed 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
@@ -15,6 +15,8 @@
  */
 package com.google.cloud.dataflow.sdk.runners.inprocess;
 
+import com.google.cloud.dataflow.sdk.io.BoundedSource;
+import com.google.cloud.dataflow.sdk.io.BoundedSource.BoundedReader;
 import com.google.cloud.dataflow.sdk.io.Read.Bounded;
 import com.google.cloud.dataflow.sdk.io.Source.Reader;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
@@ -78,8 +80,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
   @SuppressWarnings("unchecked")
   private  Queue 
getTransformEvaluatorQueue(
   final AppliedPTransform, Bounded> 
transform,
-  final InProcessEvaluationContext evaluationContext)
-  throws IOException {
+  final InProcessEvaluationContext evaluationContext) {
 // Key by the application and the context the evaluation is occurring in 
(which call to
 // Pipeline#run).
 EvaluatorKey key = new EvaluatorKey(transform, evaluationContext);
@@ -101,21 +102,25 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 return evaluatorQueue;
   }
 
+  /**
+   * A {@link BoundedReadEvaluator} produces elements from an underlying 
{@link BoundedSource},
+   * discarding all input elements. Within the call to {@link 
#finishBundle()}, the evaluator
+   * creates the {@link BoundedReader} and consumes all available input.
+   *
+   * A {@link BoundedReadEvaluator} should only be created once per {@link 
BoundedSource}, and
+   * each evaluator should only be called once per evaluation of the pipeline. 
Otherwise, the source
+   * may produce duplicate elements.
+   */
   private static class BoundedReadEvaluator implements 
TransformEvaluator {
 private final AppliedPTransform, Bounded> 
transform;
 private final InProcessEvaluationContext evaluationContext;
-private final Reader reader;
 private boolean contentsRemaining;
 
 public BoundedReadEvaluator(
 AppliedPTransform, Bounded> transform,
-InProcessEvaluationContext evaluationContext)
-throws IOException {
+InProcessEvaluationContext evaluationContext) {
   this.transform = transform;
   this.evaluationContext = evaluationContext;
-  reader =
-  
transform.getTransform().getSource().createReader(evaluationContext.getPipelineOptions());
-  contentsRemaining = reader.start();
 }
 
 @Override
@@ -123,17 +128,25 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 
 @Override
 public InProcessTransformResult finishBundle() throws IOException {
-  UncommittedBundle output = 
evaluationContext.createRootBundle(transform.getOutput());
-  while (contentsRemaining) {
-output.add(
-WindowedValue.timestampedValueInGlobalWindow(
-reader.getCurrent(), reader.getCurrentTimestamp()));
-contentsRemaining = reader.advance();
+  try (final Reader reader =
+  transform
+  

[jira] [Created] (BEAM-135) Utilities for "batching" elements in a DoFn

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-135:
-

 Summary: Utilities for "batching" elements in a DoFn
 Key: BEAM-135
 URL: https://issues.apache.org/jira/browse/BEAM-135
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Ben Chambers


We regularly receive questions about how to write a {{DoFn}} that operates on 
batches of elements. Example answers include:

http://stackoverflow.com/questions/35065109/can-datastore-input-in-google-dataflow-pipeline-be-processed-in-a-batch-of-n-ent/35068341#35068341

http://stackoverflow.com/questions/30177812/partition-data-coming-from-csv-so-i-can-process-larger-patches-rather-then-indiv/30178170#30178170

Possible APIs could be to wrap a {{DoFn}} and include a batch size, or to 
create a utility like {{Filter}}, {{Partition}}, etc. that takes a 
{{SerializableFunction}} or a {{SimpleFunction}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-127) Support for Unbounded sinks in Streaming

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202872#comment-15202872
 ] 

Maximilian Michels commented on BEAM-127:
-

+1 Adding a simple wrapper should be fairly straight forward. How about using 
the {{Write.Bound}} interface? In addition to Flink sources, we could also wrap 
custom sources through that interface.

> Support for Unbounded sinks in Streaming
> 
>
> Key: BEAM-127
> URL: https://issues.apache.org/jira/browse/BEAM-127
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>
> Currently the Flink runner for streaming does not support any sinks apart 
> from writing on the local filesystems of the task managers. This has to 
> change. A possible way is to add support for UnboundedFlinkSinks that wrap 
> around the currently supported sinks in Flink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-83] Move Java 8 tests to their ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: Fix AfterWatermark Early and Late jav...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/54


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-119) Integrate DisplayData into DataflowPipelineRunner

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-119:
-

 Summary: Integrate DisplayData into DataflowPipelineRunner
 Key: BEAM-119
 URL: https://issues.apache.org/jira/browse/BEAM-119
 Project: Beam
  Issue Type: Sub-task
Reporter: Scott Wegner






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-133) Test flakiness in the Spark runner

[Davor Bonaci (JIRA)]

Davor Bonaci created BEAM-133:
-

 Summary: Test flakiness in the Spark runner
 Key: BEAM-133
 URL: https://issues.apache.org/jira/browse/BEAM-133
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Davor Bonaci
Assignee: Amit Sela


Jenkins shows some flakiness in the Spark runner in the context of an unrelated 
pre-commit test.

{code}
Results :

Tests in error: 
  AvroPipelineTest.testGeneric:75 » Runtime java.io.IOException: Could not 
creat...
  NumShardsTest.testText:77 » Runtime java.io.IOException: Could not create 
File...
  HadoopFileFormatPipelineTest.testSequenceFile:83 » Runtime 
java.io.IOException...
  
TransformTranslatorTest.testTextIOReadAndWriteTransforms:76->runPipeline:96 » 
Runtime
  KafkaStreamingTest.testRun:121 » Runtime java.io.IOException: failure to 
login...

Tests run: 27, Failures: 0, Errors: 5, Skipped: 0

[ERROR] There are test failures.
{code}

https://builds.apache.org/job/beam_PreCommit/98/console

Amit, does this sounds like a test code issue or the infrastructure issue?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: Improve the reliability of `mvn verif...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/57


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-123) Skip header row from csv

[Davin Pidoto (JIRA)]

Davin Pidoto created BEAM-123:
-

 Summary: Skip header row from csv 
 Key: BEAM-123
 URL: https://issues.apache.org/jira/browse/BEAM-123
 Project: Beam
  Issue Type: New Feature
Reporter: Davin Pidoto
Priority: Minor


Add functionality to skip header rows when reading from a csv file.

http://stackoverflow.com/questions/28450554/skipping-header-rows-is-it-possible-with-cloud-dataflow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [flink] fix UnboundedFlinkSource wrapper

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master a9c46057e -> 0f137169e


[flink] fix UnboundedFlinkSource wrapper

- remove unnecessary PipelineOptions cache
- use the correct interface types
- improve Kafka example


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0f137169
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0f137169
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0f137169

Branch: refs/heads/master
Commit: 0f137169e4c2cd8d3e5a86c91bc2f401d276e8ed
Parents: a9c4605
Author: Maximilian Michels 
Authored: Thu Mar 17 12:26:03 2016 +0100
Committer: Maximilian Michels 
Committed: Thu Mar 17 12:27:37 2016 +0100

--
 .../KafkaWindowedWordCountExample.java  |  7 ++--
 .../streaming/io/UnboundedFlinkSource.java  | 37 ++--
 2 files changed, 22 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0f137169/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
index 55cdc22..3942d0d 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/KafkaWindowedWordCountExample.java
@@ -22,7 +22,6 @@ import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.Unbounded
 import com.google.cloud.dataflow.sdk.Pipeline;
 import com.google.cloud.dataflow.sdk.io.Read;
 import com.google.cloud.dataflow.sdk.io.TextIO;
-import com.google.cloud.dataflow.sdk.io.UnboundedSource;
 import com.google.cloud.dataflow.sdk.options.Default;
 import com.google.cloud.dataflow.sdk.options.Description;
 import com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory;
@@ -30,7 +29,7 @@ import com.google.cloud.dataflow.sdk.transforms.*;
 import com.google.cloud.dataflow.sdk.transforms.windowing.*;
 import com.google.cloud.dataflow.sdk.values.KV;
 import com.google.cloud.dataflow.sdk.values.PCollection;
-import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082;
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08;
 import org.apache.flink.streaming.util.serialization.SimpleStringSchema;
 import org.joda.time.Duration;
 
@@ -121,12 +120,12 @@ public class KafkaWindowedWordCountExample {
 
 // this is the Flink consumer that reads the input to
 // the program from a kafka topic.
-FlinkKafkaConsumer082 kafkaConsumer = new FlinkKafkaConsumer082<>(
+FlinkKafkaConsumer08 kafkaConsumer = new FlinkKafkaConsumer08<>(
 options.getKafkaTopic(),
 new SimpleStringSchema(), p);
 
 PCollection words = pipeline
-.apply(Read.from(new UnboundedFlinkSource(options, 
kafkaConsumer)).named("StreamingWordCount"))
+
.apply(Read.named("StreamingWordCount").from(UnboundedFlinkSource.of(kafkaConsumer)))
 .apply(ParDo.of(new ExtractWordsFn()))
 
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(options.getWindowSize(
 
.triggering(AfterWatermark.pastEndOfWindow()).withAllowedLateness(Duration.ZERO)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0f137169/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
index 2857efd..05a8956 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedFlinkSource.java
@@ -22,7 +22,9 @@ import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.io.UnboundedSource;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.common.base.Preconditions;
+import org.apache.flink.streaming.api.functions.source.ParallelSourceFunction;
 import 

[jira] [Created] (BEAM-125) PubSubIO.Read should be implemented by an UnboundedSource

[Mark Shields (JIRA)]

Mark Shields created BEAM-125:
-

 Summary: PubSubIO.Read should be implemented by an UnboundedSource
 Key: BEAM-125
 URL: https://issues.apache.org/jira/browse/BEAM-125
 Project: Beam
  Issue Type: Bug
Reporter: Mark Shields


So any runner can stream messages from pub/sub, we need an UnboundendSource 
implementation of PubSubIO.Read.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-113) Update Spark runner README

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-113.
--
Resolution: Fixed

Closed PR-55

> Update Spark runner README
> --
>
> Key: BEAM-113
> URL: https://issues.apache.org/jira/browse/BEAM-113
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> The runner is now a part of Beam and the runner's README should be updated 
> accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-117) Implement the API for Static Display Metadata

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers updated BEAM-117:
--
Issue Type: New Feature  (was: Bug)

> Implement the API for Static Display Metadata
> -
>
> Key: BEAM-117
> URL: https://issues.apache.org/jira/browse/BEAM-117
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Scott Wegner
>
> As described in the following doc, we would like the SDK to allow associating 
> display metadata with PTransforms.
> https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[7/8] incubator-beam git commit: Fix typo in javadoc

[kenn]

Fix typo in javadoc


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a2a2c7b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a2a2c7b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a2a2c7b9

Branch: refs/heads/master
Commit: a2a2c7b9026779424abded31af3888e54e28ab8f
Parents: e607237
Author: Kenneth Knowles 
Authored: Thu Mar 17 13:25:38 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Mar 17 13:25:38 2016 -0700

--
 .../google/cloud/dataflow/examples/complete/game/GameStats.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a2a2c7b9/java8examples/src/main/java/com/google/cloud/dataflow/examples/complete/game/GameStats.java
--
diff --git 
a/java8examples/src/main/java/com/google/cloud/dataflow/examples/complete/game/GameStats.java
 
b/java8examples/src/main/java/com/google/cloud/dataflow/examples/complete/game/GameStats.java
index bc79180..7c67d10 100644
--- 
a/java8examples/src/main/java/com/google/cloud/dataflow/examples/complete/game/GameStats.java
+++ 
b/java8examples/src/main/java/com/google/cloud/dataflow/examples/complete/game/GameStats.java
@@ -75,8 +75,8 @@ import java.util.TimeZone;
  * subsequent fixed windowing. (This could be used to tell us what games are 
giving us greater
  * user retention).
  *
- *  Run {@literal 
com.google.cloud.dataflow.examples.complete.game.injector.Injector} to generate
- * pubsub data for this pipeline. The {@literal Injector} documentation 
provides more detail.
+ *  Run {@code 
com.google.cloud.dataflow.examples.complete.game.injector.Injector} to generate
+ * pubsub data for this pipeline. The {@code Injector} documentation provides 
more detail.
  *
  *  To execute this pipeline using the Dataflow service, specify the 
pipeline configuration
  * like this:

[GitHub] incubator-beam pull request: [BEAM-96] Add composed combine functi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/23


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-53) PubSubIO: reimplement in Java

[Mark Shields (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201823#comment-15201823
 ] 

Mark Shields commented on BEAM-53:
--

Working on this now.
First goal is to support custom timestamps, 'reasonable' watermark tracking and 
good perf.
Other 'while were there' items mentioned by Dan will come later.

> PubSubIO: reimplement in Java
> -
>
> Key: BEAM-53
> URL: https://issues.apache.org/jira/browse/BEAM-53
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Mark Shields
>Priority: Minor
>
> PubSubIO is currently only partially implemented in Java: the 
> DirectPipelineRunner uses a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path 
> implemented in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in 
> a scalable way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than 
> from a message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-36) TimestampedValueInMultipleWindows should use a more compact set representation

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers closed BEAM-36.

Resolution: Won't Fix

As per discussion on https://github.com/apache/incubator-beam/pull/43:

"We looked some more at the original Jira issue and realized that it is likely 
a non-issue. It was created to track the fact we needed to examine our usage of 
a HashSet there, since we ran into problems with the over-allocation of a hash 
set (eg., 64 slots to hold 23 items, etc.). When we have 1000 of these in 
memory at a time, the over-allocation starts to hurt.

Upon further scrutiny, those WindowedValues should only be getting turned into 
a Set when we need to do equals or hashCode, to make sure we get an 
order-independent comparison. Assuming this is limited to tests, we can 
probably resolve the Jira issue as won't fix."

> TimestampedValueInMultipleWindows should use a more compact set representation
> --
>
> Key: BEAM-36
> URL: https://issues.apache.org/jira/browse/BEAM-36
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Trivial
>  Labels: Windowing
>
> Today TimestampedValueInMultipleWindows converts its collection of windows to 
> a LinkedHashSet for comparisons and hashing. Since it is an immutable set, 
> more compact representations are available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200967#comment-15200967
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/52


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam-site git commit: Capability matrix page + blog post: - Content as discussed in this thread: http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/%3CCAB8MnHW6mE3

[takidau]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site a8ebbad3e -> b04b002ca


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/931d7f51/content/blog/index.html
--
diff --git a/content/blog/index.html b/content/blog/index.html
index 9513347..66fa6fe 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -45,6 +45,7 @@
   Documentation 
   
 Getting Started
+   Capability Matrix
 https://goo.gl/ps8twC;>Technical Docs
 https://goo.gl/nk5OM0;>Technical Vision
   
@@ -86,8 +87,795 @@
 This is the blog for the Apache Beam project. This blog contains news 
and updates
 for the project.
 
+Clarifying 
 Formalizing Runner Capabilities
+Mar 17, 2016 •  Frances Perry [https://twitter.com/francesjperry;>@francesjperry]  Tyler Akidau 
[https://twitter.com/takidau;>@takidau]
+
+
+With initial code drops complete (https://github.com/apache/incubator-beam/pull/1;>Dataflow SDK and 
Runner, https://github.com/apache/incubator-beam/pull/12;>Flink 
Runner, https://github.com/apache/incubator-beam/pull/42;>Spark 
Runner) and expressed interest in runner implementations for https://issues.apache.org/jira/browse/BEAM-9;>Storm, https://issues.apache.org/jira/browse/BEAM-19;>Hadoop, and https://issues.apache.org/jira/browse/BEAM-79;>Gearpump (amongst 
others), we wanted to start addressing a big question in the Apache Beam 
(incubating) community: what capabilities will each runner be able to 
support?
+
+While we’d love to have a world where all runners support the full suite 
of semantics included in the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf;>Dataflow Model), 
practically speaking, there will always be certain features that some runners 
can’t provide. For example, a Hadoop-based runner would be inherently 
batch-based and may be unable to (easily) implement support for unbounded 
collections. However, that doesn’t prevent it from being extremely useful for 
a large set of uses. In other cases, the implementations provided by one runner 
may have slightly different semantics that those provided by another (e.g. even 
though the current suite of runners all support exactly-once delivery 
guarantees, an http://samza.apache.org/;>Apache Samza runner, 
which would be a welcome addition, would currently only support 
at-least-once).
+
+To help clarify things, we’ve been working on enumerating the key 
features of the Beam model in a capability 
matrix for all existing runners, categorized around the four key questions 
addressed by the model: What / Where / When 
/ How (if you’re not familiar with those 
questions, you might want to read through http://oreilly.com/ideas/the-world-beyond-batch-streaming-102;>Streaming 
102 for an overview). This table will be maintained over time as the model 
evolves, our understanding grows, and runners are created or features added.
+
+Included below is a summary snapshot of our current understanding of the 
capabilities of the existing runners (see the live version for full details, descriptions, and 
Jira links); since integration is still under way, the system as whole isn’t 
yet in a completely stable, usable state. But that should be changing in the 
near future, and we’ll be updating loud and clear on this blog when the first 
supported Beam 1.0 release happens.
+
+In the meantime, these tables should help clarify where we expect to be in 
the very near term, and help guide expectations about what existing runners are 
capable of, and what features runner implementers will be tackling next.
+
+
+  function ToggleTables(showDetails, anchor) {
+document.getElementById("cap-summary").style.display = showDetails ? 
"none" : "block";
+document.getElementById("cap-full").style.display = showDetails ? "block" 
: "none";
+location.hash = anchor;
+  }
+
+
+
+
+
+
+  
+  
+What is being computed?
+  
+  
+
+  
+Beam Model
+  
+Google Cloud Dataflow
+  
+Apache Flink
+  
+Apache Spark
+  
+  
+  
+  
+ParDo
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+  
+  
+GroupByKey
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+~
+
+
+  
+  
+  
+Flatten
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+  
+  
+Combine
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+  
+  
+Composite Transforms
+
+
+
+
+
+
+
+
+~
+
+
+
+
+~
+
+
+
+
+~
+
+
+  
+  
+  
+Side Inputs
+
+
+
+
+
+
+
+
+
+
+
+
+
+~
+
+
+
+
+~
+
+
+  
+  
+  
+Source API
+
+
+
+
+
+
+
+
+
+
+
+
+
+~
+
+
+

[jira] [Updated] (BEAM-53) PubSubIO: reimplement in Java

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-53:

Assignee: Mark Shields

> PubSubIO: reimplement in Java
> -
>
> Key: BEAM-53
> URL: https://issues.apache.org/jira/browse/BEAM-53
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Mark Shields
>Priority: Minor
>
> PubSubIO is currently only partially implemented in Java: the 
> DirectPipelineRunner uses a non-scalable API in a single-threaded manner.
> In contrast, the DataflowPipelineRunner uses an entirely different code path 
> implemented in the Google Cloud Dataflow service.
> We need to reimplement PubSubIO in Java in order to support other runners in 
> a scalable way.
> Additionally, we can take this opportunity to add new features:
> * getting timestamp from an arbitrary lambda in arbitrary formats rather than 
> from a message attribute in only 2 formats.
> * exposing metadata and attributes in the elements produced by PubSubIO.Read
> * setting metadata and attributes in the messages written by PubSubIO.Write



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-134) Investigate use of AutoValue

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-134:
--
Assignee: Scott Wegner  (was: Davor Bonaci)

> Investigate use of AutoValue
> 
>
> Key: BEAM-134
> URL: https://issues.apache.org/jira/browse/BEAM-134
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> The initial PR for [BEAM-118] added a dependency on AutoValue to 
> auto-implement equality semantics for a new POJO. We decided to remove the 
> dependency because the cost of adding the dependency for this feature may not 
> be worth it for the value.
> However, we could use AutoValue for all of our POJO's, it might be worth it. 
> The proposal here is to follow-up with an investigation on whether we would 
> gain significant value to porting our code to use AutoValue instead of 
> hand-written POJO's.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-116) Change runners artifactId to runners-parent

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201317#comment-15201317
 ] 

Maximilian Michels commented on BEAM-116:
-

Frankly, I don't think that is possible with Maven. Even if there were no 
parent relationship but merely a dependency, we still would have to create a 
project for this dependency.

There are some ways around this using the Shade plugin's dependencyReducedPOM 
option which generates a new pom with shaded dependencies removed. This really 
gets hacky though :) Better to have a good README for users to get started with 
their Maven project.

> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build became unstable: beam_MavenVerify #14

[Davor Bonaci]

Adding Jason explicitly. This is somewhat expected.

Thanks Andreas.

On Thu, Mar 17, 2016 at 8:08 AM, Andreas Veithen 
wrote:

> It looks like nowadays, when enabling "Use private Maven repository", you
> also need to select a strategy. If you want to shield the build against
> pollution of the local Maven repository by other builds (which I highly
> recommend, speaking from experience), you should use "Local to the
> workspace". That's currently not the case for that build: it's set to
> "Default (~/.m2/respository)", which doesn't sound very private...
>
> Andreas
>
> On Thu, Mar 17, 2016 at 2:57 PM, Amit Sela  wrote:
>
>> Looks like the cached MapR Hadoop dependencies issue is back..
>> I didn't push anything lately, and the only commit was unrelated.
>>
>> On Thu, Mar 17, 2016 at 4:54 PM Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>>
>> > See 
>> >
>> >
>>
>
>

[5/8] incubator-beam git commit: Move Java 8 examples to their own module

[kenn]

Move Java 8 examples to their own module

This allows easy setting of the compiler source and target version to
1.8 without any fragile plugin configuration. It also allows the
import into Eclipse to easily set separate compliance levels, as this
is done per-Eclipse-project.

This parallels the approach taken with tests to support java 8.

Also fixed some Checkstyle and unused variable warnings in the examples.
These were previously hidden since the code wasn't in a directory like
`src/{main,test}/java`. The warnings got fixed while I had the files
open.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e5c7cf88
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e5c7cf88
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e5c7cf88

Branch: refs/heads/master
Commit: e5c7cf88078cbb9c2236b38d854128abb0c18c41
Parents: e8b7100
Author: bchambers 
Authored: Wed Mar 16 14:05:41 2016 -0700
Committer: bchambers 
Committed: Thu Mar 17 13:11:37 2016 -0700

--
 examples/pom.xml| 127 --
 .../examples/MinimalWordCountJava8.java |  68 ---
 .../examples/complete/game/GameStats.java   | 347 ---
 .../examples/complete/game/HourlyTeamScore.java | 193 -
 .../examples/complete/game/LeaderBoard.java | 237 ---
 .../dataflow/examples/complete/game/README.md   | 119 --
 .../examples/complete/game/UserScore.java   | 239 ---
 .../complete/game/injector/Injector.java| 417 ---
 .../complete/game/injector/InjectorUtils.java   | 101 -
 .../injector/RetryHttpInitializerWrapper.java   | 127 --
 .../complete/game/utils/WriteToBigQuery.java| 134 --
 .../game/utils/WriteWindowedToBigQuery.java |  76 
 .../examples/MinimalWordCountJava8Test.java | 103 -
 .../examples/complete/game/GameStatsTest.java   |  99 -
 .../complete/game/HourlyTeamScoreTest.java  | 121 --
 .../examples/complete/game/UserScoreTest.java   | 156 ---
 java8examples/pom.xml   | 278 +
 .../examples/MinimalWordCountJava8.java |  68 +++
 .../examples/complete/game/GameStats.java   | 339 +++
 .../examples/complete/game/HourlyTeamScore.java | 193 +
 .../examples/complete/game/LeaderBoard.java | 237 +++
 .../dataflow/examples/complete/game/README.md   | 113 +
 .../examples/complete/game/UserScore.java   | 239 +++
 .../complete/game/injector/Injector.java| 415 ++
 .../complete/game/injector/InjectorUtils.java   | 101 +
 .../injector/RetryHttpInitializerWrapper.java   | 126 ++
 .../complete/game/utils/WriteToBigQuery.java| 134 ++
 .../game/utils/WriteWindowedToBigQuery.java |  76 
 .../examples/MinimalWordCountJava8Test.java | 103 +
 .../examples/complete/game/GameStatsTest.java   |  76 
 .../complete/game/HourlyTeamScoreTest.java  | 111 +
 .../examples/complete/game/UserScoreTest.java   | 154 +++
 pom.xml |   9 +
 33 files changed, 2772 insertions(+), 2664 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e5c7cf88/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index c15f73f..ea44b01 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -45,133 +45,6 @@
 both
   
 
-
-
-  java8
-  
-[1.8,)
-  
-
-  
-
-  
-  
-org.codehaus.mojo
-build-helper-maven-plugin
-
-  
-add-java8-main-source
-initialize
-
-  add-source
-
-
-  
-${project.basedir}/src/main/java8
-  
-
-  
-
-  
-add-java8-test-source
-initialize
-
-  add-test-source
-
-
-  
-${project.basedir}/src/test/java8
-  
-
-  
-
-  
-
-  
-org.apache.maven.plugins
-maven-compiler-plugin
-
-
-  
-  
-default-testCompile
-test-compile
-
-  testCompile
-
-
-  1.7
-  1.7
-  
-
-**/*Java8Test.java
-

[jira] [Commented] (BEAM-136) PipelineOptionsFactory should detect all runners

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202429#comment-15202429
 ] 

Thomas Groh commented on BEAM-136:
--

The default collection of runners is obtained by using 
ServiceLoader.load(PipelineRunnerRegistrar.class); within the SDK, we use the 
@AutoService annotation (com.google.auto.service.AutoService, from Guava) to 
register the DataflowPipelineRunner, BlockingDataflowPipelineRunner, and 
DirectPipelineRunner, but should automatically pick up any other registrars 
located on the classpath.

PR 61 (https://github.com/apache/incubator-beam/pull/61) fixes looking up 
runners via a fully qualified class name.

> PipelineOptionsFactory should detect all runners
> 
>
> Key: BEAM-136
> URL: https://issues.apache.org/jira/browse/BEAM-136
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
>Priority: Minor
>
> Even with flink-runner jar on classpath, we didn't find it.
> See linked thread here: 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/%3CCAA8k_FLekCURJKYO8qab%3DR%3DG-sRrTmYc7_NQXcrY1kHyATAtBw%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-116) Change runners artifactId to runners-parent

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-116:
--
Component/s: runner-core

> Change runners artifactId to runners-parent
> ---
>
> Key: BEAM-116
> URL: https://issues.apache.org/jira/browse/BEAM-116
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> I've seen regular cases where users use a root ArtifactId, e.g. {{runners}}, 
> presumably because they think it contains all the runners' code. In fact, it 
> just contains no code. 
> I'd propose to rename the runners ArtifactId to {{runners-parent}} which 
> makes it explicit that it is the parent of the sub modules. This is also 
> along the lines of the sdk, i.e. {{google-cloud-dataflow-java-sdk-parent}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)