[GitHub] incubator-beam pull request #1089: [BEAM-742] Replicate triggers to runners-...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1089


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable

[Andrew Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572443#comment-15572443
 ] 

Andrew Martin commented on BEAM-741:


[~kenn] After investigating further, it seems like the coder for the output of 
the values transform is not inferred correctly because of what appears to be 
some loss of type information in the type descriptor - the output of the 
`Values` transform should be of type Iterable but the raw type is just 
Object during the inference process, so the default coder provider is used 
(which we set in our own code).

I'm part of a team at Spotify developing Scio (https://github.com/spotify/scio) 
and we have a work-in-progress branch for beam porting, and it is some tests in 
there that fail. I'd like to have a failing test written in the pure beam API 
so you can take a look - that being said, is it possible to invoke the 
@RunnableOnService tests locally using the direct runner? 

> Values transform does not use the correct output coder when values is an 
> Iterable
> 
>
> Key: BEAM-741
> URL: https://issues.apache.org/jira/browse/BEAM-741
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Andrew Martin
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-747) Text checksum verifier is not resilient to eventually consistent filesystems

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-747:


 Summary: Text checksum verifier is not resilient to eventually 
consistent filesystems
 Key: BEAM-747
 URL: https://issues.apache.org/jira/browse/BEAM-747
 Project: Beam
  Issue Type: Bug
  Components: testing
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Mark Liu


Example 1: 
https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console

Here it looks like we need to retry listing files, at least a little bit, if 
none are found. They did show up:

{code}
gsutil ls 
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
{code}

Example 2: 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-257) fix ConcurrentModificationException in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi resolved BEAM-257.
---
   Resolution: Fixed
Fix Version/s: 0.1.0-incubating

https://github.com/apache/incubator-beam/pull/290

> fix ConcurrentModificationException in KafkaIO
> --
>
> Key: BEAM-257
> URL: https://issues.apache.org/jira/browse/BEAM-257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
> Fix For: 0.1.0-incubating
>
>
> [~tgroh] reported the following exception with KafkaIO while using with 
> DirectRunner. Fixing it in https://github.com/apache/incubator-beam/pull/290
> {code}
>  java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
> multi-threaded access
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1324)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1255)
>   at com.google.common.io.Closeables.close(Closeables.java:79)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaReader.close(KafkaIO.java:1050)
>   at 
> org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.finishBundle(UnboundedReadEvaluatorFactory.java:167)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.finishBundle(TransformExecutor.java:161)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.call(TransformExecutor.java:116)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.call(TransformExecutor.java:41)
>   at java.util.concurrent.FutureTask.run(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>   at java.lang.Thread.run(Unknown Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-748) Issues with local processes in MongoDbIO and JdbcIO

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-748:
-
Description: 
Hey JB, Can you take a look at these issues? Causing flakes in precommit 
testing.

https://builds.apache.org/job/beam_PreCommit_MavenVerify/3931/

A few different types, likely due to issues setting up databases and reusing 
them across tests?

{code}
java.net.ConnectException : Error connecting to server localhost on port 1,527 
with message Connection refused.
{code}

{code}
A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,42)
  Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
  Granted XID : {185, S} 
Lock : ROW, SYSTABLES, (1,30)
  Waiting XID : {185, X} , APP, drop table BEAM
  Granted XID : {185, S} , {187, S} 
. The selected victim is XID : 187.
Stacktrace

java.sql.SQLTransactionRollbackException: 
A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,42)
  Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
  Granted XID : {185, S} 
Lock : ROW, SYSTABLES, (1,30)
  Waiting XID : {185, X} , APP, drop table BEAM
  Granted XID : {185, S} , {187, S} 
. The selected victim is XID : 187.
at 
org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.client.am.SqlException.getSQLException(Unknown 
Source)
at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown 
Source)
at org.apache.beam.sdk.io.jdbc.JdbcIOTest.initTable(JdbcIOTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Suppressed: java.sql.SQLException: Cannot close a connection while a 
transaction is still active.
at 
org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
at 

[jira] [Comment Edited] (BEAM-688) Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572415#comment-15572415
 ] 

Ven Bijjam edited comment on BEAM-688 at 10/13/16 4:30 PM:
---

I am seeing this on the initial build itself. I downloaded the code from master 
branch today 10/13/2016.

mvn clean install -DskipTests -Dcheckstyle.skip=true -DskipIT

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
(default-integration-test) on project beam-sdks-java-maven-archetype
s-starter:
[ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :beam-sdks-java-maven-archetypes-starter


was (Author: venbijjam):
I am seeing this on the initial build itself. I downloaded the code from master 
branch today 10/13/2016.

mvn clean install -DskipTests -Dcheckstyle.skip=true -DskipIT

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
(default-integration-test) on project beam-sdks-java-maven-archetype
s-starter:
[ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :beam-sdks-java-maven-archetypes-starter

> Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with 
> undeclared dependency error
> 
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When building repeatedly using {{mvn install}}, the second build will fail on 
> target beam-sdks-java-maven-archetypes-starter with error:
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
> For some reason this only happens on the second build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-742) Move trigger state machines to runners-core, convert triggers to AST

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572593#comment-15572593
 ] 

ASF GitHub Bot commented on BEAM-742:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1089


> Move trigger state machines to runners-core, convert triggers to AST
> 
>
> Key: BEAM-742
> URL: https://issues.apache.org/jira/browse/BEAM-742
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Update to compile in eclipse again

[dkulp]

Repository: incubator-beam
Updated Branches:
  refs/heads/eclipse2 [created] 83f0b6436


Update to compile in eclipse again


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a8cbd246
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a8cbd246
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a8cbd246

Branch: refs/heads/eclipse2
Commit: a8cbd246c5f74e2c6066f424db19d84ad316b06e
Parents: 7322616
Author: Daniel Kulp 
Authored: Thu Oct 13 12:41:32 2016 -0400
Committer: Daniel Kulp 
Committed: Thu Oct 13 12:41:32 2016 -0400

--
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java   | 3 ++-
 .../apache/beam/runners/direct/TestStreamEvaluatorFactory.java | 2 +-
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java | 5 +++--
 .../main/java/org/apache/beam/runners/spark/io/SourceRDD.java  | 6 --
 4 files changed, 10 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 326a535..8e06c3f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -33,6 +33,7 @@ import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 
 /**
@@ -131,7 +132,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 private  Collection createInitialSplits(
-AppliedPTransform> transform) {
+AppliedPTransform transform) {
   BoundedSource source = transform.getTransform().getSource();
   return Collections.singleton(
   evaluationContext

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
index 4a48a58..fc4f7c5 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
@@ -204,7 +204,7 @@ class TestStreamEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 private  Collection createInputBundle(
-AppliedPTransform> transform) {
+AppliedPTransform transform) {
   CommittedBundle initialBundle =
   evaluationContext
   .createRootBundle()

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a8cbd246/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index 08dc286..183225f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -37,6 +37,7 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.joda.time.Instant;
 
@@ -70,7 +71,7 @@ class UnboundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
   }
 
   private  TransformEvaluator createEvaluator(
-  AppliedPTransform, Read.Unbounded> 
application) {
+  

[2/2] incubator-beam git commit: Attempt to add eclipse-jdt profile to a travis build

[dkulp]

Attempt to add eclipse-jdt profile to a travis build


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/83f0b643
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/83f0b643
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/83f0b643

Branch: refs/heads/eclipse2
Commit: 83f0b6436fb5a800f7ef0794aa65eb4d5a029069
Parents: a8cbd24
Author: Daniel Kulp 
Authored: Wed Sep 14 13:21:34 2016 -0400
Committer: Daniel Kulp 
Committed: Thu Oct 13 12:41:44 2016 -0400

--
 .travis.yml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/83f0b643/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 786b370..5133a43 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,8 @@ matrix:
 - os: linux
   env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 - os: linux
+  env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m' -Peclipse-jdt"
+- os: linux
   env: CUSTOM_JDK="openjdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 
 before_install:

[jira] [Commented] (BEAM-591) Better handling watermark in KafkaIO

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572540#comment-15572540
 ] 

Raghu Angadi commented on BEAM-591:
---


PubSubIO does advance the watermark to current time if there haven't been any 
records recently ([line 
996|https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubUnboundedSource.java#L996]).
 

PubSub tracks last one minute of timestamps since pubsub might deliver out of 
order. Kafka does not have that issue. In addition KafkaIO knows if it has 
caught up with the latest records. 

For the default watermark case (i.e. KafkaIO processing time), I propose 
KafkaIO should advance to current time when the backlog is zero (back log is 
updated every 5 seconds). This will cover most of the use cases.

This policy would be fine for custom timestamps too (as in PubSubIO). If the 
users want more control, we could invoke watermark function with out the the 
kafka record so that user can return current timestamp. 

> Better handling watermark in KafkaIO
> 
>
> Key: BEAM-591
> URL: https://issues.apache.org/jira/browse/BEAM-591
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>
> Right now default watermark in KafkaIO is same as timestamp of the record. 
> The main problem with this is that watermark does not change if there n't any 
> new records on the topic. This can hold up many open windows. 
> The record timestamp by default is set to processing time (i.e. when the 
> runner reads a record from Kafka reader).
> A user can provide functions to calculate watermark and record timestamps. 
> There are a few concerns with current design:
> * What should happen when a kafka topic is idle:
>   ** in default case, I think watermark should advance to current time.
>   ** What should happen when user has provided a function to calculate record 
> timestamp? 
>*** Should the watermark stay same as record timestamp?
>*** same when user has provided own watermark function? 
> * Are the current semantics of user provided watermark function correct?
>   ** -it is run once for each record read-.
>   ** -Should it instead be run inside {{getWatermark()}} called by the runner 
> (we could still provide the last user record, and its timestamp)-.
>   ** It does run inside {{getWatermark()}}. should we pass current record 
> timestamp in addition to the record?
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-593:

Summary: Support unblocking run() in FlinkRunner and cancel() and 
waitUntilFinish() in FlinkRunnerResult  (was: Support cancel() and 
waitUntilFinish() in FlinkRunnerResult)

> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-741) Values transform does not use the correct output coder when values is an Iterable

[Andrew Martin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Martin reopened BEAM-741:


> Values transform does not use the correct output coder when values is an 
> Iterable
> 
>
> Key: BEAM-741
> URL: https://issues.apache.org/jira/browse/BEAM-741
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Andrew Martin
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-748) Issues with local processes in MongoDbIO and JdbcIO

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-748:


 Summary: Issues with local processes in MongoDbIO and JdbcIO
 Key: BEAM-748
 URL: https://issues.apache.org/jira/browse/BEAM-748
 Project: Beam
  Issue Type: Bug
  Components: testing
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Jean-Baptiste Onofré


Hey JB, Can you take a look at these issues?

https://builds.apache.org/job/beam_PreCommit_MavenVerify/3931/

A few different types, likely due to issues setting up databases and reusing 
them across tests?

{code}
java.net.ConnectException : Error connecting to server localhost on port 1,527 
with message Connection refused.
{code}

{code}
A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,42)
  Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
  Granted XID : {185, S} 
Lock : ROW, SYSTABLES, (1,30)
  Waiting XID : {185, X} , APP, drop table BEAM
  Granted XID : {185, S} , {187, S} 
. The selected victim is XID : 187.
Stacktrace

java.sql.SQLTransactionRollbackException: 
A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
Lock : ROW, SYSCOLUMNS, (4,42)
  Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
  Granted XID : {185, S} 
Lock : ROW, SYSTABLES, (1,30)
  Waiting XID : {185, X} , APP, drop table BEAM
  Granted XID : {185, S} , {187, S} 
. The selected victim is XID : 187.
at 
org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.client.am.SqlException.getSQLException(Unknown 
Source)
at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown 
Source)
at org.apache.beam.sdk.io.jdbc.JdbcIOTest.initTable(JdbcIOTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
at 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at 
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Suppressed: java.sql.SQLException: Cannot close a connection 

[GitHub] incubator-beam pull request #1094: Build in eclipse/eclipse-jdt

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1094

Build in eclipse/eclipse-jdt

This just updates the code to build with eclipse-jdt (again) and adds the 
travis.xml changes back in .

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam eclipse2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1094


commit a8cbd246c5f74e2c6066f424db19d84ad316b06e
Author: Daniel Kulp 
Date:   2016-10-13T16:41:32Z

Update to compile in eclipse again

commit 83f0b6436fb5a800f7ef0794aa65eb4d5a029069
Author: Daniel Kulp 
Date:   2016-09-14T17:21:34Z

Attempt to add eclipse-jdt profile to a travis build




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-688) Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with undeclared dependency error

[Venkata Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572415#comment-15572415
 ] 

Venkata Bijjam commented on BEAM-688:
-

I am seeing this on the initial build itself. I downloaded the code from master 
branch today 10/13/2016.

mvn clean install -DskipTests -Dcheckstyle.skip=true -DskipIT

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
(default-integration-test) on project beam-sdks-java-maven-archetype
s-starter:
[ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :beam-sdks-java-maven-archetypes-starter

> Repeated 'mvn install' fails on beam-sdks-java-maven-archetypes-starter with 
> undeclared dependency error
> 
>
> Key: BEAM-688
> URL: https://issues.apache.org/jira/browse/BEAM-688
> Project: Beam
>  Issue Type: Bug
>  Components: build-system, sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When building repeatedly using {{mvn install}}, the second build will fail on 
> target beam-sdks-java-maven-archetypes-starter with error:
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-sdks-java-maven-archetypes-starter ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.7.14:runtime
> {noformat}
> For some reason this only happens on the second build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[incubator-beam] Git Push Summary

[dkulp]

Repository: incubator-beam
Updated Branches:
  refs/heads/eclipse2 [deleted] 83f0b6436

[jira] [Created] (BEAM-749) Syntax highlight on website

[Frances Perry (JIRA)]

Frances Perry created BEAM-749:
--

 Summary: Syntax highlight on website
 Key: BEAM-749
 URL: https://issues.apache.org/jira/browse/BEAM-749
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Frances Perry
Assignee: James Malone


We should able to enable rouge on the website in order to get syntax 
highlighting in the programming guide, walkthroughs, etc.

https://jekyllrb.com/docs/templates/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #662: [BEAM-115] WIP: JSON Schema definition of ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/662


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572807#comment-15572807
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/662


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-520) Update Python SDK example tests to use assert_that

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573085#comment-15573085
 ] 

Ven Bijjam commented on BEAM-520:
-

I would like to work on this. How do I get started?

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-520) Update Python SDK example tests to use assert_that

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-520:
-
Assignee: (was: Frances Perry)

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-520) Update Python SDK example tests to use assert_that

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-520:
-
Assignee: Frances Perry

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Frances Perry
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-595:

Summary: Support non-blocking run() in SparkRunner and cancel() and 
waitUntilFinish() in Spark EvaluationContext  (was: Support cancel() and 
waitUntilFinish() in Spark EvaluationContext)

> Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() 
> in Spark EvaluationContext
> ---
>
> Key: BEAM-595
> URL: https://issues.apache.org/jira/browse/BEAM-595
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Pei He
>Assignee: Amit Sela
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationExcedption in Spark runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-748) Issues with local processes in MongoDbIO and JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573247#comment-15573247
 ] 

Jean-Baptiste Onofré commented on BEAM-748:
---

Yup, let me take a look.

> Issues with local processes in MongoDbIO and JdbcIO
> ---
>
> Key: BEAM-748
> URL: https://issues.apache.org/jira/browse/BEAM-748
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Hey JB, Can you take a look at these issues? Causing flakes in precommit 
> testing.
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3931/
> A few different types, likely due to issues setting up databases and reusing 
> them across tests?
> {code}
> java.net.ConnectException : Error connecting to server localhost on port 
> 1,527 with message Connection refused.
> {code}
> {code}
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
> Stacktrace
> java.sql.SQLTransactionRollbackException: 
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
>   at 
> org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
>   at org.apache.derby.client.am.SqlException.getSQLException(Unknown 
> Source)
>   at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown 
> Source)
>   at org.apache.beam.sdk.io.jdbc.JdbcIOTest.initTable(JdbcIOTest.java:106)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
>   

[GitHub] incubator-beam pull request #1097: [BEAM-443] Update Beam batch examples to ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1097

[BEAM-443] Update Beam batch examples to call waitUntilFinish()





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam waitInExample

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1097.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1097


commit 6ac9e9930208dfdf0789a066243048e5a01c9f5e
Author: Pei He 
Date:   2016-10-13T21:44:13Z

[BEAM-443] Update Beam batch examples to call waitUntilFinish()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-147) Introduce an easy API pipeline metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573440#comment-15573440
 ] 

ASF GitHub Bot commented on BEAM-147:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1024


> Introduce an easy API pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1099: Move the step output ids to use a flat na...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1099

Move the step output ids to use a flat namespace.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Also add a logical mapping from tuple tag to the flat namespace for DoFns.
Note that this is expected to break the Dataflow integration tests until 
the worker image is updated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam flatid

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1099.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1099


commit 0f9b6cae01cead77369e336e2c6d1a39b69e8fe5
Author: Luke Cwik 
Date:   2016-10-13T22:33:49Z

Move the step output ids to use a flat namespace.
Also add a logical mapping from tuple tag to the flat namespace for DoFns.

commit f2d2ab825f4e378e45b8b24c773ecc8e40f83338
Author: Luke Cwik 
Date:   2016-10-13T22:40:05Z

!fixup Fix test to validate that ids are valid int64s.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1096: [BEAM-595] avoid throwing Exception in wa...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1096

[BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam avoid-throw

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1096.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1096


commit f7bf497e8cb035bb3b89f3dca64998f28720704f
Author: Pei He 
Date:   2016-10-13T21:14:15Z

[BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-570) Update AvroSource to support more compression types

[Konstantinos Katsiapis (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573243#comment-15573243
 ] 

Konstantinos Katsiapis commented on BEAM-570:
-

Thanks Frances,

PR https://github.com/apache/incubator-beam/pull/1007 has been merged into beam 
and my https://github.com/apache/incubator-beam/pull/1053 is now also ready for 
a merge.

> Update AvroSource to support more compression types
> ---
>
> Key: BEAM-570
> URL: https://issues.apache.org/jira/browse/BEAM-570
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Konstantinos Katsiapis
>
> Python AvroSource [1] currently only support 'deflate' compression. We should 
> update it to support other compression types supported by the Avro library 
> (e.g.: snappy, bzip2).
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/avroio.py



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1024: [BEAM-147] Prototype of Metrics API for J...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1024


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/5] incubator-beam git commit: Initial Metrics API for Beam Java

[bchambers]

Initial Metrics API for Beam Java

This includes a simple Counter metric and a Distribution metric that
reports the SUM, COUNT, MIN, MAX and MEAN of the reported values.

The API is labeled @Experimental since metrics will only be reported
and queryable with the DirectRunner, and the API may change as it is
implemented on other runners.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8524ed95
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8524ed95
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8524ed95

Branch: refs/heads/master
Commit: 8524ed9545f5af4bdeb54601f333549b34eb35aa
Parents: e969f3d
Author: bchambers 
Authored: Wed Oct 12 10:29:50 2016 -0700
Committer: bchambers 
Committed: Thu Oct 13 15:29:29 2016 -0700

--
 .../beam/sdk/annotations/Experimental.java  |   3 +
 .../org/apache/beam/sdk/metrics/Counter.java|  40 +
 .../apache/beam/sdk/metrics/CounterCell.java|  76 ++
 .../org/apache/beam/sdk/metrics/DirtyState.java |  98 
 .../apache/beam/sdk/metrics/Distribution.java   |  30 
 .../beam/sdk/metrics/DistributionCell.java  |  58 +++
 .../beam/sdk/metrics/DistributionData.java  |  59 
 .../beam/sdk/metrics/DistributionResult.java|  42 ++
 .../org/apache/beam/sdk/metrics/Metric.java |  24 +++
 .../org/apache/beam/sdk/metrics/MetricCell.java |  47 ++
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  40 +
 .../org/apache/beam/sdk/metrics/MetricName.java |  46 ++
 .../beam/sdk/metrics/MetricNameFilter.java  |  60 
 .../beam/sdk/metrics/MetricQueryResults.java|  33 
 .../apache/beam/sdk/metrics/MetricResult.java   |  45 ++
 .../apache/beam/sdk/metrics/MetricResults.java  |  34 +
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  72 +
 .../org/apache/beam/sdk/metrics/Metrics.java| 110 ++
 .../beam/sdk/metrics/MetricsContainer.java  | 150 +++
 .../beam/sdk/metrics/MetricsEnvironment.java|  85 +++
 .../apache/beam/sdk/metrics/MetricsFilter.java  |  86 +++
 .../org/apache/beam/sdk/metrics/MetricsMap.java |  86 +++
 .../apache/beam/sdk/metrics/package-info.java   |  28 
 .../beam/sdk/metrics/CounterCellTest.java   |  55 +++
 .../apache/beam/sdk/metrics/DirtyStateTest.java |  56 +++
 .../beam/sdk/metrics/DistributionCellTest.java  |  53 +++
 .../apache/beam/sdk/metrics/MetricMatchers.java |  99 
 .../beam/sdk/metrics/MetricsContainerTest.java  | 129 
 .../sdk/metrics/MetricsEnvironmentTest.java |  63 
 .../apache/beam/sdk/metrics/MetricsMapTest.java | 103 +
 .../apache/beam/sdk/metrics/MetricsTest.java|  98 
 31 files changed, 2008 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8524ed95/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
index 789f4b2..14d2358 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java
@@ -83,5 +83,8 @@ public @interface Experimental {
  * Do not use: API is unstable and runner support is incomplete.
  */
 SPLITTABLE_DO_FN,
+
+/** Metrics-related experimental APIs. */
+METRICS
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8524ed95/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/Counter.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/Counter.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/Counter.java
new file mode 100644
index 000..9f48016
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/Counter.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the 

[5/5] incubator-beam git commit: Closes #1024

[bchambers]

Closes #1024


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3c731707
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3c731707
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3c731707

Branch: refs/heads/master
Commit: 3c731707b2d986e7a460907c8f64bbebf1dff714
Parents: e969f3d 834933c
Author: bchambers 
Authored: Thu Oct 13 15:29:30 2016 -0700
Committer: bchambers 
Committed: Thu Oct 13 15:29:30 2016 -0700

--
 .../beam/runners/direct/DirectMetrics.java  | 331 +++
 .../beam/runners/direct/DirectRunner.java   |  11 +-
 .../beam/runners/direct/EvaluationContext.java  |  10 +
 .../direct/ExecutorServiceParallelExecutor.java |   1 +
 .../direct/ImmutableListBundleFactory.java  |  10 +
 .../runners/direct/StepTransformResult.java |  49 ++-
 .../beam/runners/direct/TransformExecutor.java  |  35 +-
 .../beam/runners/direct/TransformResult.java|  12 +
 .../beam/runners/direct/DirectMetricsTest.java  | 133 
 .../beam/runners/direct/DirectRunnerTest.java   |  36 ++
 .../runners/direct/TransformExecutorTest.java   |  12 +
 .../beam/runners/flink/FlinkRunnerResult.java   |   6 +
 .../runners/dataflow/DataflowPipelineJob.java   |   7 +
 .../spark/translation/EvaluationContext.java|   6 +
 .../org/apache/beam/sdk/PipelineResult.java |  11 +
 .../beam/sdk/annotations/Experimental.java  |   3 +
 .../org/apache/beam/sdk/metrics/Counter.java|  40 +++
 .../apache/beam/sdk/metrics/CounterCell.java|  76 +
 .../org/apache/beam/sdk/metrics/DirtyState.java |  98 ++
 .../apache/beam/sdk/metrics/Distribution.java   |  30 ++
 .../beam/sdk/metrics/DistributionCell.java  |  58 
 .../beam/sdk/metrics/DistributionData.java  |  59 
 .../beam/sdk/metrics/DistributionResult.java|  42 +++
 .../org/apache/beam/sdk/metrics/Metric.java |  24 ++
 .../org/apache/beam/sdk/metrics/MetricCell.java |  47 +++
 .../org/apache/beam/sdk/metrics/MetricKey.java  |  40 +++
 .../org/apache/beam/sdk/metrics/MetricName.java |  46 +++
 .../beam/sdk/metrics/MetricNameFilter.java  |  60 
 .../beam/sdk/metrics/MetricQueryResults.java|  33 ++
 .../apache/beam/sdk/metrics/MetricResult.java   |  45 +++
 .../apache/beam/sdk/metrics/MetricResults.java  |  34 ++
 .../apache/beam/sdk/metrics/MetricUpdates.java  |  72 
 .../org/apache/beam/sdk/metrics/Metrics.java| 110 ++
 .../beam/sdk/metrics/MetricsContainer.java  | 150 +
 .../beam/sdk/metrics/MetricsEnvironment.java|  85 +
 .../apache/beam/sdk/metrics/MetricsFilter.java  |  86 +
 .../org/apache/beam/sdk/metrics/MetricsMap.java |  86 +
 .../apache/beam/sdk/metrics/package-info.java   |  28 ++
 .../beam/sdk/metrics/CounterCellTest.java   |  55 +++
 .../apache/beam/sdk/metrics/DirtyStateTest.java |  56 
 .../beam/sdk/metrics/DistributionCellTest.java  |  53 +++
 .../apache/beam/sdk/metrics/MetricMatchers.java |  99 ++
 .../beam/sdk/metrics/MetricsContainerTest.java  | 129 
 .../sdk/metrics/MetricsEnvironmentTest.java |  63 
 .../apache/beam/sdk/metrics/MetricsMapTest.java | 103 ++
 .../apache/beam/sdk/metrics/MetricsTest.java|  98 ++
 46 files changed, 2645 insertions(+), 33 deletions(-)
--

[3/5] incubator-beam git commit: Initial Metrics API for Beam Java

[bchambers]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8524ed95/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsMapTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsMapTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsMapTest.java
new file mode 100644
index 000..4104f8d
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsMapTest.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.metrics;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.hamcrest.Matchers.sameInstance;
+import static org.junit.Assert.assertThat;
+
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicLong;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+/**
+ * Tests for {@link MetricsMap}.
+ */
+@RunWith(JUnit4.class)
+public class MetricsMapTest {
+
+  public MetricsMap metricsMap =
+  new MetricsMap<>(new MetricsMap.Factory() {
+@Override
+public AtomicLong createInstance(String unusedKey) {
+  return new AtomicLong();
+}
+  });
+
+  @Test
+  public void testCreateSeparateInstances() {
+AtomicLong foo = metricsMap.get("foo");
+AtomicLong bar = metricsMap.get("bar");
+
+assertThat(foo, not(sameInstance(bar)));
+  }
+
+  @Test
+  public void testReuseInstances() {
+AtomicLong foo1 = metricsMap.get("foo");
+AtomicLong foo2 = metricsMap.get("foo");
+
+assertThat(foo1, sameInstance(foo2));
+  }
+
+  @Test
+  public void testGet() {
+assertThat(metricsMap.tryGet("foo"), nullValue(AtomicLong.class));
+
+AtomicLong foo = metricsMap.get("foo");
+assertThat(metricsMap.tryGet("foo"), sameInstance(foo));
+  }
+
+  @Test
+  public void testGetEntries() {
+AtomicLong foo = metricsMap.get("foo");
+AtomicLong bar = metricsMap.get("bar");
+assertThat(metricsMap.entries(), containsInAnyOrder(
+hasEntry("foo", foo),
+hasEntry("bar", bar)));
+  }
+
+  private static Matcher> hasEntry(
+  final String key, final AtomicLong value) {
+return new TypeSafeMatcher>() {
+
+  @Override
+  public void describeTo(Description description) {
+description
+.appendText("Map.Entry{key=").appendValue(key)
+.appendText(", value=").appendValue(value)
+.appendText("}");
+  }
+
+  @Override
+  protected boolean matchesSafely(Entry item) {
+return Objects.equals(key, item.getKey())
+&& Objects.equals(value, item.getValue());
+  }
+};
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8524ed95/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
new file mode 100644
index 000..d11b44d
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is 

[1/5] incubator-beam git commit: Implement Metrics in the DirectRunner

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e969f3d38 -> 3c731707b


Implement Metrics in the DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/834933c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/834933c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/834933c5

Branch: refs/heads/master
Commit: 834933c520997b4f83cf8b04219c2c63dac61e61
Parents: 51fee39
Author: bchambers 
Authored: Wed Oct 12 10:55:53 2016 -0700
Committer: bchambers 
Committed: Thu Oct 13 15:29:29 2016 -0700

--
 .../beam/runners/direct/DirectMetrics.java  | 331 +++
 .../beam/runners/direct/DirectRunner.java   |   8 +-
 .../beam/runners/direct/EvaluationContext.java  |  10 +
 .../direct/ExecutorServiceParallelExecutor.java |   1 +
 .../direct/ImmutableListBundleFactory.java  |  10 +
 .../runners/direct/StepTransformResult.java |  49 ++-
 .../beam/runners/direct/TransformExecutor.java  |  35 +-
 .../beam/runners/direct/TransformResult.java|  12 +
 .../beam/runners/direct/DirectMetricsTest.java  | 133 
 .../beam/runners/direct/DirectRunnerTest.java   |  36 ++
 .../runners/direct/TransformExecutorTest.java   |  12 +
 11 files changed, 602 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/834933c5/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
new file mode 100644
index 000..a749a76
--- /dev/null
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
@@ -0,0 +1,331 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct;
+
+import static java.util.Arrays.asList;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.base.Objects;
+import com.google.common.collect.ImmutableList;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
+import org.apache.beam.sdk.metrics.DistributionData;
+import org.apache.beam.sdk.metrics.DistributionResult;
+import org.apache.beam.sdk.metrics.MetricKey;
+import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricNameFilter;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.MetricResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.sdk.metrics.MetricUpdates;
+import org.apache.beam.sdk.metrics.MetricUpdates.MetricUpdate;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.sdk.metrics.MetricsMap;
+
+/**
+ * Implementation of {@link MetricResults} for the Direct Runner.
+ */
+class DirectMetrics extends MetricResults {
+
+  // TODO: (BEAM-723) Create a shared ExecutorService for maintenance tasks in 
the DirectRunner.
+  private static final ExecutorService COUNTER_COMMITTER = 
Executors.newCachedThreadPool();
+
+  private interface MetricAggregation {
+UpdateT zero();
+UpdateT combine(Iterable updates);
+ResultT extract(UpdateT data);
+  }
+
+  /**
+   * Implementation of a metric in the direct runner.
+   *
+   * @param  The type of raw data received and aggregated across 
updates.
+   * @param  The type of result extracted from the data.
+   */
+  private static class DirectMetric {
+private 

[2/5] incubator-beam git commit: Add the ability to query metrics on PipelineResult

[bchambers]

Add the ability to query metrics on PipelineResult

All runners currently implement this by throwing an
UnsupportedOperationException.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/51fee39b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/51fee39b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/51fee39b

Branch: refs/heads/master
Commit: 51fee39b7bc66d7f60ea2e0ce31e3cb516a89305
Parents: 8524ed9
Author: bchambers 
Authored: Wed Oct 12 10:55:05 2016 -0700
Committer: bchambers 
Committed: Thu Oct 13 15:29:29 2016 -0700

--
 .../org/apache/beam/runners/direct/DirectRunner.java |  7 +++
 .../org/apache/beam/runners/flink/FlinkRunnerResult.java |  6 ++
 .../beam/runners/dataflow/DataflowPipelineJob.java   |  7 +++
 .../runners/spark/translation/EvaluationContext.java |  6 ++
 .../main/java/org/apache/beam/sdk/PipelineResult.java| 11 +++
 5 files changed, 37 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51fee39b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index a72f7ae..e13046d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -42,6 +42,7 @@ import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.Write;
+import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.testing.TestStream;
@@ -380,6 +381,12 @@ public class DirectRunner
   };
 }
 
+@Override
+public MetricResults metrics() {
+  throw new UnsupportedOperationException(
+  "The DirectRunner does not currently support metrics.");
+}
+
 /**
  * Blocks until the {@link Pipeline} execution represented by this
  * {@link DirectPipelineResult} is complete, returning the terminal state.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51fee39b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
index 90bb64d..6b15485 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
@@ -23,6 +23,7 @@ import java.util.Map;
 import org.apache.beam.sdk.AggregatorRetrievalException;
 import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.joda.time.Duration;
 
@@ -86,4 +87,9 @@ public class FlinkRunnerResult implements PipelineResult {
   public State waitUntilFinish(Duration duration) {
 throw new UnsupportedOperationException("FlinkRunnerResult does not 
support waitUntilFinish.");
   }
+
+  @Override
+  public MetricResults metrics() {
+throw new UnsupportedOperationException("The FlinkRunner does not 
currently support metrics.");
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51fee39b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index 269b824..bbcf11f 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -42,6 +42,7 @@ import org.apache.beam.runners.dataflow.util.MonitoringUtil;
 import org.apache.beam.sdk.AggregatorRetrievalException;
 import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.PipelineResult;
+import 

[jira] [Updated] (BEAM-147) Introduce an easy API for pipeline metrics

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers updated BEAM-147:
--
Summary: Introduce an easy API for pipeline metrics  (was: Introduce an 
easy API pipeline metrics)

> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-520) Update Python SDK example tests to use assert_that

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573113#comment-15573113
 ] 

Ahmet Altay commented on BEAM-520:
--

Ven, thank you for your interest.

A partial cleanup was already done in an earlier PR 
(https://github.com/apache/incubator-beam/pull/650). Take a look at that. 
Similar changes needs to happen in the remaining tests (e.g. 
multiple_output_pardo_test.py). Convert as many tests/examples as possible to 
the assert_that pattern and send a PR.

If you are not familiar, you can also look at Beam contribution guide 
(http://beam.incubator.apache.org/contribute/contribution-guide/) for the 
general workflow of working with Beam.

> Update Python SDK example tests to use assert_that
> --
>
> Key: BEAM-520
> URL: https://issues.apache.org/jira/browse/BEAM-520
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
>
> Most of our examples use assert_that to test examples:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/complete/autocomplete_test.py#L38
> Some of our examples use this pattern:
> 1) Create a path(s)
> 2) Construct fake command line arguments using these paths
> 3) Construct an argparse object to parse these flags
> 4) Do the (often trivial logic)
> 5) Write to a file
> 6) Manually open and read the file
> 7) Compare results. 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
> As well as being cumbersome, this obscures the core of what is being 
> illustrated and tested. As many as possible tests should be updated to use 
> assert_that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573234#comment-15573234
 ] 

Ven Bijjam commented on BEAM-361:
-

I am thinking along these lines:: in
{code}PubsubUnboundedSource{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}

> Batch Pub/Sub sink does not set idLabel
> ---
>
> Key: BEAM-361
> URL: https://issues.apache.org/jira/browse/BEAM-361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Mills
>  Labels: newbie, starter
>
> The transform should generate a unique token (must be stable across retries), 
> and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-596) Support cancel() and waitUntilFinish() in DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573352#comment-15573352
 ] 

ASF GitHub Bot commented on BEAM-596:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1098

[BEAM-596] Replace DirectResult#awaitCompletion with waitUntilFinish

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This is the PipelineResult interface method, rather than the
DirectRunner-specific method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam wait_until_finsish

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1098.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1098


commit 851b447584015e9e0e864554d1ba507f09157eb8
Author: Thomas Groh 
Date:   2016-10-13T22:07:51Z

Replace DirectResult#awaitCompletion with waitUntilFinish

This is the PipelineResult interface method, rather than the
DirectRunner-specific method.




> Support cancel() and waitUntilFinish() in DirectRunner
> --
>
> Key: BEAM-596
> URL: https://issues.apache.org/jira/browse/BEAM-596
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Pei He
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1098: [BEAM-596] Replace DirectResult#awaitComp...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1098

[BEAM-596] Replace DirectResult#awaitCompletion with waitUntilFinish

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This is the PipelineResult interface method, rather than the
DirectRunner-specific method.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam wait_until_finsish

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1098.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1098


commit 851b447584015e9e0e864554d1ba507f09157eb8
Author: Thomas Groh 
Date:   2016-10-13T22:07:51Z

Replace DirectResult#awaitCompletion with waitUntilFinish

This is the PipelineResult interface method, rather than the
DirectRunner-specific method.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573207#comment-15573207
 ] 

ASF GitHub Bot commented on BEAM-595:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1096

[BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam avoid-throw

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1096.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1096


commit f7bf497e8cb035bb3b89f3dca64998f28720704f
Author: Pei He 
Date:   2016-10-13T21:14:15Z

[BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner




> Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() 
> in Spark EvaluationContext
> ---
>
> Key: BEAM-595
> URL: https://issues.apache.org/jira/browse/BEAM-595
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Pei He
>Assignee: Amit Sela
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationExcedption in Spark runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-742) Move trigger state machines to runners-core, convert triggers to AST

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573390#comment-15573390
 ] 

ASF GitHub Bot commented on BEAM-742:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1083


> Move trigger state machines to runners-core, convert triggers to AST
> 
>
> Key: BEAM-742
> URL: https://issues.apache.org/jira/browse/BEAM-742
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[14/17] incubator-beam git commit: Restore prior trigger files, for temporary compatibility

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64acb2f/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
new file mode 100644
index 000..7e6e938
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.windowing;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.sdk.util.TriggerTester;
+import org.apache.beam.sdk.util.TriggerTester.SimpleTriggerTester;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests the {@link AfterSynchronizedProcessingTime}.
+ */
+@RunWith(JUnit4.class)
+public class AfterSynchronizedProcessingTimeTest {
+
+  private Trigger underTest = new AfterSynchronizedProcessingTime();
+
+  @Test
+  public void testAfterProcessingTimeWithFixedWindows() throws Exception {
+Duration windowDuration = Duration.millis(10);
+SimpleTriggerTester tester = TriggerTester.forTrigger(
+AfterProcessingTime
+.pastFirstElementInPane()
+.plusDelayOf(Duration.millis(5)),
+FixedWindows.of(windowDuration));
+
+tester.advanceProcessingTime(new Instant(10));
+
+// Timer at 15
+tester.injectElements(1);
+IntervalWindow firstWindow = new IntervalWindow(new Instant(0), new 
Instant(10));
+tester.advanceProcessingTime(new Instant(12));
+assertFalse(tester.shouldFire(firstWindow));
+
+// Load up elements in the next window, timer at 17 for them
+tester.injectElements(11, 12, 13);
+IntervalWindow secondWindow = new IntervalWindow(new Instant(10), new 
Instant(20));
+assertFalse(tester.shouldFire(secondWindow));
+
+// Not quite time to fire
+tester.advanceProcessingTime(new Instant(14));
+assertFalse(tester.shouldFire(firstWindow));
+assertFalse(tester.shouldFire(secondWindow));
+
+// Timer at 19 for these in the first window; it should be ignored since 
the 15 will fire first
+tester.injectElements(2, 3);
+
+// Advance past the first timer and fire, finishing the first window
+tester.advanceProcessingTime(new Instant(16));
+assertTrue(tester.shouldFire(firstWindow));
+assertFalse(tester.shouldFire(secondWindow));
+tester.fireIfShouldFire(firstWindow);
+assertTrue(tester.isMarkedFinished(firstWindow));
+
+// The next window fires and finishes now
+tester.advanceProcessingTime(new Instant(18));
+assertTrue(tester.shouldFire(secondWindow));
+tester.fireIfShouldFire(secondWindow);
+assertTrue(tester.isMarkedFinished(secondWindow));
+  }
+
+  @Test
+  public void testAfterProcessingTimeWithMergingWindow() throws Exception {
+Duration windowDuration = Duration.millis(10);
+SimpleTriggerTester tester = TriggerTester.forTrigger(
+AfterProcessingTime
+.pastFirstElementInPane()
+.plusDelayOf(Duration.millis(5)),
+Sessions.withGapDuration(windowDuration));
+
+tester.advanceProcessingTime(new Instant(10));
+tester.injectElements(1); // in [1, 11), timer for 15
+IntervalWindow firstWindow = new IntervalWindow(new Instant(1), new 
Instant(11));
+assertFalse(tester.shouldFire(firstWindow));
+
+tester.advanceProcessingTime(new Instant(12));
+tester.injectElements(3); // in [3, 13), timer for 17
+IntervalWindow secondWindow = new IntervalWindow(new Instant(3), new 
Instant(13));
+assertFalse(tester.shouldFire(secondWindow));
+
+tester.mergeWindows();
+IntervalWindow mergedWindow = new IntervalWindow(new Instant(1), new 
Instant(13));
+
+

[05/17] incubator-beam git commit: Rename runners-core Trigger to TriggerStateMachine

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Repeatedly.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Repeatedly.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Repeatedly.java
deleted file mode 100644
index 8858798..000
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Repeatedly.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms.windowing;
-
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.sdk.util.ExecutableTrigger;
-import org.joda.time.Instant;
-
-/**
- * Repeat a trigger, either until some condition is met or forever.
- *
- * For example, to fire after the end of the window, and every time late 
data arrives:
- *  {@code
- * Repeatedly.forever(AfterWatermark.isPastEndOfWindow());
- * } 
- *
- * {@code Repeatedly.forever(someTrigger)} behaves like an infinite
- * {@code AfterEach.inOrder(someTrigger, someTrigger, someTrigger, ...)}.
- */
-public class Repeatedly extends Trigger {
-
-  private static final int REPEATED = 0;
-
-  /**
-   * Create a composite trigger that repeatedly executes the trigger {@code 
repeated}, firing each
-   * time it fires and ignoring any indications to finish.
-   *
-   * Unless used with {@link Trigger#orFinally} the composite trigger will 
never finish.
-   *
-   * @param repeated the trigger to execute repeatedly.
-   */
-  public static Repeatedly forever(Trigger repeated) {
-return new Repeatedly(repeated);
-  }
-
-  private Repeatedly(Trigger repeated) {
-super(Arrays.asList(repeated));
-  }
-
-
-  @Override
-  public void onElement(OnElementContext c) throws Exception {
-getRepeated(c).invokeOnElement(c);
-  }
-
-  @Override
-  public void onMerge(OnMergeContext c) throws Exception {
-getRepeated(c).invokeOnMerge(c);
-  }
-
-  @Override
-  public Instant getWatermarkThatGuaranteesFiring(BoundedWindow window) {
-// This trigger fires once the repeated trigger fires.
-return subTriggers.get(REPEATED).getWatermarkThatGuaranteesFiring(window);
-  }
-
-  @Override
-  public Trigger getContinuationTrigger(List continuationTriggers) {
-return new Repeatedly(continuationTriggers.get(REPEATED));
-  }
-
-  @Override
-  public boolean shouldFire(Trigger.TriggerContext context) throws Exception {
-return getRepeated(context).invokeShouldFire(context);
-  }
-
-  @Override
-  public void onFire(TriggerContext context) throws Exception {
-getRepeated(context).invokeOnFire(context);
-
-if (context.trigger().isFinished(REPEATED)) {
-  // Reset tree will recursively clear the finished bits, and invoke clear.
-  context.forTrigger(getRepeated(context)).trigger().resetTree();
-}
-  }
-
-  @Override
-  public String toString() {
-return String.format("Repeatedly.forever(%s)", subTriggers.get(REPEATED));
-  }
-
-  private ExecutableTrigger getRepeated(TriggerContext context) {
-return context.trigger().subTrigger(REPEATED);
-  }
-}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/ReshuffleTrigger.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/ReshuffleTrigger.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/ReshuffleTrigger.java
deleted file mode 100644
index 9e2c27d..000
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/ReshuffleTrigger.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * 

[12/17] incubator-beam git commit: Move triggers to runners-core

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4398e1e/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/FinishedTriggersSet.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/FinishedTriggersSet.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/FinishedTriggersSet.java
new file mode 100644
index 000..a9feb73
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/FinishedTriggersSet.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.google.common.collect.Sets;
+import java.util.Set;
+
+/**
+ * An implementation of {@link FinishedTriggers} atop a user-provided mutable 
{@link Set}.
+ */
+public class FinishedTriggersSet implements FinishedTriggers {
+
+  private final Set finishedTriggers;
+
+  private FinishedTriggersSet(Set finishedTriggers) {
+this.finishedTriggers = finishedTriggers;
+  }
+
+  public static FinishedTriggersSet fromSet(Set 
finishedTriggers) {
+return new FinishedTriggersSet(finishedTriggers);
+  }
+
+  /**
+   * Returns a mutable {@link Set} of the underlying triggers that are 
finished.
+   */
+  public Set getFinishedTriggers() {
+return finishedTriggers;
+  }
+
+  @Override
+  public boolean isFinished(ExecutableTrigger trigger) {
+return finishedTriggers.contains(trigger);
+  }
+
+  @Override
+  public void setFinished(ExecutableTrigger trigger, boolean value) {
+if (value) {
+  finishedTriggers.add(trigger);
+} else {
+  finishedTriggers.remove(trigger);
+}
+  }
+
+  @Override
+  public void clearRecursively(ExecutableTrigger trigger) {
+finishedTriggers.remove(trigger);
+for (ExecutableTrigger subTrigger : trigger.subTriggers()) {
+  clearRecursively(subTrigger);
+}
+  }
+
+  @Override
+  public FinishedTriggersSet copy() {
+return fromSet(Sets.newHashSet(finishedTriggers));
+  }
+
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4398e1e/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Never.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Never.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Never.java
new file mode 100644
index 000..5f20465
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/reactors/Never.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.windowing;
+
+import java.util.List;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
+import org.joda.time.Instant;
+
+/**
+ * A trigger which never fires.
+ *
+ * Using this trigger will only produce output when the watermark passes 
the end of the
+ * {@link BoundedWindow window} plus the {@link Window#withAllowedLateness 
allowed
+ * lateness}.
+ */
+public final class Never {
+  /**
+   * Returns a trigger which never fires. Output will be produced from the 
using {@link GroupByKey}
+   * when the {@link BoundedWindow} closes.
+   */
+  public static OnceTrigger ever() {
+// NeverTrigger ignores all inputs and is Window-type 

[08/17] incubator-beam git commit: Move triggers to runners-core

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4398e1e/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
deleted file mode 100644
index 7e6e938..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTimeTest.java
+++ /dev/null
@@ -1,121 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms.windowing;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
-import org.apache.beam.sdk.util.TriggerTester;
-import org.apache.beam.sdk.util.TriggerTester.SimpleTriggerTester;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/**
- * Tests the {@link AfterSynchronizedProcessingTime}.
- */
-@RunWith(JUnit4.class)
-public class AfterSynchronizedProcessingTimeTest {
-
-  private Trigger underTest = new AfterSynchronizedProcessingTime();
-
-  @Test
-  public void testAfterProcessingTimeWithFixedWindows() throws Exception {
-Duration windowDuration = Duration.millis(10);
-SimpleTriggerTester tester = TriggerTester.forTrigger(
-AfterProcessingTime
-.pastFirstElementInPane()
-.plusDelayOf(Duration.millis(5)),
-FixedWindows.of(windowDuration));
-
-tester.advanceProcessingTime(new Instant(10));
-
-// Timer at 15
-tester.injectElements(1);
-IntervalWindow firstWindow = new IntervalWindow(new Instant(0), new 
Instant(10));
-tester.advanceProcessingTime(new Instant(12));
-assertFalse(tester.shouldFire(firstWindow));
-
-// Load up elements in the next window, timer at 17 for them
-tester.injectElements(11, 12, 13);
-IntervalWindow secondWindow = new IntervalWindow(new Instant(10), new 
Instant(20));
-assertFalse(tester.shouldFire(secondWindow));
-
-// Not quite time to fire
-tester.advanceProcessingTime(new Instant(14));
-assertFalse(tester.shouldFire(firstWindow));
-assertFalse(tester.shouldFire(secondWindow));
-
-// Timer at 19 for these in the first window; it should be ignored since 
the 15 will fire first
-tester.injectElements(2, 3);
-
-// Advance past the first timer and fire, finishing the first window
-tester.advanceProcessingTime(new Instant(16));
-assertTrue(tester.shouldFire(firstWindow));
-assertFalse(tester.shouldFire(secondWindow));
-tester.fireIfShouldFire(firstWindow);
-assertTrue(tester.isMarkedFinished(firstWindow));
-
-// The next window fires and finishes now
-tester.advanceProcessingTime(new Instant(18));
-assertTrue(tester.shouldFire(secondWindow));
-tester.fireIfShouldFire(secondWindow);
-assertTrue(tester.isMarkedFinished(secondWindow));
-  }
-
-  @Test
-  public void testAfterProcessingTimeWithMergingWindow() throws Exception {
-Duration windowDuration = Duration.millis(10);
-SimpleTriggerTester tester = TriggerTester.forTrigger(
-AfterProcessingTime
-.pastFirstElementInPane()
-.plusDelayOf(Duration.millis(5)),
-Sessions.withGapDuration(windowDuration));
-
-tester.advanceProcessingTime(new Instant(10));
-tester.injectElements(1); // in [1, 11), timer for 15
-IntervalWindow firstWindow = new IntervalWindow(new Instant(1), new 
Instant(11));
-assertFalse(tester.shouldFire(firstWindow));
-
-tester.advanceProcessingTime(new Instant(12));
-tester.injectElements(3); // in [3, 13), timer for 17
-IntervalWindow secondWindow = new IntervalWindow(new Instant(3), new 
Instant(13));
-assertFalse(tester.shouldFire(secondWindow));
-
-tester.mergeWindows();
-IntervalWindow mergedWindow = new IntervalWindow(new Instant(1), new 
Instant(13));
-
-

[jira] [Comment Edited] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573234#comment-15573234
 ] 

Ven Bijjam edited comment on BEAM-361 at 10/13/16 9:29 PM:
---

I am thinking along these lines:: in
{code} @VisibleForTesting
  PubsubUnboundedSource(
  Clock clock,
  PubsubClientFactory pubsubFactory,
  @Nullable ProjectPath project,
  @Nullable TopicPath topic,
  @Nullable SubscriptionPath subscription,
  Coder elementCoder,
  @Nullable String timestampLabel,
  @Nullable String idLabel) {
checkArgument((topic == null) != (subscription == null),
  "Exactly one of topic and subscription must be given");
checkArgument((topic == null) == (project == null),
  "Project must be given if topic is given");
this.clock = clock;
this.pubsubFactory = checkNotNull(pubsubFactory);
this.project = project;
this.topic = topic;
this.subscription = subscription;
this.elementCoder = checkNotNull(elementCoder);
this.timestampLabel = timestampLabel;
this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel; 
//Geetha instead of idLabel;
  }{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}


was (Author: venbijjam):
I am thinking along these lines:: in
{code}PubsubUnboundedSource{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}

> Batch Pub/Sub sink does not set idLabel
> ---
>
> Key: BEAM-361
> URL: https://issues.apache.org/jira/browse/BEAM-361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Mills
>  Labels: newbie, starter
>
> The transform should generate a unique token (must be stable across retries), 
> and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-443) PipelineResult needs waitToFinish() and cancel()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573284#comment-15573284
 ] 

ASF GitHub Bot commented on BEAM-443:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1097

[BEAM-443] Update Beam batch examples to call waitUntilFinish()





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam waitInExample

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1097.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1097


commit 6ac9e9930208dfdf0789a066243048e5a01c9f5e
Author: Pei He 
Date:   2016-10-13T21:44:13Z

[BEAM-443] Update Beam batch examples to call waitUntilFinish()




> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-443
> URL: https://issues.apache.org/jira/browse/BEAM-443
> Project: Beam
>  Issue Type: New Feature
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.2.0-incubating
>
>
> waitToFinish() and cancel() are two most common operations for users to 
> interact with a started pipeline.
> Right now, they are only available in DataflowPipelineJob. But, it is better 
> to move them to the common interface, so people can start implement them in 
> other runners, and runner agnostic code can interact with PipelineResult 
> better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1522

[Apache Jenkins Server]

See 


--
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform 
0.3.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 15 source files to 

[INFO] 
:
 

 uses or overrides a deprecated API.
[INFO] 
:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 15 source files to 

[INFO] 
:
 Some input files use or override a deprecated API.
[INFO] 
:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ 
beam-sdks-java-io-google-cloud-platform ---
[INFO] Surefire report directory: 

[INFO] Using configured provider 
org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO] parallel='none', perCoreThreadCount=true, threadCount=0, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true

---
 T E S T S
---
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.914 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.442 sec - in 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableInserterTest
Tests run: 4, Failures: 0, Errors: 

[jira] [Commented] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573125#comment-15573125
 ] 

Ven Bijjam commented on BEAM-361:
-

Hi
I would like to take this up. But I am not sure how to contribute as this is my 
first time getting involved in open source project.

> Batch Pub/Sub sink does not set idLabel
> ---
>
> Key: BEAM-361
> URL: https://issues.apache.org/jira/browse/BEAM-361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Mills
>  Labels: newbie, starter
>
> The transform should generate a unique token (must be stable across retries), 
> and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573234#comment-15573234
 ] 

Ven Bijjam edited comment on BEAM-361 at 10/13/16 9:30 PM:
---

I am thinking along these lines:: in
{code} @VisibleForTesting
  PubsubUnboundedSource(
  Clock clock,
  PubsubClientFactory pubsubFactory,
  @Nullable ProjectPath project,
  @Nullable TopicPath topic,
  @Nullable SubscriptionPath subscription,
  Coder elementCoder,
  @Nullable String timestampLabel,
  @Nullable String idLabel) {
checkArgument((topic == null) != (subscription == null),
  "Exactly one of topic and subscription must be given");
checkArgument((topic == null) == (project == null),
  "Project must be given if topic is given");
this.clock = clock;
this.pubsubFactory = checkNotNull(pubsubFactory);
this.project = project;
this.topic = topic;
this.subscription = subscription;
this.elementCoder = checkNotNull(elementCoder);
this.timestampLabel = timestampLabel;
this.idLabel = idLabel; 
  }{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}


was (Author: venbijjam):
I am thinking along these lines:: in
{code} @VisibleForTesting
  PubsubUnboundedSource(
  Clock clock,
  PubsubClientFactory pubsubFactory,
  @Nullable ProjectPath project,
  @Nullable TopicPath topic,
  @Nullable SubscriptionPath subscription,
  Coder elementCoder,
  @Nullable String timestampLabel,
  @Nullable String idLabel) {
checkArgument((topic == null) != (subscription == null),
  "Exactly one of topic and subscription must be given");
checkArgument((topic == null) == (project == null),
  "Project must be given if topic is given");
this.clock = clock;
this.pubsubFactory = checkNotNull(pubsubFactory);
this.project = project;
this.topic = topic;
this.subscription = subscription;
this.elementCoder = checkNotNull(elementCoder);
this.timestampLabel = timestampLabel;
this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel; 
  }{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}

> Batch Pub/Sub sink does not set idLabel
> ---
>
> Key: BEAM-361
> URL: https://issues.apache.org/jira/browse/BEAM-361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Mills
>  Labels: newbie, starter
>
> The transform should generate a unique token (must be stable across retries), 
> and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Ven Bijjam (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573234#comment-15573234
 ] 

Ven Bijjam edited comment on BEAM-361 at 10/13/16 9:29 PM:
---

I am thinking along these lines:: in
{code} @VisibleForTesting
  PubsubUnboundedSource(
  Clock clock,
  PubsubClientFactory pubsubFactory,
  @Nullable ProjectPath project,
  @Nullable TopicPath topic,
  @Nullable SubscriptionPath subscription,
  Coder elementCoder,
  @Nullable String timestampLabel,
  @Nullable String idLabel) {
checkArgument((topic == null) != (subscription == null),
  "Exactly one of topic and subscription must be given");
checkArgument((topic == null) == (project == null),
  "Project must be given if topic is given");
this.clock = clock;
this.pubsubFactory = checkNotNull(pubsubFactory);
this.project = project;
this.topic = topic;
this.subscription = subscription;
this.elementCoder = checkNotNull(elementCoder);
this.timestampLabel = timestampLabel;
this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel; 
  }{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}


was (Author: venbijjam):
I am thinking along these lines:: in
{code} @VisibleForTesting
  PubsubUnboundedSource(
  Clock clock,
  PubsubClientFactory pubsubFactory,
  @Nullable ProjectPath project,
  @Nullable TopicPath topic,
  @Nullable SubscriptionPath subscription,
  Coder elementCoder,
  @Nullable String timestampLabel,
  @Nullable String idLabel) {
checkArgument((topic == null) != (subscription == null),
  "Exactly one of topic and subscription must be given");
checkArgument((topic == null) == (project == null),
  "Project must be given if topic is given");
this.clock = clock;
this.pubsubFactory = checkNotNull(pubsubFactory);
this.project = project;
this.topic = topic;
this.subscription = subscription;
this.elementCoder = checkNotNull(elementCoder);
this.timestampLabel = timestampLabel;
this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel; 
//Geetha instead of idLabel;
  }{code} 
{code}..\sdks\java\core\src\main\java\org\apache\beam\sdk\io\PubsubUnboundedSource.java{code}

change the following line {code}this.idLabel = idLabel; {code} 
to the following:
{code}this.idLabel = idLabel == null ? 
UUID.nameUUIDFromBytes(timestampLabel.getBytes()).toString() : idLabel;{code}

> Batch Pub/Sub sink does not set idLabel
> ---
>
> Key: BEAM-361
> URL: https://issues.apache.org/jira/browse/BEAM-361
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Mills
>  Labels: newbie, starter
>
> The transform should generate a unique token (must be stable across retries), 
> and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[15/17] incubator-beam git commit: Restore prior trigger files, for temporary compatibility

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64acb2f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
new file mode 100644
index 000..a960aa4
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
@@ -0,0 +1,527 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.windowing;
+
+import com.google.common.base.Joiner;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.util.ExecutableTrigger;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.apache.beam.sdk.util.state.MergingStateAccessor;
+import org.apache.beam.sdk.util.state.StateAccessor;
+import org.joda.time.Instant;
+
+/**
+ * {@code Trigger}s control when the elements for a specific key and window 
are output. As elements
+ * arrive, they are put into one or more windows by a {@link Window} transform 
and its associated
+ * {@link WindowFn}, and then passed to the associated {@code Trigger} to 
determine if the
+ * {@code Window}s contents should be output.
+ *
+ * See {@link org.apache.beam.sdk.transforms.GroupByKey} and {@link Window}
+ * for more information about how grouping with windows works.
+ *
+ * The elements that are assigned to a window since the last time it was 
fired (or since the
+ * window was created) are placed into the current window pane. Triggers are 
evaluated against the
+ * elements as they are added. When the root trigger fires, the elements in 
the current pane will be
+ * output. When the root trigger finishes (indicating it will never fire 
again), the window is
+ * closed and any new elements assigned to that window are discarded.
+ *
+ * Several predefined {@code Trigger}s are provided:
+ * 
+ *{@link AfterWatermark} for firing when the watermark passes a 
timestamp determined from
+ *   either the end of the window or the arrival of the first element in a 
pane.
+ *{@link AfterProcessingTime} for firing after some amount of 
processing time has elapsed
+ *   (typically since the first element in a pane).
+ *{@link AfterPane} for firing off a property of the elements in the 
current pane, such as
+ *   the number of elements that have been assigned to the current pane.
+ * 
+ *
+ * In addition, {@code Trigger}s can be combined in a variety of ways:
+ * 
+ *{@link Repeatedly#forever} to create a trigger that executes 
forever. Any time its
+ *   argument finishes it gets reset and starts over. Can be combined with
+ *   {@link Trigger#orFinally} to specify a condition that causes the 
repetition to stop.
+ *{@link AfterEach#inOrder} to execute each trigger in sequence, 
firing each (and every)
+ *   time that a trigger fires, and advancing to the next trigger in the 
sequence when it finishes.
+ *{@link AfterFirst#of} to create a trigger that fires after at least 
one of its arguments
+ *   fires. An {@link AfterFirst} trigger finishes after it fires once.
+ *{@link AfterAll#of} to create a trigger that fires after all least 
one of its arguments
+ *   have fired at least once. An {@link AfterAll} trigger finishes after it 
fires once.
+ * 
+ *
+ * Each trigger tree is instantiated per-key and per-window. Every trigger 
in the tree is in one
+ * of the following states:
+ * 
+ *Never Existed - before the trigger has started executing, there is 
no state associated
+ *   with it anywhere in the system. A trigger moves to the executing state as 
soon as it
+ *   processes in the current pane.
+ *Executing - while the trigger is receiving items and may fire. While 
it is in this state,
+ *   it may persist book-keeping information to persisted state, set timers, 
etc.
+ *Finished - after a trigger finishes, all of its book-keeping data is 
cleaned up, and 

[13/17] incubator-beam git commit: Move triggers to runners-core

[kenn]

Move triggers to runners-core

This commit is effectively a "git hint" that all the files
in runners-core are moved from prior files. The moved files
will be replaced.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e4398e1e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e4398e1e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e4398e1e

Branch: refs/heads/master
Commit: e4398e1e7b08ca5098e422eb62c2d8cb139d906b
Parents: 7322616
Author: Kenneth Knowles 
Authored: Thu Jun 23 20:05:27 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 13 14:34:34 2016 -0700

--
 .../apache/beam/runners/core/TriggerRunner.java | 247 -
 .../beam/runners/core/reactors/AfterAll.java| 122 +
 .../reactors/AfterDelayFromFirstElement.java| 335 
 .../beam/runners/core/reactors/AfterEach.java   | 141 +
 .../beam/runners/core/reactors/AfterFirst.java  | 124 +
 .../beam/runners/core/reactors/AfterPane.java   | 144 +
 .../core/reactors/AfterProcessingTime.java  | 102 
 .../AfterSynchronizedProcessingTime.java|  73 +++
 .../runners/core/reactors/AfterWatermark.java   | 355 +
 .../runners/core/reactors/DefaultTrigger.java   |  92 
 .../core/reactors/ExecutableTrigger.java| 159 ++
 .../runners/core/reactors/FinishedTriggers.java |  44 ++
 .../core/reactors/FinishedTriggersBitSet.java   |  67 +++
 .../core/reactors/FinishedTriggersSet.java  |  72 +++
 .../beam/runners/core/reactors/Never.java   |  75 +++
 .../runners/core/reactors/OrFinallyTrigger.java | 105 
 .../beam/runners/core/reactors/Repeatedly.java  | 101 
 .../runners/core/reactors/ReshuffleTrigger.java |  66 +++
 .../beam/runners/core/reactors/Trigger.java | 527 +++
 .../core/reactors/TriggerContextFactory.java| 507 ++
 .../runners/core/reactors/TriggerRunner.java| 247 +
 .../runners/core/reactors/AfterAllTest.java | 156 ++
 .../runners/core/reactors/AfterEachTest.java| 132 +
 .../runners/core/reactors/AfterFirstTest.java   | 181 +++
 .../runners/core/reactors/AfterPaneTest.java| 132 +
 .../core/reactors/AfterProcessingTimeTest.java  | 187 +++
 .../AfterSynchronizedProcessingTimeTest.java| 121 +
 .../core/reactors/AfterWatermarkTest.java   | 380 +
 .../core/reactors/DefaultTriggerTest.java   | 176 +++
 .../core/reactors/ExecutableTriggerTest.java| 127 +
 .../reactors/FinishedTriggersBitSetTest.java|  55 ++
 .../reactors/FinishedTriggersProperties.java| 110 
 .../core/reactors/FinishedTriggersSetTest.java  |  60 +++
 .../beam/runners/core/reactors/NeverTest.java   |  56 ++
 .../core/reactors/OrFinallyTriggerTest.java | 215 
 .../runners/core/reactors/RepeatedlyTest.java   | 224 
 .../core/reactors/ReshuffleTriggerTest.java |  67 +++
 .../beam/runners/core/reactors/StubTrigger.java |  70 +++
 .../beam/runners/core/reactors/TriggerTest.java | 118 +
 .../runners/core/reactors/TriggerTester.java| 410 +++
 .../beam/sdk/transforms/windowing/AfterAll.java | 122 -
 .../windowing/AfterDelayFromFirstElement.java   | 335 
 .../sdk/transforms/windowing/AfterEach.java | 141 -
 .../sdk/transforms/windowing/AfterFirst.java| 124 -
 .../sdk/transforms/windowing/AfterPane.java | 144 -
 .../windowing/AfterProcessingTime.java  | 102 
 .../AfterSynchronizedProcessingTime.java|  73 ---
 .../transforms/windowing/AfterWatermark.java| 355 -
 .../transforms/windowing/DefaultTrigger.java|  92 
 .../beam/sdk/transforms/windowing/Never.java|  75 ---
 .../transforms/windowing/OrFinallyTrigger.java  | 105 
 .../sdk/transforms/windowing/Repeatedly.java| 101 
 .../beam/sdk/transforms/windowing/Trigger.java  | 527 ---
 .../apache/beam/sdk/util/ExecutableTrigger.java | 159 --
 .../apache/beam/sdk/util/FinishedTriggers.java  |  44 --
 .../beam/sdk/util/FinishedTriggersBitSet.java   |  67 ---
 .../beam/sdk/util/FinishedTriggersSet.java  |  72 ---
 .../apache/beam/sdk/util/ReshuffleTrigger.java  |  66 ---
 .../beam/sdk/util/TriggerContextFactory.java| 507 --
 .../sdk/transforms/windowing/AfterAllTest.java  | 156 --
 .../sdk/transforms/windowing/AfterEachTest.java | 132 -
 .../transforms/windowing/AfterFirstTest.java| 181 ---
 .../sdk/transforms/windowing/AfterPaneTest.java | 132 -
 .../windowing/AfterProcessingTimeTest.java  | 187 ---
 .../AfterSynchronizedProcessingTimeTest.java| 121 -
 .../windowing/AfterWatermarkTest.java   | 380 -
 .../windowing/DefaultTriggerTest.java   | 176 ---
 

[16/17] incubator-beam git commit: Restore prior trigger files, for temporary compatibility

[kenn]

Restore prior trigger files, for temporary compatibility


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a64acb2f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a64acb2f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a64acb2f

Branch: refs/heads/master
Commit: a64acb2f84ac26bd1a3f297085477f13b0252570
Parents: 69b1efd
Author: Kenneth Knowles 
Authored: Tue Oct 11 21:35:02 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 13 14:34:35 2016 -0700

--
 .../apache/beam/runners/core/TriggerRunner.java | 247 +
 .../beam/sdk/transforms/windowing/AfterAll.java | 122 +
 .../windowing/AfterDelayFromFirstElement.java   | 335 
 .../sdk/transforms/windowing/AfterEach.java | 141 +
 .../sdk/transforms/windowing/AfterFirst.java| 124 +
 .../sdk/transforms/windowing/AfterPane.java | 144 +
 .../windowing/AfterProcessingTime.java  | 102 
 .../AfterSynchronizedProcessingTime.java|  73 +++
 .../transforms/windowing/AfterWatermark.java| 355 +
 .../transforms/windowing/DefaultTrigger.java|  92 
 .../beam/sdk/transforms/windowing/Never.java|  75 +++
 .../transforms/windowing/OrFinallyTrigger.java  | 105 
 .../sdk/transforms/windowing/Repeatedly.java| 101 
 .../beam/sdk/transforms/windowing/Trigger.java  | 527 +++
 .../apache/beam/sdk/util/ExecutableTrigger.java | 159 ++
 .../apache/beam/sdk/util/FinishedTriggers.java  |  44 ++
 .../beam/sdk/util/FinishedTriggersBitSet.java   |  67 +++
 .../beam/sdk/util/FinishedTriggersSet.java  |  72 +++
 .../apache/beam/sdk/util/ReshuffleTrigger.java  |  66 +++
 .../beam/sdk/util/TriggerContextFactory.java| 507 ++
 .../sdk/transforms/windowing/AfterAllTest.java  | 156 ++
 .../sdk/transforms/windowing/AfterEachTest.java | 132 +
 .../transforms/windowing/AfterFirstTest.java| 181 +++
 .../sdk/transforms/windowing/AfterPaneTest.java | 132 +
 .../windowing/AfterProcessingTimeTest.java  | 187 +++
 .../AfterSynchronizedProcessingTimeTest.java| 121 +
 .../windowing/AfterWatermarkTest.java   | 380 +
 .../windowing/DefaultTriggerTest.java   | 176 +++
 .../sdk/transforms/windowing/NeverTest.java |  56 ++
 .../windowing/OrFinallyTriggerTest.java | 215 
 .../transforms/windowing/RepeatedlyTest.java| 224 
 .../sdk/transforms/windowing/StubTrigger.java   |  70 +++
 .../sdk/transforms/windowing/TriggerTest.java   | 118 +
 .../org/apache/beam/sdk/util/TriggerTester.java | 410 +++
 34 files changed, 6016 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a64acb2f/runners/core-java/src/main/java/org/apache/beam/runners/core/TriggerRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/TriggerRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/TriggerRunner.java
new file mode 100644
index 000..8d0f322
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/TriggerRunner.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.util.BitSet;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.util.BitSetCoder;
+import org.apache.beam.sdk.util.ExecutableTrigger;
+import org.apache.beam.sdk.util.FinishedTriggers;

[GitHub] incubator-beam pull request #1083: [BEAM-742] Replicate triggers to runners-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1083


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[04/17] incubator-beam git commit: Rename runners-core Trigger to TriggerStateMachine

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
new file mode 100644
index 000..2490463
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.triggers;
+
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.joda.time.Instant;
+
+/**
+ * {@code AfterProcessingTime} triggers fire based on the current processing 
time. They operate in
+ * the real-time domain.
+ *
+ * The time at which to fire the timer can be adjusted via the methods in 
{@link
+ * AfterDelayFromFirstElementStateMachine}, such as {@link
+ * AfterDelayFromFirstElementStateMachine#plusDelayOf} or {@link
+ * AfterDelayFromFirstElementStateMachine#alignedTo}.
+ */
+@Experimental(Experimental.Kind.TRIGGER)
+public class AfterProcessingTimeStateMachine extends 
AfterDelayFromFirstElementStateMachine {
+
+  @Override
+  @Nullable
+  public Instant getCurrentTime(TriggerStateMachine.TriggerContext context) {
+return context.currentProcessingTime();
+  }
+
+  private AfterProcessingTimeStateMachine(List> transforms) {
+super(TimeDomain.PROCESSING_TIME, transforms);
+  }
+
+  /**
+   * Creates a trigger that fires when the current processing time passes the 
processing time
+   * at which this trigger saw the first element in a pane.
+   */
+  public static AfterProcessingTimeStateMachine pastFirstElementInPane() {
+return new AfterProcessingTimeStateMachine(IDENTITY);
+  }
+
+  @Override
+  protected AfterProcessingTimeStateMachine newWith(
+  List> transforms) {
+return new AfterProcessingTimeStateMachine(transforms);
+  }
+
+  @Override
+  public String toString() {
+StringBuilder builder = new 
StringBuilder("AfterProcessingTime.pastFirstElementInPane()");
+for (SerializableFunction delayFn : timestampMappers) {
+  builder
+  .append(".plusDelayOf(")
+  .append(delayFn)
+  .append(")");
+}
+
+return builder.toString();
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+if (this == obj) {
+  return true;
+}
+if (!(obj instanceof AfterProcessingTimeStateMachine)) {
+  return false;
+}
+AfterProcessingTimeStateMachine that = (AfterProcessingTimeStateMachine) 
obj;
+return Objects.equals(this.timestampMappers, that.timestampMappers);
+  }
+
+  @Override
+  public int hashCode() {
+return Objects.hash(getClass(), this.timestampMappers);
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
new file mode 100644
index 000..000f6e7
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this 

[01/17] incubator-beam git commit: Rename runners-core Trigger to TriggerStateMachine

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 73226168a -> e969f3d38


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
new file mode 100644
index 000..119c937
--- /dev/null
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachineTest.java
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.triggers;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.when;
+
+import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
+import 
org.apache.beam.runners.core.triggers.TriggerStateMachineTester.SimpleTriggerStateMachineTester;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+
+/**
+ * Tests the {@link AfterWatermarkStateMachine} triggers.
+ */
+@RunWith(JUnit4.class)
+public class AfterWatermarkStateMachineTest {
+
+  @Mock private OnceTriggerStateMachine mockEarly;
+  @Mock private OnceTriggerStateMachine mockLate;
+
+  private SimpleTriggerStateMachineTester tester;
+  private static TriggerStateMachine.TriggerContext anyTriggerContext() {
+return Mockito.any();
+  }
+  private static TriggerStateMachine.OnElementContext anyElementContext() {
+return Mockito.any();
+  }
+
+  private void injectElements(int... elements) throws Exception {
+for (int element : elements) {
+  doNothing().when(mockEarly).onElement(anyElementContext());
+  doNothing().when(mockLate).onElement(anyElementContext());
+  tester.injectElements(element);
+}
+  }
+
+  @Before
+  public void setUp() {
+MockitoAnnotations.initMocks(this);
+  }
+
+  public void testRunningAsTrigger(OnceTriggerStateMachine mockTrigger, 
IntervalWindow window)
+  throws Exception {
+
+// Don't fire due to mock saying no
+when(mockTrigger.shouldFire(anyTriggerContext())).thenReturn(false);
+assertFalse(tester.shouldFire(window)); // not ready
+
+// Fire due to mock trigger; early trigger is required to be a OnceTrigger
+when(mockTrigger.shouldFire(anyTriggerContext())).thenReturn(true);
+assertTrue(tester.shouldFire(window)); // ready
+tester.fireIfShouldFire(window);
+assertFalse(tester.isMarkedFinished(window));
+  }
+
+  @Test
+  public void testEarlyAndAtWatermark() throws Exception {
+tester = TriggerStateMachineTester.forTrigger(
+AfterWatermarkStateMachine.pastEndOfWindow()
+.withEarlyFirings(mockEarly),
+FixedWindows.of(Duration.millis(100)));
+
+injectElements(1);
+IntervalWindow window = new IntervalWindow(new Instant(0), new 
Instant(100));
+
+testRunningAsTrigger(mockEarly, window);
+
+// Fire due to watermark
+when(mockEarly.shouldFire(anyTriggerContext())).thenReturn(false);
+tester.advanceInputWatermark(new Instant(100));
+assertTrue(tester.shouldFire(window));
+tester.fireIfShouldFire(window);
+assertTrue(tester.isMarkedFinished(window));
+  }
+
+  @Test
+  public void testAtWatermarkAndLate() throws Exception {
+tester = TriggerStateMachineTester.forTrigger(
+AfterWatermarkStateMachine.pastEndOfWindow()
+.withLateFirings(mockLate),
+FixedWindows.of(Duration.millis(100)));
+
+

[03/17] incubator-beam git commit: Rename runners-core Trigger to TriggerStateMachine

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/69b1efda/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
new file mode 100644
index 000..0ffbbca
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.triggers;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import java.util.BitSet;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.BitSetCoder;
+import org.apache.beam.sdk.util.Timers;
+import org.apache.beam.sdk.util.state.MergingStateAccessor;
+import org.apache.beam.sdk.util.state.StateAccessor;
+import org.apache.beam.sdk.util.state.StateTag;
+import org.apache.beam.sdk.util.state.StateTags;
+import org.apache.beam.sdk.util.state.ValueState;
+import org.joda.time.Instant;
+
+/**
+ * Executes a trigger while managing persistence of information about which 
subtriggers are
+ * finished. Subtriggers include all recursive trigger expressions as well as 
the entire trigger.
+ *
+ * Specifically, the responsibilities are:
+ *
+ * 
+ *   Invoking the trigger's methods via its {@link 
ExecutableTriggerStateMachine} wrapper by
+ *   constructing the appropriate trigger contexts.
+ *   Committing a record of which subtriggers are finished to persistent 
state.
+ *   Restoring the record of which subtriggers are finished from 
persistent state.
+ *   Clearing out the persisted finished set when a caller indicates
+ *   (via {#link #clearFinished}) that it is no longer needed.
+ * 
+ *
+ * These responsibilities are intertwined: trigger contexts include mutable 
information about
+ * which subtriggers are finished. This class provides the information when 
building the contexts
+ * and commits the information when the method of the {@link 
ExecutableTriggerStateMachine} returns.
+ *
+ * @param  The kind of windows being processed.
+ */
+public class TriggerStateMachineRunner {
+  @VisibleForTesting
+  static final StateTag FINISHED_BITS_TAG =
+  StateTags.makeSystemTagInternal(StateTags.value("closed", 
BitSetCoder.of()));
+
+  private final ExecutableTriggerStateMachine rootTrigger;
+  private final TriggerStateMachineContextFactory contextFactory;
+
+  public TriggerStateMachineRunner(
+  ExecutableTriggerStateMachine rootTrigger,
+  TriggerStateMachineContextFactory contextFactory) {
+checkState(rootTrigger.getTriggerIndex() == 0);
+this.rootTrigger = rootTrigger;
+this.contextFactory = contextFactory;
+  }
+
+  private FinishedTriggersBitSet readFinishedBits(ValueState state) {
+if (!isFinishedSetNeeded()) {
+  // If no trigger in the tree will ever have finished bits, then we don't 
need to read them.
+  // So that the code can be agnostic to that fact, we create a BitSet 
that is all 0 (not
+  // finished) for each trigger in the tree.
+  return 
FinishedTriggersBitSet.emptyWithCapacity(rootTrigger.getFirstIndexAfterSubtree());
+}
+
+BitSet bitSet = state.read();
+return bitSet == null
+? 
FinishedTriggersBitSet.emptyWithCapacity(rootTrigger.getFirstIndexAfterSubtree())
+: FinishedTriggersBitSet.fromBitSet(bitSet);
+  }
+
+
+  private void clearFinishedBits(ValueState state) {
+if (!isFinishedSetNeeded()) {
+  // Nothing to clear.
+  return;
+}
+state.clear();
+  }
+
+  /** Return true if the trigger is closed in the window corresponding to the 
specified state. */
+  public boolean isClosed(StateAccessor state) {
+return 

[11/17] incubator-beam git commit: Move triggers to runners-core

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4398e1e/runners/core-java/src/test/java/org/apache/beam/runners/core/reactors/AfterPaneTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/reactors/AfterPaneTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/reactors/AfterPaneTest.java
new file mode 100644
index 000..38d030e
--- /dev/null
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/reactors/AfterPaneTest.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.windowing;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.sdk.util.TriggerTester;
+import org.apache.beam.sdk.util.TriggerTester.SimpleTriggerTester;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link AfterPane}.
+ */
+@RunWith(JUnit4.class)
+public class AfterPaneTest {
+
+  SimpleTriggerTester tester;
+  /**
+   * Tests that the trigger does fire when enough elements are in a window, 
and that it only
+   * fires that window (no leakage).
+   */
+  @Test
+  public void testAfterPaneElementCountFixedWindows() throws Exception {
+tester = TriggerTester.forTrigger(
+AfterPane.elementCountAtLeast(2),
+FixedWindows.of(Duration.millis(10)));
+
+tester.injectElements(1); // [0, 10)
+IntervalWindow window = new IntervalWindow(new Instant(0), new 
Instant(10));
+assertFalse(tester.shouldFire(window));
+
+tester.injectElements(2); // [0, 10)
+tester.injectElements(11); // [10, 20)
+
+assertTrue(tester.shouldFire(window)); // ready to fire
+tester.fireIfShouldFire(window); // and finished
+assertTrue(tester.isMarkedFinished(window));
+
+// But don't finish the other window
+assertFalse(tester.isMarkedFinished(new IntervalWindow(new Instant(10), 
new Instant(20;
+  }
+
+  @Test
+  public void testClear() throws Exception {
+SimpleTriggerTester tester = TriggerTester.forTrigger(
+AfterPane.elementCountAtLeast(2),
+FixedWindows.of(Duration.millis(10)));
+
+tester.injectElements(1, 2, 3);
+IntervalWindow window = new IntervalWindow(new Instant(0), new 
Instant(10));
+tester.clearState(window);
+tester.assertCleared(window);
+  }
+
+  @Test
+  public void testAfterPaneElementCountSessions() throws Exception {
+tester = TriggerTester.forTrigger(
+AfterPane.elementCountAtLeast(2),
+Sessions.withGapDuration(Duration.millis(10)));
+
+tester.injectElements(
+1, // in [1, 11)
+2); // in [2, 12)
+
+assertFalse(tester.shouldFire(new IntervalWindow(new Instant(1), new 
Instant(11;
+assertFalse(tester.shouldFire(new IntervalWindow(new Instant(2), new 
Instant(12;
+
+tester.mergeWindows();
+
+IntervalWindow mergedWindow = new IntervalWindow(new Instant(1), new 
Instant(12));
+assertTrue(tester.shouldFire(mergedWindow));
+tester.fireIfShouldFire(mergedWindow);
+assertTrue(tester.isMarkedFinished(mergedWindow));
+
+// Because we closed the previous window, we don't have it around to merge 
with. So there
+// will be a new FIRE_AND_FINISH result.
+tester.injectElements(
+7,  // in [7, 17)
+9); // in [9, 19)
+
+tester.mergeWindows();
+
+IntervalWindow newMergedWindow = new IntervalWindow(new Instant(7), new 
Instant(19));
+assertTrue(tester.shouldFire(newMergedWindow));
+tester.fireIfShouldFire(newMergedWindow);
+assertTrue(tester.isMarkedFinished(newMergedWindow));
+  }
+
+  @Test
+  public void testFireDeadline() throws Exception {
+assertEquals(BoundedWindow.TIMESTAMP_MAX_VALUE,
+AfterPane.elementCountAtLeast(1).getWatermarkThatGuaranteesFiring(
+new IntervalWindow(new Instant(0), new Instant(10;
+  }
+
+  @Test
+  public void testContinuation() throws Exception {
+assertEquals(
+

[17/17] incubator-beam git commit: This closes #1083

[kenn]

This closes #1083


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e969f3d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e969f3d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e969f3d3

Branch: refs/heads/master
Commit: e969f3d388dda4b991d36e5aa37d5ada71f593e3
Parents: 7322616 a64acb2
Author: Kenneth Knowles 
Authored: Thu Oct 13 15:22:08 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 13 15:22:08 2016 -0700

--
 .../core/triggers/AfterAllStateMachine.java | 104 
 .../AfterDelayFromFirstElementStateMachine.java | 322 
 .../core/triggers/AfterEachStateMachine.java| 125 +
 .../core/triggers/AfterFirstStateMachine.java   | 106 
 .../core/triggers/AfterPaneStateMachine.java| 132 +
 .../AfterProcessingTimeStateMachine.java|  93 
 ...rSynchronizedProcessingTimeStateMachine.java |  63 +++
 .../triggers/AfterWatermarkStateMachine.java| 325 
 .../triggers/DefaultTriggerStateMachine.java|  81 +++
 .../triggers/ExecutableTriggerStateMachine.java | 160 ++
 .../runners/core/triggers/FinishedTriggers.java |  44 ++
 .../core/triggers/FinishedTriggersBitSet.java   |  67 +++
 .../core/triggers/FinishedTriggersSet.java  |  72 +++
 .../core/triggers/NeverStateMachine.java|  60 +++
 .../core/triggers/OrFinallyStateMachine.java|  85 
 .../core/triggers/RepeatedlyStateMachine.java   |  88 
 .../triggers/ReshuffleTriggerStateMachine.java  |  50 ++
 .../core/triggers/TriggerStateMachine.java  | 487 ++
 .../TriggerStateMachineContextFactory.java  | 509 +++
 .../triggers/TriggerStateMachineRunner.java | 234 +
 .../runners/core/triggers/package-info.java |  23 +
 .../core/triggers/AfterAllStateMachineTest.java | 140 +
 .../triggers/AfterEachStateMachineTest.java | 108 
 .../triggers/AfterFirstStateMachineTest.java| 159 ++
 .../triggers/AfterPaneStateMachineTest.java | 117 +
 .../AfterProcessingTimeStateMachineTest.java| 172 +++
 ...chronizedProcessingTimeStateMachineTest.java | 110 
 .../AfterWatermarkStateMachineTest.java | 382 ++
 .../DefaultTriggerStateMachineTest.java | 165 ++
 .../ExecutableTriggerStateMachineTest.java  | 108 
 .../triggers/FinishedTriggersBitSetTest.java|  55 ++
 .../triggers/FinishedTriggersProperties.java| 115 +
 .../core/triggers/FinishedTriggersSetTest.java  |  60 +++
 .../core/triggers/NeverStateMachineTest.java|  59 +++
 .../triggers/OrFinallyStateMachineTest.java | 177 +++
 .../triggers/RepeatedlyStateMachineTest.java| 200 
 .../ReshuffleTriggerStateMachineTest.java   |  68 +++
 .../core/triggers/StubTriggerStateMachine.java  |  60 +++
 .../core/triggers/TriggerStateMachineTest.java  |  98 
 .../triggers/TriggerStateMachineTester.java | 431 
 .../beam/sdk/util/ExecutableTriggerTest.java| 127 -
 .../sdk/util/FinishedTriggersBitSetTest.java|  55 --
 .../sdk/util/FinishedTriggersProperties.java| 110 
 .../beam/sdk/util/FinishedTriggersSetTest.java  |  60 ---
 .../beam/sdk/util/ReshuffleTriggerTest.java |  67 ---
 45 files changed, 6014 insertions(+), 419 deletions(-)
--

[07/17] incubator-beam git commit: Move triggers to runners-core

[kenn]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4398e1e/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TriggerTester.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TriggerTester.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TriggerTester.java
deleted file mode 100644
index 5fe17ad..000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/TriggerTester.java
+++ /dev/null
@@ -1,410 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkNotNull;
-import static org.junit.Assert.assertTrue;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.Trigger;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.util.ActiveWindowSet.MergeCallback;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
-import org.apache.beam.sdk.util.WindowingStrategy.AccumulationMode;
-import org.apache.beam.sdk.util.state.InMemoryTimerInternals;
-import org.apache.beam.sdk.util.state.StateInternals;
-import org.apache.beam.sdk.util.state.StateNamespace;
-import org.apache.beam.sdk.util.state.StateNamespaces;
-import 
org.apache.beam.sdk.util.state.StateNamespaces.WindowAndTriggerNamespace;
-import org.apache.beam.sdk.util.state.StateNamespaces.WindowNamespace;
-import org.apache.beam.sdk.util.state.TestInMemoryStateInternals;
-import org.apache.beam.sdk.util.state.TimerCallback;
-import org.apache.beam.sdk.values.TimestampedValue;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/**
- * Test utility that runs a {@link Trigger}, using in-memory stub 
implementation to provide
- * the {@link StateInternals}.
- *
- * @param  The type of windows being used.
- */
-public class TriggerTester {
-
-  /**
-   * A {@link TriggerTester} specialized to {@link Integer} values, so 
elements and timestamps
-   * can be conflated. Today, triggers should not observed the element type, 
so this is the
-   * only trigger tester that needs to be used.
-   */
-  public static class SimpleTriggerTester
-  extends TriggerTester {
-
-private SimpleTriggerTester(WindowingStrategy 
windowingStrategy) throws Exception {
-  super(windowingStrategy);
-}
-
-public void injectElements(int... values) throws Exception {
-  List timestampedValues =
-  Lists.newArrayListWithCapacity(values.length);
-  for (int value : values) {
-timestampedValues.add(TimestampedValue.of(value, new Instant(value)));
-  }
-  injectElements(timestampedValues);
-}
-
-public SimpleTriggerTester withAllowedLateness(Duration 
allowedLateness) throws Exception {
-  return new SimpleTriggerTester<>(
-  windowingStrategy.withAllowedLateness(allowedLateness));
-}
-  }
-
-  protected final WindowingStrategy windowingStrategy;
-
-  private final TestInMemoryStateInternals stateInternals =
-  new TestInMemoryStateInternals(null /* key */);
-  private final InMemoryTimerInternals timerInternals = new 
InMemoryTimerInternals();
-  private final TriggerContextFactory contextFactory;
-  private final WindowFn windowFn;
-  private final ActiveWindowSet activeWindows;
-  private final Map windowToMergeResult;
-
-  /**
-   * An {@link ExecutableTrigger} built from the {@link Trigger} or {@link 
Trigger}
-   * under test.
-   */
-  private final ExecutableTrigger executableTrigger;
-
-  /**
-   * A map from a window 

Re: Build failed in Jenkins: beam_PostCommit_MavenVerify #1522

[Kenneth Knowles]

FYI I am watching the subsequent results. This error was caused by the JVM
on the Jenkins worker getting OOM-killed, did not seem to relate to recent
changes. But I'll watch to make sure.

On Thu, Oct 13, 2016 at 3:35 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1522/changes>
>
> Changes:
>
> [klk] Move triggers to runners-core
>
> [klk] Rename runners-core Trigger to TriggerStateMachine
>
> [klk] Restore prior trigger files, for temporary compatibility
>
> --
> [...truncated 3079 lines...]
> [INFO] Compiling 15 source files to <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/target/classes
> >
> [INFO] <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java>:
> <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java>
> uses or overrides a deprecated API.
> [INFO] <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java>:
> Recompile with -Xlint:deprecation for details.
> [INFO]
> [INFO] --- maven-resources-plugin:2.7:testResources
> (default-testResources) @ beam-sdks-java-io-google-cloud-platform ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/test/resources
> >
> [INFO] Copying 3 resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @
> beam-sdks-java-io-google-cloud-platform ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 15 source files to <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/target/test-classes
> >
> [INFO] <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java>:
> Some input files use or override a deprecated API.
> [INFO] <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestUtil.java>:
> Recompile with -Xlint:deprecation for details.
> [INFO]
> [INFO] --- maven-checkstyle-plugin:2.17:check (default) @
> beam-sdks-java-io-google-cloud-platform ---
> [INFO] Starting audit...
> Audit done.
> [INFO]
> [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @
> beam-sdks-java-io-google-cloud-platform ---
> [INFO] Surefire report directory: <
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/ws/sdks/java/io/google-cloud-platform/target/surefire-reports
> >
> [INFO] Using configured provider
> org.apache.maven.surefire.junitcore.JUnitCoreProvider
> [INFO] parallel='none', perCoreThreadCount=true, threadCount=0,
> useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0,
> threadCountMethods=0, parallelOptimized=true
>
> ---
>  T E S T S
> ---
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.914 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableRowIteratorTest
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.442 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryAvroUtilsTest
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableInserterTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.577 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryTableInserterTest
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest
> Tests run: 62, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 4.094 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilTest
> Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.062 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtilTest
> Running org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImplTest
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 sec
> - in org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImplTest
> Running org.apache.beam.sdk.io.gcp.bigtable.BigtableIOTest
> Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.991 sec
> - in org.apache.beam.sdk.io.gcp.bigtable.BigtableIOTest
> Running org.apache.beam.sdk.io.gcp.datastore.DatastoreV1Test
> Tests run: 43, Failures: 0, Errors: 0, 

[1/2] incubator-beam git commit: KafkaIO reader should set consumedOffset even before reading the first record.

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3c731707b -> 93d2e374c


KafkaIO reader should set consumedOffset even before reading the first record.

* Distinguish between uninitialized consumed offset and consumed offset for
  an empty partition.

* Add test to verify we handle empty partitions better.

* Fix how we were using MockConsumer. checkpoint test was actually doing what 
it was supposed to.

* Avoid cpu spinning in case of a test failure


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/08c2f1c3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/08c2f1c3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/08c2f1c3

Branch: refs/heads/master
Commit: 08c2f1c361eb17f47794e805df910c3dad6a9d43
Parents: 3c73170
Author: Raghu Angadi 
Authored: Fri Oct 7 17:18:25 2016 -0700
Committer: Dan Halperin 
Committed: Thu Oct 13 17:21:52 2016 -0700

--
 .../beam/sdk/io/kafka/KafkaCheckpointMark.java  |   8 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   |  60 +++
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 162 +++
 3 files changed, 165 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/08c2f1c3/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
index 664bb6f..4f9e96f 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
@@ -56,19 +56,19 @@ public class KafkaCheckpointMark implements 
UnboundedSource.CheckpointMark, Seri
*/
   public static class PartitionMark implements Serializable {
 private final TopicPartition topicPartition;
-private final long offset;
+private final long nextOffset;
 
 public PartitionMark(TopicPartition topicPartition, long offset) {
   this.topicPartition = topicPartition;
-  this.offset = offset;
+  this.nextOffset = offset;
 }
 
 public TopicPartition getTopicPartition() {
   return topicPartition;
 }
 
-public long getOffset() {
-  return offset;
+public long getNextOffset() {
+  return nextOffset;
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/08c2f1c3/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index e26f7c5..2030789 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -782,6 +782,8 @@ public class KafkaIO {
 Executors.newSingleThreadScheduledExecutor();
 private static final int OFFSET_UPDATE_INTERVAL_SECONDS = 5;
 
+private static final long UNINITIALIZED_OFFSET = -1;
+
 /** watermark before any records have been read. */
 private static Instant initialWatermark = new Instant(Long.MIN_VALUE);
 
@@ -792,7 +794,7 @@ public class KafkaIO {
 // maintains state of each assigned partition (buffered records, consumed 
offset, etc)
 private static class PartitionState {
   private final TopicPartition topicPartition;
-  private long consumedOffset;
+  private long nextOffset;
   private long latestOffset;
   private Iterator> recordIter = 
Collections.emptyIterator();
 
@@ -800,15 +802,15 @@ public class KafkaIO {
   private double avgRecordSize = 0;
   private static final int movingAvgWindow = 1000; // very roughly avg of 
last 1000 elements
 
-  PartitionState(TopicPartition partition, long offset) {
+  PartitionState(TopicPartition partition, long nextOffset) {
 this.topicPartition = partition;
-this.consumedOffset = offset;
-this.latestOffset = -1;
+this.nextOffset = nextOffset;
+this.latestOffset = UNINITIALIZED_OFFSET;
   }
 
   // update consumedOffset and avgRecordSize
   void recordConsumed(long offset, int size) {
-consumedOffset = offset;
+nextOffset = offset + 1;
 
 // this is always updated from single thread. probably not worth 
making it an AtomicDouble
 if (avgRecordSize <= 0) {
@@ -825,14 +827,10 @@ public 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1525

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1071: KafkaIO reader should set consumedOffset ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1071


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1009: [BEAM-551] Add ValueProvider class (Take ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1009


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574304#comment-15574304
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1009


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1100: Windowed side inputs

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/1100

Windowed side inputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam windowed-side-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1100.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1100


commit a002d25729d2ed53a33c422d84b188bced7d8178
Author: Robert Bradshaw 
Date:   2016-10-07T22:28:50Z

Dissallow (unimplemented) windowed side inputs.

commit c04dcc2c12f6d91275acca2aba93cd697c7b5464
Author: Robert Bradshaw 
Date:   2016-10-07T23:17:47Z

Fix tests.

commit b040bc31b9773a5569e16917d4fbc6dcd8f96ca5
Author: Robert Bradshaw 
Date:   2016-10-07T23:31:20Z

Add windowed side inputs failure test.

commit d00c09c8a165d0bafa5851ac73706cd99b76f488
Author: Robert Bradshaw 
Date:   2016-10-11T23:43:41Z

Windowed side input test.

commit 7d902978db511d8e39a045f54bddfc0bd048cc7d
Author: Robert Bradshaw 
Date:   2016-10-11T22:27:58Z

Implement windowed side inputs for direct runner.

commit 40d131d9bb94a345e114b02fe45fa46ce7805e62
Author: Robert Bradshaw 
Date:   2016-10-13T00:37:52Z

Implement windowed side inputs for InProcess runner.

commit 711017b27e5948494282eb8a0ec18a2e39459886
Author: Robert Bradshaw 
Date:   2016-10-13T22:15:06Z

Fix tests expecting list from AsIter.

commit 2568f5906ae10e284603daaede085b75916ce565
Author: Robert Bradshaw 
Date:   2016-10-13T23:39:24Z

remove debugging

commit 8d513ca3a6cf4e1c37b3670d5d27bdedb5c950e4
Author: Robert Bradshaw 
Date:   2016-10-14T01:02:12Z

More complicated window tests.

commit d3b5988c78682989f1db17c09dc51157377ddd7c
Author: Robert Bradshaw 
Date:   2016-10-14T01:17:30Z

Optimize globally windowed side input case




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify #1525

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Add DoFn.StateId annotation and validation on fields

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5bfeb958d -> a0f649eac


Add DoFn.StateId annotation and validation on fields


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/add34518
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/add34518
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/add34518

Branch: refs/heads/master
Commit: add34518bbfb6668a421157e3c1cfaa119a6031b
Parents: 7322616
Author: Kenneth Knowles 
Authored: Mon Oct 10 21:16:37 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 13 19:31:20 2016 -0700

--
 .../org/apache/beam/sdk/transforms/DoFn.java|  44 ++
 .../org/apache/beam/sdk/transforms/ParDo.java   |  32 -
 .../sdk/transforms/reflect/DoFnSignature.java   |  25 
 .../sdk/transforms/reflect/DoFnSignatures.java  | 129 +++--
 .../apache/beam/sdk/transforms/ParDoTest.java   |  27 
 .../transforms/reflect/DoFnSignaturesTest.java  | 143 +++
 6 files changed, 388 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/add34518/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 62da28c..c86693b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -44,6 +44,8 @@ import 
org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.WindowingInternals;
+import org.apache.beam.sdk.util.state.State;
+import org.apache.beam.sdk.util.state.StateSpec;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
@@ -394,6 +396,48 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
 
   /
 
+  /**
+   * Annotation for declaring and dereferencing state cells.
+   *
+   * Not currently supported by any runner.
+   *
+   * To declare a state cell, create a field of type {@link StateSpec} 
annotated with a {@link
+   * StateId}. To use the cell during processing, add a parameter of the 
appropriate {@link State}
+   * subclass to your {@link ProcessElement @ProcessElement} method, and 
annotate it with {@link
+   * StateId}. See the following code for an example:
+   *
+   * {@code
+   * new DoFn, Baz>() {
+   *   @StateId("my-state-id")
+   *   private final StateSpec myStateSpec =
+   *   StateSpecs.value(new MyStateCoder());
+   *
+   *   @ProcessElement
+   *   public void processElement(
+   *   ProcessContext c,
+   *   @StateId("my-state-id") ValueState myState) {
+   * myState.read();
+   * myState.write(...);
+   *   }
+   * }
+   * }
+   *
+   * State is subject to the following validity conditions:
+   *
+   * 
+   * Each state ID must be declared at most once.
+   * Any state referenced in a parameter must be declared with the same 
state type.
+   * State declarations must be final.
+   * 
+   */
+  @Documented
+  @Retention(RetentionPolicy.RUNTIME)
+  @Target({ElementType.FIELD, ElementType.PARAMETER})
+  @Experimental(Kind.STATE)
+  public @interface StateId {
+/** The state ID. */
+String value();
+  }
 
   /**
* Annotation for the method to use to prepare an instance for processing 
bundles of elements. The

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/add34518/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index fdef908..c5a80c6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -29,6 +29,7 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import 

[jira] [Updated] (BEAM-747) Text checksum verifier is not resilient to eventually consistent filesystems

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-747:
-
Description: 
Example 1: 
https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console

Here it looks like we need to retry listing files, at least a little bit, if 
none are found. They did show up:

{code}
gsutil ls 
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
{code}

Example 2: 
https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1525/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/

Here it looks like we need to fill in the shard template if the filesystem does 
not give us a consistent result:

{code}
Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
readLines
INFO: [0 of 1] Read 162 lines from file: 
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-0-of-3
Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
readLines
INFO: [1 of 1] Read 144 lines from file: 
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-2-of-3
Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
matchesSafely
INFO: Generated checksum for output data: 
aec68948b2515e6ea35fd1ed7649c267a10a01e5
{code}

We missed shard 1-of-3 and hence got the wrong checksum.

  was:
Example 1: 
https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console

Here it looks like we need to retry listing files, at least a little bit, if 
none are found. They did show up:

{code}
gsutil ls 
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
{code}

Example 2: 


> Text checksum verifier is not resilient to eventually consistent filesystems
> 
>
> Key: BEAM-747
> URL: https://issues.apache.org/jira/browse/BEAM-747
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> Example 1: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3934/org.apache.beam$beam-examples-java/console
> Here it looks like we need to retry listing files, at least a little bit, if 
> none are found. They did show up:
> {code}
> gsutil ls 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results\*
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-0-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-1-of-3
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-13-12-37-02-467/output/results-2-of-3
> {code}
> Example 2: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1525/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> Here it looks like we need to fill in the shard template if the filesystem 
> does not give us a consistent result:
> {code}
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [0 of 1] Read 162 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-0-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> readLines
> INFO: [1 of 1] Read 144 lines from file: 
> gs://temp-storage-for-end-to-end-tests/WordCountIT-2016-10-14-00-25-55-609/output/results-2-of-3
> Oct 14, 2016 12:31:16 AM org.apache.beam.sdk.testing.FileChecksumMatcher 
> matchesSafely
> INFO: Generated checksum for output data: 
> aec68948b2515e6ea35fd1ed7649c267a10a01e5
> {code}
> We missed shard 1-of-3 and hence got the wrong checksum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build became unstable: beam_PostCommit_MavenVerify #1525

[Dan Halperin]

Filed https://issues.apache.org/jira/browse/BEAM-747

On Thu, Oct 13, 2016 at 5:33 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
>

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574331#comment-15574331
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1086


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1086: [BEAM-25] Add DoFn.StateId annotation and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1086


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574335#comment-15574335
 ] 

Daniel Halperin edited comment on BEAM-697 at 10/14/16 5:55 AM:


Thanks for this. Mirroring from discussion on that PR, prefer to separate out 
this (awesome, and welcome) fix as its own PR.


was (Author: dhalp...@google.com):
Thanks for this. Mirroring from discussion on that PR, prefer to separate out 
this (awesome, welcome) fix as its own PR.

> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574335#comment-15574335
 ] 

Daniel Halperin commented on BEAM-697:
--

Thanks for this. Mirroring from discussion on that PR, prefer to separate out 
this (awesome, welcome) fix as its own PR.

> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build became unstable: beam_PostCommit_MavenVerify #1525

[Jean-Baptiste Onofré]

Thx !

Regards
JB

On 10/14/2016 07:52 AM, Dan Halperin wrote:

Filed https://issues.apache.org/jira/browse/BEAM-747

On Thu, Oct 13, 2016 at 5:33 PM, Apache Jenkins Server
> wrote:

See >




--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[GitHub] incubator-beam pull request #908: Fix inconsistent in formatting flink logs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/908


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix inconsistent in formatting logs: leaveCompositeTransform always decrement depth, but enterCompositeTransform increment depth only on ENTER_TRANSFORM

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 13b45895e -> 73226168a


Fix inconsistent in formatting logs: leaveCompositeTransform always decrement 
depth, but enterCompositeTransform increment depth only on ENTER_TRANSFORM


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cea201ea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cea201ea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cea201ea

Branch: refs/heads/master
Commit: cea201eaaea24d8cc1e117645d1c81f379beeb41
Parents: 98da6e8
Author: Alexey Diomin 
Authored: Wed Aug 31 18:17:01 2016 +0400
Committer: Alexey Diomin 
Committed: Wed Aug 31 18:17:54 2016 +0400

--
 .../runners/flink/translation/FlinkBatchPipelineTranslator.java| 2 +-
 .../flink/translation/FlinkStreamingPipelineTranslator.java| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cea201ea/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
index 66c48b0..1cb604f 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
@@ -65,6 +65,7 @@ public class FlinkBatchPipelineTranslator extends 
FlinkPipelineTranslator {
   @Override
   public CompositeBehavior enterCompositeTransform(TransformTreeNode node) {
 LOG.info(genSpaces(this.depth) + "enterCompositeTransform- " + 
formatNodeName(node));
+this.depth++;
 
 BatchTransformTranslator translator = getTranslator(node);
 
@@ -73,7 +74,6 @@ public class FlinkBatchPipelineTranslator extends 
FlinkPipelineTranslator {
   LOG.info(genSpaces(this.depth) + "translated-" + formatNodeName(node));
   return CompositeBehavior.DO_NOT_ENTER_TRANSFORM;
 } else {
-  this.depth++;
   return CompositeBehavior.ENTER_TRANSFORM;
 }
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cea201ea/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingPipelineTranslator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingPipelineTranslator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingPipelineTranslator.java
index 284cd23..e5c0d76 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingPipelineTranslator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingPipelineTranslator.java
@@ -52,6 +52,7 @@ public class FlinkStreamingPipelineTranslator extends 
FlinkPipelineTranslator {
   @Override
   public CompositeBehavior enterCompositeTransform(TransformTreeNode node) {
 LOG.info(genSpaces(this.depth) + "enterCompositeTransform- " + 
formatNodeName(node));
+this.depth++;
 
 PTransform transform = node.getTransform();
 if (transform != null) {
@@ -64,7 +65,6 @@ public class FlinkStreamingPipelineTranslator extends 
FlinkPipelineTranslator {
 return CompositeBehavior.DO_NOT_ENTER_TRANSFORM;
   }
 }
-this.depth++;
 return CompositeBehavior.ENTER_TRANSFORM;
   }
 

[2/2] incubator-beam git commit: This closes #908

[mxm]

This closes #908


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/73226168
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/73226168
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/73226168

Branch: refs/heads/master
Commit: 73226168a436f88bf650e36b45434c2dbe399ae2
Parents: 13b4589 cea201e
Author: Maximilian Michels 
Authored: Thu Oct 13 10:43:30 2016 +0200
Committer: Maximilian Michels 
Committed: Thu Oct 13 10:43:30 2016 +0200

--
 .../runners/flink/translation/FlinkBatchPipelineTranslator.java| 2 +-
 .../flink/translation/FlinkStreamingPipelineTranslator.java| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/73226168/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchPipelineTranslator.java
--

[jira] [Commented] (BEAM-593) Support cancel() and waitUntilFinish() in FlinkRunnerResult

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571724#comment-15571724
 ] 

ASF GitHub Bot commented on BEAM-593:
-

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/1093

[BEAM-593] avoid throwing Exception in waitUntilFinish

The current implementation of Flink's `PipelineResult` assumes that the 
pipeline has already been processed. Hence, we can return State.Done when 
`wailUntilFinish()` is called.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-593

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1093.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1093


commit c4b78841a82951b5f7c3e1c7763e078317bb3a2f
Author: Maximilian Michels 
Date:   2016-10-13T12:01:06Z

[BEAM-593] avoid throwing Exception in waitUntilFinish

The current implementation of Flink's PipelineResult assumes that the
pipeline has already been processed. Hence, we can return State.Done
when wailUntilFinished is called.




> Support cancel() and waitUntilFinish() in FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1093: [BEAM-593] avoid throwing Exception in wa...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/1093

[BEAM-593] avoid throwing Exception in waitUntilFinish

The current implementation of Flink's `PipelineResult` assumes that the 
pipeline has already been processed. Hence, we can return State.Done when 
`wailUntilFinish()` is called.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-593

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1093.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1093


commit c4b78841a82951b5f7c3e1c7763e078317bb3a2f
Author: Maximilian Michels 
Date:   2016-10-13T12:01:06Z

[BEAM-593] avoid throwing Exception in waitUntilFinish

The current implementation of Flink's PipelineResult assumes that the
pipeline has already been processed. Hence, we can return State.Done
when wailUntilFinished is called.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---