[jira] [Updated] (BEAM-651) Making TypedPValue.setTypeDescriptorInternal no longer Internal

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-651:
-
Summary: Making TypedPValue.setTypeDescriptorInternal no longer Internal  
(was: Consider making TypedPValue.setTypeDescriptorInternal no longer Internal)

> Making TypedPValue.setTypeDescriptorInternal no longer Internal
> ---
>
> Key: BEAM-651
> URL: https://issues.apache.org/jira/browse/BEAM-651
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 0.4.0-incubating
>
>
> This would give fairly pithy answers to StackOverflow questions sometimes.
> When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a 
> transform/DoFn we often choose the type, so the coder registry can do its 
> thing.
> This would also give a similar choice between .setCoder(...) and 
> .setTypeDescriptor(...).
> And anyhow we have the intention of removing our practice of the "*Internal" 
> suffix, so this one might be most easily solved by making it public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-651) Add public TypedPValue.setTypeDescriptor

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-651:
-
Summary: Add public TypedPValue.setTypeDescriptor  (was: Making 
TypedPValue.setTypeDescriptorInternal no longer Internal)

> Add public TypedPValue.setTypeDescriptor
> 
>
> Key: BEAM-651
> URL: https://issues.apache.org/jira/browse/BEAM-651
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 0.4.0-incubating
>
>
> This would give fairly pithy answers to StackOverflow questions sometimes.
> When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a 
> transform/DoFn we often choose the type, so the coder registry can do its 
> thing.
> This would also give a similar choice between .setCoder(...) and 
> .setTypeDescriptor(...).
> And anyhow we have the intention of removing our practice of the "*Internal" 
> suffix, so this one might be most easily solved by making it public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-35) Remove timer multiplexing and give timers identifiers

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-35:

Issue Type: Improvement  (was: New Feature)

> Remove timer multiplexing and give timers identifiers
> -
>
> Key: BEAM-35
> URL: https://issues.apache.org/jira/browse/BEAM-35
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 0.4.0-incubating
>
>
> Today, timers set by the runner core of the SDK are identified by timestamp, 
> so multiple timers for the same time result in only one backend timer. This 
> is runner-specific and obsolete, and makes it unsafe to support timer 
> deletion because backend timers have no real owner. User-facing timers will 
> require explicit identifiers anyhow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-864) RAT check fails on DEPENDENCIES

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-864:
-
Summary: RAT check fails on DEPENDENCIES  (was: Update to latest Apache 
Maven-Parent)

> RAT check fails on DEPENDENCIES
> ---
>
> Key: BEAM-864
> URL: https://issues.apache.org/jira/browse/BEAM-864
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Aljoscha Krettek
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> The release plugin creates a DEPENDENCIES file that is not properly excluded 
> from the rat check with the current version of the Apache Maven-Parent that 
> we are using.
> This is the relevant RAT Jira issue: 
> https://issues.apache.org/jira/browse/RAT-184



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-808) Increase "spark.port.maxRetries" to avoid BindException in ROS.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-808:
-
Issue Type: Bug  (was: Improvement)

> Increase "spark.port.maxRetries" to avoid BindException in ROS.
> ---
>
> Key: BEAM-808
> URL: https://issues.apache.org/jira/browse/BEAM-808
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.4.0-incubating
>
>
> The default is 16, and it's hard to know which port failed to bind since 
> there's no logging.
> Let's start with 64.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-830) Launcher for ApexRunner execution on YARN cluster

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-830:
-
Issue Type: New Feature  (was: Improvement)

> Launcher for ApexRunner execution on YARN cluster 
> --
>
> Key: BEAM-830
> URL: https://issues.apache.org/jira/browse/BEAM-830
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
> Fix For: 0.4.0-incubating
>
>
> Currently the ApexRunner only support execution in embedded mode. Add the 
> support to package the dependencies and run the Apex app on a YARN cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1060) Make DoFnTester use new DoFn

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1060:
--
Issue Type: New Feature  (was: Improvement)

> Make DoFnTester use new DoFn
> 
>
> Key: BEAM-1060
> URL: https://issues.apache.org/jira/browse/BEAM-1060
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-252) Make Regex Transform

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-252:
-
Issue Type: New Feature  (was: Improvement)

> Make Regex Transform
> 
>
> Key: BEAM-252
> URL: https://issues.apache.org/jira/browse/BEAM-252
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
> Fix For: 0.4.0-incubating
>
>
> There needs to be an easier way to run Regular Expressions as part of a 
> transform. This will make string-based ETL much easier.
> The transform should support using the matches and find methods. The 
> transform should allow you to choose a group in the regex to output. The 
> transform should allow single strings to be output or KV's of strings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1060) Make DoFnTester use new DoFn

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1060:
--
Issue Type: Improvement  (was: New Feature)

> Make DoFnTester use new DoFn
> 
>
> Key: BEAM-1060
> URL: https://issues.apache.org/jira/browse/BEAM-1060
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-939) New credentials code broke Dataflow runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-939:
-
Issue Type: Bug  (was: New Feature)

> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-651) Consider making TypedPValue.setTypeDescriptorInternal no longer Internal

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-651:
-
Issue Type: New Feature  (was: Wish)

> Consider making TypedPValue.setTypeDescriptorInternal no longer Internal
> 
>
> Key: BEAM-651
> URL: https://issues.apache.org/jira/browse/BEAM-651
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 0.4.0-incubating
>
>
> This would give fairly pithy answers to StackOverflow questions sometimes.
> When choosing between .getOutputCoder() and .getOutputTypeDescriptor() for a 
> transform/DoFn we often choose the type, so the coder registry can do its 
> thing.
> This would also give a similar choice between .setCoder(...) and 
> .setTypeDescriptor(...).
> And anyhow we have the intention of removing our practice of the "*Internal" 
> suffix, so this one might be most easily solved by making it public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-947) WindowedWordCountIT fails due to not setting "--output" pipeline options

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-947:
-
Issue Type: Bug  (was: New Feature)

> WindowedWordCountIT fails due to not setting "--output" pipeline options
> 
>
> Key: BEAM-947
> URL: https://issues.apache.org/jira/browse/BEAM-947
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.4.0-incubating
>
>
> Example failure: 
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4793/org.apache.beam$beam-examples-java/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInStreaming/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-261) Apache Apex Runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-261:
-
Issue Type: New Feature  (was: Wish)

> Apache Apex Runner
> --
>
> Key: BEAM-261
> URL: https://issues.apache.org/jira/browse/BEAM-261
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-apex
>Reporter: Suminda Dharmasena
>Assignee: Thomas Weise
> Fix For: 0.4.0-incubating
>
>
> Like Spark, Flink and GearPump, Apache Apex also does have advantages. Is it 
> possible to have a runner for Apache Apex?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1176) Make our test suites use @Rule TestPipeline

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1176:
-

 Summary: Make our test suites use @Rule TestPipeline
 Key: BEAM-1176
 URL: https://issues.apache.org/jira/browse/BEAM-1176
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Stas Levin
Priority: Minor


Now that [~staslev] has made {{TestPipeline}} a JUnit rule that performs useful 
sanity checks, we should port all of our tests to it so that they set a good 
example for users. Maybe we'll even catch some straggling tests with errors :-)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-85) PAssert needs sanity check that it's used correctly

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-85?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-85.
-
   Resolution: Fixed
Fix Version/s: 0.5.0-incubating

When used as a `@Rule` this is now the case. Now we need to port the tests, 
which I'd say fits more into BEAM0-298 or potentially a new ticket.

> PAssert needs sanity check that it's used correctly
> ---
>
> Key: BEAM-85
> URL: https://issues.apache.org/jira/browse/BEAM-85
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Stas Levin
> Fix For: 0.5.0-incubating
>
>
> We should validate two things:
> # DataflowAssert is not added to a pipeline that has already been run.
> # The pipeline is run after the DataflowAssert is added.
> If either of these are not validated, then it is possible that the test 
> doesn't actually verify anything.
> This code should throw an assertion error or fail in some other way.
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> p.run();
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}
> but it would pass silently.
> similarly, this code wills pass silently:
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1595: [BEAM-85,BEAM-298] Introducing TestPipeli...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1595


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #1595, makes TestPipeline a JUnit rule

[kenn]

This closes #1595, makes TestPipeline a JUnit rule

  Empty TestPipeline need not be run
  [BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1c9bf8d6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1c9bf8d6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1c9bf8d6

Branch: refs/heads/master
Commit: 1c9bf8d665979961478160c384b57ae42847e7b1
Parents: c28957d 0d54a4c
Author: Kenneth Knowles 
Authored: Sat Dec 17 14:12:37 2016 -0800
Committer: Kenneth Knowles 
Committed: Sat Dec 17 14:12:37 2016 -0800

--
 .../apache/beam/sdk/testing/TestPipeline.java   | 225 ---
 .../beam/sdk/testing/TestPipelineTest.java  | 186 +--
 2 files changed, 365 insertions(+), 46 deletions(-)
--

[1/3] incubator-beam git commit: [BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c28957d16 -> 1c9bf8d66


[BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/443b25a4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/443b25a4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/443b25a4

Branch: refs/heads/master
Commit: 443b25a4d11201fb88f40da437ec7aab4b3e273f
Parents: 33b7ca7
Author: Stas Levin 
Authored: Tue Dec 13 19:27:41 2016 +0200
Committer: Kenneth Knowles 
Committed: Sat Dec 17 14:11:39 2016 -0800

--
 .../apache/beam/sdk/testing/TestPipeline.java   | 207 ---
 .../beam/sdk/testing/TestPipelineTest.java  | 183 ++--
 2 files changed, 344 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/443b25a4/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index 493d4cc..49ac3af 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -23,12 +23,17 @@ import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.ObjectNode;
 import com.google.common.base.Optional;
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
 import com.google.common.base.Strings;
+import com.google.common.collect.FluentIterable;
 import com.google.common.collect.Iterators;
 import java.io.IOException;
 import java.lang.reflect.Method;
 import java.util.ArrayList;
 import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
 import java.util.Map.Entry;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.Pipeline;
@@ -39,34 +44,39 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.runners.PipelineRunner;
+import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.util.TestCredential;
 import org.junit.experimental.categories.Category;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
 
 /**
- * A creator of test pipelines that can be used inside of tests that can be
- * configured to run locally or against a remote pipeline runner.
+ * A creator of test pipelines that can be used inside of tests that can be 
configured to run
+ * locally or against a remote pipeline runner.
  *
- * It is recommended to tag hand-selected tests for this purpose using the
- * {@link RunnableOnService} {@link Category} annotation, as each test run 
against a pipeline runner
- * will utilize resources of that pipeline runner.
+ * It is recommended to tag hand-selected tests for this purpose using the 
{@link
+ * RunnableOnService} {@link Category} annotation, as each test run against a 
pipeline runner will
+ * utilize resources of that pipeline runner.
  *
  * In order to run tests on a pipeline runner, the following conditions 
must be met:
+ *
  * 
- *   System property "beamTestPipelineOptions" must contain a JSON 
delimited list of pipeline
- *   options. For example:
- *   {@code [
+ * System property "beamTestPipelineOptions" must contain a JSON delimited 
list of pipeline
+ * options. For example:
+ * {@code [
  * "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner",
  * "--project=mygcpproject",
  * "--stagingLocation=gs://mygcsbucket/path"
  * ]}
  * Note that the set of pipeline options required is pipeline runner 
specific.
- *   
- *   Jars containing the SDK and test classes must be available on the 
classpath.
+ * Jars containing the SDK and test classes must be available on the 
classpath.
  * 
  *
  * Use {@link PAssert} for tests, as it integrates with this test harness 
in both direct and
  * remote execution modes. For example:
+ *
  * {@code
  * Pipeline p = TestPipeline.create();
  * PCollection output = ...
@@ -76,19 +86,136 @@ import org.junit.experimental.categories.Category;
  * p.run();
  * }
  *
- * For pipeline runners, it is required that they must throw an {@link 
AssertionError}
- * containing the message from the {@link PAssert} that failed.
+ * For pipeline 

[jira] [Commented] (BEAM-85) PAssert needs sanity check that it's used correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15757718#comment-15757718
 ] 

ASF GitHub Bot commented on BEAM-85:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1595


> PAssert needs sanity check that it's used correctly
> ---
>
> Key: BEAM-85
> URL: https://issues.apache.org/jira/browse/BEAM-85
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Stas Levin
>
> We should validate two things:
> # DataflowAssert is not added to a pipeline that has already been run.
> # The pipeline is run after the DataflowAssert is added.
> If either of these are not validated, then it is possible that the test 
> doesn't actually verify anything.
> This code should throw an assertion error or fail in some other way.
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> p.run();
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}
> but it would pass silently.
> similarly, this code wills pass silently:
> {code}
> Pipeline p = TestPipeline.create();
> PCollection value = p.apply(Create.of(Boolean.FALSE));
> DataflowAssert.thatSingleton(value).isEqualTo(true);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] incubator-beam git commit: Empty TestPipeline need not be run

[kenn]

Empty TestPipeline need not be run


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0d54a4c6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0d54a4c6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0d54a4c6

Branch: refs/heads/master
Commit: 0d54a4c6708516170cb045b5e0d1d33443973c6e
Parents: 443b25a
Author: Kenneth Knowles 
Authored: Sat Dec 17 13:57:07 2016 -0800
Committer: Kenneth Knowles 
Committed: Sat Dec 17 14:12:23 2016 -0800

--
 .../apache/beam/sdk/testing/TestPipeline.java   | 20 +++-
 .../beam/sdk/testing/TestPipelineTest.java  |  3 +++
 2 files changed, 22 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0d54a4c6/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index 49ac3af..b707a81 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -169,7 +169,12 @@ public class TestPipeline extends Pipeline implements 
TestRule {
   throw new AbandonedNodeException("The pipeline contains abandoned 
PTransform(s).");
 }
   } else if (runVisitedNodes == null && !enableAutoRunIfMissing) {
-throw new PipelineRunMissingException("The pipeline has not been 
run.");
+IsEmptyVisitor isEmptyVisitor = new IsEmptyVisitor();
+pipeline.traverseTopologically(isEmptyVisitor);
+
+if (!isEmptyVisitor.isEmpty()) {
+  throw new PipelineRunMissingException("The pipeline has not been 
run.");
+}
   }
 }
 
@@ -389,4 +394,17 @@ public class TestPipeline extends Pipeline implements 
TestRule {
 }
 return firstInstanceAfterTestPipeline;
   }
+
+  private static class IsEmptyVisitor extends PipelineVisitor.Defaults {
+private boolean empty = true;
+
+public boolean isEmpty() {
+  return empty;
+}
+
+@Override
+public void visitPrimitiveTransform(TransformHierarchy.Node node) {
+  empty = false;
+}
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0d54a4c6/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
index d1797e7..f484566 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
@@ -74,6 +74,9 @@ public class TestPipelineTest implements Serializable {
   @Rule public transient RuleChain ruleOrder = 
RuleChain.outerRule(exception).around(pipeline);
 
   @Test
+  public void testNoTestPipelineUsed() { }
+
+  @Test
   public void testCreationUsingDefaults() {
 assertNotNull(TestPipeline.create());
   }

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15757625#comment-15757625
 ] 

ASF GitHub Bot commented on BEAM-27:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1612


> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/5] incubator-beam git commit: Make TimerSpec and StateSpec fields accessible

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5255a3381 -> c28957d16


Make TimerSpec and StateSpec fields accessible


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c1e1017d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c1e1017d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c1e1017d

Branch: refs/heads/master
Commit: c1e1017d6e4d75aee7f32cc3d08b9e2a7c21dbb2
Parents: ffe3ab3
Author: Kenneth Knowles 
Authored: Wed Dec 7 20:09:06 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 20:10:37 2016 -0800

--
 .../org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java   | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c1e1017d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index e3ba966..d72cea4 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
@@ -1040,6 +1040,8 @@ public class DoFnSignatures {
   ErrorReporter errors, Class fnClazz) {
 Map declarations = new HashMap<>();
 for (Field field : declaredFieldsWithAnnotation(DoFn.TimerId.class, 
fnClazz, DoFn.class)) {
+  // TimerSpec fields may generally be private, but will be accessed via 
the signature
+  field.setAccessible(true);
   String id = field.getAnnotation(DoFn.TimerId.class).value();
   validateTimerField(errors, declarations, id, field);
   declarations.put(id, DoFnSignature.TimerDeclaration.create(id, field));
@@ -1205,6 +1207,8 @@ public class DoFnSignatures {
 Map declarations = new HashMap<>();
 
 for (Field field : declaredFieldsWithAnnotation(DoFn.StateId.class, 
fnClazz, DoFn.class)) {
+  // StateSpec fields may generally be private, but will be accessed via 
the signature
+  field.setAccessible(true);
   String id = field.getAnnotation(DoFn.StateId.class).value();
 
   if (declarations.containsKey(id)) {

[3/5] incubator-beam git commit: Add timer support to DoFnRunner(s)

[kenn]

Add timer support to DoFnRunner(s)


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8af13b01
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8af13b01
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8af13b01

Branch: refs/heads/master
Commit: 8af13b0102cda6c68601efa4119723900d12ca5c
Parents: c1e1017
Author: Kenneth Knowles 
Authored: Wed Nov 23 14:21:40 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 20:14:19 2016 -0800

--
 .../apache/beam/runners/core/DoFnRunner.java|   9 +
 .../core/LateDataDroppingDoFnRunner.java|   7 +
 .../core/PushbackSideInputDoFnRunner.java   |   8 +
 .../beam/runners/core/SimpleDoFnRunner.java | 236 +-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   8 +
 .../core/PushbackSideInputDoFnRunnerTest.java   |  41 +++
 .../beam/runners/core/SimpleDoFnRunnerTest.java | 247 +++
 7 files changed, 555 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8af13b01/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
index 501667e..7c73a34 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunner.java
@@ -20,8 +20,11 @@ package org.apache.beam.runners.core;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
+import org.joda.time.Instant;
 
 /**
  * An wrapper interface that represents the execution of a {@link DoFn}.
@@ -39,6 +42,12 @@ public interface DoFnRunner {
   void processElement(WindowedValue elem);
 
   /**
+   * Calls a {@link DoFn DoFn's} {@link DoFn.OnTimer @OnTimer} method for the 
given timer
+   * in the given window.
+   */
+  void onTimer(String timerId, BoundedWindow window, Instant timestamp, 
TimeDomain timeDomain);
+
+  /**
* Calls a {@link DoFn DoFn's} {@link DoFn.FinishBundle @FinishBundle} 
method and performs
* additional tasks, such as flushing in-memory states.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8af13b01/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
index 9bfe9ae..290171a 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
@@ -24,6 +24,7 @@ import com.google.common.collect.Iterables;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowTracing;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -73,6 +74,12 @@ public class LateDataDroppingDoFnRunner

[GitHub] incubator-beam pull request #1612: [BEAM-27] Support timer setting and recei...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1612


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/5] incubator-beam git commit: Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle

[kenn]

Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b4c7d10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3b4c7d10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3b4c7d10

Branch: refs/heads/master
Commit: 3b4c7d103c07e73d30b2ad534a17b3059232dbda
Parents: 8af13b0
Author: Kenneth Knowles 
Authored: Fri Dec 16 13:43:54 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 20:14:19 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunnerTest.java | 53 
 1 file changed, 53 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3b4c7d10/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
index f068c19..837a162 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
@@ -151,6 +151,49 @@ public class SimpleDoFnRunnerTest {
 TimeDomain.EVENT_TIME);
   }
 
+  @Test
+  public void testStartBundleExceptionsWrappedAsUserCodeException() {
+ThrowingDoFn fn = new ThrowingDoFn();
+DoFnRunner runner =
+new SimpleDoFnRunner<>(
+null,
+fn,
+null,
+null,
+null,
+Collections.emptyList(),
+mockStepContext,
+null,
+WindowingStrategy.of(new GlobalWindows()));
+
+thrown.expect(UserCodeException.class);
+thrown.expectCause(is(fn.exceptionToThrow));
+
+runner.startBundle();
+  }
+
+  @Test
+  public void testFinishBundleExceptionsWrappedAsUserCodeException() {
+ThrowingDoFn fn = new ThrowingDoFn();
+DoFnRunner runner =
+new SimpleDoFnRunner<>(
+null,
+fn,
+null,
+null,
+null,
+Collections.emptyList(),
+mockStepContext,
+null,
+WindowingStrategy.of(new GlobalWindows()));
+
+thrown.expect(UserCodeException.class);
+thrown.expectCause(is(fn.exceptionToThrow));
+
+runner.finishBundle();
+  }
+
+
   /**
* Tests that {@link SimpleDoFnRunner#onTimer} properly dispatches to the 
underlying
* {@link DoFn}.
@@ -200,6 +243,16 @@ public class SimpleDoFnRunnerTest {
 @TimerId(TIMER_ID)
 private static final TimerSpec timer = 
TimerSpecs.timer(TimeDomain.EVENT_TIME);
 
+@StartBundle
+public void startBundle(Context c) throws Exception {
+  throw exceptionToThrow;
+}
+
+@FinishBundle
+public void finishBundle(Context c) throws Exception {
+  throw exceptionToThrow;
+}
+
 @ProcessElement
 public void processElement(ProcessContext c) throws Exception {
   throw exceptionToThrow;

[4/5] incubator-beam git commit: Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest

[kenn]

Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b78aa669
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b78aa669
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b78aa669

Branch: refs/heads/master
Commit: b78aa669831154f82266eb12ab795442c02f8977
Parents: 3b4c7d1
Author: Kenneth Knowles 
Authored: Fri Dec 16 20:57:06 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 20:57:06 2016 -0800

--
 .../apache/beam/runners/core/SimpleDoFnRunnerTest.java | 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b78aa669/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
index 837a162..ec5d375 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
@@ -33,6 +33,7 @@ import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.BaseExecutionContext.StepContext;
+import org.apache.beam.sdk.util.NullSideInputReader;
 import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.TimerInternals;
@@ -77,7 +78,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),
@@ -98,7 +99,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),
@@ -129,7 +130,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),
@@ -158,7 +159,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),
@@ -179,7 +180,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),
@@ -206,7 +207,7 @@ public class SimpleDoFnRunnerTest {
 new SimpleDoFnRunner<>(
 null,
 fn,
-null,
+NullSideInputReader.empty(),
 null,
 null,
 Collections.emptyList(),

[5/5] incubator-beam git commit: This closes #1612: [BEAM-27] Support timer setting and receiving in SimpleDoFnRunner

[kenn]

This closes #1612: [BEAM-27] Support timer setting and receiving in 
SimpleDoFnRunner

  Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest
  Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle
  Add timer support to DoFnRunner(s)
  Make TimerSpec and StateSpec fields accessible


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c28957d1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c28957d1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c28957d1

Branch: refs/heads/master
Commit: c28957d16fb0f63f82f578cf904df61bf7bb63e5
Parents: 5255a33 b78aa66
Author: Kenneth Knowles 
Authored: Sat Dec 17 12:59:25 2016 -0800
Committer: Kenneth Knowles 
Committed: Sat Dec 17 12:59:25 2016 -0800

--
 .../apache/beam/runners/core/DoFnRunner.java|   9 +
 .../core/LateDataDroppingDoFnRunner.java|   7 +
 .../core/PushbackSideInputDoFnRunner.java   |   8 +
 .../beam/runners/core/SimpleDoFnRunner.java | 236 ++-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   8 +
 .../core/PushbackSideInputDoFnRunnerTest.java   |  41 +++
 .../beam/runners/core/SimpleDoFnRunnerTest.java | 301 +++
 .../sdk/transforms/reflect/DoFnSignatures.java  |   4 +
 8 files changed, 613 insertions(+), 1 deletion(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2134

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-853) Force streaming execution on batch pipelines for testing.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-853.

   Resolution: Fixed
Fix Version/s: 0.5.0-incubating

> Force streaming execution on batch pipelines for testing.
> -
>
> Key: BEAM-853
> URL: https://issues.apache.org/jira/browse/BEAM-853
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0-incubating
>
>
> The SDK's {{streaming}} tests actually use a 
> {{BoundedReadFromUnboundedSource}} while "forcing" a streaming pipeline 
> (although the input is {{Read.Bounded}}).
> The {{TestSparkRunner}} should be able to use the underlying 
> {{UnboundedSource}} to execute a streaming pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1846

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1846

[Apache Jenkins Server]

See