Updated per PR comments

Signed-off-by: Jason Kuster <ja...@google.com>


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ec1ef53b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ec1ef53b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ec1ef53b

Branch: refs/heads/asf-site
Commit: ec1ef53b8bbc20bcf5f61b6985468b15f376a3a7
Parents: 6ad7ff0
Author: Jason Kuster <ja...@google.com>
Authored: Tue Sep 20 17:11:37 2016 -0700
Committer: Jason Kuster <ja...@google.com>
Committed: Tue Sep 20 17:11:37 2016 -0700

----------------------------------------------------------------------
 contribute/testing.md | 228 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 190 insertions(+), 38 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ec1ef53b/contribute/testing.md
----------------------------------------------------------------------
diff --git a/contribute/testing.md b/contribute/testing.md
index 21d1e87..d8891bd 100644
--- a/contribute/testing.md
+++ b/contribute/testing.md
@@ -1,11 +1,27 @@
+---
+layout: default
+title: 'Beam Testing Guide'
+redirect_from: '/docs/testing'
+---
+
 # Beam Testing Documentation
 
-# Overview
+## Overview
 
-Apache Beam (incubating) is a rapidly-maturing software project with a strong 
commitment to testing. Consequently, it has many testing-related needs. It 
requires precommit tests to ensure code going into the repository meets a 
certain quality bar and it requires ongoing postcommit tests to make sure that 
more subtle changes which escape precommit are nonetheless caught. This 
document outlines how to write tests, which tests are appropriate where, and 
when tests are run, with some additional information about the testing systems 
at the bottom.
+Apache Beam (incubating) is a rapidly-maturing software project with a strong
+commitment to testing. Consequently, it has many testing-related needs. It
+requires precommit tests to ensure code going into the repository meets a
+certain quality bar and it requires ongoing postcommit tests to make sure that
+more subtle changes which escape precommit are nonetheless caught. This 
document
+outlines how to write tests, which tests are appropriate where, and when tests
+are run, with some additional information about the testing systems at the
+bottom.
 
-## Beam Java Testing Matrix
+If you’re writing tests, take a look at the testing matrix first, find what 
you
+want to test, then look into the “Scenarios” and “Types” sections 
below for more
+details on those testing types.
 
+## Beam Java Testing Matrix
 
 <table>
   <tr>
@@ -17,6 +33,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><strong>Link to Example</strong>
    </td>
+   <td><strong>Type</strong>
+   </td>
+   <td><strong>Runs In</strong>
+   </td>
   </tr>
   <tr>
    <td>DoFn
@@ -27,6 +47,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java#L1325";>BigQueryIOTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit, Postcommit
+   </td>
   </tr>
   <tr>
    <td>BoundedSource
@@ -37,6 +61,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L972";>TextIOTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit, Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -47,6 +75,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/8b1e64a668489297e11926124c4eee6c8f69a3a7/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java#L339";>BigtableTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit, Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -57,6 +89,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java#L1021";>TextIOTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit, Postcommit
+   </td>
   </tr>
   <tr>
    <td>Transform
@@ -67,6 +103,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java#L1199";>ParDoTest</a>
    </td>
+   <td>@NeedsRunner
+   </td>
+   <td>
+   </td>
   </tr>
   <tr>
    <td>Example Pipeline
@@ -77,6 +117,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L76";>WordCountIT</a>
    </td>
+   <td>E2E
+   </td>
+   <td>Postcommit (Except WordCountIT)
+   </td>
   </tr>
   <tr>
    <td>Source/Sink with external resource
@@ -87,6 +131,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java#L646";>FakeBigtableService
 in BigtableTest</a>
    </td>
+   <td>Unit / @NeedsRunner
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -97,6 +145,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/84a0dd1714028370befa80dea16f720edce05252/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadIT.java#L40";>BigtableReadIT</a>
    </td>
+   <td>E2E
+   </td>
+   <td>Postcommit
+   </td>
   </tr>
   <tr>
    <td>Runner
@@ -107,6 +159,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java#L78";>WordCountIT</a>,
 <a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java";>ParDoTest</a>
    </td>
+   <td>E2E, @RunnableonService
+   </td>
+   <td>Postcommit
+   </td>
   </tr>
   <tr>
    <td>Triggers
@@ -117,6 +173,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java";>AfterWatermarkTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>Coders
@@ -127,6 +187,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java";>NullableCoderTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -137,6 +201,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java";>NullableCoderTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -147,6 +215,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java";>NullableCoderTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -157,6 +229,10 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java";>NullableCoderTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
   <tr>
    <td>
@@ -167,64 +243,140 @@ Apache Beam (incubating) is a rapidly-maturing software 
project with a strong co
    </td>
    <td><a 
href="https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java";>NullableCoderTest</a>
    </td>
+   <td>Unit
+   </td>
+   <td>Precommit / Postcommit
+   </td>
   </tr>
 </table>
 
-
 ## Beam Python Testing Matrix
 
-Coming soon...
-
-# Testing Scenarios
-
-By using the above utilities, we have a good set of tools which we can use to 
verify Beam correctness. To ensure an ongoing high quality of code, we use 
precommit and postcommit testing.
-
-## Precommit
-
-For precommit testing, Beam uses Jenkins, Travis, and a code coverage tool 
called Coveralls, hooked up to Github, to ensure that pull requests meet a 
certain quality bar. These precommits verify correctness via two of the above 
testing tools: unit tests (with coverage monitored by Coveralls) and E2E tests. 
We run the full slate of unit tests in precommit, ensuring correctness at a 
basic level, and then run the WordCount E2E test in both batch and streaming 
(coming soon!) against each supported SDK / runner combination as a smoke test, 
to verify that a basic level of functionality exists. We think that this hits 
the appropriate tradeoff between a desire for short (ideally less than 30m) 
precommit times and a desire to verify that pull requests going into Beam 
function in the way in which they are intended.
-
-Precommit tests are kicked off when a user makes a Pull Request against the 
apache/incubator-beam repository and the Travis, Jenkins, and Coveralls 
statuses are displayed at the bottom of the pull request page. Clicking on 
“Details” will open the status page in the selected tool; there, test 
status and output can be viewed.
-
-## Postcommit
-
-Running in postcommit removes as stringent of a time constraint, which gives 
us the ability to do some more comprehensive testing. In postcommit we have a 
test suite running the RunnableOnService tests against each supported runner, 
and another for running the full set of E2E tests against each runner. 
Currently-supported runners are Dataflow, Flink, Spark, and Gearpump, with 
others soon to follow. Work is ongoing to enable Flink, Spark, and Gearpump in 
the E2E framework, with full support targeted for end of August 2016. 
Postcommit tests run periodically, with timing defined in their Jenkins 
configurations.
-
-Adding new postcommit E2E tests is generally as easy as adding a *IT.java file 
to the repository - Failsafe will notice it and run it - but if you want to do 
more interesting things, take a look at 
[WordCountIT.java](https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java).
-
-# Testing Types
+Beam Python Testing Matrix coming soon. Python is currently tested in 
postcommit
+by a Jenkins build; precommit testing will be coming soon.
+
+## Testing Scenarios
+
+With the tools at our disposal, we have a good set of utilities which we can 
use
+to verify Beam correctness. To ensure an ongoing high quality of code, we use
+precommit and postcommit testing.
+
+### Precommit
+
+For precommit testing, Beam uses
+[Jenkins](https://builds.apache.org/view/Beam/),
+[Travis](http://travis-ci.org/apache/incubator-beam), and a code coverage tool
+called [Coveralls](https://coveralls.io/github/apache/incubator-beam), hooked 
up
+to [Github](https://github.com/apache/incubator-beam), to ensure that pull
+requests meet a certain quality bar. These precommits verify correctness via 
two
+of the below testing tools: unit tests (with coverage monitored by Coveralls)
+and E2E tests. We run the full slate of unit tests in precommit, ensuring
+correctness at a basic level, and then run the WordCount E2E test in both batch
+and streaming (coming soon!) against each supported SDK / runner combination as
+a smoke test, to verify that a basic level of functionality exists. We think
+that this hits the appropriate tradeoff between a desire for short (ideally
+\<30m) precommit times and a desire to verify that pull requests going into 
Beam
+function in the way in which they are intended.
+
+Precommit tests are kicked off when a user makes a Pull Request against the
+apache/incubator-beam repository and the Travis, Jenkins, and Coveralls 
statuses
+are displayed at the bottom of the pull request page. Clicking on 
“Details” will
+open the status page in the selected tool; there, test status and output can be
+viewed.
+
+### Postcommit
+
+Running in postcommit removes as stringent of a time constraint, which gives us
+the ability to do some more comprehensive testing. In postcommit we have a test
+suite running the RunnableOnService tests against each supported runner, and
+another for running the full set of E2E tests against each runner.
+Currently-supported runners are Dataflow, Flink, Spark, and Gearpump, with
+others soon to follow. Work is ongoing to enable Flink, Spark, and Gearpump in
+the E2E framework, with full support targeted for end of August 2016. 
Postcommit
+tests run periodically, with timing defined in their Jenkins configurations.
+
+Adding new postcommit E2E tests is generally as easy as adding a \*IT.java file
+to the repository - Failsafe will notice it and run it - but if you want to do
+more interesting things, take a look at
+[WordCountIT.java](https://github.com/apache/incubator-beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java).
+
+Postcommit test results can be found in
+[Jenkins](https://builds.apache.org/view/Beam/).
+
+## Testing Types
 
 ### Unit
 
-Unit tests are, in Beam as everywhere else, the first line of defense in 
ensuring software correctness. As all of the contributors to Beam understand 
the importance of testing, Beam has a robust set of unit tests, as well as 
testing coverage measurement tools, which protect the codebase from simple to 
moderate breakages. Beam Java unit tests are written in JUnit.
+Unit tests are, in Beam as everywhere else, the first line of defense in
+ensuring software correctness. As all of the contributors to Beam understand 
the
+importance of testing, Beam has a robust set of unit tests, as well as testing
+coverage measurement tools, which protect the codebase from simple to moderate
+breakages. Beam Java unit tests are written in JUnit.
 
 ### RunnableOnService (Working Title)
 
-RunnableOnService tests contain components of both component and end-to-end 
tests. They fulfill the typical purpose of a component test - they are meant to 
test a well-scoped piece of Beam functionality or the interactions between two 
such pieces and can be run in a component-test-type fashion against the 
DirectRunner. Additionally, they are built with the ability to run in an 
end-to-end fashion against a runner, allowing them to verify not only core Beam 
functionality, but runner functionality as well. They are more lightweight than 
a traditional end-to-end test and, because of their well-scoped nature, provide 
good signal as to what exactly is working or broken against a particular runner.
-
-The name “RunnableOnService” is an artifact of when Beam was still the 
Google Cloud Dataflow SDK and will be changing to something more indicative of 
its use in the coming months.
+RunnableOnService tests contain components of both component and end-to-end
+tests. They fulfill the typical purpose of a component test - they are meant to
+test a well-scoped piece of Beam functionality or the interactions between two
+such pieces and can be run in a component-test-type fashion against the
+DirectRunner. Additionally, they are built with the ability to run in an
+end-to-end fashion against a runner, allowing them to verify not only core Beam
+functionality, but runner functionality as well. They are more lightweight than
+a traditional end-to-end test and, because of their well-scoped nature, provide
+good signal as to what exactly is working or broken against a particular 
runner.
+
+The name “RunnableOnService” is an artifact of when Beam was still the 
Google
+Cloud Dataflow SDK and [will be
+changing](https://issues.apache.org/jira/browse/BEAM-655) to something more
+indicative of its use in the coming months.
 
 ### E2E
 
-End-to-End tests are meant to verify at the very highest level that the Beam 
codebase is working as intended. Because they are implemented as a thin wrapper 
around existing pipelines, they can be used to prove that the core Beam 
functionality is available. They will be used to verify runner correctness, but 
they can also be used for IO connectors and other core functionality.
+End-to-End tests are meant to verify at the very highest level that the Beam
+codebase is working as intended. Because they are implemented as a thin wrapper
+around existing pipelines, they can be used to prove that the core Beam
+functionality is available. They will be used to verify runner correctness, but
+they can also be used for IO connectors and other core functionality.
 
-# Testing Systems
+## Testing Systems
 
 ### E2E Testing Framework
 
-The Beam end-to-end testing framework is a framework designed in a 
runner-agnostic fashion to exercise the entire lifecycle of a Beam pipeline. We 
run a pipeline as a user would and allow it to run to completion in the same 
way, verifying after completion that it behaved how we expected. Using 
pipelines from the Beam examples, or custom-built pipelines, the framework will 
provide hooks during several pipeline lifecycle events, e.g., pipeline 
creation, pipeline success, and pipeline failure, to allow verification of 
pipeline state.
-
-The E2E testing framework is currently built to hook into the [Maven Failsafe 
Integration Test 
plugin](http://maven.apache.org/surefire/maven-failsafe-plugin/), which means 
it is tightly integrated with the overall build process. Once it is determined 
how Python and other future languages will integrate into the overall 
build/test system (via Maven or otherwise) we will adjust this. The framework 
provides a wrapper around actual Beam pipelines, enabling those pipelines to be 
run in an environment which facilitates verification of pipeline results and 
details.
+The Beam end-to-end testing framework is a framework designed in a
+runner-agnostic fashion to exercise the entire lifecycle of a Beam pipeline. We
+run a pipeline as a user would and allow it to run to completion in the same
+way, verifying after completion that it behaved how we expected. Using 
pipelines
+from the Beam examples, or custom-built pipelines, the framework will provide
+hooks during several pipeline lifecycle events, e.g., pipeline creation,
+pipeline success, and pipeline failure, to allow verification of pipeline 
state.
+
+The E2E testing framework is currently built to hook into the [Maven Failsafe
+Integration Test
+plugin](http://maven.apache.org/surefire/maven-failsafe-plugin/), which means 
it
+is tightly integrated with the overall build process. Once it is determined how
+Python and other future languages will integrate into the overall build/test
+system (via Maven or otherwise) we will adjust this. The framework provides a
+wrapper around actual Beam pipelines, enabling those pipelines to be run in an
+environment which facilitates verification of pipeline results and details.
 
 Verifiers include:
 
+*   Output verification. Output verifiers ensure that the pipeline has produced
+    the expected output. Current verifiers check text-based output, but future
+    verifiers could support other output such as BigQuery and Datastore.
+*   Aggregator verification. Aggregator verifiers ensure that the user-defined
+    aggregators present in the pipelines under test finish in the expected
+    state.
 
-
-*   Output verification. Output verifiers ensure that the pipeline has 
produced the expected output. Current verifiers check text-based output, but 
future verifiers could support other output such as BigQuery and Datastore.
-*   Aggregator verification. Aggregator verifiers ensure that the user-defined 
aggregators present in the pipelines under test finish in the expected state.
-
-The E2E framework will support running on various different configurations of 
environments. We currently provide the ability to run against the DirectRunner, 
against a local Spark instance, a local Flink instance, and against the Google 
Cloud Dataflow service.
+The E2E framework will support running on various different configurations of
+environments. We currently provide the ability to run against the DirectRunner,
+against a local Spark instance, a local Flink instance, and against the Google
+Cloud Dataflow service.
 
 ### RunnableOnService Tests
 
-RunnableOnService tests are tests built to use the Beam TestPipeline class, 
which enables test authors to write simple functionality verification. They are 
meant to use some of the built-in utilities of the SDK, namely PAssert, to 
verify that the simple pipelines they run end in the correct state.
+RunnableOnService tests are tests built to use the Beam TestPipeline class,
+which enables test authors to write simple functionality verification. They are
+meant to use some of the built-in utilities of the SDK, namely PAssert, to
+verify that the simple pipelines they run end in the correct state.
 

Reply via email to