Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #565

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-181) Make TextIOTest and AvroIOTest runner-agnostic

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337323#comment-15337323
 ] 

Thomas Groh commented on BEAM-181:
--

The idea is to first convert all the reads and writes that use the Java File 
API to use IOChannelUtil instead, with the existing temporary locations (which 
can still be obtained the same way; just written to differently). After that is 
completed, a location can be provided to the test by the test configuration, 
which can differ depending on the runner.

> Make TextIOTest and AvroIOTest runner-agnostic
> --
>
> Key: BEAM-181
> URL: https://issues.apache.org/jira/browse/BEAM-181
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: Minor
>
> PipelineOptions contains a tempLocation property, and IOChannelUtils should 
> be capable of handling arbitrary file locations that are used as a 
> tempLocation. 
> The read and write tests for TextIO and AvroIO can use these properties to be 
> written in a runner-agnostic fashion, and then be marked as 
> RunnableOnService. Doing so allows all runners integrated with 
> RunnableOnService to benefit from the existing tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #494: Use item equality in apply_to_list test

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/494

Use item equality in apply_to_list test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Orders of PCollections are not guaranteed and this test depends on
the current DirectRunner implementation for output to have the same
order as input.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam pttest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #494


commit 0ba5b6559d1779e30ca712e371a81788cb4812a7
Author: Ahmet Altay 
Date:   2016-06-17T23:57:11Z

Use item equality in apply_to_list test

Orders of PCollections are not guaranteed and this test depends on
the current DirectRunner implementation for output to have the same
order as input.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337220#comment-15337220
 ] 

ASF GitHub Bot commented on BEAM-234:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/493

[BEAM-234] Remove Pipeline from TestDataflowPipelineRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam df_pr_cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #493


commit 8dadcb65f6dc25f2979aab1918b6fd4b7f6c1c40
Author: Thomas Groh 
Date:   2016-06-17T23:36:22Z

Remove Pipeline from TestDataflowPipelineRunner




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #493: [BEAM-234] Remove Pipeline from TestDatafl...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/493

[BEAM-234] Remove Pipeline from TestDataflowPipelineRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam df_pr_cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #493


commit 8dadcb65f6dc25f2979aab1918b6fd4b7f6c1c40
Author: Thomas Groh 
Date:   2016-06-17T23:36:22Z

Remove Pipeline from TestDataflowPipelineRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #492: Run Pipelines in tests that expect them

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/492

Run Pipelines in tests that expect them

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
run_pipelines_where_assertions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #492


commit 916bf3a84776c45f679bc27dfbaa6e7eec205ef7
Author: Thomas Groh 
Date:   2016-06-17T23:06:37Z

Run Pipelines in tests that expect them




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-287) Add new beam groupIds

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-287.
-
Resolution: Fixed

> Add new beam groupIds
> -
>
> Key: BEAM-287
> URL: https://issues.apache.org/jira/browse/BEAM-287
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.1.0-incubating
>
>
> Now, all beam modules use the same groupId: {{org.apache.beam}}.
> If it's convenient when using {{search.maven.org}}, browsing the repository 
> is not easy as all artifacts are in the same location.
> As proposed initially, I will introduce new groupIds (like 
> {{org.apache.beam.io}}, {{org.apache.beam.runner}}, etc). We discussed about 
> that with [~davor] last week.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-319) Attach src distribution artifact to be take by deploy

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-319.
-
Resolution: Fixed

> Attach src distribution artifact to be take by deploy
> -
>
> Key: BEAM-319
> URL: https://issues.apache.org/jira/browse/BEAM-319
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #482: Turn on failOnWarning and set runners depe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/482


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Turn on failOnWarning and ignore unused runners modules in example.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8a9dcc71f -> 69a4141a5


Turn on failOnWarning and ignore unused runners modules in example.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/95308fbb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/95308fbb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/95308fbb

Branch: refs/heads/master
Commit: 95308fbbfa3accc5a1be5c87db729bcc41255156
Parents: 8a9dcc7
Author: Pei He 
Authored: Thu Jun 16 15:50:21 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 15:26:45 2016 -0700

--
 examples/java/pom.xml | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95308fbb/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 9458962..5167810 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -59,6 +59,14 @@
   
 org.apache.maven.plugins
 maven-dependency-plugin
+
+  
+analyze-only
+
+  true
+
+  
+
   
 
   
@@ -216,6 +224,7 @@
   org.apache.beam
   beam-runners-direct-java
   ${project.version}
+  runtime
 
 
 

[2/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
new file mode 100644
index 000..e094d0d
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -0,0 +1,1417 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.startsWith;
+import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsList;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsMap;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsMultimap;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsSingleton;
+import org.apache.beam.runners.dataflow.DataflowRunner.TransformedMap;
+import org.apache.beam.runners.dataflow.internal.IsmFormat;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
+import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
+import org.apache.beam.sdk.coders.BigEndianLongCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.io.AvroSource;
+import org.apache.beam.sdk.io.BigQueryIO;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.runners.TransformTreeNode;
+import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFnTester;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.GcsUtil;
+import org.apache.beam.sdk.util.NoopPathValidator;
+import org.apache.beam.sdk.util.ReleaseInfo;
+import org.apache.beam.sdk.util.TestCredential;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
+import org.apache.beam.sdk.util.WindowingStrategy;
+import org.apache.beam.sdk.util.gcsfs.GcsPath;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import 

[1/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 81dcd5278 -> 8a9dcc71f


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
deleted file mode 100644
index fbaf116..000
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
+++ /dev/null
@@ -1,601 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow.testing;
-
-import static org.hamcrest.Matchers.containsString;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertSame;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Mockito.atLeastOnce;
-import static org.mockito.Mockito.doCallRealMethod;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.runners.dataflow.DataflowPipelineJob;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil.JobMessagesHandler;
-import org.apache.beam.runners.dataflow.util.TimeUtil;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.SerializableMatcher;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.NoopPathValidator;
-import org.apache.beam.sdk.util.TestCredential;
-import org.apache.beam.sdk.util.Transport;
-import org.apache.beam.sdk.values.PCollection;
-
-import com.google.api.client.http.LowLevelHttpResponse;
-import com.google.api.client.json.Json;
-import com.google.api.client.testing.http.MockHttpTransport;
-import com.google.api.client.testing.http.MockLowLevelHttpRequest;
-import com.google.api.client.testing.http.MockLowLevelHttpResponse;
-import com.google.api.services.dataflow.Dataflow;
-import com.google.api.services.dataflow.model.JobMessage;
-import com.google.api.services.dataflow.model.JobMetrics;
-import com.google.api.services.dataflow.model.MetricStructuredName;
-import com.google.api.services.dataflow.model.MetricUpdate;
-import com.google.common.base.Optional;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Lists;
-
-import org.hamcrest.BaseMatcher;
-import org.hamcrest.Description;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.MockitoAnnotations;
-import org.mockito.invocation.InvocationOnMock;
-import org.mockito.stubbing.Answer;
-
-import java.io.IOException;
-import java.math.BigDecimal;
-import java.util.Arrays;
-import java.util.concurrent.TimeUnit;
-
-/** Tests for {@link TestDataflowPipelineRunner}. */
-@RunWith(JUnit4.class)
-public class TestDataflowPipelineRunnerTest {
-  @Rule public ExpectedException expectedException = ExpectedException.none();
-  @Mock private MockHttpTransport transport;
-  @Mock private MockLowLevelHttpRequest request;
-  @Mock private GcsUtil mockGcsUtil;
-
-  private 

[9/9] incubator-beam git commit: Closes #487

[dhalperi]

Closes #487


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8a9dcc71
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8a9dcc71
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8a9dcc71

Branch: refs/heads/master
Commit: 8a9dcc71f08aa72a008344b75c30949703e7f6f5
Parents: 81dcd52 3841f41
Author: Dan Halperin 
Authored: Fri Jun 17 15:13:39 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jun 17 15:13:39 2016 -0700

--
 .../beam/examples/DebuggingWordCount.java   |2 +-
 .../apache/beam/examples/MinimalWordCount.java  |4 +-
 .../apache/beam/examples/WindowedWordCount.java |2 +-
 .../org/apache/beam/examples/WordCount.java |2 +-
 .../examples/common/DataflowExampleUtils.java   |   18 +-
 .../beam/examples/complete/AutoComplete.java|4 +-
 .../apache/beam/examples/complete/TfIdf.java|2 +-
 .../examples/complete/TopWikipediaSessions.java |2 +-
 .../examples/cookbook/BigQueryTornadoes.java|2 +-
 .../cookbook/CombinePerKeyExamples.java |2 +-
 .../examples/cookbook/DatastoreWordCount.java   |4 +-
 .../beam/examples/cookbook/DeDupExample.java|2 +-
 .../beam/examples/cookbook/FilterExamples.java  |2 +-
 .../beam/examples/cookbook/JoinExamples.java|2 +-
 .../examples/cookbook/MaxPerKeyExamples.java|2 +-
 .../beam/examples/MinimalWordCountJava8.java|4 +-
 .../beam/examples/complete/game/GameStats.java  |2 +-
 .../examples/complete/game/HourlyTeamScore.java |2 +-
 .../examples/complete/game/LeaderBoard.java |2 +-
 .../beam/examples/complete/game/UserScore.java  |2 +-
 .../beam/runners/flink/FlinkPipelineRunner.java |2 +-
 .../BlockingDataflowPipelineRunner.java |  186 -
 .../dataflow/BlockingDataflowRunner.java|  186 +
 .../dataflow/DataflowJobCancelledException.java |2 +-
 .../dataflow/DataflowJobExecutionException.java |2 +-
 .../dataflow/DataflowJobUpdatedException.java   |2 +-
 .../runners/dataflow/DataflowPipelineJob.java   |2 +-
 .../dataflow/DataflowPipelineRegistrar.java |8 +-
 .../dataflow/DataflowPipelineRunner.java| 3229 --
 .../dataflow/DataflowPipelineRunnerHooks.java   |   39 -
 .../dataflow/DataflowPipelineTranslator.java|   10 +-
 .../beam/runners/dataflow/DataflowRunner.java   | 3229 ++
 .../runners/dataflow/DataflowRunnerHooks.java   |   39 +
 .../runners/dataflow/internal/package-info.java |2 +-
 .../BlockingDataflowPipelineOptions.java|6 +-
 .../options/DataflowPipelineOptions.java|4 +-
 .../DataflowPipelineWorkerPoolOptions.java  |6 +-
 .../testing/TestDataflowPipelineRunner.java |   10 +-
 .../BlockingDataflowPipelineRunnerTest.java |  304 --
 .../dataflow/BlockingDataflowRunnerTest.java|  304 ++
 .../dataflow/DataflowPipelineRegistrarTest.java |4 +-
 .../dataflow/DataflowPipelineRunnerTest.java| 1417 
 .../DataflowPipelineTranslatorTest.java |   50 +-
 .../runners/dataflow/DataflowRunnerTest.java| 1417 
 .../runners/dataflow/io/DataflowAvroIOTest.java |4 +-
 .../dataflow/io/DataflowPubsubIOTest.java   |4 +-
 .../runners/dataflow/io/DataflowTextIOTest.java |4 +-
 .../testing/TestDataflowPipelineRunnerTest.java |  601 
 .../testing/TestDataflowRunnerTest.java |  601 
 .../DataflowDisplayDataEvaluator.java   |   10 +-
 .../transforms/DataflowGroupByKeyTest.java  |8 +-
 .../dataflow/transforms/DataflowViewTest.java   |8 +-
 .../util/DataflowPathValidatorTest.java |8 +-
 .../beam/runners/spark/SimpleWordCountTest.java |2 +-
 .../spark/translation/SerializationTest.java|2 +-
 .../apache/beam/sdk/runners/package-info.java   |   12 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |2 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |2 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java|2 +-
 .../src/main/java/DebuggingWordCount.java   |2 +-
 .../src/main/java/MinimalWordCount.java |4 +-
 .../src/main/java/WindowedWordCount.java|2 +-
 .../src/main/java/WordCount.java|2 +-
 .../main/java/common/DataflowExampleUtils.java  |   14 +-
 .../src/main/java/StarterPipeline.java  |2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |2 +-
 sdks/java/pom.xml   |2 +-
 68 files changed, 5913 insertions(+), 5913 deletions(-)
--

[5/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
new file mode 100644
index 000..91e34ac
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -0,0 +1,3229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
+import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
+import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification;
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TransformTranslator;
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TranslationContext;
+import org.apache.beam.runners.dataflow.internal.AssignWindows;
+import org.apache.beam.runners.dataflow.internal.DataflowAggregatorTransforms;
+import org.apache.beam.runners.dataflow.internal.IsmFormat;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
+import org.apache.beam.runners.dataflow.internal.ReadTranslator;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
+import org.apache.beam.runners.dataflow.util.DataflowTransport;
+import org.apache.beam.runners.dataflow.util.MonitoringUtil;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.BigEndianLongCoder;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.coders.StandardCoder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.io.BigQueryIO;
+import org.apache.beam.sdk.io.FileBasedSink;
+import org.apache.beam.sdk.io.PubsubIO;
+import org.apache.beam.sdk.io.PubsubUnboundedSink;
+import org.apache.beam.sdk.io.PubsubUnboundedSource;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.ShardNameTemplate;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.Write;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.runners.AggregatorPipelineExtractor;
+import org.apache.beam.sdk.runners.PipelineRunner;
+import org.apache.beam.sdk.runners.TransformTreeNode;
+import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.apache.beam.sdk.transforms.Create;

[7/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
deleted file mode 100644
index 1eb39ad..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
+++ /dev/null
@@ -1,3229 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
-import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
-import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkState;
-
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TransformTranslator;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TranslationContext;
-import org.apache.beam.runners.dataflow.internal.AssignWindows;
-import org.apache.beam.runners.dataflow.internal.DataflowAggregatorTransforms;
-import org.apache.beam.runners.dataflow.internal.IsmFormat;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
-import org.apache.beam.runners.dataflow.internal.ReadTranslator;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
-import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
-import org.apache.beam.runners.dataflow.util.DataflowTransport;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.Pipeline.PipelineVisitor;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.coders.AvroCoder;
-import org.apache.beam.sdk.coders.BigEndianLongCoder;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.ListCoder;
-import org.apache.beam.sdk.coders.MapCoder;
-import org.apache.beam.sdk.coders.SerializableCoder;
-import org.apache.beam.sdk.coders.StandardCoder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.io.AvroIO;
-import org.apache.beam.sdk.io.BigQueryIO;
-import org.apache.beam.sdk.io.FileBasedSink;
-import org.apache.beam.sdk.io.PubsubIO;
-import org.apache.beam.sdk.io.PubsubUnboundedSink;
-import org.apache.beam.sdk.io.PubsubUnboundedSource;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.io.ShardNameTemplate;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.io.UnboundedSource;
-import org.apache.beam.sdk.io.Write;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.sdk.options.StreamingOptions;
-import org.apache.beam.sdk.runners.AggregatorPipelineExtractor;
-import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.runners.TransformTreeNode;
-import org.apache.beam.sdk.transforms.Aggregator;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import 

[4/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
new file mode 100644
index 000..05297ec
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import org.apache.beam.sdk.annotations.Experimental;
+
+import com.google.api.services.dataflow.model.Environment;
+
+/**
+ * An instance of this class can be passed to the
+ * {@link DataflowRunner} to add user defined hooks to be
+ * invoked at various times during pipeline execution.
+ */
+@Experimental
+public class DataflowRunnerHooks {
+  /**
+   * Allows the user to modify the environment of their job before their job 
is submitted
+   * to the service for execution.
+   *
+   * @param environment The environment of the job. Users can make change to 
this instance in order
+   * to change the environment with which their job executes on the 
service.
+   */
+  public void modifyEnvironmentBeforeSubmission(Environment environment) {}
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
index f2e8459..ea83bfb 100755
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
@@ -16,6 +16,6 @@
  * limitations under the License.
  */
 /**
- * Implementation of the {@link 
org.apache.beam.runners.dataflow.DataflowPipelineRunner}.
+ * Implementation of the {@link 
org.apache.beam.runners.dataflow.DataflowRunner}.
  */
 package org.apache.beam.runners.dataflow.internal;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
index 7fa5ad6..809df35 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.dataflow.options;
 
-import org.apache.beam.runners.dataflow.BlockingDataflowPipelineRunner;
+import org.apache.beam.runners.dataflow.BlockingDataflowRunner;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.DefaultValueFactory;
 import org.apache.beam.sdk.options.Description;
@@ -29,9 +29,9 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 import java.io.PrintStream;
 
 /**
- * Options that are used to configure the {@link 
BlockingDataflowPipelineRunner}.
+ * Options that are used to configure the {@link BlockingDataflowRunner}.
  */
-@Description("Configure options on the BlockingDataflowPipelineRunner.")
+@Description("Configure options on the BlockingDataflowRunner.")
 public interface BlockingDataflowPipelineOptions 

[8/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

Rename DataflowPipelineRunner to DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3841f411
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3841f411
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3841f411

Branch: refs/heads/master
Commit: 3841f41133219272e1e88d6ba56b0d158d6bd59d
Parents: 81dcd52
Author: Thomas Groh 
Authored: Fri Jun 17 11:42:19 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jun 17 15:13:12 2016 -0700

--
 .../beam/examples/DebuggingWordCount.java   |2 +-
 .../apache/beam/examples/MinimalWordCount.java  |4 +-
 .../apache/beam/examples/WindowedWordCount.java |2 +-
 .../org/apache/beam/examples/WordCount.java |2 +-
 .../examples/common/DataflowExampleUtils.java   |   18 +-
 .../beam/examples/complete/AutoComplete.java|4 +-
 .../apache/beam/examples/complete/TfIdf.java|2 +-
 .../examples/complete/TopWikipediaSessions.java |2 +-
 .../examples/cookbook/BigQueryTornadoes.java|2 +-
 .../cookbook/CombinePerKeyExamples.java |2 +-
 .../examples/cookbook/DatastoreWordCount.java   |4 +-
 .../beam/examples/cookbook/DeDupExample.java|2 +-
 .../beam/examples/cookbook/FilterExamples.java  |2 +-
 .../beam/examples/cookbook/JoinExamples.java|2 +-
 .../examples/cookbook/MaxPerKeyExamples.java|2 +-
 .../beam/examples/MinimalWordCountJava8.java|4 +-
 .../beam/examples/complete/game/GameStats.java  |2 +-
 .../examples/complete/game/HourlyTeamScore.java |2 +-
 .../examples/complete/game/LeaderBoard.java |2 +-
 .../beam/examples/complete/game/UserScore.java  |2 +-
 .../beam/runners/flink/FlinkPipelineRunner.java |2 +-
 .../BlockingDataflowPipelineRunner.java |  186 -
 .../dataflow/BlockingDataflowRunner.java|  186 +
 .../dataflow/DataflowJobCancelledException.java |2 +-
 .../dataflow/DataflowJobExecutionException.java |2 +-
 .../dataflow/DataflowJobUpdatedException.java   |2 +-
 .../runners/dataflow/DataflowPipelineJob.java   |2 +-
 .../dataflow/DataflowPipelineRegistrar.java |8 +-
 .../dataflow/DataflowPipelineRunner.java| 3229 --
 .../dataflow/DataflowPipelineRunnerHooks.java   |   39 -
 .../dataflow/DataflowPipelineTranslator.java|   10 +-
 .../beam/runners/dataflow/DataflowRunner.java   | 3229 ++
 .../runners/dataflow/DataflowRunnerHooks.java   |   39 +
 .../runners/dataflow/internal/package-info.java |2 +-
 .../BlockingDataflowPipelineOptions.java|6 +-
 .../options/DataflowPipelineOptions.java|4 +-
 .../DataflowPipelineWorkerPoolOptions.java  |6 +-
 .../testing/TestDataflowPipelineRunner.java |   10 +-
 .../BlockingDataflowPipelineRunnerTest.java |  304 --
 .../dataflow/BlockingDataflowRunnerTest.java|  304 ++
 .../dataflow/DataflowPipelineRegistrarTest.java |4 +-
 .../dataflow/DataflowPipelineRunnerTest.java| 1417 
 .../DataflowPipelineTranslatorTest.java |   50 +-
 .../runners/dataflow/DataflowRunnerTest.java| 1417 
 .../runners/dataflow/io/DataflowAvroIOTest.java |4 +-
 .../dataflow/io/DataflowPubsubIOTest.java   |4 +-
 .../runners/dataflow/io/DataflowTextIOTest.java |4 +-
 .../testing/TestDataflowPipelineRunnerTest.java |  601 
 .../testing/TestDataflowRunnerTest.java |  601 
 .../DataflowDisplayDataEvaluator.java   |   10 +-
 .../transforms/DataflowGroupByKeyTest.java  |8 +-
 .../dataflow/transforms/DataflowViewTest.java   |8 +-
 .../util/DataflowPathValidatorTest.java |8 +-
 .../beam/runners/spark/SimpleWordCountTest.java |2 +-
 .../spark/translation/SerializationTest.java|2 +-
 .../apache/beam/sdk/runners/package-info.java   |   12 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |2 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |2 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java|2 +-
 .../src/main/java/DebuggingWordCount.java   |2 +-
 .../src/main/java/MinimalWordCount.java |4 +-
 .../src/main/java/WindowedWordCount.java|2 +-
 .../src/main/java/WordCount.java|2 +-
 .../main/java/common/DataflowExampleUtils.java  |   14 +-
 .../src/main/java/StarterPipeline.java  |2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |2 +-
 sdks/java/pom.xml   |2 +-
 68 files changed, 5913 insertions(+), 5913 deletions(-)
--

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337099#comment-15337099
 ] 

ASF GitHub Bot commented on BEAM-234:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/487


> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[6/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
deleted file mode 100644
index 4d37966..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import org.apache.beam.sdk.annotations.Experimental;
-
-import com.google.api.services.dataflow.model.Environment;
-
-/**
- * An instance of this class can be passed to the
- * {@link DataflowPipelineRunner} to add user defined hooks to be
- * invoked at various times during pipeline execution.
- */
-@Experimental
-public class DataflowPipelineRunnerHooks {
-  /**
-   * Allows the user to modify the environment of their job before their job 
is submitted
-   * to the service for execution.
-   *
-   * @param environment The environment of the job. Users can make change to 
this instance in order
-   * to change the environment with which their job executes on the 
service.
-   */
-  public void modifyEnvironmentBeforeSubmission(Environment environment) {}
-}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index f5fefc0..e15b9d2 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -30,7 +30,7 @@ import static org.apache.beam.sdk.util.Structs.getString;
 
 import static com.google.common.base.Preconditions.checkArgument;
 
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.GroupByKeyAndSortValuesOnly;
+import 
org.apache.beam.runners.dataflow.DataflowRunner.GroupByKeyAndSortValuesOnly;
 import org.apache.beam.runners.dataflow.internal.ReadTranslator;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.runners.dataflow.util.DoFnInfo;
@@ -139,7 +139,7 @@ public class DataflowPipelineTranslator {
*/
   public JobSpecification translate(
   Pipeline pipeline,
-  DataflowPipelineRunner runner,
+  DataflowRunner runner,
   List packages) {
 
 Translator translator = new Translator(pipeline, runner);
@@ -224,7 +224,7 @@ public class DataflowPipelineTranslator {
 
   /**
* The interface provided to registered callbacks for interacting
-   * with the {@link DataflowPipelineRunner}, including reading and writing the
+   * with the {@link DataflowRunner}, including reading and writing the
* values of {@link PCollection}s and side inputs ({@link PCollectionView}s).
*/
   public interface TranslationContext {
@@ -350,7 +350,7 @@ public class DataflowPipelineTranslator {
 private final Pipeline pipeline;
 
 /** The runner which will execute the pipeline. */
-private final DataflowPipelineRunner runner;
+private final DataflowRunner runner;
 
 /** The Cloud Dataflow Job representation. */
 private final Job job = new Job();
@@ -385,7 +385,7 @@ public class DataflowPipelineTranslator {
  * Constructs a Translator that will translate the specified
  * Pipeline into Dataflow objects.
  */
-public Translator(Pipeline pipeline, DataflowPipelineRunner runner) {
+public Translator(Pipeline pipeline, 

[3/9] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[dhalperi]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3841f411/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
deleted file mode 100644
index 38d4c96..000
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
+++ /dev/null
@@ -1,1417 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.containsString;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasItem;
-import static org.hamcrest.Matchers.instanceOf;
-import static org.hamcrest.Matchers.startsWith;
-import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Matchers.eq;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsList;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsMap;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsMultimap;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsSingleton;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.TransformedMap;
-import org.apache.beam.runners.dataflow.internal.IsmFormat;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.Pipeline.PipelineVisitor;
-import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
-import org.apache.beam.sdk.coders.BigEndianLongCoder;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.io.AvroIO;
-import org.apache.beam.sdk.io.AvroSource;
-import org.apache.beam.sdk.io.BigQueryIO;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.runners.TransformTreeNode;
-import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFnTester;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.NoopPathValidator;
-import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.beam.sdk.util.TestCredential;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.gcsfs.GcsPath;
-import org.apache.beam.sdk.values.KV;
-import 

[GitHub] incubator-beam pull request #487: [BEAM-234] Rename DataflowPipelineRunner t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337095#comment-15337095
 ] 

ASF GitHub Bot commented on BEAM-234:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/490


> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #490: [BEAM-234] Finish removing DirectPipelineR...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/490


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #490

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 47dd61a55 -> 81dcd5278


Closes #490


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/81dcd527
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/81dcd527
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/81dcd527

Branch: refs/heads/master
Commit: 81dcd527888be7bcfde09e47a3f512aeafd4a962
Parents: 47dd61a 6ebc8ab
Author: Dan Halperin 
Authored: Fri Jun 17 15:11:01 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jun 17 15:11:01 2016 -0700

--
 .../examples/common/DataflowExampleUtils.java   |  4 +-
 .../examples/cookbook/DatastoreWordCount.java   |  4 +-
 .../translation/TransformTranslatorTest.java|  2 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   |  2 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |  2 +-
 .../beam/sdk/options/DirectPipelineOptions.java | 74 
 .../beam/sdk/options/PipelineOptions.java   |  7 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |  2 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java| 24 ---
 .../main/java/common/DataflowExampleUtils.java  |  2 +-
 .../src/main/java/StarterPipeline.java  |  2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |  2 +-
 13 files changed, 28 insertions(+), 101 deletions(-)
--

[2/2] incubator-beam git commit: Finish removing DirectPipelineRunner references

[dhalperi]

Finish removing DirectPipelineRunner references


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6ebc8abe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6ebc8abe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6ebc8abe

Branch: refs/heads/master
Commit: 6ebc8abe83b2417df9abe11c691a4f6cec4e66b6
Parents: 47dd61a
Author: Thomas Groh 
Authored: Fri Jun 17 13:22:26 2016 -0700
Committer: Dan Halperin 
Committed: Fri Jun 17 15:11:01 2016 -0700

--
 .../examples/common/DataflowExampleUtils.java   |  4 +-
 .../examples/cookbook/DatastoreWordCount.java   |  4 +-
 .../translation/TransformTranslatorTest.java|  2 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   |  2 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |  2 +-
 .../beam/sdk/options/DirectPipelineOptions.java | 74 
 .../beam/sdk/options/PipelineOptions.java   |  7 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |  2 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java| 24 ---
 .../main/java/common/DataflowExampleUtils.java  |  2 +-
 .../src/main/java/StarterPipeline.java  |  2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |  2 +-
 13 files changed, 28 insertions(+), 101 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ebc8abe/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
index 5b1af6d..46b8af3 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
@@ -315,7 +315,7 @@ public class DataflowExampleUtils {
   }
 
   /**
-   * Do some runner setup: check that the DirectPipelineRunner is not used in 
conjunction with
+   * Do some runner setup: check that the DirectRunner is not used in 
conjunction with
* streaming, and if streaming is specified, use the DataflowPipelineRunner.
*/
   public void setupRunner() {
@@ -413,7 +413,7 @@ public class DataflowExampleUtils {
   }
 } else {
   // Do nothing if the given PipelineResult doesn't support waitToFinish(),
-  // such as EvaluationResults returned by DirectPipelineRunner.
+  // such as EvaluationResults returned by DirectRunner.
   tearDown();
   printPendingMessages();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ebc8abe/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
index bfaecdf..2d1f88c 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
@@ -193,7 +193,7 @@ public class DatastoreWordCount {
 
   /**
* An example that creates a pipeline to populate DatastoreIO from a
-   * text input.  Forces use of DirectPipelineRunner for local execution mode.
+   * text input.  Forces use of DirectRunner for local execution mode.
*/
   public static void writeDataToDatastore(Options options) {
   Pipeline p = Pipeline.create(options);
@@ -247,7 +247,7 @@ public class DatastoreWordCount {
 
   /**
* An example to demo how to use {@link DatastoreIO}.  The runner here is
-   * customizable, which means users could pass either {@code 
DirectPipelineRunner}
+   * customizable, which means users could pass either {@code DirectRunner}
* or {@code DataflowPipelineRunner} in the pipeline options.
*/
   public static void main(String args[]) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ebc8abe/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
index 5fdfb49..b593316 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
+++ 

[1/2] incubator-beam git commit: Remove references to javax.servlet.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 336ae2e7e -> 47dd61a55


Remove references to javax.servlet.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cdfab9ba
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cdfab9ba
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cdfab9ba

Branch: refs/heads/master
Commit: cdfab9ba95c3d0b32a8a46aa24b4895a9d261904
Parents: 336ae2e
Author: Pei He 
Authored: Fri Jun 17 13:02:58 2016 -0700
Committer: Pei He 
Committed: Fri Jun 17 13:03:04 2016 -0700

--
 examples/java/pom.xml  | 6 --
 .../org/apache/beam/examples/common/DataflowExampleUtils.java  | 6 +++---
 2 files changed, 3 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cdfab9ba/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 5211b80..9458962 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -285,12 +285,6 @@
   runtime
 
 
-
-  javax.servlet
-  javax.servlet-api
-  3.1.0
-
-
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cdfab9ba/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
index a0b7319..5b1af6d 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
@@ -62,8 +62,6 @@ import java.util.List;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
-import javax.servlet.http.HttpServletResponse;
-
 /**
  * The utility class that sets up and tears down external resources, starts 
the Google Cloud Pub/Sub
  * injector, and cancels the streaming and the injector pipelines once the 
program terminates.
@@ -72,6 +70,8 @@ import javax.servlet.http.HttpServletResponse;
  */
 public class DataflowExampleUtils {
 
+  private static final int SC_NOT_FOUND = 404;
+
   private final DataflowPipelineOptions options;
   private Bigquery bigQueryClient = null;
   private Pubsub pubsubClient = null;
@@ -481,7 +481,7 @@ public class DataflowExampleUtils {
 try {
   return request.execute();
 } catch (GoogleJsonResponseException e) {
-  if (e.getStatusCode() == HttpServletResponse.SC_NOT_FOUND) {
+  if (e.getStatusCode() == SC_NOT_FOUND) {
 return null;
   } else {
 throw e;

[2/2] incubator-beam git commit: This closes #486

[davor]

This closes #486


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/47dd61a5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/47dd61a5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/47dd61a5

Branch: refs/heads/master
Commit: 47dd61a553a0440125eb2994a514904d739677cd
Parents: 336ae2e cdfab9b
Author: Davor Bonaci 
Authored: Fri Jun 17 13:53:59 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 13:53:59 2016 -0700

--
 examples/java/pom.xml  | 6 --
 .../org/apache/beam/examples/common/DataflowExampleUtils.java  | 6 +++---
 2 files changed, 3 insertions(+), 9 deletions(-)
--

[jira] [Commented] (BEAM-356) Custom commands not executed on pip install for user workflow package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336922#comment-15336922
 ] 

ASF GitHub Bot commented on BEAM-356:
-

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/491

[BEAM-356] Update juliaset example to support custom commands on "pip 
install"

This change updates the juliaset example to support custom commands run 
when the workflow package is installed on a worker with "pip install".

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam 
pip-setup-commands

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #491


commit 368f5e1e266c271db0d24f5b5d76a5521f878f47
Author: Charles Chen 
Date:   2016-06-17T20:36:32Z

Update juliaset example to support "pip install"




> Custom commands not executed on pip install for user workflow package
> -
>
> Key: BEAM-356
> URL: https://issues.apache.org/jira/browse/BEAM-356
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Minor
>
> Custom commands in setup.py stopped working after Dataflow switched from 
> using "easy_install" to "pip install" for installing user workflow packages.
> Please see https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/20.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #491: [BEAM-356] Update juliaset example to supp...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/491

[BEAM-356] Update juliaset example to support custom commands on "pip 
install"

This change updates the juliaset example to support custom commands run 
when the workflow package is installed on a worker with "pip install".

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam 
pip-setup-commands

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #491


commit 368f5e1e266c271db0d24f5b5d76a5521f878f47
Author: Charles Chen 
Date:   2016-06-17T20:36:32Z

Update juliaset example to support "pip install"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-356) Custom commands not executed on pip install for user workflow package

[Charles Chen (JIRA)]

Charles Chen created BEAM-356:
-

 Summary: Custom commands not executed on pip install for user 
workflow package
 Key: BEAM-356
 URL: https://issues.apache.org/jira/browse/BEAM-356
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Charles Chen
Assignee: Charles Chen
Priority: Minor


Custom commands in setup.py stopped working after Dataflow switched from using 
"easy_install" to "pip install" for installing user workflow packages.

Please see https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/20.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336888#comment-15336888
 ] 

ASF GitHub Bot commented on BEAM-234:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/490

[BEAM-234] Finish removing DirectPipelineRunner references

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam finish_direct_rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit 3bc4a13a6d5c730483181abf729068242544f815
Author: Thomas Groh 
Date:   2016-06-17T20:22:26Z

Finish removing DirectPipelineRunner references




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #490: [BEAM-234] Finish removing DirectPipelineR...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/490

[BEAM-234] Finish removing DirectPipelineRunner references

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam finish_direct_rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit 3bc4a13a6d5c730483181abf729068242544f815
Author: Thomas Groh 
Date:   2016-06-17T20:22:26Z

Finish removing DirectPipelineRunner references




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #488: [BEAM-234] Rename SparkPipelineRunner to S...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/488

[BEAM-234] Rename SparkPipelineRunner to SparkRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rename_spark_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #488


commit b5487e0ca33475290065ce610ca9e33f96de3b5f
Author: Thomas Groh 
Date:   2016-06-17T17:45:09Z

Rename SparkPipelineRunner to SparkRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336864#comment-15336864
 ] 

ASF GitHub Bot commented on BEAM-234:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/489

[BEAM-234] Rename FlinkPipelineRunner to FlinkRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rename_flink_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/489.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #489


commit 0fc67447799dad3deeb7aff9943f67c155461398
Author: Thomas Groh 
Date:   2016-06-17T16:57:25Z

Rename FlinkPipelineRunner to FlinkRunner




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336863#comment-15336863
 ] 

ASF GitHub Bot commented on BEAM-234:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/488

[BEAM-234] Rename SparkPipelineRunner to SparkRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rename_spark_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #488


commit b5487e0ca33475290065ce610ca9e33f96de3b5f
Author: Thomas Groh 
Date:   2016-06-17T17:45:09Z

Rename SparkPipelineRunner to SparkRunner




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #487: [BEAM-234] Rename DataflowPipelineRunner t...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/487

[BEAM-234] Rename DataflowPipelineRunner to DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rename_dataflow_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #487


commit 816e3ff27d665c75e3f560b6b89a1af80fe16ef3
Author: Thomas Groh 
Date:   2016-06-17T18:42:19Z

Rename DataflowPipelineRunner to DataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336862#comment-15336862
 ] 

ASF GitHub Bot commented on BEAM-234:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/487

[BEAM-234] Rename DataflowPipelineRunner to DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rename_dataflow_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #487


commit 816e3ff27d665c75e3f560b6b89a1af80fe16ef3
Author: Thomas Groh 
Date:   2016-06-17T18:42:19Z

Rename DataflowPipelineRunner to DataflowRunner




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-181) Make TextIOTest and AvroIOTest runner-agnostic

[Andrew Psaltis (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336824#comment-15336824
 ] 

Andrew Psaltis commented on BEAM-181:
-

[~tgroh] I am looking at doing the work for this and trying to understand the 
desired changes. Are you suggesting that the tempLocation property gets added 
to the IOChannelUtils and then the read/write tests in TextIO and AvroIO use 
those properties instead of using "tmpFolder.newFile(.)" ? 

> Make TextIOTest and AvroIOTest runner-agnostic
> --
>
> Key: BEAM-181
> URL: https://issues.apache.org/jira/browse/BEAM-181
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: Minor
>
> PipelineOptions contains a tempLocation property, and IOChannelUtils should 
> be capable of handling arbitrary file locations that are used as a 
> tempLocation. 
> The read and write tests for TextIO and AvroIO can use these properties to be 
> written in a runner-agnostic fashion, and then be marked as 
> RunnableOnService. Doing so allows all runners integrated with 
> RunnableOnService to benefit from the existing tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #486: Remove references to javax.servlet.

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/486

Remove references to javax.servlet.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam rm-javax

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/486.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #486


commit cdfab9ba95c3d0b32a8a46aa24b4895a9d261904
Author: Pei He 
Date:   2016-06-17T20:02:58Z

Remove references to javax.servlet.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-234) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336678#comment-15336678
 ] 

ASF GitHub Bot commented on BEAM-234:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/484


> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-234
> URL: https://issues.apache.org/jira/browse/BEAM-234
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The fact that a PipelineRunner runs a Pipeline is provided by its 
> implementation of the PipelineRunner abstract class, so all the inclusion of 
> "Pipeline" is makes it inconvenient to type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Remove last vestige of the words DirectPipeline

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master da324bf52 -> 336ae2e7e


Remove last vestige of the words DirectPipeline


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c666d4ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c666d4ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c666d4ee

Branch: refs/heads/master
Commit: c666d4eedf9b0b85a24445bbe8bc69a0f26686f5
Parents: da324bf
Author: Thomas Groh 
Authored: Fri Jun 17 10:25:44 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 11:45:43 2016 -0700

--
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c666d4ee/README.md
--
diff --git a/README.md b/README.md
index fa00e82..b34d933 100644
--- a/README.md
+++ b/README.md
@@ -66,7 +66,7 @@ Have ideas for new SDKs or DSLs? See the 
[Jira](https://issues.apache.org/jira/b
 
 Beam supports executing programs on multiple distributed processing backends 
through PipelineRunners. Currently, the following PipelineRunners are available:
 
-- The `DirectPipelineRunner` runs the pipeline on your local machine.
+- The `DirectRunner` runs the pipeline on your local machine.
 - The `DataflowPipelineRunner` submits the pipeline to the [Google Cloud 
Dataflow](http://cloud.google.com/dataflow/).
 - The `FlinkPipelineRunner` runs the pipeline on an Apache Flink cluster. The 
code has been donated from 
[dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) 
and is now part of Beam.
 - The `SparkPipelineRunner` runs the pipeline on an Apache Spark cluster. The 
code has been donated from 
[cloudera/spark-dataflow](https://github.com/cloudera/spark-dataflow) and is 
now part of Beam.

[2/2] incubator-beam git commit: This closes #484

[davor]

This closes #484


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/336ae2e7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/336ae2e7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/336ae2e7

Branch: refs/heads/master
Commit: 336ae2e7e3aba43e16a32f0b6b4dabbf92e8ff02
Parents: da324bf c666d4e
Author: Davor Bonaci 
Authored: Fri Jun 17 11:45:49 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 11:45:49 2016 -0700

--
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #484: [BEAM-234] Remove last vestige of the word...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #478: Remove the beam.examples dependency from f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/478


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Remove the beam.examples dependency from flink.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f57e66c48 -> da324bf52


Remove the beam.examples dependency from flink.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c4684bff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c4684bff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c4684bff

Branch: refs/heads/master
Commit: c4684bffc39ca3c40d2ef333502e0dd08d310591
Parents: f57e66c
Author: Pei He 
Authored: Thu Jun 16 11:38:51 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 11:40:45 2016 -0700

--
 runners/flink/runner/pom.xml | 12 
 1 file changed, 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c4684bff/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 2fdf61d..8933457 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -106,18 +106,6 @@
 
 
 
-  org.apache.beam
-  beam-examples-java
-  
-
-  org.slf4j
-  slf4j-jdk14
-
-  
-  test
-
-
-
   org.apache.flink
   flink-streaming-java_2.10
   ${flink.version}

[2/2] incubator-beam git commit: This closes #478

[davor]

This closes #478


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/da324bf5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/da324bf5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/da324bf5

Branch: refs/heads/master
Commit: da324bf522bdef7b990f79cd30fa6566029a748f
Parents: f57e66c c4684bf
Author: Davor Bonaci 
Authored: Fri Jun 17 11:40:54 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 11:40:54 2016 -0700

--
 runners/flink/runner/pom.xml | 12 
 1 file changed, 12 deletions(-)
--

[jira] [Updated] (BEAM-288) Improve javadoc for UnboundedSource

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-288:
--
Component/s: sdk-java-core

> Improve javadoc for UnboundedSource
> ---
>
> Key: BEAM-288
> URL: https://issues.apache.org/jira/browse/BEAM-288
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> While working on the pubsub source noticed the UnboundedSource and associated 
> Reader / Checkpoint API needed some important clarrification. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-279) Make sure we unit test bounded sessions

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-279:
--
Component/s: sdk-java-core

> Make sure we unit test bounded sessions
> ---
>
> Key: BEAM-279
> URL: https://issues.apache.org/jira/browse/BEAM-279
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Shields
>
> A few customers have been using Window.into(Sessions...) and of course 
> quickly realize they are exposed to unbounded sessions.
> We should have unit tests to confirm various combinations of 
> AfterPane.elementCountAtLeast and AfterProcessingTime... correctly force 
> sessions to be broken apart.
> We should also check this all works with repeated messages with the same 
> timestamp (since they will create the exact same session window and can thus 
> see trigger state from previous sessions).
> At some point we may may flow into reworking bounded sessions to be done 
> directly rather than via Sessions plus triggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-348) Clean temp_dir usage in _stage_extra_packages

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-348.
--
Resolution: Fixed

> Clean temp_dir usage in _stage_extra_packages
> -
>
> Key: BEAM-348
> URL: https://issues.apache.org/jira/browse/BEAM-348
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-348) Clean temp_dir usage in _stage_extra_packages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336562#comment-15336562
 ] 

ASF GitHub Bot commented on BEAM-348:
-

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/475


> Clean temp_dir usage in _stage_extra_packages
> -
>
> Key: BEAM-348
> URL: https://issues.apache.org/jira/browse/BEAM-348
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #475: [BEAM-348] Clean up usage of temp director...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/475


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #485: Back port HDFS enhancements

[nevillelyh]

GitHub user nevillelyh opened a pull request:

https://github.com/apache/incubator-beam/pull/485

Back port HDFS enhancements



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nevillelyh/incubator-beam neville/hdfs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/485.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #485


commit 763a339bbaf6599534f9a1d8069f630be8f81e1e
Author: Neville Li 
Date:   2016-06-17T17:46:35Z

add AvroHDFSFileSource

commit d233d98222e6f830e78dac446d2017e85d051426
Author: Neville Li 
Date:   2016-06-17T17:50:11Z

add HDFSFileSink




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #475

[davor]

This closes #475


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0e5c662b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0e5c662b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0e5c662b

Branch: refs/heads/python-sdk
Commit: 0e5c662b48725664cc448cdb7fe18e6d8c41b315
Parents: 20687e8 c5a49a7
Author: Davor Bonaci 
Authored: Fri Jun 17 10:49:33 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 10:49:33 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 37 
 1 file changed, 12 insertions(+), 25 deletions(-)
--

[1/2] incubator-beam git commit: Clean up usage of temp directories in _stage_extra_packages

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 20687e83a -> 0e5c662b4


Clean up usage of temp directories in _stage_extra_packages

* temp_dir is a required parameter, clarified the doc string
* Renamed similarly named tempdir to temp_dir and creates this new
temp directory under the temp_dir. A separate temp directory is
needed for listing only the downloaded resources at this stage.
* Removed the tempdir clean up stage. Caller is giving the tempdir
and responsible for cleaning it. Only caller (stage_job_resources)
already does this.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c5a49a79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c5a49a79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c5a49a79

Branch: refs/heads/python-sdk
Commit: c5a49a797c104021126205e546a3e20e6d5d9709
Parents: 20687e8
Author: Ahmet Altay 
Authored: Wed Jun 15 18:10:34 2016 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 17 10:49:05 2016 -0700

--
 sdks/python/apache_beam/utils/dependency.py | 37 
 1 file changed, 12 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c5a49a79/sdks/python/apache_beam/utils/dependency.py
--
diff --git a/sdks/python/apache_beam/utils/dependency.py 
b/sdks/python/apache_beam/utils/dependency.py
index 05fcfd5..1c6ad9c 100644
--- a/sdks/python/apache_beam/utils/dependency.py
+++ b/sdks/python/apache_beam/utils/dependency.py
@@ -116,19 +116,18 @@ def _dependency_file_download(from_url, to_folder):
   return local_download_file
 
 
-def _stage_extra_packages(extra_packages,
-  staging_location,
-  file_copy=_dependency_file_copy, temp_dir=None):
+def _stage_extra_packages(extra_packages, staging_location, temp_dir,
+  file_copy=_dependency_file_copy):
   """Stages a list of local extra packages.
 
   Args:
 extra_packages: Ordered list of local paths to extra packages to be staged.
 staging_location: Staging location for the packages.
+temp_dir: Temporary folder where the resource building can happen. Caller
+  is responsible for cleaning up this folder after this function returns.
 file_copy: Callable for copying files. The default version will copy from
   a local file to a GCS location using the gsutil tool available in the
   Google Cloud SDK package.
-temp_dir: Temporary folder where the resource building can happen. If None
-  then a unique temp directory will be created. Used only for testing.
 
   Returns:
 A list of file names (no paths) for the resources staged. All the files
@@ -139,7 +138,7 @@ def _stage_extra_packages(extra_packages,
   name patterns.
   """
   resources = []
-  tempdir = None
+  staging_temp_dir = None
   local_packages = []
   for package in extra_packages:
 if not os.path.basename(package).endswith('.tar.gz'):
@@ -149,11 +148,11 @@ def _stage_extra_packages(extra_packages,
 
 if not os.path.isfile(package):
   if package.startswith('gs://'):
-if not tempdir:
-  tempdir = tempfile.mkdtemp()
+if not staging_temp_dir:
+  staging_temp_dir = tempfile.mkdtemp(dir=temp_dir)
 logging.info('Downloading extra package: %s locally before staging',
  package)
-_dependency_file_copy(package, tempdir)
+_dependency_file_copy(package, staging_temp_dir)
   else:
 raise RuntimeError(
 'The file %s cannot be found. It was specified in the '
@@ -161,9 +160,10 @@ def _stage_extra_packages(extra_packages,
 else:
   local_packages.append(package)
 
-  if tempdir:
+  if staging_temp_dir:
 local_packages.extend(
-[utils.path.join(tempdir, f) for f in os.listdir(tempdir)])
+[utils.path.join(staging_temp_dir, f) for f in os.listdir(
+staging_temp_dir)])
 
   for package in local_packages:
 basename = os.path.basename(package)
@@ -186,18 +186,6 @@ def _stage_extra_packages(extra_packages,
   file_copy(os.path.join(temp_dir, EXTRA_PACKAGES_FILE), staged_path)
   resources.append(EXTRA_PACKAGES_FILE)
 
-  # Remove temp files created by downloading packages from GCS.
-  if tempdir:
-try:
-  temp_files = os.listdir(tempdir)
-  for temp_file in temp_files:
-os.remove(utils.path.join(tempdir, temp_file))
-  os.rmdir(tempdir)
-except OSError as e:
-  logging.info(
-  '%s: (Ignored) Failed to delete all temporary files in %s.',
-  e, tempdir)
-
   return resources
 
 
@@ -311,8 +299,7 @@ def stage_job_resources(
 resources.extend(

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.20150602 [deleted] bdde37492

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.20150727 [deleted] 39ba98739

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.20150710 [deleted] ffb182ee3

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150211 [deleted] 5a11734d6

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150326 [deleted] 8192a7ef6

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150227 [deleted] 0aa4d9d53

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150210 [deleted] 0f05e56cb

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20141216 [deleted] 71b4fe2ff

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.6.0 [deleted] 79e9b3164

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.3.20150109 [deleted] adadb6103

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.2.1 [deleted] eac7a85b0

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.1.0 [deleted] 469dd0b83

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.5.0 [deleted] 05755bb18

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.3.0 [deleted] 0c637d38a

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v1.2.0 [deleted] 8e1de2021

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.1.0-incubating [created] 9287077f5
  refs/tags/v0.3.20141216 [created] 71b4fe2ff
  refs/tags/v0.3.20150109 [created] adadb6103
  refs/tags/v0.3.20150210 [created] 0f05e56cb
  refs/tags/v0.3.20150211 [created] 5a11734d6
  refs/tags/v0.3.20150227 [created] 0aa4d9d53
  refs/tags/v0.3.20150326 [created] 8192a7ef6
  refs/tags/v0.4.20150414 [created] 843eec882
  refs/tags/v0.4.20150602 [created] bdde37492
  refs/tags/v0.4.20150710 [created] ffb182ee3
  refs/tags/v0.4.20150727 [created] 39ba98739
  refs/tags/v1.0.0 [created] f971e4be4
  refs/tags/v1.1.0 [created] 469dd0b83
  refs/tags/v1.2.0 [created] 8e1de2021
  refs/tags/v1.2.1 [created] eac7a85b0
  refs/tags/v1.3.0 [created] 0c637d38a
  refs/tags/v1.4.0 [created] 02acad47e
  refs/tags/v1.5.0 [created] 05755bb18
  refs/tags/v1.5.1 [created] beb506c32
  refs/tags/v1.6.0 [created] 79e9b3164

[GitHub] incubator-beam pull request #484: Remove last vestige of the words DirectPip...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/484

Remove last vestige of the words DirectPipeline

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam finish_direct_rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/484.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #484


commit 3c6805b1cd0f3f8dcc1d17d0aebd1ca08bd2e084
Author: Thomas Groh 
Date:   2016-06-17T17:25:44Z

Remove last vestige of the words DirectPipeline




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #483: Use native Flink rebalance for Reshuffle T...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/483

Use native Flink rebalance for Reshuffle Transform

R: @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam flink/rebalance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/483.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #483


commit 8d49f03ae3c8446c8b5d11e9d50f877ede2781f9
Author: Aljoscha Krettek 
Date:   2016-06-17T13:16:19Z

Use native Flink rebalance for Reshuffle Transform




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[incubator-beam] Git Push Summary

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/runners-spark2 [created] f57e66c48