[jira] [Created] (BEAM-257) fix ConcurrentModificationException in KafkaIO

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-257:
-

 Summary: fix ConcurrentModificationException in KafkaIO
 Key: BEAM-257
 URL: https://issues.apache.org/jira/browse/BEAM-257
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: James Malone


[~tgroh] reported the following exception with KafkaIO while using with 
DirectRunner. Fixing it in https://github.com/apache/incubator-beam/pull/290

{code}
 java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
multi-threaded access
at 
org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1324)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1255)
at com.google.common.io.Closeables.close(Closeables.java:79)
at 
org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaReader.close(KafkaIO.java:1050)
at 
org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.finishBundle(UnboundedReadEvaluatorFactory.java:167)
at 
org.apache.beam.runners.direct.TransformExecutor.finishBundle(TransformExecutor.java:161)
at 
org.apache.beam.runners.direct.TransformExecutor.call(TransformExecutor.java:116)
at 
org.apache.beam.runners.direct.TransformExecutor.call(TransformExecutor.java:41)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: - after shutting down consumer poll t...

[rangadi]

GitHub user rangadi opened a pull request:

https://github.com/apache/incubator-beam/pull/290

 - after shutting down consumer poll thread (and offset updater thread), 
wait for them to actually exit.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - handle a possible race between poll thread and close().

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/incubator-beam shutdown

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #290


commit 8cdb8fbd22ec4e1d806417ef28d8a69fcdf6a1a5
Author: Raghu Angadi 
Date:   2016-05-05T01:34:20Z

 - after shutting down, wait for the threads to actually finish
 - handle a possible race between poll thread and close().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: [BEAM-22] Mark CheckpointMark as vola...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/289

[BEAM-22] Mark CheckpointMark as volatile in UnboundedReadEvaluator

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The evaluator may be reused in a different thread, and updates to the
checkpoint must be visible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
unbounded_evaluator_close_before_requeue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #289


commit e6ba15d451c9e3574a29ac0d51e6275893c9ee60
Author: Thomas Groh 
Date:   2016-05-05T00:44:59Z

Mark CheckpointMark as volatile in UnboundedReadEvaluator

The evaluator may be reused in a different thread, and updates to the
checkpoint must be visible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271716#comment-15271716
 ] 

ASF GitHub Bot commented on BEAM-22:


GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/289

[BEAM-22] Mark CheckpointMark as volatile in UnboundedReadEvaluator

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The evaluator may be reused in a different thread, and updates to the
checkpoint must be visible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
unbounded_evaluator_close_before_requeue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #289


commit e6ba15d451c9e3574a29ac0d51e6275893c9ee60
Author: Thomas Groh 
Date:   2016-05-05T00:44:59Z

Mark CheckpointMark as volatile in UnboundedReadEvaluator

The evaluator may be reused in a different thread, and updates to the
checkpoint must be visible.




> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #278

[Apache Jenkins Server]

See 

[1/2] incubator-beam-site git commit: Added the /docs/ directory and a landing page at docs/index.md. Added coming-soon.md as a placeholder for missing content. Modified main landing page with SDK and

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 3b54e40ce -> c90dbf184


Added the /docs/ directory and a landing page at docs/index.md.
Added coming-soon.md as a placeholder for missing content.
Modified main landing page with SDK and Runner status tables and descriptions.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/34ae3e35
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/34ae3e35
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/34ae3e35

Branch: refs/heads/asf-site
Commit: 34ae3e35c71e63318fe1071e471bf52d78077463
Parents: 3b54e40
Author: Devin Donnelly 
Authored: Wed Apr 27 14:18:58 2016 -0700
Committer: Devin Donnelly 
Committed: Wed May 4 16:59:22 2016 -0700

--
 coming-soon.md|  17 
 content/capability-matrix/index.html  |   2 +-
 content/coming-soon.html  | 126 +
 content/contribution-guide/index.html |   4 +-
 content/docs/index.html   | 144 +
 content/feed.xml  |   6 +-
 content/index.html|  82 
 docs/index.md |  29 ++
 index.md  |  70 +++---
 9 files changed, 442 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/coming-soon.md
--
diff --git a/coming-soon.md b/coming-soon.md
new file mode 100644
index 000..370af20
--- /dev/null
+++ b/coming-soon.md
@@ -0,0 +1,17 @@
+---
+layout: default
+---
+
+  
+  
+  
+  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
+  
+
+
+# Documentation Coming Soon
+
+You've reached a page that's still in draft, or otherwise being developed! 
Please bear with us as we improve the documentation for Apache Beam.
+
+[Go Back](/) to the main Beam site.
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index 543f899..e4d0c01 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -89,7 +89,7 @@
 
   
 Apache Beam Capability 
Matrix
-Last updated: 2016-04-29 14:54 
PDT
+Last updated: 2016-05-04 16:48 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf;>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/34ae3e35/content/coming-soon.html
--
diff --git a/content/coming-soon.html b/content/coming-soon.html
new file mode 100644
index 000..5491f6b
--- /dev/null
+++ b/content/coming-soon.html
@@ -0,0 +1,126 @@
+
+
+
+  
+  
+  
+  
+
+  Apache Beam (incubating)
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/coming-soon.html;>
+  http://beam.incubator.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+
+
+  
+
+  Documentation 
+  
+Guides
+Getting Started
+
+Technical Documentation
+Capability Matrix
+https://goo.gl/ps8twC;>Technical Docs
+https://goo.gl/nk5OM0;>Technical Vision
+  
+
+
+  Community 
+  
+Community
+

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #277

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Speed up non-release builds

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 928640df2 -> e63311fa4


Speed up non-release builds


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/119812a6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/119812a6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/119812a6

Branch: refs/heads/master
Commit: 119812a66bceb9e8b91dcd7afb26bc9b7ee5eee7
Parents: 6819dff
Author: Kenneth Knowles 
Authored: Tue May 3 21:11:39 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed May 4 14:23:56 2016 -0700

--
 pom.xml | 141 ++
 runners/direct-java/pom.xml |   8 --
 sdks/java/core/pom.xml  | 180 +++
 sdks/pom.xml|  21 +
 4 files changed, 197 insertions(+), 153 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/119812a6/pom.xml
--
diff --git a/pom.xml b/pom.xml
index b0b258b..03534ee 100644
--- a/pom.xml
+++ b/pom.xml
@@ -132,19 +132,79 @@
 
   
 sdks
-
+
 runners/google-cloud-dataflow-java
 runners
-
+
 sdks/java/maven-archetypes
 examples
   
 
   
+
+
+
+  release
+  
+
+  
+
+  org.apache.maven.plugins
+  maven-javadoc-plugin
+  
+
+  javadoc
+  package
+  
+jar
+  
+
+  
+
+
+
+  org.apache.maven.plugins
+  maven-source-plugin
+  
+
+  attach-sources
+  package
+  
+jar-no-fork
+  
+
+
+  attach-test-sources
+  package
+  
+test-jar-no-fork
+  
+
+  
+
+
+
+  org.apache.maven.plugins
+  maven-jar-plugin
+  
+
+  default-test-jar
+  
+test-jar
+  
+
+  
+
+  
+
+  
+
+
 
   doclint-java8-disable
   
@@ -154,6 +214,7 @@
 -Xdoclint:-missing
   
 
+
 
   src
   
@@ -329,8 +390,8 @@
 google-api-services-storage
 ${storage.version}
 
-  
+  
   
 com.google.guava
 guava-jdk5
@@ -343,8 +404,8 @@
 google-api-services-datastore-protobuf
 ${datastore.version}
 
-  
+  
   
 com.google.guava
 guava-jdk5
@@ -407,8 +468,8 @@
 google-api-services-dataflow
 ${dataflow.version}
 
-  
+  
   
 com.google.guava
 guava-jdk5
@@ -421,8 +482,8 @@
 google-api-services-clouddebugger
 ${clouddebugger.version}
 
-  
+  
   
 com.google.guava
 guava-jdk5
@@ -490,6 +551,7 @@
 test-jar
 test
   
+
   
 org.apache.beam
 java-sdk-all
@@ -599,7 +661,7 @@
   -Xlint:-try
   -Xlint:-unchecked
   -Xlint:-varargs
-  
+  
   
   
 
@@ -620,12 +682,6 @@
 jar
   
 
-
-  default-test-jar
-  
-test-jar
-  
-
   
 
 
@@ -633,37 +689,12 @@
   org.apache.maven.plugins
   maven-source-plugin
   2.4
-  
-
-  attach-sources
-  package
-  
-jar-no-fork
-  
-
-
-  attach-test-sources
-  package
-  
-test-jar-no-fork
-  
-
-  
 
 
 
   org.apache.maven.plugins
   maven-javadoc-plugin
   2.10.3
-  
-
-  javadoc
-  package
-  
-jar
-  
-
-  
 
 
 
@@ -717,11 +748,11 @@
   
 
   
-
-  
-**/AutoValue_*.class
-  
-
+  
+
+  **/AutoValue_*.class
+
+  

[GitHub] incubator-beam pull request: Speed up non-release builds

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/286


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #286

[kenn]

This closes #286


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e63311fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e63311fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e63311fa

Branch: refs/heads/master
Commit: e63311fa4a1ae72f901900bcac0f6a13e200f6fc
Parents: 928640d 119812a
Author: Kenneth Knowles 
Authored: Wed May 4 14:24:47 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed May 4 14:24:47 2016 -0700

--
 pom.xml | 141 ++
 runners/direct-java/pom.xml |   8 --
 sdks/java/core/pom.xml  | 180 +++
 sdks/pom.xml|  21 +
 4 files changed, 197 insertions(+), 153 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e63311fa/pom.xml
--

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e63311fa/runners/direct-java/pom.xml
--

[GitHub] incubator-beam pull request: Refactor BigQueryServices to support ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/285


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-48] Refactor BigQueryServices to support extract and query jobs

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 97945648c -> 928640df2


[BEAM-48] Refactor BigQueryServices to support extract and query jobs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2fe1ebc8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2fe1ebc8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2fe1ebc8

Branch: refs/heads/master
Commit: 2fe1ebc8ebbabe8836b73d3c7c5a251dc2d746bf
Parents: 9794564
Author: Pei He 
Authored: Tue May 3 17:55:11 2016 -0700
Committer: Dan Halperin 
Committed: Wed May 4 14:10:13 2016 -0700

--
 .../java/org/apache/beam/sdk/io/BigQueryIO.java |  40 -
 .../apache/beam/sdk/util/BigQueryServices.java  |  40 +++--
 .../beam/sdk/util/BigQueryServicesImpl.java | 148 ---
 .../org/apache/beam/sdk/io/BigQueryIOTest.java  |  79 ++
 .../beam/sdk/util/BigQueryServicesImplTest.java |  79 +-
 5 files changed, 252 insertions(+), 134 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2fe1ebc8/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
index 4a0bdac..7785298 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
@@ -47,7 +47,7 @@ import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.BigQueryServices;
-import org.apache.beam.sdk.util.BigQueryServices.LoadService;
+import org.apache.beam.sdk.util.BigQueryServices.JobService;
 import org.apache.beam.sdk.util.BigQueryServicesImpl;
 import org.apache.beam.sdk.util.BigQueryTableInserter;
 import org.apache.beam.sdk.util.BigQueryTableRowIterator;
@@ -68,7 +68,9 @@ import org.apache.beam.sdk.values.PInput;
 
 import com.google.api.client.json.JsonFactory;
 import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Job;
 import com.google.api.services.bigquery.model.JobConfigurationLoad;
+import com.google.api.services.bigquery.model.JobStatus;
 import com.google.api.services.bigquery.model.QueryRequest;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
@@ -1168,6 +1170,10 @@ public class BigQueryIO {
   // The maximum number of retry load jobs.
   private static final int MAX_RETRY_LOAD_JOBS = 3;
 
+  // The maximum number of retries to poll the status of a load job.
+  // It sets to {@code Integer.MAX_VALUE} to block until the BigQuery job 
finishes.
+  private static final int MAX_JOB_STATUS_POLL_RETRIES = Integer.MAX_VALUE;
+
   private final BigQuerySink bigQuerySink;
 
   private BigQueryWriteOperation(BigQuerySink sink) {
@@ -1191,7 +1197,7 @@ public class BigQueryIO {
   }
   if (!tempFiles.isEmpty()) {
   load(
-  bigQuerySink.bqServices.getLoadService(bqOptions),
+  bigQuerySink.bqServices.getJobService(bqOptions),
   bigQuerySink.jobIdToken,
   fromJsonString(bigQuerySink.jsonTable, TableReference.class),
   tempFiles,
@@ -1215,7 +1221,7 @@ public class BigQueryIO {
* If a load job failed, it will try another load job with a 
different job id.
*/
   private void load(
-  LoadService loadService,
+  JobService jobService,
   String jobIdPrefix,
   TableReference ref,
   List gcsUris,
@@ -1238,8 +1244,9 @@ public class BigQueryIO {
 LOG.info("Previous load jobs failed, retrying.");
   }
   LOG.info("Starting BigQuery load job: {}", jobId);
-  loadService.startLoadJob(jobId, loadConfig);
-  BigQueryServices.Status jobStatus = 
loadService.pollJobStatus(projectId, jobId);
+  jobService.startLoadJob(jobId, loadConfig);
+  Status jobStatus = parseStatus(
+  jobService.pollJob(projectId, jobId, 
MAX_JOB_STATUS_POLL_RETRIES));
   switch (jobStatus) {
 case SUCCEEDED:
   return;
@@ -1669,6 +1676,29 @@ public class BigQueryIO {
 }
   }
 
+  /**
+   * Status of a BigQuery job or request.
+   */
+  enum Status {
+SUCCEEDED,
+FAILED,
+UNKNOWN,
+  }
+
+  private static Status parseStatus(@Nullable Job job) {
+if (job == null) {
+  return Status.UNKNOWN;
+}
+

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271078#comment-15271078
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/258


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-22] Add PushbackDoFnRunner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/258


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam git commit: Move ReadyCheckingSideInputReader to util

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 32970c985 -> 97945648c


Move ReadyCheckingSideInputReader to util

This SideInputReader allows callers to check for a side input being
available before attempting to read the contents


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3e8df24a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3e8df24a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3e8df24a

Branch: refs/heads/master
Commit: 3e8df24a2ced82be7ebe26837f96f651acc1ac06
Parents: b9116ac
Author: Thomas Groh 
Authored: Mon May 2 10:03:43 2016 -0700
Committer: Thomas Groh 
Committed: Mon May 2 10:25:25 2016 -0700

--
 .../direct/InProcessEvaluationContext.java  | 11 +--
 .../direct/InProcessSideInputContainer.java |  2 +-
 .../direct/InProcessSideInputContainerTest.java |  2 +-
 .../sdk/util/ReadyCheckingSideInputReader.java  | 34 
 4 files changed, 37 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3e8df24a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
index d9a7ff0..92e5aa5 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
@@ -31,6 +31,7 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.util.ExecutionContext;
+import org.apache.beam.sdk.util.ReadyCheckingSideInputReader;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -333,16 +334,6 @@ class InProcessEvaluationContext {
 return sideInputContainer.createReaderForViews(sideInputs);
   }
 
-  /**
-   * A {@link SideInputReader} that allows callers to check to see if a {@link 
PCollectionView} has
-   * had its contents set in a window.
-   */
-  static interface ReadyCheckingSideInputReader extends SideInputReader {
-/**
- * Returns true if the {@link PCollectionView} is ready in the provided 
{@link BoundedWindow}.
- */
-boolean isReady(PCollectionView view, BoundedWindow window);
-  }
 
   /**
* Create a {@link CounterSet} for this {@link Pipeline}. The {@link 
CounterSet} is independent

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3e8df24a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
index f4980ef..d0f29ff 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
@@ -19,10 +19,10 @@ package org.apache.beam.runners.direct;
 
 import static com.google.common.base.Preconditions.checkArgument;
 
-import 
org.apache.beam.runners.direct.InProcessEvaluationContext.ReadyCheckingSideInputReader;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.PCollectionViewWindow;
+import org.apache.beam.sdk.util.ReadyCheckingSideInputReader;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3e8df24a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessSideInputContainerTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessSideInputContainerTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessSideInputContainerTest.java
index d8a78f2..8f89e70 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessSideInputContainerTest.java
+++ 

[3/3] incubator-beam git commit: This closes #258

[kenn]

This closes #258


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/97945648
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/97945648
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/97945648

Branch: refs/heads/master
Commit: 97945648c8efbaaea6780ec1b6027c6b5be7527f
Parents: 32970c9 f57c1dc
Author: Kenneth Knowles 
Authored: Wed May 4 10:52:19 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed May 4 10:52:19 2016 -0700

--
 .../direct/InProcessEvaluationContext.java  |  11 +-
 .../direct/InProcessSideInputContainer.java |   2 +-
 .../direct/InProcessSideInputContainerTest.java |   2 +-
 .../sdk/util/PushbackSideInputDoFnRunner.java   | 115 +
 .../sdk/util/ReadyCheckingSideInputReader.java  |  34 +++
 .../sdk/util/IdentitySideInputWindowFn.java |  54 +
 .../util/PushbackSideInputDoFnRunnerTest.java   | 234 +++
 7 files changed, 440 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/97945648/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
--

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/97945648/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessSideInputContainer.java
--

[jira] [Commented] (BEAM-77) Reorganize Directory structure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271033#comment-15271033
 ] 

ASF GitHub Bot commented on BEAM-77:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/281


> Reorganize Directory structure
> --
>
> Key: BEAM-77
> URL: https://issues.apache.org/jira/browse/BEAM-77
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Reporter: Frances Perry
>Assignee: Jean-Baptiste Onofré
>
> Now that we've done the initial Dataflow code drop, we will restructure 
> directories to provide space for additional SDKs and Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-77] Create runners/core module ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/281


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #281

[kenn]

This closes #281


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/32970c98
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/32970c98
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/32970c98

Branch: refs/heads/master
Commit: 32970c985f5b557101533ba2bb2d3d6352e284fa
Parents: 1ef53b1 78798ad
Author: Kenneth Knowles 
Authored: Wed May 4 10:27:27 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed May 4 10:27:27 2016 -0700

--
 pom.xml |   6 +
 runners/core-java/pom.xml   | 208 +++
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java | 112 ++
 runners/core-java/src/test/java/.placeholder|   0
 runners/direct-java/pom.xml |   5 +
 .../direct/GroupByKeyEvaluatorFactory.java  |   2 +-
 runners/flink/runner/pom.xml|  11 +
 .../FlinkGroupAlsoByWindowWrapper.java  |   8 +-
 runners/pom.xml |   1 +
 .../util/GroupAlsoByWindowViaWindowSetDoFn.java | 105 --
 10 files changed, 348 insertions(+), 110 deletions(-)
--

[2/3] incubator-beam git commit: Create runners/core module for artifact org.apache.beam:runners-core

[kenn]

Create runners/core module for artifact org.apache.beam:runners-core

This is strictly creating the module and moving one easy class to it.
Many of the utilities in org.apache.beam.util and subpackages should
move as developments allow.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/892ead2c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/892ead2c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/892ead2c

Branch: refs/heads/master
Commit: 892ead2c3b8bf7a53ee2d7570ba587453c186009
Parents: 6819dff
Author: Kenneth Knowles 
Authored: Tue May 3 13:22:59 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed May 4 09:44:05 2016 -0700

--
 pom.xml |   6 +
 runners/core-java/pom.xml   | 208 +++
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java | 112 ++
 runners/core-java/src/test/java/.placeholder|   0
 runners/direct-java/pom.xml |  21 ++
 .../direct/GroupByKeyEvaluatorFactory.java  |   2 +-
 runners/flink/runner/pom.xml|  11 +
 .../FlinkGroupAlsoByWindowWrapper.java  |   8 +-
 runners/pom.xml |   1 +
 .../util/GroupAlsoByWindowViaWindowSetDoFn.java | 105 --
 10 files changed, 364 insertions(+), 110 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/892ead2c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index b0b258b..27787da 100644
--- a/pom.xml
+++ b/pom.xml
@@ -196,6 +196,12 @@
 
   
 org.apache.beam
+runners-core
+${project.version}
+  
+
+  
+org.apache.beam
 google-cloud-dataflow-java-runner
 ${project.version}
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/892ead2c/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
new file mode 100644
index 000..b6f6f29
--- /dev/null
+++ b/runners/core-java/pom.xml
@@ -0,0 +1,208 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+  4.0.0
+
+  
+org.apache.beam
+runners-parent
+0.1.0-incubating-SNAPSHOT
+../pom.xml
+  
+
+  runners-core
+  Apache Beam :: Runners :: Core
+  Beam Runners Core provides utilities to aid runner 
authors.
+
+  jar
+
+  
+
+  
+org.apache.maven.plugins
+maven-compiler-plugin
+  
+
+  
+org.apache.maven.plugins
+maven-dependency-plugin
+
+  
+analyze-only
+
+  true
+
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-checkstyle-plugin
+  
+
+  
+org.apache.maven.plugins
+maven-jar-plugin
+  
+
+  
+  
+org.apache.maven.plugins
+maven-source-plugin
+  
+
+  
+org.apache.maven.plugins
+maven-javadoc-plugin
+
+  Beam Runners Core utilities ${project.version} 
API
+  Beam Runners Core utilities for Java, version 
${project.version}
+  ../javadoc/overview.html
+
+  org.apache.beam.runners.core
+  false
+  true
+  
+
+
+  
+
+  jar
+
+package
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-shade-plugin
+2.4.1
+
+  
+  
+bundle-and-repackage
+package
+
+  shade
+
+
+  true
+  
+
+  com.google.guava:guava
+
+  
+  
+
+  *:*
+  
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+  
+
+
+  com.google.common
+  
org.apache.beam.sdk.repackaged.com.google.common
+
+
+  com.google.thirdparty
+  
org.apache.beam.sdk.repackaged.com.google.thirdparty
+
+  
+
+  
+
+  
+  
+bundle-rest-without-repackaging
+package
+
+  shade
+
+  

[jira] [Commented] (BEAM-254) Add maven-rat-plugin execution

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270899#comment-15270899
 ] 

Jean-Baptiste Onofré commented on BEAM-254:
---

Good point. Actually, I think it's better to exclude {{**/*.java}} in the rat 
configuration. Thanks to that, rat will only consider "non" java files.

> Add maven-rat-plugin execution
> --
>
> Key: BEAM-254
> URL: https://issues.apache.org/jira/browse/BEAM-254
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> In order to check the license headers and validate we are "Apache compliant", 
> I will add the maven-rat-plugin execution in the main parent POM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-254) Add maven-rat-plugin execution

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270888#comment-15270888
 ] 

Scott Wegner commented on BEAM-254:
---

I see; I hadn't considered that we need a header on non-source files.

When you add this, should we remove the checkstyle Header check? As long as 
it's redundant, removing it should be safe and help speed up the build a bit.

> Add maven-rat-plugin execution
> --
>
> Key: BEAM-254
> URL: https://issues.apache.org/jira/browse/BEAM-254
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> In order to check the license headers and validate we are "Apache compliant", 
> I will add the maven-rat-plugin execution in the main parent POM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)