[jira] [Updated] (BEAM-752) infrastructure for toggling code snippets in documentation

[Frances Perry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frances Perry updated BEAM-752:
---
Assignee: Abdullah Bashir  (was: James Malone)

> infrastructure for toggling code snippets in documentation
> --
>
> Key: BEAM-752
> URL: https://issues.apache.org/jira/browse/BEAM-752
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Abdullah Bashir
>  Labels: starter
>
> Once the python sdk gets merged to the master branch, a lot of our 
> documentation (programming guide, walkthroughs, etc) will need to support 
> multiple languages.
> The hope is that the vast bulk of the prose can be written about Beam 
> concepts in a language independent way. But for code snippets it would be 
> great to be able to toggle languages.
> Goals:
> * Support tabbed language toggles for both code and small sections of text.
> * Support easily changing the default per-user-visit so that the entire file 
> (or even better entire site) defaults to showing a specific language



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-752) infrastructure for toggling code snippets in documentation

[Abdullah Bashir (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627742#comment-15627742
 ] 

Abdullah Bashir commented on BEAM-752:
--

Hi i have submitted a pull request over here. 

https://github.com/apache/incubator-beam-site/pull/59

Can you review it.

Thanks.

> infrastructure for toggling code snippets in documentation
> --
>
> Key: BEAM-752
> URL: https://issues.apache.org/jira/browse/BEAM-752
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: James Malone
>  Labels: starter
>
> Once the python sdk gets merged to the master branch, a lot of our 
> documentation (programming guide, walkthroughs, etc) will need to support 
> multiple languages.
> The hope is that the vast bulk of the prose can be written about Beam 
> concepts in a language independent way. But for code snippets it would be 
> great to be able to toggle languages.
> Goals:
> * Support tabbed language toggles for both code and small sections of text.
> * Support easily changing the default per-user-visit so that the entire file 
> (or even better entire site) defaults to showing a specific language



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-501) Update website skin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627735#comment-15627735
 ] 

ASF GitHub Bot commented on BEAM-501:
-

GitHub user mabdullah353 opened a pull request:

https://github.com/apache/incubator-beam-site/pull/59

[BEAM-501] Infrastructure for toggling code snippets in documentation.

## Working

This commit will enable mark down language to support two languages `java` 
and `python`, By adding a nav bar on top of code blocks.

Add html class as `language-python` or `language-java` and the HTML 
elements visibility will be controlled based on user selected language 
preferences.

Code blocks in markdown languages starting with *```java* or *```python* 
have by default `langauge-java` or `language-python` classes so no special 
attributes required to apply this effect.

## Example

1) Toggles for code block.

Add python code below the java code in markdown language, It will be 
converted into selectable code block div.

> ```java "some-code-in-java"

> ```python "sam-code-in-python"

And user will see block like this.

https://cloud.githubusercontent.com/assets/1429432/19917036/70f0fcc8-a0e1-11e6-9a10-270927936d7f.png;>


2) Section of text.

Wrap a text with attributes having class `language-java` or 
`language-python` and it's visibility will be changed based on user preferences.

> To use Beam, your driver program must first create an instance of the 
Beam SDK class `Pipeline` (typically in the 
`main()` function)(typically in 
`def`).

When `java` is selected then this will render as:
https://cloud.githubusercontent.com/assets/1429432/19917102/045e7242-a0e2-11e6-9b02-615421a5fec4.png;>

And for `python` it will render as:

https://cloud.githubusercontent.com/assets/1429432/19917113/1d2dfee6-a0e2-11e6-80b2-ba55f07f544e.png;>


3) User preference persistence:

The language preference will be stored on HTML5 local storage, keeping it 
persistence per user bases.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mabdullah353/incubator-beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit 2d80cee3be9da2515e927afd67541899b2dde258
Author: Abdullah Bashir 
Date:   2016-11-02T04:04:41Z

[BEAM-501] Infrastructure for toggling code snippets in documentation




> Update website skin
> ---
>
> Key: BEAM-501
> URL: https://issues.apache.org/jira/browse/BEAM-501
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: Jean-Baptiste Onofré
>
> Update the main landing page and website skin as discussed here
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #59: [BEAM-501] Infrastructure for toggling...

[mabdullah353]

GitHub user mabdullah353 opened a pull request:

https://github.com/apache/incubator-beam-site/pull/59

[BEAM-501] Infrastructure for toggling code snippets in documentation.

## Working

This commit will enable mark down language to support two languages `java` 
and `python`, By adding a nav bar on top of code blocks.

Add html class as `language-python` or `language-java` and the HTML 
elements visibility will be controlled based on user selected language 
preferences.

Code blocks in markdown languages starting with *```java* or *```python* 
have by default `langauge-java` or `language-python` classes so no special 
attributes required to apply this effect.

## Example

1) Toggles for code block.

Add python code below the java code in markdown language, It will be 
converted into selectable code block div.

> ```java "some-code-in-java"

> ```python "sam-code-in-python"

And user will see block like this.

https://cloud.githubusercontent.com/assets/1429432/19917036/70f0fcc8-a0e1-11e6-9a10-270927936d7f.png;>


2) Section of text.

Wrap a text with attributes having class `language-java` or 
`language-python` and it's visibility will be changed based on user preferences.

> To use Beam, your driver program must first create an instance of the 
Beam SDK class `Pipeline` (typically in the 
`main()` function)(typically in 
`def`).

When `java` is selected then this will render as:
https://cloud.githubusercontent.com/assets/1429432/19917102/045e7242-a0e2-11e6-9b02-615421a5fec4.png;>

And for `python` it will render as:

https://cloud.githubusercontent.com/assets/1429432/19917113/1d2dfee6-a0e2-11e6-80b2-ba55f07f544e.png;>


3) User preference persistence:

The language preference will be stored on HTML5 local storage, keeping it 
persistence per user bases.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mabdullah353/incubator-beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/59.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #59


commit 2d80cee3be9da2515e927afd67541899b2dde258
Author: Abdullah Bashir 
Date:   2016-11-02T04:04:41Z

[BEAM-501] Infrastructure for toggling code snippets in documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-879) Renaming DeDupExample to DistinctExample

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian updated BEAM-879:
-
Description: 
In BEAM-239, we renamed DeDupExampleTest to DistinctExampleTest.
Need to modify DeDupExample to DistinctExample as well.

> Renaming DeDupExample to DistinctExample
> 
>
> Key: BEAM-879
> URL: https://issues.apache.org/jira/browse/BEAM-879
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Neelesh Srinivas Salian
>Assignee: Neelesh Srinivas Salian
>Priority: Trivial
>
> In BEAM-239, we renamed DeDupExampleTest to DistinctExampleTest.
> Need to modify DeDupExample to DistinctExample as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-879) Renaming DeDupExample to DistinctExample

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian updated BEAM-879:
-
Summary: Renaming DeDupExample to DistinctExample  (was: Renaming 
DeDupExample.java to DistinctExample.java)

> Renaming DeDupExample to DistinctExample
> 
>
> Key: BEAM-879
> URL: https://issues.apache.org/jira/browse/BEAM-879
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Neelesh Srinivas Salian
>Assignee: Neelesh Srinivas Salian
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627669#comment-15627669
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1249


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-879) Renaming DeDupExample.java to DistinctExample.java

[Neelesh Srinivas Salian (JIRA)]

Neelesh Srinivas Salian created BEAM-879:


 Summary: Renaming DeDupExample.java to DistinctExample.java
 Key: BEAM-879
 URL: https://issues.apache.org/jira/browse/BEAM-879
 Project: Beam
  Issue Type: Sub-task
Reporter: Neelesh Srinivas Salian
Assignee: Neelesh Srinivas Salian
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Add templateRunner option to Dataflow runner

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8883877ae -> 2c0d0f476


Add templateRunner option to Dataflow runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/be4c0256
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/be4c0256
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/be4c0256

Branch: refs/heads/master
Commit: be4c0256f9be0813692674ba931579a72f9cc15c
Parents: 8883877
Author: Sam McVeety 
Authored: Tue Nov 1 09:27:44 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 21:31:55 2016 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 45 ++---
 .../options/DataflowPipelineOptions.java|  8 +++
 .../dataflow/util/DataflowTemplateJob.java  | 70 
 .../runners/dataflow/DataflowRunnerTest.java| 45 +
 4 files changed, 158 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/be4c0256/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index ce126db..841b13f 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.dataflow;
 
+import static com.google.common.base.MoreObjects.firstNonNull;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
@@ -49,7 +50,6 @@ import com.google.common.collect.Iterables;
 import com.google.common.collect.Maps;
 import com.google.common.collect.Multimap;
 import java.io.File;
-import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
@@ -58,6 +58,8 @@ import java.io.Serializable;
 import java.net.URISyntaxException;
 import java.net.URL;
 import java.net.URLClassLoader;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
@@ -86,6 +88,7 @@ import 
org.apache.beam.runners.dataflow.internal.ReadTranslator;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
+import org.apache.beam.runners.dataflow.util.DataflowTemplateJob;
 import org.apache.beam.runners.dataflow.util.DataflowTransport;
 import org.apache.beam.runners.dataflow.util.MonitoringUtil;
 import org.apache.beam.sdk.Pipeline;
@@ -140,6 +143,7 @@ import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.util.InstanceBuilder;
+import org.apache.beam.sdk.util.MimeTypes;
 import org.apache.beam.sdk.util.PCollectionViews;
 import org.apache.beam.sdk.util.PathValidator;
 import org.apache.beam.sdk.util.PropertyNames;
@@ -550,16 +554,37 @@ public class DataflowRunner extends 
PipelineRunner {
   hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
 }
 
-if (!isNullOrEmpty(options.getDataflowJobFile())) {
-  try (PrintWriter printWriter = new PrintWriter(
-  new File(options.getDataflowJobFile( {
-String workSpecJson = DataflowPipelineTranslator.jobToString(newJob);
+if (!isNullOrEmpty(options.getDataflowJobFile())
+|| !isNullOrEmpty(options.getTemplateLocation())) {
+  boolean isTemplate = !isNullOrEmpty(options.getTemplateLocation());
+  if (isTemplate) {
+checkArgument(isNullOrEmpty(options.getDataflowJobFile()),
+"--dataflowJobFile and --templateLocation are mutually 
exclusive.");
+  }
+  String fileLocation = firstNonNull(
+  options.getTemplateLocation(), options.getDataflowJobFile());
+  checkArgument(fileLocation.startsWith("/") || 
fileLocation.startsWith("gs://"),
+  String.format(
+  "Location must be local or on Cloud Storage, got {}.", 
fileLocation));
+  String workSpecJson = DataflowPipelineTranslator.jobToString(newJob);
+  try (
+  WritableByteChannel writer =
+  

[GitHub] incubator-beam pull request #1249: [BEAM-551] Add --templateRunner option to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1249


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1249

[dhalperi]

Closes #1249


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2c0d0f47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2c0d0f47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2c0d0f47

Branch: refs/heads/master
Commit: 2c0d0f476ba92bf3b18bcc96f3dac47da176575d
Parents: 8883877 be4c025
Author: Dan Halperin 
Authored: Tue Nov 1 21:31:56 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 21:31:56 2016 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 45 ++---
 .../options/DataflowPipelineOptions.java|  8 +++
 .../dataflow/util/DataflowTemplateJob.java  | 70 
 .../runners/dataflow/DataflowRunnerTest.java| 45 +
 4 files changed, 158 insertions(+), 10 deletions(-)
--

[GitHub] incubator-beam pull request #1256: On timer invoker

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1256

On timer invoker

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff 

This comes after #1252 but is otherwise independent. You can review just 
the additional commit.

I have a few more thoughts here that I would love feedback on. I am trying 
to avoid major overhauls. I will leave my concerns as a self-review so they are 
contextualized.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam OnTimerInvoker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1256.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1256


commit 15a8901ade07c1649378a0b72238eb514877c61c
Author: Kenneth Knowles 
Date:   2016-11-01T21:48:54Z

Switch DoFnSignature, etc, from TypeToken to TypeDescriptor

commit 6de216695c05e4d9b283f0842aed8713075be7aa
Author: Kenneth Knowles 
Date:   2016-11-01T21:50:24Z

DoFnSignature: Make TypeDescriptor-returning methods public

commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32
Author: Kenneth Knowles 
Date:   2016-11-01T22:38:01Z

Add TypeDescriptor#getTypes

commit f2c2f0736d7904474041ed16f782a1c19ca6264e
Author: Kenneth Knowles 
Date:   2016-11-01T04:30:40Z

Refactor and reuse parameter analysis in DoFnSignatures

commit a0f5a1521c3b0d8d195f63d15dcc1779640edd8b
Author: Kenneth Knowles 
Date:   2016-11-02T02:58:35Z

fixup! use mutable style for contexts

commit aa31ebcfcb51078c39de715c63b01c33f8b0db46
Author: Kenneth Knowles 
Date:   2016-11-01T02:26:38Z

Add OnTimerInvoker(s), for invoking DoFn @OnTimer methods

OnTimerInvoker encapsulates the dispatch from onTimer()
to a call to the DoFn method annotated with @OnTimer().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-59) IOChannelFactory rethinking/redesign

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627552#comment-15627552
 ] 

ASF GitHub Bot commented on BEAM-59:


GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1255

[BEAM-59] Use ServiceLoader to register IOChannelFactories.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam io-channel-register

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1255


commit 18e204b7a0b90705e4b0fb41093a15341bffb996
Author: Pei He 
Date:   2016-11-01T00:58:31Z

[BEAM-59] Add use Factory interface in Gcs/File/IOChannelFactory.

commit fd940fa32cc96933c85b2cf93306b6572b783d1a
Author: Pei He 
Date:   2016-11-01T01:01:41Z

[BEAM-59] Create IOChannelFactoryFactoryRegistrar interface and its 
gcs/file implementations.

commit f8b30dda94555e9dada829e7d626a2d3da952804
Author: Pei He 
Date:   2016-11-01T01:02:49Z

[BEAM-59] Use ServiceLoader to register IOChannelFactories in 
IOChannelUtils.




> IOChannelFactory rethinking/redesign
> 
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Pei He
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1255: [BEAM-59] Use ServiceLoader to register I...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1255

[BEAM-59] Use ServiceLoader to register IOChannelFactories.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam io-channel-register

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1255


commit 18e204b7a0b90705e4b0fb41093a15341bffb996
Author: Pei He 
Date:   2016-11-01T00:58:31Z

[BEAM-59] Add use Factory interface in Gcs/File/IOChannelFactory.

commit fd940fa32cc96933c85b2cf93306b6572b783d1a
Author: Pei He 
Date:   2016-11-01T01:01:41Z

[BEAM-59] Create IOChannelFactoryFactoryRegistrar interface and its 
gcs/file implementations.

commit f8b30dda94555e9dada829e7d626a2d3da952804
Author: Pei He 
Date:   2016-11-01T01:02:49Z

[BEAM-59] Use ServiceLoader to register IOChannelFactories in 
IOChannelUtils.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-239) Rename RemoveDuplicates to Distinct

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627444#comment-15627444
 ] 

ASF GitHub Bot commented on BEAM-239:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1164


> Rename RemoveDuplicates to Distinct
> ---
>
> Key: BEAM-239
> URL: https://issues.apache.org/jira/browse/BEAM-239
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: backward-incompatible, newbie, starter
>
> I had a really tough time finding this transform in the docs. I suggest 
> changing this class' name to Distinct instead of RemoveDuplicates. At the 
> very least, the JavaDoc for RemoveDuplicates should have the word distinct in 
> it to make this more findable/searchable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: BEAM-239: Refactored the project to replace RemoveDuplicates to Distinct

[jesseanderson]

Repository: incubator-beam
Updated Branches:
  refs/heads/master facf096e5 -> 8883877ae


BEAM-239: Refactored the project to replace RemoveDuplicates to Distinct


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8183ac82
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8183ac82
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8183ac82

Branch: refs/heads/master
Commit: 8183ac825d506a56421f41adc2e25b544f1bb80f
Parents: facf096
Author: Neelesh Srinivas Salian 
Authored: Sat Oct 22 13:05:48 2016 -0700
Committer: Jesse Anderson 
Committed: Tue Nov 1 19:15:12 2016 -0600

--
 .../apache/beam/examples/complete/TfIdf.java|   6 +-
 .../beam/examples/cookbook/DeDupExample.java|   8 +-
 .../org/apache/beam/examples/cookbook/README.md |   2 +-
 .../beam/examples/complete/TfIdfTest.java   |   4 +-
 .../examples/cookbook/DeDupExampleTest.java |  82 --
 .../examples/cookbook/DistinctExampleTest.java  |  82 ++
 .../UnboundedReadFromBoundedSourceTest.java |   4 +-
 .../beam/runners/flink/examples/TFIDF.java  |   6 +-
 .../streaming/KafkaStreamingTest.java   |   4 +-
 .../sdk/io/BoundedReadFromUnboundedSource.java  |   4 +-
 .../apache/beam/sdk/transforms/Distinct.java| 158 ++
 .../beam/sdk/transforms/RemoveDuplicates.java   | 159 ---
 .../apache/beam/sdk/io/CountingInputTest.java   |   6 +-
 .../apache/beam/sdk/io/CountingSourceTest.java  |   8 +-
 .../sdk/transforms/ApproximateUniqueTest.java   |   2 +-
 .../beam/sdk/transforms/DistinctTest.java   | 130 +++
 .../sdk/transforms/RemoveDuplicatesTest.java| 130 ---
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   |   6 +-
 .../beam/sdk/transforms/DistinctJava8Test.java  | 104 
 .../transforms/RemoveDuplicatesJava8Test.java   | 104 
 20 files changed, 504 insertions(+), 505 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8183ac82/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
index c0ba1e9..edf48e7 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
@@ -36,12 +36,12 @@ import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.Validation;
 import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Distinct;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.Keys;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.RemoveDuplicates;
 import org.apache.beam.sdk.transforms.Values;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.WithKeys;
@@ -209,7 +209,7 @@ public class TfIdf {
   final PCollectionView totalDocuments =
   uriToContent
   .apply("GetURIs", Keys.create())
-  .apply("RemoveDuplicateDocs", RemoveDuplicates.create())
+  .apply("DistinctDocs", Distinct.create())
   .apply(Count.globally())
   .apply(View.asSingleton());
 
@@ -238,7 +238,7 @@ public class TfIdf {
   // Compute a mapping from each word to the total
   // number of documents in which it appears.
   PCollection> wordToDocCount = uriToWords
-  .apply("RemoveDuplicateWords", RemoveDuplicates.>create())
+  .apply("DistinctWords", Distinct.>create())
   .apply(Values.create())
   .apply("CountDocs", Count.perElement());
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8183ac82/examples/java/src/main/java/org/apache/beam/examples/cookbook/DeDupExample.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DeDupExample.java
 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DeDupExample.java
index 0883815..34fb901 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DeDupExample.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DeDupExample.java
@@ -24,16 +24,16 @@ import org.apache.beam.sdk.options.DefaultValueFactory;
 import 

[GitHub] incubator-beam pull request #1164: [BEAM-239]: Rename RemoveDuplicates to Di...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1164


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1164

[jesseanderson]

This closes #1164


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8883877a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8883877a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8883877a

Branch: refs/heads/master
Commit: 8883877ae91fa42963181d419d8f3cbfea8b314b
Parents: facf096 8183ac8
Author: Jesse Anderson 
Authored: Tue Nov 1 19:25:27 2016 -0600
Committer: Jesse Anderson 
Committed: Tue Nov 1 19:25:27 2016 -0600

--
 .../apache/beam/examples/complete/TfIdf.java|   6 +-
 .../beam/examples/cookbook/DeDupExample.java|   8 +-
 .../org/apache/beam/examples/cookbook/README.md |   2 +-
 .../beam/examples/complete/TfIdfTest.java   |   4 +-
 .../examples/cookbook/DeDupExampleTest.java |  82 --
 .../examples/cookbook/DistinctExampleTest.java  |  82 ++
 .../UnboundedReadFromBoundedSourceTest.java |   4 +-
 .../beam/runners/flink/examples/TFIDF.java  |   6 +-
 .../streaming/KafkaStreamingTest.java   |   4 +-
 .../sdk/io/BoundedReadFromUnboundedSource.java  |   4 +-
 .../apache/beam/sdk/transforms/Distinct.java| 158 ++
 .../beam/sdk/transforms/RemoveDuplicates.java   | 159 ---
 .../apache/beam/sdk/io/CountingInputTest.java   |   6 +-
 .../apache/beam/sdk/io/CountingSourceTest.java  |   8 +-
 .../sdk/transforms/ApproximateUniqueTest.java   |   2 +-
 .../beam/sdk/transforms/DistinctTest.java   | 130 +++
 .../sdk/transforms/RemoveDuplicatesTest.java| 130 ---
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   |   6 +-
 .../beam/sdk/transforms/DistinctJava8Test.java  | 104 
 .../transforms/RemoveDuplicatesJava8Test.java   | 104 
 20 files changed, 504 insertions(+), 505 deletions(-)
--

[GitHub] incubator-beam pull request #1254: [BEAM-833] Exercise Dynamic Splitting in ...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1254

[BEAM-833] Exercise Dynamic Splitting in the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

For sources that are above a certain size, the DirectRunner will run a
Thread which will split off half of the remaining work. This exercises
the concurrent behavior for splitAtFraction and getFractionConsumed

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam dynamic_splitting

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1254.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1254


commit 4b1549c5e8d7bd759fda2aaae758bb29aa65663a
Author: Thomas Groh 
Date:   2016-11-02T00:23:03Z

Exercise Dynamic Splitting in the DirectRunner

For sources that are above a certain size, the DirectRunner will run a
Thread which will split off half of the remaining work. This exercises
the concurrent behavior for splitAtFraction and getFractionConsumed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-833) Exercise Dynamic Splitting in the DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627267#comment-15627267
 ] 

ASF GitHub Bot commented on BEAM-833:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1254

[BEAM-833] Exercise Dynamic Splitting in the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

For sources that are above a certain size, the DirectRunner will run a
Thread which will split off half of the remaining work. This exercises
the concurrent behavior for splitAtFraction and getFractionConsumed

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam dynamic_splitting

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1254.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1254


commit 4b1549c5e8d7bd759fda2aaae758bb29aa65663a
Author: Thomas Groh 
Date:   2016-11-02T00:23:03Z

Exercise Dynamic Splitting in the DirectRunner

For sources that are above a certain size, the DirectRunner will run a
Thread which will split off half of the remaining work. This exercises
the concurrent behavior for splitAtFraction and getFractionConsumed




> Exercise Dynamic Splitting in the DirectRunner
> --
>
> Key: BEAM-833
> URL: https://issues.apache.org/jira/browse/BEAM-833
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This primarily means calling the {{BoundedSource#splitAtFraction}} API during 
> the evaluation of a BoundedReadEvaluatorFactory - this can return the work 
> via addUnprocessedElements. 
> At a later point, the {{MonitorRunnable}} within 
> {{ExecutorServiceParallelExecutor}} can examine currently evaluating work and 
> invoke a {{TransformExecutor#split}} method, which can return to the runner 
> some amount of input, which can be scheduled to use more of the available 
> threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-878) ApiSurfaceTest improvements and reusability

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-878:


 Summary: ApiSurfaceTest improvements and reusability
 Key: BEAM-878
 URL: https://issues.apache.org/jira/browse/BEAM-878
 Project: Beam
  Issue Type: Improvement
  Components: testing
Affects Versions: Not applicable
Reporter: Daniel Halperin
 Fix For: Not applicable


{{ApiSurfaceTest}} in the {{sdks/java/core}} is the class responsible for 
protecting our public API surface.

This test walks the public signatures of all modules and explicitly verifies 
that everything is on a whitelist. This is how we control what dependencies we 
expose to our users, so that Beam can keep a tight, stable API surface.

We should improve this functionality to be reusable across modules.

Some desiderata for reuse:

* test automatically infers the module on which it should seed the API search.
* only 2 things in the file -- a whitelist and a ~1-line test that passes the 
whitelist as a parameter to some helper.
* test fails if there are things in the whitelist that are not exposed.

Any other cool things that are similar to what Maven enforcer does?

As an example of what you have to do without this functionality, see 
https://github.com/apache/incubator-beam/pull/1183



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1702

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1702

[Apache Jenkins Server]

See 

[1/3] incubator-beam git commit: This closes #1203

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7e0cfe5b9 -> facf096e5


This closes #1203


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/facf096e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/facf096e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/facf096e

Branch: refs/heads/master
Commit: facf096e5007eba1ad46b63e52104331e163d756
Parents: 7e0cfe5 726998a
Author: Thomas Groh 
Authored: Tue Nov 1 16:45:07 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 16:45:07 2016 -0700

--
 .../beam/runners/direct/ParDoEvaluator.java |  18 +-
 .../runners/direct/ParDoEvaluatorFactory.java   | 126 ++
 .../direct/ParDoMultiEvaluatorFactory.java  | 106 -
 .../direct/ParDoMultiEvaluatorHooks.java|  54 +++
 .../direct/ParDoSingleEvaluatorFactory.java | 109 -
 .../direct/ParDoSingleEvaluatorHooks.java   |  57 +++
 .../direct/TransformEvaluatorRegistry.java  |  10 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |  10 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  | 439 ---
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 +++
 .../direct/ParDoSingleEvaluatorFactoryTest.java | 335 --
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 ++
 12 files changed, 1030 insertions(+), 1008 deletions(-)
--

[GitHub] incubator-beam pull request #1203: Deduplicates ParDo{Single,Multi}Evaluator...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1203


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Deduplicates ParDo{Single, Multi}EvaluatorFactory

[tgroh]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/726998ae/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
deleted file mode 100644
index d22643a..000
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,335 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.not;
-import static org.hamcrest.Matchers.nullValue;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-import java.io.Serializable;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.TimeDomain;
-import org.apache.beam.sdk.util.Timer;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
-import org.apache.beam.sdk.util.TimerSpec;
-import org.apache.beam.sdk.util.TimerSpecs;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.state.BagState;
-import org.apache.beam.sdk.util.state.StateNamespace;
-import org.apache.beam.sdk.util.state.StateNamespaces;
-import org.apache.beam.sdk.util.state.StateSpec;
-import org.apache.beam.sdk.util.state.StateSpecs;
-import org.apache.beam.sdk.util.state.StateTag;
-import org.apache.beam.sdk.util.state.StateTags;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TupleTag;
-import org.hamcrest.Matchers;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Ignore;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/**
- * Tests for {@link ParDoSingleEvaluatorFactory}.
- */
-@RunWith(JUnit4.class)
-public class ParDoSingleEvaluatorFactoryTest implements Serializable {
-  private transient BundleFactory bundleFactory = 
ImmutableListBundleFactory.create();
-
-  @Test
-  public void testParDoInMemoryTransformEvaluator() throws Exception {
-TestPipeline p = TestPipeline.create();
-
-PCollection input = p.apply(Create.of("foo", "bara", "bazam"));
-PCollection collection =
-input.apply(
-ParDo.of(
-new DoFn() {
-  @ProcessElement
-  public void processElement(ProcessContext c) {
-c.output(c.element().length());
-  }
-}));
-CommittedBundle inputBundle =
-bundleFactory.createBundle(input).commit(Instant.now());
-
-EvaluationContext evaluationContext = mock(EvaluationContext.class);
-UncommittedBundle outputBundle = 
bundleFactory.createBundle(collection);
-when(evaluationContext.createBundle(collection)).thenReturn(outputBundle);
-DirectExecutionContext executionContext =
-new DirectExecutionContext(null, null, null, null);
-
when(evaluationContext.getExecutionContext(collection.getProducingTransformInternal(),
-inputBundle.getKey())).thenReturn(executionContext);
-AggregatorContainer container = AggregatorContainer.create();
-

[3/3] incubator-beam git commit: Deduplicates ParDo{Single, Multi}EvaluatorFactory

[tgroh]

Deduplicates ParDo{Single,Multi}EvaluatorFactory

This is in preparation for adding a third one for a future ParDo-like primitive
transform to be introduced inside SplittableParDo.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/726998ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/726998ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/726998ae

Branch: refs/heads/master
Commit: 726998ae68ee99eb10bf43ff8aae1a5f121728a4
Parents: 7e0cfe5
Author: Eugene Kirpichov 
Authored: Wed Oct 26 16:34:47 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 16:45:07 2016 -0700

--
 .../beam/runners/direct/ParDoEvaluator.java |  18 +-
 .../runners/direct/ParDoEvaluatorFactory.java   | 126 ++
 .../direct/ParDoMultiEvaluatorFactory.java  | 106 -
 .../direct/ParDoMultiEvaluatorHooks.java|  54 +++
 .../direct/ParDoSingleEvaluatorFactory.java | 109 -
 .../direct/ParDoSingleEvaluatorHooks.java   |  57 +++
 .../direct/TransformEvaluatorRegistry.java  |  10 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |  10 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  | 439 ---
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 +++
 .../direct/ParDoSingleEvaluatorFactoryTest.java | 335 --
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 ++
 12 files changed, 1030 insertions(+), 1008 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/726998ae/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
index ff49b60..5913379 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
@@ -39,8 +39,8 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 
-class ParDoEvaluator implements TransformEvaluator {
-  public static  ParDoEvaluator create(
+class ParDoEvaluator implements TransformEvaluator {
+  public static  ParDoEvaluator create(
   EvaluationContext evaluationContext,
   DirectStepContext stepContext,
   AppliedPTransform application,
@@ -86,17 +86,17 @@ class ParDoEvaluator implements TransformEvaluator {
 
   

 
-  private final PushbackSideInputDoFnRunner fnRunner;
-  private final AppliedPTransform transform;
+  private final PushbackSideInputDoFnRunner fnRunner;
+  private final AppliedPTransform transform;
   private final AggregatorContainer.Mutator aggregatorChanges;
   private final Collection outputBundles;
   private final DirectStepContext stepContext;
 
-  private final ImmutableList.Builder unprocessedElements;
+  private final ImmutableList.Builder 
unprocessedElements;
 
   private ParDoEvaluator(
-  PushbackSideInputDoFnRunner fnRunner,
-  AppliedPTransform transform,
+  PushbackSideInputDoFnRunner fnRunner,
+  AppliedPTransform transform,
   AggregatorContainer.Mutator aggregatorChanges,
   Collection outputBundles,
   DirectStepContext stepContext) {
@@ -109,9 +109,9 @@ class ParDoEvaluator implements TransformEvaluator {
   }
 
   @Override
-  public void processElement(WindowedValue element) {
+  public void processElement(WindowedValue element) {
 try {
-  Iterable unprocessed = 
fnRunner.processElementInReadyWindows(element);
+  Iterable unprocessed = 
fnRunner.processElementInReadyWindows(element);
   unprocessedElements.addAll(unprocessed);
 } catch (Exception e) {
   throw UserCodeException.wrap(e);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/726998ae/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
 

[GitHub] incubator-beam pull request #1253: Optimize WindowedValueCoder

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/1253

Optimize WindowedValueCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam windowed-value-coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1253.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1253


commit 00492a02d98dc4707ca841c2922ef20c7195e629
Author: Robert Bradshaw 
Date:   2016-11-01T23:12:19Z

Optimize WindowedValueCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1701

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1701

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1701

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626917#comment-15626917
 ] 

ASF GitHub Bot commented on BEAM-27:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1252

[BEAM-27] Factor of parameter analysis in DoFnSignatures; use it for 
@OnTimer methods

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff 

This might be easiest to review as individual commits, though the last one 
is the goal of this PR.

This PR is a prefix of ongoing work to invoke `@OnTimer` methods. I've 
peeled it off and fixed it up and added a smidge of testing. Since the code is 
entirely shared, just making sure the right path is taken provides moderate 
coverage. (a further refactor could make this clearer, by isolating the shared 
code into a class with its own unit tests, absolving its callers of proving 
they exercise the code path, but I thought this baby step was better)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DoFnSignatures

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1252.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1252


commit 5946ec175d55f9164f9b1b34a06539da3c24f5ae
Author: Kenneth Knowles 
Date:   2016-11-01T04:30:40Z

Refactor and reuse parameter analysis in DoFnSignatures

commit 15a8901ade07c1649378a0b72238eb514877c61c
Author: Kenneth Knowles 
Date:   2016-11-01T21:48:54Z

Switch DoFnSignature, etc, from TypeToken to TypeDescriptor

commit 6de216695c05e4d9b283f0842aed8713075be7aa
Author: Kenneth Knowles 
Date:   2016-11-01T21:50:24Z

DoFnSignature: Make TypeDescriptor-returning methods public

commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32
Author: Kenneth Knowles 
Date:   2016-11-01T22:38:01Z

Add TypeDescriptor#getTypes




> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1252: [BEAM-27] Factor of parameter analysis in...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1252

[BEAM-27] Factor of parameter analysis in DoFnSignatures; use it for 
@OnTimer methods

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff 

This might be easiest to review as individual commits, though the last one 
is the goal of this PR.

This PR is a prefix of ongoing work to invoke `@OnTimer` methods. I've 
peeled it off and fixed it up and added a smidge of testing. Since the code is 
entirely shared, just making sure the right path is taken provides moderate 
coverage. (a further refactor could make this clearer, by isolating the shared 
code into a class with its own unit tests, absolving its callers of proving 
they exercise the code path, but I thought this baby step was better)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DoFnSignatures

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1252.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1252


commit 5946ec175d55f9164f9b1b34a06539da3c24f5ae
Author: Kenneth Knowles 
Date:   2016-11-01T04:30:40Z

Refactor and reuse parameter analysis in DoFnSignatures

commit 15a8901ade07c1649378a0b72238eb514877c61c
Author: Kenneth Knowles 
Date:   2016-11-01T21:48:54Z

Switch DoFnSignature, etc, from TypeToken to TypeDescriptor

commit 6de216695c05e4d9b283f0842aed8713075be7aa
Author: Kenneth Knowles 
Date:   2016-11-01T21:50:24Z

DoFnSignature: Make TypeDescriptor-returning methods public

commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32
Author: Kenneth Knowles 
Date:   2016-11-01T22:38:01Z

Add TypeDescriptor#getTypes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-597) Provide type information from Coders

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626865#comment-15626865
 ] 

ASF GitHub Bot commented on BEAM-597:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/968


> Provide type information from Coders
> 
>
> Key: BEAM-597
> URL: https://issues.apache.org/jira/browse/BEAM-597
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jeremie Lenfant-Engelmann
>Assignee: Jeremie Lenfant-Engelmann
>Priority: Minor
>
> The Coder interface should return a TypeDescriptor describing the type that 
> is currently encoded/decoded by the Coder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #968: [BEAM-597] Added public where method to sp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/968


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #968

[kenn]

This closes #968


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7e0cfe5b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7e0cfe5b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7e0cfe5b

Branch: refs/heads/master
Commit: 7e0cfe5b9c2ac2d0aeaf833a5e3097ce25a40c03
Parents: f6ad58d a36f1fa
Author: Kenneth Knowles 
Authored: Tue Nov 1 15:22:04 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 1 15:22:04 2016 -0700

--
 .../apache/beam/sdk/values/TypeDescriptor.java  | 35 ---
 .../apache/beam/sdk/values/TypeDescriptors.java |  1 -
 .../apache/beam/sdk/values/TypeParameter.java   | 62 
 .../beam/sdk/values/TypeDescriptorTest.java | 12 
 4 files changed, 100 insertions(+), 10 deletions(-)
--

[1/3] incubator-beam git commit: Added a where method on TypeDescriptor to allow substituting a type parameter with a TypeDescriptor. In the process introduced a TypeParameter class which represents t

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f6ad58d60 -> 7e0cfe5b9


Added a where method on TypeDescriptor to allow substituting a type
parameter with a TypeDescriptor. In the process introduced a TypeParameter
class which represents the parameter type.
This is useful when having a type such as Set to be able to specify
what T is as part of the TypeDescriptor information.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/699075e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/699075e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/699075e2

Branch: refs/heads/master
Commit: 699075e250973447a9cdce1eb9c52d3406788212
Parents: f6ad58d
Author: Jeremie Lenfant-Engelmann 
Authored: Thu Oct 20 19:29:39 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 1 15:14:24 2016 -0700

--
 .../apache/beam/sdk/values/TypeDescriptor.java  | 35 +++
 .../apache/beam/sdk/values/TypeDescriptors.java |  1 -
 .../apache/beam/sdk/values/TypeParameter.java   | 45 
 .../beam/sdk/values/TypeDescriptorTest.java | 12 ++
 4 files changed, 83 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/699075e2/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
index 724b8b6..6eabf42 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
@@ -20,7 +20,7 @@ package org.apache.beam.sdk.values;
 import com.google.common.collect.Lists;
 import com.google.common.reflect.Invokable;
 import com.google.common.reflect.Parameter;
-import com.google.common.reflect.TypeParameter;
+import com.google.common.reflect.TypeResolver;
 import com.google.common.reflect.TypeToken;
 import java.io.Serializable;
 import java.lang.reflect.Field;
@@ -171,14 +171,6 @@ public abstract class TypeDescriptor implements 
Serializable {
   }
 
   /**
-   * Creates a new {@link SimpleTypeDescriptor} using the {@link #token}.
-   * Package visible so this isn't abused.
-   */
-   TypeDescriptor where(TypeParameter typeParam, TypeDescriptor 
typeDescriptor) {
-return new SimpleTypeDescriptor<>(token.where(typeParam, 
typeDescriptor.token));
-  }
-
-  /**
* Returns the {@link Type} represented by this {@link TypeDescriptor}.
*/
   public Type getType() {
@@ -322,6 +314,31 @@ public abstract class TypeDescriptor implements 
Serializable {
 return classes;
   }
 
+  /**
+   * Returns a new {@code TypeDescriptor} where type variables represented by
+   * {@code typeParameter} are substituted by {@code typeDescriptor}. For 
example, it can be used to
+   * construct {@code Map} for any {@code K} and {@code V} type:  
{@code
+   *   static  TypeDescriptor> mapOf(
+   *   TypeDescriptor keyType, TypeDescriptor valueType) {
+   * return new TypeDescriptor>() {}
+   * .where(new TypeParameter() {}, keyType)
+   * .where(new TypeParameter() {}, valueType);
+   *   }}
+   *
+   * @param  The parameter type
+   * @param typeParameter the parameter type variable
+   * @param typeDescriptor the actual type to substitute
+   */
+  @SuppressWarnings("unchecked")
+  public  TypeDescriptor where(TypeParameter typeParameter,
+  TypeDescriptor typeDescriptor) {
+TypeResolver resolver =
+new TypeResolver()
+.where(
+typeParameter.typeVariable, typeDescriptor.getType());
+return (TypeDescriptor) 
TypeDescriptor.of(resolver.resolveType(token.getType()));
+  }
+
   @Override
   public String toString() {
 return token.toString();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/699075e2/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
index 7a78131..b6b1a1a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptors.java
@@ -17,7 +17,6 @@
  */
 package org.apache.beam.sdk.values;
 
-import com.google.common.reflect.TypeParameter;
 import java.math.BigDecimal;
 import java.math.BigInteger;
 import java.util.List;

[2/3] incubator-beam git commit: Add header

[kenn]

Add header


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a36f1fa8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a36f1fa8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a36f1fa8

Branch: refs/heads/master
Commit: a36f1fa89079686cb1ce25a3c0abd3dd4f328c92
Parents: 699075e
Author: Kenneth Knowles 
Authored: Tue Nov 1 15:21:10 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 1 15:21:10 2016 -0700

--
 .../org/apache/beam/sdk/values/TypeParameter.java  | 17 +
 1 file changed, 17 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a36f1fa8/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
index a9a658e..dd9d59c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
 package org.apache.beam.sdk.values;
 
 import static com.google.common.base.Preconditions.checkArgument;

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1700

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1700

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1251: [BEAM-788] Implement GroupAlsoByWindowsEv...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1251

[BEAM-788] Implement GroupAlsoByWindowsEvaluator Directly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Invoke the ReduceFnRunner methods directly rather than by constructing a
GroupAlsoByWindowDoFn and a ParDoEvaluator. Instead, construct a
ReduceFnRunner, ReduceFn, and invoke the methods directly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam gabw_implemented_directly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1251


commit 1cd3dbc5fd35748751c27e17def6100d591ff043
Author: Thomas Groh 
Date:   2016-11-01T21:29:48Z

Implement GroupAlsoByWindowsEvaluator Directly

Invoke the ReduceFnRunner methods directly rather than by constructing a
GroupAlsoByWindowDoFn and a ParDoEvaluator. Instead, construct a
ReduceFnRunner, ReduceFn, and invoke the methods directly.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-788) Execute ReduceFn directly, not via OldDoFn wrapper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626783#comment-15626783
 ] 

ASF GitHub Bot commented on BEAM-788:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1251

[BEAM-788] Implement GroupAlsoByWindowsEvaluator Directly

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Invoke the ReduceFnRunner methods directly rather than by constructing a
GroupAlsoByWindowDoFn and a ParDoEvaluator. Instead, construct a
ReduceFnRunner, ReduceFn, and invoke the methods directly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam gabw_implemented_directly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1251.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1251


commit 1cd3dbc5fd35748751c27e17def6100d591ff043
Author: Thomas Groh 
Date:   2016-11-01T21:29:48Z

Implement GroupAlsoByWindowsEvaluator Directly

Invoke the ReduceFnRunner methods directly rather than by constructing a
GroupAlsoByWindowDoFn and a ParDoEvaluator. Instead, construct a
ReduceFnRunner, ReduceFn, and invoke the methods directly.




> Execute ReduceFn directly, not via OldDoFn wrapper
> --
>
> Key: BEAM-788
> URL: https://issues.apache.org/jira/browse/BEAM-788
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>
> Right now, a ReduceFn is executed as an OldDoFn, depending on deprecated 
> methods. We can execute it directly to allow removal of OldDoFn.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-877) Allow disabling flattening of results when using BigQuery source

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-877:
---

 Summary: Allow disabling flattening of results when using BigQuery 
source
 Key: BEAM-877
 URL: https://issues.apache.org/jira/browse/BEAM-877
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Sourabh Bajaj


Java SDK supports disabling results flattening when creating a BQ source using 
a query.
https://github.com/apache/incubator-beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L477

Python SDK should be updated to support this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1201: Makes PTransformOverrideFactory type-safe

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1201


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Makes PTransformOverrideFactory type-safe

[tgroh]

Makes PTransformOverrideFactory type-safe


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bb661437
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bb661437
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bb661437

Branch: refs/heads/master
Commit: bb66143754cc405435db734fb7629423b9ee292c
Parents: 1283308
Author: Eugene Kirpichov 
Authored: Wed Oct 26 16:14:51 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 14:06:35 2016 -0700

--
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 14 +++---
 .../direct/DirectGroupByKeyOverrideFactory.java | 25 --
 .../GroupAlsoByWindowEvaluatorFactory.java  |  1 -
 .../direct/PTransformOverrideFactory.java   |  8 ++--
 .../beam/runners/direct/ParDoEvaluator.java |  5 --
 .../direct/ParDoMultiEvaluatorFactory.java  |  1 -
 .../runners/direct/ParDoOverrideFactory.java| 14 +++---
 .../direct/ParDoSingleEvaluatorFactory.java |  1 -
 .../direct/TestStreamEvaluatorFactory.java  | 14 ++
 .../direct/UncommittedBundleOutputManager.java  | 50 
 .../runners/direct/ViewEvaluatorFactory.java| 19 +++-
 .../direct/WriteWithShardingFactory.java| 15 ++
 .../beam/runners/direct/ParDoEvaluatorTest.java |  1 -
 .../direct/WriteWithShardingFactoryTest.java| 26 ++
 14 files changed, 47 insertions(+), 147 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bb661437/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
index e232552..b63e23b 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
@@ -28,16 +28,16 @@ import org.apache.beam.sdk.util.ReifyTimestampsAndWindows;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PInput;
-import org.apache.beam.sdk.values.POutput;
 
 /** Provides an implementation of {@link GBKIntoKeyedWorkItems} for the Direct 
Runner. */
-class DirectGBKIntoKeyedWorkItemsOverrideFactory implements 
PTransformOverrideFactory {
+class DirectGBKIntoKeyedWorkItemsOverrideFactory
+implements PTransformOverrideFactory<
+PCollection>, PCollection>,
+GBKIntoKeyedWorkItems> {
   @Override
-  @SuppressWarnings("unchecked")
-  public  PTransform override(
-  PTransform transform) {
-return new DirectGBKIntoKeyedWorkItems(transform.getName());
+  public PTransform>, 
PCollection>>
+  override(GBKIntoKeyedWorkItems transform) {
+return new DirectGBKIntoKeyedWorkItems<>(transform.getName());
   }
 
   /** The Direct Runner specific implementation of {@link 
GBKIntoKeyedWorkItems}. */

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bb661437/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
index c64f3f0..9acf5e9 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
@@ -19,23 +19,16 @@ package org.apache.beam.runners.direct;
 
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.PInput;
-import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
 
-/**
- * A {@link PTransformOverrideFactory} for {@link GroupByKey} PTransforms.
- */
-final class DirectGroupByKeyOverrideFactory
-implements PTransformOverrideFactory {
+/** A {@link PTransformOverrideFactory} for {@link GroupByKey} PTransforms. */
+final 

[jira] [Created] (BEAM-876) Support schemaUpdateOption in BigQueryIO

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-876:
-

 Summary: Support schemaUpdateOption in BigQueryIO
 Key: BEAM-876
 URL: https://issues.apache.org/jira/browse/BEAM-876
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Eugene Kirpichov
Assignee: Daniel Halperin


BigQuery recently added support for updating the schema as a side effect of the 
load job.

Here is the relevant API method in JobConfigurationLoad: 
https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setSchemaUpdateOptions(java.util.List)

BigQueryIO should support this too. See user request for this: 
http://stackoverflow.com/questions/40333245/is-it-possible-to-update-schema-while-doing-a-load-into-an-existing-bigquery-tab



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-875) Inconsistency between time types between the inprocess and dataflow runner for BQ

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626545#comment-15626545
 ] 

ASF GitHub Bot commented on BEAM-875:
-

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1250

[BEAM-875] Make the BQ input for dataflow runner and local runner is 
identical

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Time related datatypes were not all supported for the local runner as well 
as timestamps were being serialized different so changing the local runner to 
match the big query export environment now. 

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-875-datetime-inconsistency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1250


commit 28ab78d87393d7b7d57980e2dfdccdc366d213b2
Author: Sourabh Bajaj 
Date:   2016-11-01T20:06:10Z

Make the BQ input for dataflow runner and local runner is identical for 
time related types




> Inconsistency between time types between the inprocess and dataflow runner 
> for BQ
> -
>
> Key: BEAM-875
> URL: https://issues.apache.org/jira/browse/BEAM-875
> Project: Beam
>  Issue Type: Bug
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> investigate if we support datatypes TIMESTAMP, DATE, TIME, and DATETIME 
> properly and in a consistent matter across InProcess and Dataflow runners for 
> BQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1250: [BEAM-875] Make the BQ input for dataflow...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1250

[BEAM-875] Make the BQ input for dataflow runner and local runner is 
identical

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Time related datatypes were not all supported for the local runner as well 
as timestamps were being serialized different so changing the local runner to 
match the big query export environment now. 

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-875-datetime-inconsistency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1250


commit 28ab78d87393d7b7d57980e2dfdccdc366d213b2
Author: Sourabh Bajaj 
Date:   2016-11-01T20:06:10Z

Make the BQ input for dataflow runner and local runner is identical for 
time related types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-875) Inconsistency between time types between the inprocess and dataflow runner for BQ

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-875:
--

 Summary: Inconsistency between time types between the inprocess 
and dataflow runner for BQ
 Key: BEAM-875
 URL: https://issues.apache.org/jira/browse/BEAM-875
 Project: Beam
  Issue Type: Bug
Reporter: Sourabh Bajaj
Assignee: Sourabh Bajaj
Priority: Minor


investigate if we support datatypes TIMESTAMP, DATE, TIME, and DATETIME 
properly and in a consistent matter across InProcess and Dataflow runners for BQ



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #762: [BEAM-443] Remove BlockingDataflowRunner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/762


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-443) PipelineResult needs waitUntilFinish() and cancel()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626490#comment-15626490
 ] 

ASF GitHub Bot commented on BEAM-443:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/762


> PipelineResult needs waitUntilFinish() and cancel()
> ---
>
> Key: BEAM-443
> URL: https://issues.apache.org/jira/browse/BEAM-443
> Project: Beam
>  Issue Type: New Feature
>Reporter: Pei He
>Assignee: Pei He
>
> waitToFinish() and cancel() are two most common operations for users to 
> interact with a started pipeline.
> Right now, they are only available in DataflowPipelineJob. But, it is better 
> to move them to the common interface, so people can start implement them in 
> other runners, and runner agnostic code can interact with PipelineResult 
> better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Remove BlockingDataflowRunner

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8390a2212 -> 1283308e2


Remove BlockingDataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4c0fab0b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4c0fab0b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4c0fab0b

Branch: refs/heads/master
Commit: 4c0fab0b3a36d184c6d1fe060d60dd9b6678daf1
Parents: 8390a22
Author: Pei He 
Authored: Fri Jul 29 15:40:07 2016 -0700
Committer: Davor Bonaci 
Committed: Tue Nov 1 12:52:09 2016 -0700

--
 .../beam/examples/common/ExampleUtils.java  |   3 +-
 .../dataflow/BlockingDataflowRunner.java| 170 ---
 .../dataflow/DataflowJobCancelledException.java |  39 ---
 .../dataflow/DataflowJobExecutionException.java |  35 ---
 .../dataflow/DataflowJobUpdatedException.java   |  51 
 .../runners/dataflow/DataflowPipelineJob.java   |  43 ++-
 .../dataflow/DataflowPipelineRegistrar.java |  11 +-
 .../BlockingDataflowPipelineOptions.java|  28 --
 .../testing/TestDataflowPipelineOptions.java|   6 +-
 .../dataflow/testing/TestDataflowRunner.java|   7 +-
 .../dataflow/BlockingDataflowRunnerTest.java| 300 ---
 .../dataflow/DataflowPipelineJobTest.java   |  30 +-
 .../dataflow/DataflowPipelineRegistrarTest.java |   7 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |   9 +-
 14 files changed, 80 insertions(+), 659 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4c0fab0b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
index 1209a67..6962571 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
@@ -272,8 +272,7 @@ public class ExampleUtils {
   }
 
   /**
-   * If {@literal DataflowRunner} or {@literal BlockingDataflowRunner} is used,
-   * waits for the pipeline to finish and cancels it (and the injector) before 
the program exists.
+   * Waits for the pipeline to finish and cancels it before the program exists.
*/
   public void waitToFinish(PipelineResult result) {
 pipelinesToCancel.add(result);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4c0fab0b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BlockingDataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BlockingDataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BlockingDataflowRunner.java
deleted file mode 100644
index 5285ade..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BlockingDataflowRunner.java
+++ /dev/null
@@ -1,170 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import javax.annotation.Nullable;
-import 
org.apache.beam.runners.dataflow.options.BlockingDataflowPipelineOptions;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.PInput;
-import org.apache.beam.sdk.values.POutput;
-import org.joda.time.Duration;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**

[2/2] incubator-beam git commit: This closes #762

[davor]

This closes #762


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1283308e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1283308e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1283308e

Branch: refs/heads/master
Commit: 1283308e2c99d1b0c2cf4640ad089f8a638d8163
Parents: 8390a22 4c0fab0
Author: Davor Bonaci 
Authored: Tue Nov 1 12:52:36 2016 -0700
Committer: Davor Bonaci 
Committed: Tue Nov 1 12:52:36 2016 -0700

--
 .../beam/examples/common/ExampleUtils.java  |   3 +-
 .../dataflow/BlockingDataflowRunner.java| 170 ---
 .../dataflow/DataflowJobCancelledException.java |  39 ---
 .../dataflow/DataflowJobExecutionException.java |  35 ---
 .../dataflow/DataflowJobUpdatedException.java   |  51 
 .../runners/dataflow/DataflowPipelineJob.java   |  43 ++-
 .../dataflow/DataflowPipelineRegistrar.java |  11 +-
 .../BlockingDataflowPipelineOptions.java|  28 --
 .../testing/TestDataflowPipelineOptions.java|   6 +-
 .../dataflow/testing/TestDataflowRunner.java|   7 +-
 .../dataflow/BlockingDataflowRunnerTest.java| 300 ---
 .../dataflow/DataflowPipelineJobTest.java   |  30 +-
 .../dataflow/DataflowPipelineRegistrarTest.java |   7 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |   9 +-
 14 files changed, 80 insertions(+), 659 deletions(-)
--

Jenkins build became unstable: beam_PostCommit_MavenVerify #1696

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1697

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1697

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1696

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-874) sdks/java/microbenchmarks instructions incorrect since benchmarks no longer run

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-874:
---
Description: 
microbenchmarks.jar is built with an empty META-INF/BenchmarkList file which 
leads to this failure upon executing
"java -jar target/microbenchmarks.jar":

No matching benchmarks. Miss-spelled regexp?
Use EXTRA verbose mode to debug the pattern matching.

Also, note that sdks/java/microbenchmarks/README.md should attempt to avoid 
usage of mvn install when possible. An alternate suggestion could be:
mvn package -pl sdks/java/microbenchmarks -am

  was:
microbenchmarks.jar is built with an empty META-INF/BenchmarkList file which 
leads to this failure upon executing
```java -jar target/microbenchmarks.jar```:

No matching benchmarks. Miss-spelled regexp?
Use EXTRA verbose mode to debug the pattern matching.

Also, note that sdks/java/microbenchmarks/README.md should attempt to avoid 
usage of mvn install when possible. An alternate suggestion could be:
mvn package -pl sdks/java/microbenchmarks -am


> sdks/java/microbenchmarks instructions incorrect since benchmarks no longer 
> run
> ---
>
> Key: BEAM-874
> URL: https://issues.apache.org/jira/browse/BEAM-874
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>
> microbenchmarks.jar is built with an empty META-INF/BenchmarkList file which 
> leads to this failure upon executing
> "java -jar target/microbenchmarks.jar":
> No matching benchmarks. Miss-spelled regexp?
> Use EXTRA verbose mode to debug the pattern matching.
> Also, note that sdks/java/microbenchmarks/README.md should attempt to avoid 
> usage of mvn install when possible. An alternate suggestion could be:
> mvn package -pl sdks/java/microbenchmarks -am



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #1239

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6a6e042f8 -> 8390a2212


Closes #1239


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8390a221
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8390a221
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8390a221

Branch: refs/heads/master
Commit: 8390a2212e58b25c042b7324381a07e695e6551d
Parents: 6a6e042 830b8c8
Author: Dan Halperin 
Authored: Tue Nov 1 11:47:04 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 11:47:04 2016 -0700

--
 runners/core-java/pom.xml   | 30 --
 runners/direct-java/pom.xml | 30 --
 runners/google-cloud-dataflow-java/pom.xml  | 31 ---
 sdks/java/core/pom.xml  | 30 --
 .../main/resources/archetype-resources/pom.xml  | 32 
 5 files changed, 153 deletions(-)
--

[GitHub] incubator-beam pull request #1239: Drop bundled jars from maven shading prof...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1239


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Drop bundled jars from maven shading profiles

[dhalperi]

Drop bundled jars from maven shading profiles

These bundled jars used to be used for testing, but are no longer
needed. They slow down the build, so let's drop them.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/830b8c86
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/830b8c86
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/830b8c86

Branch: refs/heads/master
Commit: 830b8c86073c62a3c425999684d733788f0c31c4
Parents: 6a6e042
Author: Dan Halperin 
Authored: Mon Oct 31 15:49:13 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 11:47:04 2016 -0700

--
 runners/core-java/pom.xml   | 30 --
 runners/direct-java/pom.xml | 30 --
 runners/google-cloud-dataflow-java/pom.xml  | 31 ---
 sdks/java/core/pom.xml  | 30 --
 .../main/resources/archetype-resources/pom.xml  | 32 
 5 files changed, 153 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/830b8c86/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index 0601512..e3e4fed 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -95,7 +95,6 @@
 org.apache.maven.plugins
 maven-shade-plugin
 
-  
   
 bundle-and-repackage
 package
@@ -134,35 +133,6 @@
   
 
   
-
-  
-  
-bundle-rest-without-repackaging
-package
-
-  shade
-
-
-  true
-  
${project.artifactId}-bundled-${project.version}
-  
-
-  com.google.guava:guava
-
-  
-  
-
-  *:*
-  
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-  
-
-  
-
-  
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/830b8c86/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index c182d15..84190b2 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -162,7 +162,6 @@
 org.apache.maven.plugins
 maven-shade-plugin
 
-  
   
 bundle-and-repackage
 package
@@ -205,35 +204,6 @@
   
 
   
-
-  
-  
-bundle-rest-without-repackaging
-package
-
-  shade
-
-
-  true
-  
${project.artifactId}-bundled-${project.version}
-  
-
-  com.google.guava:guava
-
-  
-  
-
-  *:*
-  
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-  
-
-  
-
-  
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/830b8c86/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index c5dd274..750f520 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -164,7 +164,6 @@
 org.apache.maven.plugins
 maven-shade-plugin
 
-  
   
 bundle-and-repackage
 package
@@ -219,36 +218,6 @@
   
 
   
-
-  
-  
-bundle-rest-without-repackaging
-package
-
-  shade
-
-
-  true
-  
${project.artifactId}-bundled-${project.version}
-  
-
-  
com.google.cloud.bigtable:bigtable-client-core
-  com.google.guava:guava
-
-  
-  
-
-  *:*
-  
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-  
-
-  
-
-  

[jira] [Created] (BEAM-874) sdks/java/microbenchmarks instructions and execution no longer function

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-874:
--

 Summary: sdks/java/microbenchmarks instructions and execution no 
longer function
 Key: BEAM-874
 URL: https://issues.apache.org/jira/browse/BEAM-874
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Davor Bonaci


microbenchmarks.jar is built with an empty META-INF/BenchmarkList file which 
leads to this failure upon executing
```java -jar target/microbenchmarks.jar```:

No matching benchmarks. Miss-spelled regexp?
Use EXTRA verbose mode to debug the pattern matching.

Also, note that sdks/java/microbenchmarks/README.md should attempt to avoid 
usage of mvn install when possible. An alternate suggestion could be:
mvn package -pl sdks/java/microbenchmarks -am



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-874) sdks/java/microbenchmarks instructions incorrect since benchmarks no longer run

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-874:
---
Summary: sdks/java/microbenchmarks instructions incorrect since benchmarks 
no longer run  (was: sdks/java/microbenchmarks instructions and execution no 
longer function)

> sdks/java/microbenchmarks instructions incorrect since benchmarks no longer 
> run
> ---
>
> Key: BEAM-874
> URL: https://issues.apache.org/jira/browse/BEAM-874
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>
> microbenchmarks.jar is built with an empty META-INF/BenchmarkList file which 
> leads to this failure upon executing
> ```java -jar target/microbenchmarks.jar```:
> No matching benchmarks. Miss-spelled regexp?
> Use EXTRA verbose mode to debug the pattern matching.
> Also, note that sdks/java/microbenchmarks/README.md should attempt to avoid 
> usage of mvn install when possible. An alternate suggestion could be:
> mvn package -pl sdks/java/microbenchmarks -am



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1219: Short-circuit side input window checks in...

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1188

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 68623e91f -> 6a6e042f8


This closes #1188


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a6e042f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a6e042f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a6e042f

Branch: refs/heads/master
Commit: 6a6e042f847911e8a567221612ff507d7fa3e82c
Parents: 68623e9 d8bfd00
Author: Thomas Groh 
Authored: Tue Nov 1 11:14:04 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 11:14:04 2016 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 128 +--
 1 file changed, 5 insertions(+), 123 deletions(-)
--

[2/2] incubator-beam git commit: Remove Window.Unbound

[tgroh]

Remove Window.Unbound

Window PTransforms always have a bound output type; equal to that of the
input PCollection type.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d8bfd00a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d8bfd00a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d8bfd00a

Branch: refs/heads/master
Commit: d8bfd00a1861a3083a815033998bd35d440cd035
Parents: 68623e9
Author: Thomas Groh 
Authored: Wed Jul 13 09:59:35 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 11:14:04 2016 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 128 +--
 1 file changed, 5 insertions(+), 123 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d8bfd00a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index 90e6a3a..5607762 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -166,7 +166,7 @@ public class Window {
* properties can be set on it first.
*/
   public static  Bound into(WindowFn fn) {
-return new Unbound().into(fn);
+return new Bound(null).into(fn);
   }
 
   /**
@@ -179,7 +179,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound triggering(Trigger trigger) {
-return new Unbound().triggering(trigger);
+return new Bound(null).triggering(trigger);
   }
 
   /**
@@ -191,7 +191,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound discardingFiredPanes() {
-return new Unbound().discardingFiredPanes();
+return new Bound(null).discardingFiredPanes();
   }
 
   /**
@@ -203,7 +203,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound accumulatingFiredPanes() {
-return new Unbound().accumulatingFiredPanes();
+return new Bound(null).accumulatingFiredPanes();
   }
 
   /**
@@ -219,125 +219,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound withAllowedLateness(Duration allowedLateness) {
-return new Unbound().withAllowedLateness(allowedLateness);
-  }
-
-  /**
-   * Override the amount of lateness allowed for data elements in the output 
{@link PCollection},
-   * and downstream {@link PCollection PCollections} until explicitly set 
again. Like
-   * the other properties on this {@link Window} operation, this will be 
applied at
-   * the next {@link GroupByKey}. Any elements that are later than this as 
decided by
-   * the system-maintained watermark will be dropped.
-   *
-   * This value also determines how long state will be kept around for old 
windows.
-   * Once no elements will be added to a window (because this duration has 
passed) any state
-   * associated with the window will be cleaned up.
-   */
-  @Experimental(Kind.TRIGGER)
-  public static  Bound withAllowedLateness(
-  Duration allowedLateness, ClosingBehavior closingBehavior) {
-return new Unbound().withAllowedLateness(allowedLateness, closingBehavior);
-  }
-
-  /**
-   * An incomplete {@code Window} transform, with unbound input/output type.
-   *
-   * Before being applied, {@link Window.Unbound#into} must be
-   * invoked to specify the {@link WindowFn} to invoke, which will also
-   * bind the input/output type of this {@code PTransform}.
-   */
-  public static class Unbound {
-String name;
-
-Unbound() {}
-
-Unbound(String name) {
-  this.name = name;
-}
-
-/**
- * Returns a new {@code Window} {@code PTransform} that's like this
- * transform but that will use the given {@link WindowFn}, and that has
- * its input and output types bound.  Does not modify this transform.  The
- * resulting {@code PTransform} is sufficiently specified to be applied,
- * but more properties can still be specified.
- */
-public  Bound into(WindowFn fn) {
-  return new Bound(name).into(fn);
-}
-
-/**
- * Sets a non-default trigger for this {@code Window} {@code PTransform}.
- * Elements that are assigned to a specific window will be output when
- * the trigger fires.
- *
- * {@link org.apache.beam.sdk.transforms.windowing.Trigger}
- * has more details on the available triggers.
- *
- * Must also specify allowed lateness using {@link 
#withAllowedLateness} and accumulation
- * mode using either {@link 

[jira] [Commented] (BEAM-447) Stop referring to types with Bound/Unbound

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626123#comment-15626123
 ] 

ASF GitHub Bot commented on BEAM-447:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1188


> Stop referring to types with Bound/Unbound
> --
>
> Key: BEAM-447
> URL: https://issues.apache.org/jira/browse/BEAM-447
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Bounded and Unbounded are used to refer to PCollections, and the overlap is 
> confusing.  These classes should be renamed to be both more specific (e.g. 
> ParDo.LackingDoFnSingleOutput, ParDo.SingleOutput, Window.AssignWindows) 
> which remove the overlap.
> examples:
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java#L658
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java#L868



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-700) Update FileBasedSource to use sampling when estimating size of file patterns

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-700.
-
Resolution: Fixed

> Update FileBasedSource to use sampling when estimating size of file patterns
> 
>
> Key: BEAM-700
> URL: https://issues.apache.org/jira/browse/BEAM-700
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Currently when estimating size of file patterns we estimate size of each file 
> and add the sizes up [1]. This can be very inefficient for file patterns that 
> expand into a large number of files. We should update FIleBasedSource to use 
> sampling similar to Java SDK [2].
> [1]https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/filebasedsource.py#L120
> [2] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java#L204



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1188: [BEAM-447] Remove Window.Unbound

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1188


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-447] Remove Window.Unbound

[tgroh]

[BEAM-447] Remove Window.Unbound

This closes #1188


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/68623e91
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/68623e91
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/68623e91

Branch: refs/heads/master
Commit: 68623e91fe46bf1aa87ce057e7cffc221d687c1f
Parents: fab7b24 5bf23ac
Author: Thomas Groh 
Authored: Tue Nov 1 10:45:25 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 10:45:25 2016 -0700

--
 .../core/PushbackSideInputDoFnRunner.java   | 28 +++-
 1 file changed, 15 insertions(+), 13 deletions(-)
--

[1/2] incubator-beam git commit: Short-circuit side input window checks in PushbackDoFnRunner

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master fab7b2402 -> 68623e91f


Short-circuit side input window checks in PushbackDoFnRunner

This uses the collection of not-ready windows to avoid checking when the
answer must be false.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5bf23ac0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5bf23ac0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5bf23ac0

Branch: refs/heads/master
Commit: 5bf23ac0d3f41a3b3e2088024996b1247d246131
Parents: fab7b24
Author: Thomas Groh 
Authored: Fri Oct 28 09:21:51 2016 -0700
Committer: Thomas Groh 
Committed: Tue Nov 1 10:44:56 2016 -0700

--
 .../core/PushbackSideInputDoFnRunner.java   | 28 +++-
 1 file changed, 15 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5bf23ac0/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
index deeac3c..8c169da 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.java
@@ -23,7 +23,6 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.HashSet;
 import java.util.Set;
-
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.ReadyCheckingSideInputReader;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -78,18 +77,7 @@ public class PushbackSideInputDoFnRunner 
implements DoFnRunner<
 ImmutableList.Builder pushedBack = 
ImmutableList.builder();
 for (WindowedValue windowElem : elem.explodeWindows()) {
   BoundedWindow mainInputWindow = 
Iterables.getOnlyElement(windowElem.getWindows());
-  boolean isReady = !notReadyWindows.contains(mainInputWindow);
-  for (PCollectionView view : views) {
-BoundedWindow sideInputWindow =
-view.getWindowingStrategyInternal()
-.getWindowFn()
-.getSideInputWindow(mainInputWindow);
-if (!sideInputReader.isReady(view, sideInputWindow)) {
-  isReady = false;
-  break;
-}
-  }
-  if (isReady) {
+  if (isReady(mainInputWindow)) {
 processElement(windowElem);
   } else {
 notReadyWindows.add(mainInputWindow);
@@ -99,6 +87,20 @@ public class PushbackSideInputDoFnRunner 
implements DoFnRunner<
 return pushedBack.build();
   }
 
+  private boolean isReady(BoundedWindow mainInputWindow) {
+if (notReadyWindows.contains(mainInputWindow)) {
+  return false;
+}
+for (PCollectionView view : views) {
+  BoundedWindow sideInputWindow =
+  
view.getWindowingStrategyInternal().getWindowFn().getSideInputWindow(mainInputWindow);
+  if (!sideInputReader.isReady(view, sideInputWindow)) {
+return false;
+  }
+}
+return true;
+  }
+
   @Override
   public void processElement(WindowedValue elem) {
 underlying.processElement(elem);

[1/2] incubator-beam git commit: Add SortValues

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master afb1e91f5 -> fab7b2402


Add SortValues

an extension that provides a PTransform which performs
local(non-distributed) sorting. It will sort in memory until the buffer
is full, then flush to disk and use external sorting.

Consumes a PCollection of KVs from primary key to iterable of secondary
key and value KVs and sorts the iterables. Would probably be called
after a GroupByKey. Uses coders to convert secondary keys and values
into byte arrays and does a lexicographical comparison on the secondary
keys.

Uses Hadoop as an external sorting library.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/49ee9299
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/49ee9299
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/49ee9299

Branch: refs/heads/master
Commit: 49ee9299636715403d1b0c17119373fc360f660b
Parents: afb1e91
Author: Mitch Shanklin 
Authored: Tue Oct 25 16:17:01 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 10:40:20 2016 -0700

--
 sdks/java/extensions/pom.xml|   1 +
 sdks/java/extensions/sorter/README.md   |  42 
 sdks/java/extensions/sorter/pom.xml | 167 ++
 .../sorter/BufferedExternalSorter.java  | 125 +++
 .../sdk/extensions/sorter/ExternalSorter.java   | 225 +++
 .../sdk/extensions/sorter/InMemorySorter.java   | 166 ++
 .../beam/sdk/extensions/sorter/SortValues.java  | 213 ++
 .../beam/sdk/extensions/sorter/Sorter.java  |  47 
 .../sdk/extensions/sorter/package-info.java |  23 ++
 .../sorter/BufferedExternalSorterTest.java  | 177 +++
 .../extensions/sorter/ExternalSorterTest.java   |  87 +++
 .../extensions/sorter/InMemorySorterTest.java   | 144 
 .../sdk/extensions/sorter/SortValuesTest.java   | 128 +++
 .../sdk/extensions/sorter/SorterTestUtils.java  | 129 +++
 14 files changed, 1674 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/49ee9299/sdks/java/extensions/pom.xml
--
diff --git a/sdks/java/extensions/pom.xml b/sdks/java/extensions/pom.xml
index 4328d3d..fc90edf 100644
--- a/sdks/java/extensions/pom.xml
+++ b/sdks/java/extensions/pom.xml
@@ -33,6 +33,7 @@
 
   
 join-library
+sorter
   
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/49ee9299/sdks/java/extensions/sorter/README.md
--
diff --git a/sdks/java/extensions/sorter/README.md 
b/sdks/java/extensions/sorter/README.md
new file mode 100644
index 000..80d2a40
--- /dev/null
+++ b/sdks/java/extensions/sorter/README.md
@@ -0,0 +1,42 @@
+
+
+#Sorter
+This module provides the SortValues transform, which takes a 
`PCollection>` has been sorted by the byte encoding of secondary key 
(`K2`). It will efficiently and scalably sort the iterables, even if they are 
large (do not fit in memory).
+
+##Caveats
+* This transform performs value-only sorting; the iterable accompanying each 
key is sorted, but *there is no relationship between different keys*, as Beam 
does not support any defined relationship between different elements in a 
PCollection.
+* Each `Iterable>` is sorted on a single worker using local memory 
and disk. This means that `SortValues` may be a performance and/or scalability 
bottleneck when used in different pipelines. For example, users are discouraged 
from using `SortValues` on a `PCollection` of a single element to globally sort 
a large `PCollection`.
+
+##Options
+* The user can customize the temporary location used if sorting requires 
spilling to disk and the maximum amount of memory to use by creating a custom 
instance of `BufferedExternalSorter.Options` to pass into `SortValues.create`.
+
+##Using `SortValues`
+
+PCollection>> input = ...
+
+// Group by primary key, bringing  pairs for the same key 
together.
+PCollection>create());
+
+// For every primary key, sort the iterable of  pairs by 
secondary key.
+PCollectioncreate(new 
BufferedExternalSorter.Options()));
+
\ No newline at end of file

[2/2] incubator-beam git commit: Closes #1199

[dhalperi]

Closes #1199


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fab7b240
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fab7b240
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fab7b240

Branch: refs/heads/master
Commit: fab7b2402fb03b20e9cad8e9438ec094283786cf
Parents: afb1e91 49ee929
Author: Dan Halperin 
Authored: Tue Nov 1 10:40:21 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 10:40:21 2016 -0700

--
 sdks/java/extensions/pom.xml|   1 +
 sdks/java/extensions/sorter/README.md   |  42 
 sdks/java/extensions/sorter/pom.xml | 167 ++
 .../sorter/BufferedExternalSorter.java  | 125 +++
 .../sdk/extensions/sorter/ExternalSorter.java   | 225 +++
 .../sdk/extensions/sorter/InMemorySorter.java   | 166 ++
 .../beam/sdk/extensions/sorter/SortValues.java  | 213 ++
 .../beam/sdk/extensions/sorter/Sorter.java  |  47 
 .../sdk/extensions/sorter/package-info.java |  23 ++
 .../sorter/BufferedExternalSorterTest.java  | 177 +++
 .../extensions/sorter/ExternalSorterTest.java   |  87 +++
 .../extensions/sorter/InMemorySorterTest.java   | 144 
 .../sdk/extensions/sorter/SortValuesTest.java   | 128 +++
 .../sdk/extensions/sorter/SorterTestUtils.java  | 129 +++
 14 files changed, 1674 insertions(+)
--

[GitHub] incubator-beam pull request #1248: [BEAM-820] Skip javadoc for real in Travi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-820] Skip javadoc for real

[dhalperi]

[BEAM-820] Skip javadoc for real


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fcba3d9d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fcba3d9d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fcba3d9d

Branch: refs/heads/master
Commit: fcba3d9d460c9c010bd6b562c5a051fb06bbf3c6
Parents: 7745c91
Author: Dan Halperin 
Authored: Tue Nov 1 09:20:48 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 10:39:39 2016 -0700

--
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fcba3d9d/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 3234f0c..9e1406c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -30,7 +30,7 @@ notifications:
 
 env:
   global:
-   - MAVEN_OVERRIDE="--settings=.travis/settings.xml -Djavadoc.skip=true 
-Dcheckstyle.skip=true -Dfindbugs.skip=true"
+   - MAVEN_OVERRIDE="--settings=.travis/settings.xml -Dmaven.javadoc.skip=true 
-Dcheckstyle.skip=true -Dfindbugs.skip=true"
- MAVEN_CONTAINER_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 
 matrix:

[jira] [Commented] (BEAM-859) Fix building with eclipse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625978#comment-15625978
 ] 

ASF GitHub Bot commented on BEAM-859:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1247


> Fix building with eclipse
> -
>
> Key: BEAM-859
> URL: https://issues.apache.org/jira/browse/BEAM-859
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> Kenn has gotten travis working more reliably, which has exposed that we broke 
> the Eclipse travis build again.
> > I'm trying to debug some errors in the Eclipse travis check: 
> > https://s3.amazonaws.com/archive.travis-ci.org/jobs/170856465/log.txt
> >
> > Some online threads made me think it could be related to Java8 vs Java7. 
> > E.g., https://jira.nuxeo.com/browse/NXP-18655
> >
> > https://github.com/apache/incubator-beam/commit/957c545eaa33c861b561418b1c7dadf4c31f92f3#diff-354f30a63fb0907d4ad57269548329e3R42
> Maybe we should move the Eclipse test to Java8 instead of Java7 and see what 
> happens.
> cc [~dkulp]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625975#comment-15625975
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1134


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #1247

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 75bfd781f -> 7745c91ba


Closes #1247


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7745c91b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7745c91b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7745c91b

Branch: refs/heads/master
Commit: 7745c91ba178fd746d4d4f0713ef3cae82b9dc53
Parents: 75bfd78 2952702
Author: Dan Halperin 
Authored: Tue Nov 1 09:51:55 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 09:51:55 2016 -0700

--
 .travis.yml | 4 ++--
 pom.xml | 3 +--
 2 files changed, 3 insertions(+), 4 deletions(-)
--

[2/2] incubator-beam git commit: [BEAM-859] Move Eclipse precommit build to Java 8

[dhalperi]

[BEAM-859] Move Eclipse precommit build to Java 8

Also reorder env variables so it's clear what the precommit is.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/29527027
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/29527027
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/29527027

Branch: refs/heads/master
Commit: 29527027c76adadf48d228781ca2dc7133c11dd8
Parents: 75bfd78
Author: Dan Halperin 
Authored: Fri Oct 28 09:20:17 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 09:51:55 2016 -0700

--
 .travis.yml | 4 ++--
 pom.xml | 3 +--
 2 files changed, 3 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/29527027/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 6a8b604..3234f0c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -44,9 +44,9 @@ matrix:
 - os: linux
   env: CUSTOM_JDK="oraclejdk7" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE"
 - os: linux
-  env: CUSTOM_JDK="oraclejdk7" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE -Peclipse-jdt"
-- os: linux
   env: CUSTOM_JDK="openjdk7" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE"
+- os: linux
+  env: MAVEN_OVERRIDE="-Peclipse-jdt -DskipTests $MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE" CUSTOM_JDK="oraclejdk8"
 
 before_install:
   - echo 'MAVEN_OPTS="$MAVEN_OPTS -Xmx1024m -XX:MaxPermSize=512m 
-XX:+BytecodeVerificationLocal"' >> ~/.mavenrc

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/29527027/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 2820732..51518ed 100644
--- a/pom.xml
+++ b/pom.xml
@@ -243,8 +243,7 @@
 
   org.eclipse.tycho
   tycho-compiler-jdt
-  
-  0.24.0
+  0.26.0
 
   
 

[GitHub] incubator-beam pull request #1247: [BEAM-859] Move Eclipse precommit build t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1247


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1134: [BEAM-551] Allow TextIO to accept ValuePr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1134


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1134

[dhalperi]

Closes #1134


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/75bfd781
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/75bfd781
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/75bfd781

Branch: refs/heads/master
Commit: 75bfd781fe91a03538c37166b587a64db5de3c4e
Parents: f8e26be 4ebdf0b
Author: Dan Halperin 
Authored: Tue Nov 1 09:44:30 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 09:44:30 2016 -0700

--
 .../dataflow/internal/ReadTranslator.java   | 12 ++--
 .../DataflowPipelineTranslatorTest.java | 28 ++
 .../apache/beam/sdk/io/CompressedSource.java|  2 +-
 .../org/apache/beam/sdk/io/FileBasedSource.java | 59 ++--
 .../java/org/apache/beam/sdk/io/TextIO.java | 43 +++---
 5 files changed, 116 insertions(+), 28 deletions(-)
--

[1/2] incubator-beam git commit: TextIO.Read: support ValueProvider

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f8e26be7d -> 75bfd781f


TextIO.Read: support ValueProvider

Add test


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4ebdf0bf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4ebdf0bf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4ebdf0bf

Branch: refs/heads/master
Commit: 4ebdf0bfcd17ab5df9b5d7132507c49979392721
Parents: f8e26be
Author: sammcveety 
Authored: Sun Oct 16 18:37:51 2016 -0400
Committer: Dan Halperin 
Committed: Tue Nov 1 09:43:53 2016 -0700

--
 .../dataflow/internal/ReadTranslator.java   | 12 ++--
 .../DataflowPipelineTranslatorTest.java | 28 ++
 .../apache/beam/sdk/io/CompressedSource.java|  2 +-
 .../org/apache/beam/sdk/io/FileBasedSource.java | 59 ++--
 .../java/org/apache/beam/sdk/io/TextIO.java | 43 +++---
 5 files changed, 116 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4ebdf0bf/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/ReadTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/ReadTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/ReadTranslator.java
index 83836c0..b3af165 100755
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/ReadTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/ReadTranslator.java
@@ -30,6 +30,7 @@ import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TranslationCo
 import org.apache.beam.sdk.io.FileBasedSource;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.Source;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.PropertyNames;
 import org.apache.beam.sdk.values.PValue;
@@ -50,10 +51,13 @@ public class ReadTranslator implements 
TransformTranslator {
   // TODO: Move this validation out of translation once IOChannelUtils is 
portable
   // and can be reconstructed on the worker.
   if (source instanceof FileBasedSource) {
-String filePatternOrSpec = ((FileBasedSource) 
source).getFileOrPatternSpec();
-context.getPipelineOptions()
-   .getPathValidator()
-   .validateInputFilePatternSupported(filePatternOrSpec);
+ValueProvider filePatternOrSpec =
+((FileBasedSource) source).getFileOrPatternSpecProvider();
+if (filePatternOrSpec.isAccessible()) {
+  context.getPipelineOptions()
+  .getPathValidator()
+  .validateInputFilePatternSupported(filePatternOrSpec.get());
+}
   }
 
   context.addStep(transform, "ParallelRead");

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4ebdf0bf/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index 762844b..c925454 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -67,6 +67,7 @@ import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.OldDoFn;
@@ -732,6 +733,33 @@ public class DataflowPipelineTranslatorTest implements 
Serializable {
 Collections.emptyList());
   }
 
+  private static class TestValueProvider implements ValueProvider, 
Serializable {
+@Override
+public boolean isAccessible() {
+  return false;
+}
+
+@Override
+public String get() {
+  throw new RuntimeException("Should not be called.");
+}
+  }
+
+  @Test
+  public void testInaccessibleProvider() throws Exception {
+

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625902#comment-15625902
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1249

[BEAM-551] Add --templateRunner option to Dataflow

This provides the DataflowRunner with a means of creating templates for use 
with ValueProvider. 

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/template_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1249.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1249


commit 4707450f65b40851918205dd27f0eb0d1cb7348d
Author: Sam McVeety 
Date:   2016-11-01T16:27:44Z

Add templateRunner option to Dataflow runner




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1249: [BEAM-551] Add --templateRunner option to...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1249

[BEAM-551] Add --templateRunner option to Dataflow

This provides the DataflowRunner with a means of creating templates for use 
with ValueProvider. 

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/template_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1249.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1249


commit 4707450f65b40851918205dd27f0eb0d1cb7348d
Author: Sam McVeety 
Date:   2016-11-01T16:27:44Z

Add templateRunner option to Dataflow runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-820) Travis test_wordcount.sh retrieve snapshot from repo, not Maven reactor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625865#comment-15625865
 ] 

ASF GitHub Bot commented on BEAM-820:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1248

[BEAM-820] Skip javadoc for real

R: @kennknowles 

https://maven.apache.org/plugins/maven-javadoc-plugin/javadoc-mojo.html#skip

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
disable-javadoc-for-real

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1248






> Travis test_wordcount.sh retrieve snapshot from repo, not Maven reactor
> ---
>
> Key: BEAM-820
> URL: https://issues.apache.org/jira/browse/BEAM-820
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: Not applicable
>
>
> Because of the particulars of the Maven invocation `mvn exec:java`, the 
> snapshot of the SDK that is used is pulled from the snapshot repository, not 
> the Maven reactor, so it does not correspond to the current SDK.
> In particular, this means at each release the tests will fail until a 
> snapshot is built. It is also slower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1248: [BEAM-820] Skip javadoc for real

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1248

[BEAM-820] Skip javadoc for real

R: @kennknowles 

https://maven.apache.org/plugins/maven-javadoc-plugin/javadoc-mojo.html#skip

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
disable-javadoc-for-real

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1248






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-859) Fix building with eclipse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625758#comment-15625758
 ] 

ASF GitHub Bot commented on BEAM-859:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1247

[BEAM-859] Move Eclipse precommit build to Java 8

Also reorder env variables so it's clear what the precommit is.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam beam-859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1247


commit 5f0c97998734e37a098e1d361e15256bd3452100
Author: Dan Halperin 
Date:   2016-10-28T16:20:17Z

[BEAM-859] Move Eclipse precommit build to Java 8

Also reorder env variables so it's clear what the precommit is.




> Fix building with eclipse
> -
>
> Key: BEAM-859
> URL: https://issues.apache.org/jira/browse/BEAM-859
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> Kenn has gotten travis working more reliably, which has exposed that we broke 
> the Eclipse travis build again.
> > I'm trying to debug some errors in the Eclipse travis check: 
> > https://s3.amazonaws.com/archive.travis-ci.org/jobs/170856465/log.txt
> >
> > Some online threads made me think it could be related to Java8 vs Java7. 
> > E.g., https://jira.nuxeo.com/browse/NXP-18655
> >
> > https://github.com/apache/incubator-beam/commit/957c545eaa33c861b561418b1c7dadf4c31f92f3#diff-354f30a63fb0907d4ad57269548329e3R42
> Maybe we should move the Eclipse test to Java8 instead of Java7 and see what 
> happens.
> cc [~dkulp]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1247: [BEAM-859] Move Eclipse precommit build t...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1247

[BEAM-859] Move Eclipse precommit build to Java 8

Also reorder env variables so it's clear what the precommit is.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam beam-859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1247


commit 5f0c97998734e37a098e1d361e15256bd3452100
Author: Dan Halperin 
Date:   2016-10-28T16:20:17Z

[BEAM-859] Move Eclipse precommit build to Java 8

Also reorder env variables so it's clear what the precommit is.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1246: [BEAM-626] Changes in AvroCoder serializa...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1246

[BEAM-626] Changes in AvroCoder serialization so it can serialize in Kryo

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam fix941

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1246


commit 1f770d400ca9e5506199783551c0780a1c46f77e
Author: Aviem Zur 
Date:   2016-09-08T08:21:41Z

Changes in AvroCoder serialization so it can serialize in Kryo

commit 3df987f76467c1e78ce3738a0f2c0df9cd94f82c
Author: Luke Cwik 
Date:   2016-10-31T18:38:45Z

!fixup Swap to use Serializable ThreadLocal with empty data block instead 
of memoization.

commit 3358d3d6cc036cb06fa37c6c2547909586bf048f
Author: Luke Cwik 
Date:   2016-10-31T20:21:31Z

fixup! Fix the memoization of the schema string to use 
readResolve/writeReplace with final fields for Java. Kryo ignores 
readResolve/writeReplace and serializes the Schema object directly.

commit f92c39f14b49556f8e67c22c408520af0c204cce
Author: Luke Cwik 
Date:   2016-10-31T20:24:49Z

fixup! minor whitespace clean up.

commit a5f25e2e9edf9fe0c1fb59e594d5b450062ba77f
Author: Luke Cwik 
Date:   2016-10-31T20:36:14Z

fixup! Add FB suppression on limitation of FB with Serializable objects 
containing writeReplace.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-626) AvroCoder not deserializing correctly in Kryo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625734#comment-15625734
 ] 

ASF GitHub Bot commented on BEAM-626:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1246

[BEAM-626] Changes in AvroCoder serialization so it can serialize in Kryo

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam fix941

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1246


commit 1f770d400ca9e5506199783551c0780a1c46f77e
Author: Aviem Zur 
Date:   2016-09-08T08:21:41Z

Changes in AvroCoder serialization so it can serialize in Kryo

commit 3df987f76467c1e78ce3738a0f2c0df9cd94f82c
Author: Luke Cwik 
Date:   2016-10-31T18:38:45Z

!fixup Swap to use Serializable ThreadLocal with empty data block instead 
of memoization.

commit 3358d3d6cc036cb06fa37c6c2547909586bf048f
Author: Luke Cwik 
Date:   2016-10-31T20:21:31Z

fixup! Fix the memoization of the schema string to use 
readResolve/writeReplace with final fields for Java. Kryo ignores 
readResolve/writeReplace and serializes the Schema object directly.

commit f92c39f14b49556f8e67c22c408520af0c204cce
Author: Luke Cwik 
Date:   2016-10-31T20:24:49Z

fixup! minor whitespace clean up.

commit a5f25e2e9edf9fe0c1fb59e594d5b450062ba77f
Author: Luke Cwik 
Date:   2016-10-31T20:36:14Z

fixup! Add FB suppression on limitation of FB with Serializable objects 
containing writeReplace.




> AvroCoder not deserializing correctly in Kryo
> -
>
> Key: BEAM-626
> URL: https://issues.apache.org/jira/browse/BEAM-626
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Minor
>
> Unlike with Java serialization, when deserializing AvroCoder using Kryo, the 
> resulting AvroCoder is missing all of its transient fields.
> The reason it works with Java serialization is because of the usage of 
> writeReplace and readResolve, which Kryo does not adhere to.
> In ProtoCoder for example there are also unserializable members, the way it 
> is solved there is lazy initializing these members via their getters, so they 
> are initialized in the deserialized object on first call to the member.
> It seems AvroCoder is the only class in Beam to use writeReplace convention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Fixes crash of shade plugin on SparkProcessContext

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master a38a6072d -> f8e26be7d


Fixes crash of shade plugin on SparkProcessContext

Error discovered by debugging the Maven plugin itself
(using mvnDebug) and setting a breakpoint for the exception
and then inspecting the call stack to spot a weird-looking
method signature descriptor:

`Lorg/apache/beam/sdk/transforms/windowing/WindowFn.AssignContext;`

(note the exclamation mark, which is an invalid character AFAIK!
I have no idea where it came from, but the current change makes it go away)

Further change the signature to make javap happy

When I compile beam using Eclipse Compiler in IntelliJ, this is
the only file that produces an invalid class file unless I make
the current change.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b83f4c45
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b83f4c45
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b83f4c45

Branch: refs/heads/master
Commit: b83f4c45b20fce8d2b944bbe364a3726f9d1c84b
Parents: a38a607
Author: Eugene Kirpichov 
Authored: Mon Oct 31 18:06:43 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 08:27:10 2016 -0700

--
 .../spark/translation/SparkProcessContext.java  | 46 ++--
 1 file changed, 23 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b83f4c45/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
index 2135170..99cd522 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
@@ -176,31 +176,31 @@ public abstract class SparkProcessContext
 }
   }
 
-  static  WindowedValue noElementWindowedValue(final T output,
- final Instant timestamp,
- WindowFn 
windowFn) {
-WindowFn.AssignContext assignContext = windowFn.new AssignContext() {
-
-  @Override
-  public Object element() {
-return output;
-  }
+  static  WindowedValue noElementWindowedValue(
+  final T output, final Instant timestamp, WindowFn windowFn) {
+WindowFn.AssignContext assignContext =
+windowFn.new AssignContext() {
+
+  @Override
+  public Object element() {
+return output;
+  }
 
-  @Override
-  public Instant timestamp() {
-if (timestamp != null) {
-  return timestamp;
-}
-throw new UnsupportedOperationException("outputWithTimestamp was 
called with "
-+ "null timestamp.");
-  }
+  @Override
+  public Instant timestamp() {
+if (timestamp != null) {
+  return timestamp;
+}
+throw new UnsupportedOperationException(
+"outputWithTimestamp was called with " + "null timestamp.");
+  }
 
-  @Override
-  public BoundedWindow window() {
-throw new UnsupportedOperationException("Window not available for "
-+ "start/finishBundle output.");
-  }
-};
+  @Override
+  public BoundedWindow window() {
+throw new UnsupportedOperationException(
+"Window not available for " + "start/finishBundle output.");
+  }
+};
 try {
   @SuppressWarnings("unchecked")
   Collection windows = 
windowFn.assignWindows(assignContext);

[GitHub] incubator-beam pull request #1243: Fixes crash of shade plugin on SparkProce...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1243


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1243

[dhalperi]

Closes #1243


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f8e26be7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f8e26be7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f8e26be7

Branch: refs/heads/master
Commit: f8e26be7d06e1c363112c1172e9afcdea46a7cac
Parents: a38a607 b83f4c4
Author: Dan Halperin 
Authored: Tue Nov 1 08:27:20 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 1 08:27:20 2016 -0700

--
 .../spark/translation/SparkProcessContext.java  | 46 ++--
 1 file changed, 23 insertions(+), 23 deletions(-)
--

[jira] [Commented] (BEAM-404) PubsubIO should have a mode that supports maintaining message attributes.

[Mingjian Song (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625536#comment-15625536
 ] 

Mingjian Song commented on BEAM-404:


We would like to see this issue get addressed as soon as possible. Although 
embedding our custom metadata inside the payload seems to be a viable 
workaround conceptually, it would be more natural to have them in their 
rightful place for both reading and writing.

> PubsubIO should have a mode that supports maintaining message attributes.
> -
>
> Key: BEAM-404
> URL: https://issues.apache.org/jira/browse/BEAM-404
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Daniel Halperin
>
> Right now, PubsubIO only lets uses access the message payload, decoded with 
> the user-provided coder.
> We should add a mode in which the source can return a message with the 
> metadata (attributes) as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-864) Update to latest Apache Maven-Parent

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-864.
---
Resolution: Fixed

> Update to latest Apache Maven-Parent
> 
>
> Key: BEAM-864
> URL: https://issues.apache.org/jira/browse/BEAM-864
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Aljoscha Krettek
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> The release plugin creates a DEPENDENCIES file that is not properly excluded 
> from the rat check with the current version of the Apache Maven-Parent that 
> we are using.
> This is the relevant RAT Jira issue: 
> https://issues.apache.org/jira/browse/RAT-184



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-748) Issues with local processes in MongoDbIO and JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625456#comment-15625456
 ] 

Jean-Baptiste Onofré commented on BEAM-748:
---

It could be related to BEAM-856 (when several jobs are running on the same 
machine). Let's try to see if BEAM-856 (and related) helps.

> Issues with local processes in MongoDbIO and JdbcIO
> ---
>
> Key: BEAM-748
> URL: https://issues.apache.org/jira/browse/BEAM-748
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Hey JB, Can you take a look at these issues? Causing flakes in precommit 
> testing.
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/3931/
> A few different types, likely due to issues setting up databases and reusing 
> them across tests?
> {code}
> java.net.ConnectException : Error connecting to server localhost on port 
> 1,527 with message Connection refused.
> {code}
> {code}
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
> Stacktrace
> java.sql.SQLTransactionRollbackException: 
> A lock could not be obtained due to a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSCOLUMNS, (4,42)
>   Waiting XID : {187, S} , APP, insert into BEAM values (?,?)
>   Granted XID : {185, S} 
> Lock : ROW, SYSTABLES, (1,30)
>   Waiting XID : {185, X} , APP, drop table BEAM
>   Granted XID : {185, S} , {187, S} 
> . The selected victim is XID : 187.
>   at 
> org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source)
>   at org.apache.derby.client.am.SqlException.getSQLException(Unknown 
> Source)
>   at org.apache.derby.client.am.ClientConnection.prepareStatement(Unknown 
> Source)
>   at org.apache.beam.sdk.io.jdbc.JdbcIOTest.initTable(JdbcIOTest.java:106)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> 

[jira] [Commented] (BEAM-864) Update to latest Apache Maven-Parent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625450#comment-15625450
 ] 

ASF GitHub Bot commented on BEAM-864:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1233


> Update to latest Apache Maven-Parent
> 
>
> Key: BEAM-864
> URL: https://issues.apache.org/jira/browse/BEAM-864
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Aljoscha Krettek
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> The release plugin creates a DEPENDENCIES file that is not properly excluded 
> from the rat check with the current version of the Apache Maven-Parent that 
> we are using.
> This is the relevant RAT Jira issue: 
> https://issues.apache.org/jira/browse/RAT-184



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-864) Update to latest Apache Maven-Parent

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-864:
--
Fix Version/s: 0.4.0-incubating

> Update to latest Apache Maven-Parent
> 
>
> Key: BEAM-864
> URL: https://issues.apache.org/jira/browse/BEAM-864
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Aljoscha Krettek
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> The release plugin creates a DEPENDENCIES file that is not properly excluded 
> from the rat check with the current version of the Apache Maven-Parent that 
> we are using.
> This is the relevant RAT Jira issue: 
> https://issues.apache.org/jira/browse/RAT-184



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1233: [BEAM-864] Update Apache POM parent to 18...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1233


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---