[GitHub] incubator-beam pull request #1407: Beam archetypes: enable snapshot reposito...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1407


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1407

[davor]

This closes #1407


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/249dbc04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/249dbc04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/249dbc04

Branch: refs/heads/master
Commit: 249dbc045ccfbfb19280114e973c52d48218839d
Parents: 70efa47 f3c4709
Author: Davor Bonaci 
Authored: Mon Nov 21 23:38:19 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 23:38:19 2016 -0800

--
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../test/resources/projects/basic/reference/pom.xml   | 14 ++
 3 files changed, 42 insertions(+)
--

[1/2] incubator-beam git commit: Beam archetypes: enable snapshot repositories.

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 70efa47cf -> 249dbc045


Beam archetypes: enable snapshot repositories.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3c4709f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3c4709f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3c4709f

Branch: refs/heads/master
Commit: f3c4709f6a8d6260f98ccdb1492d8094d87dd696
Parents: 70efa47
Author: Pei He 
Authored: Mon Nov 21 17:19:27 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 23:38:11 2016 -0800

--
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../test/resources/projects/basic/reference/pom.xml   | 14 ++
 3 files changed, 42 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 45bcf8a..b18c57c 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -26,6 +26,20 @@
 
   jar
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index 5287d1f..738acde 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -24,6 +24,20 @@
   ${artifactId}
   ${version}
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   

   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index cce2f99..6950ed5 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -24,6 +24,20 @@
   basic
   0.1
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   

   

[jira] [Commented] (BEAM-951) Cleanup FileBasedSink to prepare IOChannelFactory redesign.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685940#comment-15685940
 ] 

ASF GitHub Bot commented on BEAM-951:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1412

[BEAM-951] IOChannelFactory: replace toPath() with resolveSibling() t…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…o tighten the interface.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
io-channel-factory-to-path

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1412


commit aabea4aef0a4369ebd2c90ed44e6a60443019e70
Author: Pei He 
Date:   2016-11-22T07:20:59Z

[BEAM-951] IOChannelFactory: replace toPath() with resolveSibling() to 
tighten the interface.




> Cleanup FileBasedSink to prepare IOChannelFactory redesign.
> ---
>
> Key: BEAM-951
> URL: https://issues.apache.org/jira/browse/BEAM-951
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> FileBasedSink uses both FileOperations and IOChannelFactory interfaces, and 
> they need to be merged.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1412: [BEAM-951] IOChannelFactory: replace toPa...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1412

[BEAM-951] IOChannelFactory: replace toPath() with resolveSibling() t…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…o tighten the interface.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
io-channel-factory-to-path

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1412


commit aabea4aef0a4369ebd2c90ed44e6a60443019e70
Author: Pei He 
Date:   2016-11-22T07:20:59Z

[BEAM-951] IOChannelFactory: replace toPath() with resolveSibling() to 
tighten the interface.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1033) WindowedWordCountIT is flaky

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-1033:
-
Assignee: Mark Liu  (was: Davor Bonaci)

> WindowedWordCountIT is flaky
> 
>
> Key: BEAM-1033
> URL: https://issues.apache.org/jira/browse/BEAM-1033
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Mark Liu
>
> Jenkins link:
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console
> Running org.apache.beam.examples.WindowedWordCountIT
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec 
> <<< FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
> testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)  
> Time elapsed: 304.282 sec  <<< FAILURE!
> java.lang.AssertionError: 
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
>  but: Invalid BigQuery response: 
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Results :
> Failed tests: 
>   
> WindowedWordCountIT.testWindowedWordCountInBatch:59->testWindowedWordCountPipeline:88
>  
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
>  but: Invalid BigQuery response: 
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1033) WindowedWordCountIT is flaky

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685913#comment-15685913
 ] 

Pei He commented on BEAM-1033:
--

I think the BigQueryMatcher should keep retry for this case, since the the 
query job is not complete.

BigQuery response: 
{"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}

https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/BigqueryMatcher.java

> WindowedWordCountIT is flaky
> 
>
> Key: BEAM-1033
> URL: https://issues.apache.org/jira/browse/BEAM-1033
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Pei He
>Assignee: Mark Liu
>
> Jenkins link:
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console
> Running org.apache.beam.examples.WindowedWordCountIT
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec 
> <<< FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
> testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)  
> Time elapsed: 304.282 sec  <<< FAILURE!
> java.lang.AssertionError: 
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
>  but: Invalid BigQuery response: 
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
>   at 
> org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Results :
> Failed tests: 
>   
> WindowedWordCountIT.testWindowedWordCountInBatch:59->testWindowedWordCountPipeline:88
>  
> Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
>  but: Invalid BigQuery response: 
> {"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1033) WindowedWordCountIT is flaky

[Pei He (JIRA)]

Pei He created BEAM-1033:


 Summary: WindowedWordCountIT is flaky
 Key: BEAM-1033
 URL: https://issues.apache.org/jira/browse/BEAM-1033
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Pei He
Assignee: Davor Bonaci


Jenkins link:
https://builds.apache.org/job/beam_PreCommit_MavenVerify/5145/console


Running org.apache.beam.examples.WindowedWordCountIT
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 304.282 sec <<< 
FAILURE! - in org.apache.beam.examples.WindowedWordCountIT
testWindowedWordCountInBatch(org.apache.beam.examples.WindowedWordCountIT)  
Time elapsed: 304.282 sec  <<< FAILURE!
java.lang.AssertionError: 

Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
 but: Invalid BigQuery response: 
{"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
at 
org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:164)
at 
org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:93)
at 
org.apache.beam.runners.dataflow.testing.TestDataflowRunner.run(TestDataflowRunner.java:61)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
at 
org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:224)
at 
org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:88)
at 
org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatch(WindowedWordCountIT.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


Results :

Failed tests: 
  
WindowedWordCountIT.testWindowedWordCountInBatch:59->testWindowedWordCountPipeline:88
 
Expected: Expected checksum is (cd5b52939257e12428a9fa085c32a84dd209b180)
 but: Invalid BigQuery response: 
{"jobComplete":false,"jobReference":{"jobId":"job_0STNX_OD83tQOzo6MvmqXCrk61U","projectId":"apache-beam-testing"},"kind":"bigquery#queryResponse"}

Tests run: 3, Failures: 1, Errors: 0, Skipped: 0




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #94: Add maturity model assessment

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam-site/pull/94

Add maturity model assessment

R: @francesperry @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site maturity

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/94.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #94


commit 341f4612e9e5b4956004297ead2bdfc602c23bc7
Author: Davor Bonaci 
Date:   2016-11-22T06:48:40Z

Add maturity model assessment




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685840#comment-15685840
 ] 

Jean-Baptiste Onofré commented on BEAM-975:
---

Yes, I will take a look.

As {{MongoClient}} is not serializable, it's not possible to pass it directly: 
it has to be created in the IO.

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685778#comment-15685778
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1411

[BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is 
added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers   

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DataflowRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1411.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1411


commit c9640ea08e8078752a51508430c3661555c4c8a7
Author: Kenneth Knowles 
Date:   2016-11-16T05:35:03Z

Reject stateful DoFn in DataflowRunner

commit 7b7d2dd30ee7bf56fddaa221386f1046875176b1
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685776#comment-15685776
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1410

[BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is 
added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tweise  

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ApexRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1410.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1410


commit e85cea78253d2f316a18d95d65aabc1176448841
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:01Z

Reject stateful DoFn in ApexRunner

commit f8b6bb7f1ab8720ca4f2d766831d8f243dd27085
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1410: [BEAM-25] Temporarily reject stateful Par...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1410

[BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is 
added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tweise  

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ApexRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1410.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1410


commit e85cea78253d2f316a18d95d65aabc1176448841
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:01Z

Reject stateful DoFn in ApexRunner

commit f8b6bb7f1ab8720ca4f2d766831d8f243dd27085
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1411: [BEAM-25] Temporarily reject stateful Par...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1411

[BEAM-25] Temporarily reject stateful ParDo in ApexRunner (until support is 
added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers   

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DataflowRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1411.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1411


commit c9640ea08e8078752a51508430c3661555c4c8a7
Author: Kenneth Knowles 
Date:   2016-11-16T05:35:03Z

Reject stateful DoFn in DataflowRunner

commit 7b7d2dd30ee7bf56fddaa221386f1046875176b1
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685772#comment-15685772
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1409

[BEAM-25] Temporarily reject stateful ParDo in SparkRunner (until support 
is added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam SparkRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1409.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1409


commit 8d715689dd5283b7b180c0b9ec4e188abba140f5
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests

commit b0d07d74f7805ee1d30fdedf54c089790d63d898
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:13Z

Reject stateful DoFn in SparkRunner




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1409: [BEAM-25] Temporarily reject stateful Par...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1409

[BEAM-25] Temporarily reject stateful ParDo in SparkRunner (until support 
is added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @amitsela 

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam SparkRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1409.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1409


commit 8d715689dd5283b7b180c0b9ec4e188abba140f5
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests

commit b0d07d74f7805ee1d30fdedf54c089790d63d898
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:13Z

Reject stateful DoFn in SparkRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685770#comment-15685770
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1408

[BEAM-25] Temporarily reject stateful ParDo in FlinkRunner (until support 
is added)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha or @mxm 

At present, the user-facing API for stateful `DoFn` is in the codebase but 
prevented from use. It is `ParDo.of(...)` that rejects stateful `DoFn`. In 
#1399 I am removing this rejection, so I need to add it to all runners until 
they support the API.

I have also added a JUnit category so runners can exclude this from their 
`RunnableOnService` test suites.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam FlinkRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1408


commit 7de2ea30c46d076771d2759111558a01d39d8a84
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:28Z

Reject stateful DoFn in FlinkRunner

commit 75744f99d5ef0cb7c0da333c2f0651ee13901e8f
Author: Kenneth Knowles 
Date:   2016-11-21T23:41:13Z

Add JUnit category for stateful ParDo tests




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1405: Make create() also available to pure pyth...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1405


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Make create() avaialable to pure python callers

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk b4fc20e44 -> 9060f8395


Make create() avaialable to pure python callers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/081aa975
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/081aa975
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/081aa975

Branch: refs/heads/python-sdk
Commit: 081aa975c80187bdb0abaaa1e6ed2445b1e31b84
Parents: 50afc87
Author: Ahmet Altay 
Authored: Mon Nov 21 16:19:22 2016 -0800
Committer: Ahmet Altay 
Committed: Mon Nov 21 16:19:22 2016 -0800

--
 sdks/python/apache_beam/utils/windowed_value.pxd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/081aa975/sdks/python/apache_beam/utils/windowed_value.pxd
--
diff --git a/sdks/python/apache_beam/utils/windowed_value.pxd 
b/sdks/python/apache_beam/utils/windowed_value.pxd
index 41c2986..5544baf 100644
--- a/sdks/python/apache_beam/utils/windowed_value.pxd
+++ b/sdks/python/apache_beam/utils/windowed_value.pxd
@@ -34,5 +34,5 @@ cdef class WindowedValue(object):
   cdef inline bint _typed_eq(WindowedValue left, WindowedValue right) except? 
-2
 
 @cython.locals(wv=WindowedValue)
-cdef WindowedValue create(
+cpdef WindowedValue create(
   object value, int64_t timestamp_micros, object windows)

[2/2] incubator-beam git commit: This closes #1405

[kenn]

This closes #1405


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9060f839
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9060f839
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9060f839

Branch: refs/heads/python-sdk
Commit: 9060f83950a5f5f2bd663484ef0cd9e26248fe83
Parents: b4fc20e 081aa97
Author: Kenneth Knowles 
Authored: Mon Nov 21 20:23:56 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 21 20:23:56 2016 -0800

--
 sdks/python/apache_beam/utils/windowed_value.pxd | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685533#comment-15685533
 ] 

Reza Nouri commented on BEAM-975:
-

Thanks [~davor]. I guess a possible solution would be enabling this option to 
pass MongoClient to the MongoDBIO rather than manage it internally. This allows 
us to set appropriate options on MongoClient. It also, provides more 
flexibility.

Let me know if I can help with this. 

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1010) Custom FileSinks should respect AUTO compression

[Konstantinos Katsiapis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Katsiapis resolved BEAM-1010.
--
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

This was fixed by the Merge of 
https://github.com/apache/incubator-beam/pull/1392 as 
https://github.com/apache/incubator-beam/commit/8e88c7b035e76c6e15d03a79f9751c6e53786859

> Custom FileSinks should respect AUTO compression
> 
>
> Key: BEAM-1010
> URL: https://issues.apache.org/jira/browse/BEAM-1010
> Project: Beam
>  Issue Type: Bug
>Reporter: Konstantinos Katsiapis
>Assignee: Konstantinos Katsiapis
> Fix For: 0.4.0-incubating
>
>
> Currently AUTO compression is respected by Native FileSinks but not Custom 
> FileSinks.
> Also, it would be good for Beam's _CompresedFile to support usage with Python 
> "with" clause (ie implement __enter__ and __exit__).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685485#comment-15685485
 ] 

Davor Bonaci commented on BEAM-975:
---

[~jbonofre], would you mind taking a quick peek? I think you have the most 
expertise here.

[~reza.nouri], please also use the mailing lists (dev@ and user@) to expedite, 
as appropriate.

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-975) Issue with MongoDBIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-975:
--
Assignee: Jean-Baptiste Onofré

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-975) Issue with MongoDBIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-975:
--
Component/s: sdk-java-extensions

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam-site git commit: This closes #89

[davor]

This closes #89


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4215e967
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4215e967
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4215e967

Branch: refs/heads/asf-site
Commit: 4215e967b15a0aadea07670b9e929fd8918206c6
Parents: 60791ca 66f2d55
Author: Davor Bonaci 
Authored: Mon Nov 21 18:44:00 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 18:44:00 2016 -0800

--
 content/get-started/quickstart/index.html | 7 ---
 src/get-started/quickstart.md | 7 ---
 2 files changed, 8 insertions(+), 6 deletions(-)
--

[GitHub] incubator-beam-site pull request #89: Quickstart: suggest using nightly buil...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/89


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1026) User guide - "How to create Beam IO Transforms"

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk resolved BEAM-1026.

   Resolution: Duplicate
Fix Version/s: Not applicable

> User guide - "How to create Beam IO Transforms"
> ---
>
> Key: BEAM-1026
> URL: https://issues.apache.org/jira/browse/BEAM-1026
> Project: Beam
>  Issue Type: Task
>Reporter: Stephen Sisk
> Fix For: Not applicable
>
>
> Beam has javadocs for how to create a read or write transform, but no 
> friendly user guide on how to get started using BoundedSource/BoundedReader.
> This should cover:
> * background on beam's source/sink API design 
> * design patterns
> * evaluating different data sources (eg, what are the properties of a pub sub 
> system that affect how you should write your UnboundedSource? What is the 
> best design for reading from a NoSql style source?)
> * testing - how to write unit, integration (and once we have them, 
> performance tests)
> * public API recommendations
> This is related, but not strictly overlapping with: 
> https://issues.apache.org/jira/browse/BEAM-193
> - the Dataflow SDK documentation for "Custom Sources and Sinks"  contains 
> some info about writing Sources/Sinks, but it is somewhat out of date, and 
> doesn't reflect the things we've learned recently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1407: Beam archetypes: enable snapshot reposito...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1407

Beam archetypes: enable snapshot repositories.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam quick-start-nightly

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1407


commit 6e67c76a1b90d73a498a7b340d0bec4e962b7eb2
Author: Pei He 
Date:   2016-11-22T01:19:27Z

Beam archetypes: enable snapshot repositories.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b0f91e7f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b0f91e7f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b0f91e7f

Branch: refs/heads/asf-site
Commit: b0f91e7f11ec514d233f8bccb632676e37093eaa
Parents: 7f7a8b2
Author: Davor Bonaci 
Authored: Mon Nov 21 17:03:28 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 17:03:28 2016 -0800

--
 content/get-started/downloads/index.html | 33 +--
 1 file changed, 21 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b0f91e7f/content/get-started/downloads/index.html
--
diff --git a/content/get-started/downloads/index.html 
b/content/get-started/downloads/index.html
index 5b89c96..0849509 100644
--- a/content/get-started/downloads/index.html
+++ b/content/get-started/downloads/index.html
@@ -167,28 +167,37 @@ Java with the DirectRunner, add the follo
 connectors or other extensions, and additional runners to execute your pipeline
 at scale.
 
-Release Notes
+API Stability
 
-0.3.0-incubating
-https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zipaction=download;>Source
 code download
+Apache Beam uses http://semver.org/;>semantic versioning. 
Version numbers use the form major.minor.incremental and are incremented as 
follows:
 
 
-  Release notes are available https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12338051;>in
 JIRA.
+  major version for incompatible API changes
+  minor version for new functionality added in a backward-compatible 
manner
+  incremental version for forward-compatible bug fixes
 
 
+Please note that APIs marked @Experimental
+may change at any point and are not guaranteed to remain compatible across 
versions.
+
+Additionally, any API may change before the first stable release, i.e., 
between versions denoted 0.x.y.
+
+Releases
+
+0.3.0-incubating
+Official https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zipaction=download;>source
 code download.
+
+https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12338051;>Release
 notes.
+
 0.2.0-incubating
-https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zipaction=download;>Source
 code download
+Official https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zipaction=download;>source
 code download.
 
-
-  Release notes are available https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12335766;>in
 JIRA.
-
+https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12335766;>Release
 notes.
 
 0.1.0-incubating
-https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zipaction=download;>Source
 code download
+Official https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zipaction=download;>source
 code download.
 
-
-  The first incubating release of Apache Beam.
-
+The first incubating release of Apache Beam.
 
   
 

[3/3] incubator-beam-site git commit: This closes #92

[davor]

This closes #92


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/60791ca2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/60791ca2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/60791ca2

Branch: refs/heads/asf-site
Commit: 60791ca2eaec274d972f9f9d3ce4860392a7bf1b
Parents: 0c4bff7 b0f91e7
Author: Davor Bonaci 
Authored: Mon Nov 21 17:03:28 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 17:03:28 2016 -0800

--
 content/get-started/downloads/index.html | 33 +--
 src/get-started/downloads.md | 27 --
 2 files changed, 41 insertions(+), 19 deletions(-)
--

[GitHub] incubator-beam-site pull request #92: Update download page to specify backwa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/92


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam-site git commit: Update download page to specify backward-compatibility guarantees

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 0c4bff77b -> 60791ca2e


Update download page to specify backward-compatibility guarantees


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/7f7a8b2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/7f7a8b2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/7f7a8b2d

Branch: refs/heads/asf-site
Commit: 7f7a8b2d3e8672516627b6949d0cbafa44e13fc5
Parents: 0c4bff7
Author: Davor Bonaci 
Authored: Mon Nov 21 13:36:18 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 17:03:12 2016 -0800

--
 src/get-started/downloads.md | 27 ---
 1 file changed, 20 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/7f7a8b2d/src/get-started/downloads.md
--
diff --git a/src/get-started/downloads.md b/src/get-started/downloads.md
index 0015d77..af8506a 100644
--- a/src/get-started/downloads.md
+++ b/src/get-started/downloads.md
@@ -33,19 +33,32 @@ Additionally, you may want to depend on additional SDK 
modules, such as IO
 connectors or other extensions, and additional runners to execute your pipeline
 at scale.
 
-## Release Notes
+## API Stability
+
+Apache Beam uses [semantic versioning](http://semver.org/). Version numbers 
use the form `major.minor.incremental` and are incremented as follows:
+
+* major version for incompatible API changes
+* minor version for new functionality added in a backward-compatible manner
+* incremental version for forward-compatible bug fixes
+
+Please note that APIs marked [`@Experimental`]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/org/apache/beam/sdk/annotations/Experimental.html)
+may change at any point and are not guaranteed to remain compatible across 
versions.
+
+Additionally, any API may change before the first stable release, i.e., 
between versions denoted `0.x.y`.
+
+## Releases
 
 ### 0.3.0-incubating
-[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zip=download)
+Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zip=download).
 
-* Release notes are available [in 
JIRA](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051).
+[Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051).
 
 ### 0.2.0-incubating
-[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zip=download)
+Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zip=download).
 
-* Release notes are available [in 
JIRA](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12335766).
+[Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12335766).
 
 ### 0.1.0-incubating
-[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zip=download)
+Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zip=download).
 
-* The first incubating release of Apache Beam.
+The first incubating release of Apache Beam.

[jira] [Created] (BEAM-1032) Starter archetype pipeline has no runner on its classpath

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-1032:
--

 Summary: Starter archetype pipeline has no runner on its classpath
 Key: BEAM-1032
 URL: https://issues.apache.org/jira/browse/BEAM-1032
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Scott Wegner
Assignee: Scott Wegner
Priority: Minor


When generating a new maven project using the starter archetype, the pipeline 
does not execute because it doesn't have a runner jar on it's classpath:

$ mvn exec:java -Dexec.mainClass=swegner.StarterPipeline
...
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
(default-cli) on project yo: An exception occured while executing the Java 
class. null: InvocationTargetException: No Runner was specified and the 
DirectRunner was not found on the classpath.



In the examples archetype, we have the direct runner as a dependency. We should 
do the same for the starter archetype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1031) Starter archetype uses OldDoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685256#comment-15685256
 ] 

ASF GitHub Bot commented on BEAM-1031:
--

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1406

[BEAM-1031] Update StarterPipeline to use the new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam starter-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1406


commit 9f2111024e3537910a4ca2543444c415215e028c
Author: Scott Wegner 
Date:   2016-11-22T00:33:07Z

Update starter pipeline to use the new DoFn




> Starter archetype uses OldDoFn
> --
>
> Key: BEAM-1031
> URL: https://issues.apache.org/jira/browse/BEAM-1031
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> The starter archetype should give a positive first impression of Beam. The 
> starter pipeline uses OldDoFn instead of the new DoFn. We should convert it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1406: [BEAM-1031] Update StarterPipeline to use...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1406

[BEAM-1031] Update StarterPipeline to use the new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam starter-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1406


commit 9f2111024e3537910a4ca2543444c415215e028c
Author: Scott Wegner 
Date:   2016-11-22T00:33:07Z

Update starter pipeline to use the new DoFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #93: Update main page and Beam Overview pag...

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/incubator-beam-site/pull/93

Update main page and Beam Overview page

- [x] Move /src/index.md content to /src/get-started/beam-overview.md
- [x] Incorporate some of the content in v2 main page 
(/src/_layouts/v2home.html) into /src/index.md

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site main-page

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/93.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #93


commit 2d7bcc79a05b689ae2acca0ffb8a5911f1969fbc
Author: Hadar Hod 
Date:   2016-11-21T19:18:20Z

WIP - Update main landing page and Beam Overview page

commit b3ada9c9a869e0d2abbcbefd667962cb7bf51639
Author: Hadar Hod 
Date:   2016-11-22T00:31:09Z

Update get-started/beam-overview.md and src/index.md with v2 main page 
content




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #1395

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 50afc87cd -> b4fc20e44


Closes #1395


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b4fc20e4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b4fc20e4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b4fc20e4

Branch: refs/heads/python-sdk
Commit: b4fc20e4432d2b956a0333d8e0c6b9aeb472a804
Parents: 50afc87 c127f7a
Author: bchambers 
Authored: Mon Nov 21 16:21:26 2016 -0800
Committer: bchambers 
Committed: Mon Nov 21 16:21:26 2016 -0800

--
 sdks/python/apache_beam/internal/apiclient.py   |  2 +-
 .../apache_beam/internal/apiclient_test.py  |  1 +
 sdks/python/apache_beam/transforms/display.py   | 27 
 .../apache_beam/transforms/display_test.py  | 17 
 4 files changed, 46 insertions(+), 1 deletion(-)
--

[2/2] incubator-beam git commit: Fixing error with PipelineOptions DisplayData of lists

[bchambers]

Fixing error with PipelineOptions DisplayData of lists


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c127f7a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c127f7a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c127f7a9

Branch: refs/heads/python-sdk
Commit: c127f7a9e6f9885bc595e1462b27b6961c0f465d
Parents: 50afc87
Author: Pablo 
Authored: Mon Nov 21 11:15:37 2016 -0800
Committer: bchambers 
Committed: Mon Nov 21 16:21:26 2016 -0800

--
 sdks/python/apache_beam/internal/apiclient.py   |  2 +-
 .../apache_beam/internal/apiclient_test.py  |  1 +
 sdks/python/apache_beam/transforms/display.py   | 27 
 .../apache_beam/transforms/display_test.py  | 17 
 4 files changed, 46 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c127f7a9/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index 8992ec3..5612631 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -242,7 +242,7 @@ class Environment(object):
   dataflow.Environment.SdkPipelineOptionsValue.AdditionalProperty(
   key='options', value=to_json_value(options_dict)))
 
-  dd = DisplayData.create_from(options)
+  dd = DisplayData.create_from_options(options)
   items = [item.get_dict() for item in dd.items]
   self.proto.sdkPipelineOptions.additionalProperties.append(
   dataflow.Environment.SdkPipelineOptionsValue.AdditionalProperty(

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c127f7a9/sdks/python/apache_beam/internal/apiclient_test.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient_test.py 
b/sdks/python/apache_beam/internal/apiclient_test.py
index 8fddae7..66cc8db 100644
--- a/sdks/python/apache_beam/internal/apiclient_test.py
+++ b/sdks/python/apache_beam/internal/apiclient_test.py
@@ -31,5 +31,6 @@ class UtilTest(unittest.TestCase):
 pipeline_options,
 DataflowPipelineRunner.BATCH_ENVIRONMENT_MAJOR_VERSION)
 
+
 if __name__ == '__main__':
   unittest.main()

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c127f7a9/sdks/python/apache_beam/transforms/display.py
--
diff --git a/sdks/python/apache_beam/transforms/display.py 
b/sdks/python/apache_beam/transforms/display.py
index 365abaf..c38fd9b 100644
--- a/sdks/python/apache_beam/transforms/display.py
+++ b/sdks/python/apache_beam/transforms/display.py
@@ -108,6 +108,33 @@ class DisplayData(object):
   key=key))
 
   @classmethod
+  def create_from_options(cls, pipeline_options):
+""" Creates DisplayData from a PipelineOptions instance.
+
+When creating DisplayData, this method will convert the value of any
+item of a non-supported type to its string representation.
+The normal DisplayData.create_from method rejects those items.
+
+Returns:
+  A DisplayData instance with populated items.
+
+Raises:
+  ValueError: If the has_display_data argument is not an instance of
+HasDisplayData.
+"""
+from apache_beam.utils.options import PipelineOptions
+if not isinstance(pipeline_options, PipelineOptions):
+  raise ValueError(
+  'Element of class {}.{} does not subclass PipelineOptions'
+  .format(pipeline_options.__module__,
+  pipeline_options.__class__.__name__))
+
+items = {k: (v if DisplayDataItem._get_value_type(v) is not None
+ else str(v))
+ for k, v in pipeline_options.display_data().items()}
+return cls(pipeline_options._namespace(), items)
+
+  @classmethod
   def create_from(cls, has_display_data):
 """ Creates DisplayData from a HasDisplayData instance.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c127f7a9/sdks/python/apache_beam/transforms/display_test.py
--
diff --git a/sdks/python/apache_beam/transforms/display_test.py 
b/sdks/python/apache_beam/transforms/display_test.py
index d514065..fc50abe 100644
--- a/sdks/python/apache_beam/transforms/display_test.py
+++ b/sdks/python/apache_beam/transforms/display_test.py
@@ -29,6 +29,7 @@ import apache_beam as beam
 from apache_beam.transforms.display import HasDisplayData
 from apache_beam.transforms.display import DisplayData
 from apache_beam.transforms.display import 

[GitHub] incubator-beam pull request #1405: Make create() avaialable to pure python c...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1405

Make create() avaialable to pure python callers




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam cpdef

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1405


commit 081aa975c80187bdb0abaaa1e6ed2445b1e31b84
Author: Ahmet Altay 
Date:   2016-11-22T00:19:22Z

Make create() avaialable to pure python callers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1030) DoFnSignatures should validate StateSpec against key type

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1030:
-

 Summary: DoFnSignatures should validate StateSpec against key type
 Key: BEAM-1030
 URL: https://issues.apache.org/jira/browse/BEAM-1030
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1029) DoFnSignatures should validate window merging against StateSpec being auto-combinable

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1029:
-

 Summary: DoFnSignatures should validate window merging against 
StateSpec being auto-combinable
 Key: BEAM-1029
 URL: https://issues.apache.org/jira/browse/BEAM-1029
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1028) Merge content from blog post into /documentation/pipelines/test-your-pipeline.md

[Hadar Hod (JIRA)]

Hadar Hod created BEAM-1028:
---

 Summary: Merge content from blog post into 
/documentation/pipelines/test-your-pipeline.md
 Key: BEAM-1028
 URL: https://issues.apache.org/jira/browse/BEAM-1028
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Hadar Hod


blog post: http://beam.incubator.apache.org/blog/2016/10/20/test-stream.html 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1024) upgrade to protobuf-3.1.0

[Rafael Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685118#comment-15685118
 ] 

Rafael Fernandez commented on BEAM-1024:


The latter would be nice! :)

> upgrade to protobuf-3.1.0
> -
>
> Key: BEAM-1024
> URL: https://issues.apache.org/jira/browse/BEAM-1024
> Project: Beam
>  Issue Type: Wish
>Reporter: Rafael Fernandez
>
> The SDK currently uses protobuf 3.0.0-beta-1. There are critical improvements 
> to the library since (such as JsonFormat.parser().ignoringUnknownFields()).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1395: Fixing error with PipelineOptions Display...

[pabloem]

GitHub user pabloem reopened a pull request:

https://github.com/apache/incubator-beam/pull/1395

Fixing error with PipelineOptions DisplayData of lists

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam dd-options-list

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1395


commit a361cf5c0e72babbc7fae58b6401b6fdce555181
Author: Pablo 
Date:   2016-11-21T19:15:37Z

Fixing error with PipelineOptions DisplayData of lists

commit bac792a759b3269daefe4b3d47d1e4de46e5c4fe
Author: Pablo 
Date:   2016-11-21T21:52:43Z

Improving documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Remove microbenchmarks directory from Beam

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 869d082e9 -> 70efa47cf


Remove microbenchmarks directory from Beam

The microbenchmarks should not be released as part of the Beam
distribution, rather they exist for internal measurements and testing.

Given that we also don't have any automatic monitoring (and that any
such measurements are not indicative of real performance), the value of
maintaining microbenchmarks here is negligible.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1b520708
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1b520708
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1b520708

Branch: refs/heads/master
Commit: 1b5207084942e3eeefd1f35c5e34f7943704e19e
Parents: 869d082
Author: bchambers 
Authored: Mon Nov 21 13:35:39 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:32:27 2016 -0800

--
 sdks/java/microbenchmarks/README.md |  42 
 sdks/java/microbenchmarks/pom.xml   | 113 -
 .../coders/AvroCoderBenchmark.java  | 119 --
 .../coders/ByteArrayCoderBenchmark.java |  64 -
 .../coders/CoderBenchmarking.java   |  41 
 .../coders/StringUtf8CoderBenchmark.java|  70 --
 .../microbenchmarks/coders/package-info.java|  22 --
 .../transforms/DoFnInvokersBenchmark.java   | 232 ---
 .../transforms/package-info.java|  22 --
 sdks/java/pom.xml   |   1 -
 10 files changed, 726 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/README.md
--
diff --git a/sdks/java/microbenchmarks/README.md 
b/sdks/java/microbenchmarks/README.md
deleted file mode 100644
index 627e669..000
--- a/sdks/java/microbenchmarks/README.md
+++ /dev/null
@@ -1,42 +0,0 @@
-
-
-# Microbenchmarks for parts of the Beam SDK
-
-To run benchmarks:
-
- 1. Run `mvn install` in the top directory to install the SDK.
-
- 2. Build the benchmark package:
-
-cd microbenchmarks
-mvn package
-
- 3. run benchmark harness:
-
-java -jar target/microbenchmarks.jar
-
- 4. (alternate to step 3)
-to run just a subset of benchmarks, pass a regular expression that
-matches the benchmarks you want to run (this can match against the class
-name, or the method name).  E.g., to run any benchmarks with
-"DoFnReflector" in the name:
-
-java -jar target/microbenchmarks.jar ".*DoFnReflector.*"
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/pom.xml
--
diff --git a/sdks/java/microbenchmarks/pom.xml 
b/sdks/java/microbenchmarks/pom.xml
deleted file mode 100644
index 06bc4df..000
--- a/sdks/java/microbenchmarks/pom.xml
+++ /dev/null
@@ -1,113 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-
-  4.0.0
-
-  
-org.apache.beam
-beam-sdks-java-parent
-0.4.0-incubating-SNAPSHOT
-../pom.xml
-  
-
-  beam-sdks-java-microbenchmarks
-  Apache Beam :: SDKs :: Java :: Microbenchmarks
-  Microbenchmarks for components in the Beam Java 
SDK.
-  jar
-
-  
-
-  
-
-
-  org.codehaus.mojo
-  findbugs-maven-plugin
-  
-true
-  
-
-  
-
-
-
-  
-org.apache.maven.plugins
-maven-compiler-plugin
-  
-
-  
-org.apache.maven.plugins
-maven-jar-plugin
-  
-
-  
-org.apache.maven.plugins
-maven-shade-plugin
-
-  
-package
-
-  shade
-
-
-  microbenchmarks
-  
-
-  org.openjdk.jmh.Main
-
-  
-
-  
-
-  
-
-  
-
-  
-
-  org.apache.beam
-  beam-sdks-java-core
-
-
-
-  joda-time
-  joda-time
-
-
-
-  org.slf4j
-  slf4j-jdk14
-  
-  runtime
-
-
-
-  org.openjdk.jmh
-  jmh-core
-  1.0.1
-
-
-
-  org.openjdk.jmh
-  jmh-generator-annprocess
-  1.6.1
-  provided
-
-  
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/coders/AvroCoderBenchmark.java

[2/2] incubator-beam git commit: This closes #1403

[davor]

This closes #1403


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70efa47c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70efa47c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70efa47c

Branch: refs/heads/master
Commit: 70efa47cf8b37dc1ad064b698ae49616c442bf19
Parents: 869d082 1b52070
Author: Davor Bonaci 
Authored: Mon Nov 21 14:32:39 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:32:39 2016 -0800

--
 sdks/java/microbenchmarks/README.md |  42 
 sdks/java/microbenchmarks/pom.xml   | 113 -
 .../coders/AvroCoderBenchmark.java  | 119 --
 .../coders/ByteArrayCoderBenchmark.java |  64 -
 .../coders/CoderBenchmarking.java   |  41 
 .../coders/StringUtf8CoderBenchmark.java|  70 --
 .../microbenchmarks/coders/package-info.java|  22 --
 .../transforms/DoFnInvokersBenchmark.java   | 232 ---
 .../transforms/package-info.java|  22 --
 sdks/java/pom.xml   |   1 -
 10 files changed, 726 deletions(-)
--

[1/2] incubator-beam git commit: Remove dependency on findbugs annotations artifact

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cb5d216b6 -> 869d082e9


Remove dependency on findbugs annotations artifact

The findbugs annotations artifact has an incompatible license and cannot
be included within Beam. We were previously referencing it for the
@SuppressFBWarning annotation for inline FindBugs suppression.

This change moves inline suppressions out to the existing
findbugs-filter.xml file. While not ideal as it removes the suppressions
from the offending context, it allows us to drop our dependency on the
incompatible artifact.

We are also referencing the @Nullable attribute from findbugs. This is
now source from findbugs jsr305, which does have a compatible license.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d39471b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d39471b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d39471b5

Branch: refs/heads/master
Commit: d39471b51e8d7a56aab2324adc4dbc2d70d9ab40
Parents: cb5d216
Author: Scott Wegner 
Authored: Mon Nov 21 13:07:46 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:29:35 2016 -0800

--
 pom.xml |  34 +---
 runners/apex/pom.xml|   2 +-
 runners/core-java/pom.xml   |   2 +-
 .../beam/runners/core/PaneInfoTracker.java  |   3 -
 .../beam/runners/core/SystemReduceFn.java   |   3 -
 .../apache/beam/runners/core/WatermarkHold.java |   3 -
 .../AfterDelayFromFirstElementStateMachine.java |   5 -
 .../core/triggers/AfterPaneStateMachine.java|   3 -
 .../core/triggers/TriggerStateMachines.java |  15 --
 runners/direct-java/pom.xml |   2 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../src/main/resources/beam/findbugs-filter.xml | 172 ++-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/coders/AvroCoder.java   |   5 -
 .../apache/beam/sdk/coders/InstantCoder.java|   4 -
 .../org/apache/beam/sdk/testing/PAssert.java|   3 -
 .../beam/sdk/testing/SerializableMatchers.java  |   3 -
 ...AttemptAndTimeBoundedExponentialBackOff.java |   3 -
 .../sdk/util/ExposedByteArrayInputStream.java   |   2 -
 .../sdk/util/ExposedByteArrayOutputStream.java  |   3 -
 .../java/org/apache/beam/sdk/util/ZipFiles.java |   9 -
 .../beam/sdk/util/state/StateMerging.java   |   3 -
 sdks/java/io/google-cloud-platform/pom.xml  |  14 +-
 sdks/java/io/hdfs/pom.xml   |   2 +-
 sdks/java/io/jdbc/pom.xml   |   2 +-
 sdks/java/io/jms/pom.xml|   2 +-
 sdks/java/io/kafka/pom.xml  |   2 +-
 sdks/java/io/kinesis/pom.xml|   2 +-
 sdks/java/io/mongodb/pom.xml|   2 +-
 29 files changed, 178 insertions(+), 131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d39471b5/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bd9448a..e0ef47c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -390,12 +390,6 @@
 io.grpc
 grpc-core
 ${grpc.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
@@ -473,10 +467,6 @@
 com.google.guava
 guava-jdk5
   
-  
-com.google.code.findbugs
-jsr305
-  
 
   
 
@@ -501,10 +491,6 @@
 com.google.guava
 guava-jdk5
   
-  
-com.google.code.findbugs
-jsr305
-  
 
   
 
@@ -664,7 +650,7 @@
 
   
 com.google.code.findbugs
-annotations
+jsr305
 ${findbugs.version}
   
 
@@ -672,24 +658,12 @@
 com.google.cloud.bigdataoss
 gcsio
 ${google-cloud-bigdataoss.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
 com.google.cloud.bigdataoss
 util
 ${google-cloud-bigdataoss.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
@@ -829,12 +803,6 @@
 guava-testlib
 ${guava.version}
 test
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d39471b5/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 

[GitHub] incubator-beam pull request #1403: Remove microbenchmarks from pom.xml aggre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1403


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1401: Remove dependency on findbugs annotations...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1401


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1401

[davor]

This closes #1401


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/869d082e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/869d082e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/869d082e

Branch: refs/heads/master
Commit: 869d082e9c1e0807ef282f0ac617c140cd2a3181
Parents: cb5d216 d39471b
Author: Davor Bonaci 
Authored: Mon Nov 21 14:30:13 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:30:13 2016 -0800

--
 pom.xml |  34 +---
 runners/apex/pom.xml|   2 +-
 runners/core-java/pom.xml   |   2 +-
 .../beam/runners/core/PaneInfoTracker.java  |   3 -
 .../beam/runners/core/SystemReduceFn.java   |   3 -
 .../apache/beam/runners/core/WatermarkHold.java |   3 -
 .../AfterDelayFromFirstElementStateMachine.java |   5 -
 .../core/triggers/AfterPaneStateMachine.java|   3 -
 .../core/triggers/TriggerStateMachines.java |  15 --
 runners/direct-java/pom.xml |   2 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../src/main/resources/beam/findbugs-filter.xml | 172 ++-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/coders/AvroCoder.java   |   5 -
 .../apache/beam/sdk/coders/InstantCoder.java|   4 -
 .../org/apache/beam/sdk/testing/PAssert.java|   3 -
 .../beam/sdk/testing/SerializableMatchers.java  |   3 -
 ...AttemptAndTimeBoundedExponentialBackOff.java |   3 -
 .../sdk/util/ExposedByteArrayInputStream.java   |   2 -
 .../sdk/util/ExposedByteArrayOutputStream.java  |   3 -
 .../java/org/apache/beam/sdk/util/ZipFiles.java |   9 -
 .../beam/sdk/util/state/StateMerging.java   |   3 -
 sdks/java/io/google-cloud-platform/pom.xml  |  14 +-
 sdks/java/io/hdfs/pom.xml   |   2 +-
 sdks/java/io/jdbc/pom.xml   |   2 +-
 sdks/java/io/jms/pom.xml|   2 +-
 sdks/java/io/kafka/pom.xml  |   2 +-
 sdks/java/io/kinesis/pom.xml|   2 +-
 sdks/java/io/mongodb/pom.xml|   2 +-
 29 files changed, 178 insertions(+), 131 deletions(-)
--

[jira] [Created] (BEAM-1027) Hosting data stores to enable IO Transform testing

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1027:
--

 Summary: Hosting data stores to enable IO Transform testing
 Key: BEAM-1027
 URL: https://issues.apache.org/jira/browse/BEAM-1027
 Project: Beam
  Issue Type: Task
Reporter: Stephen Sisk


Currently we have a good set of unit tests for our IO Transforms - those
tend to run against in-memory versions of the data stores. However, we'd
like to further increase our test coverage to include running them against
real instances of the data stores that the IO Transforms work against (e.g.
cassandra, mongodb, kafka, etc…), which means we'll need to have real
instances of various data stores.

Additionally, if we want to do performance regression detection, it's
important to have instances of the services that behave realistically,
which isn't true of in-memory or dev versions of the services.

My proposed solution is in 
https://lists.apache.org/thread.html/367fd9669411f21c9ec1f2d27df60464f49d5ce81e6bd16de401d035@%3Cdev.beam.apache.org%3E
 
- it still needs further discussion, and (assuming we agree on the general 
idea), the beam community needs to decide which cluster management software we 
want to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1025) User guide - "How to create Beam IO Transforms"

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1025:
--

 Summary: User guide - "How to create Beam IO Transforms"
 Key: BEAM-1025
 URL: https://issues.apache.org/jira/browse/BEAM-1025
 Project: Beam
  Issue Type: Task
Reporter: Stephen Sisk


Beam has javadocs for how to create a read or write transform, but no friendly 
user guide on how to get started using BoundedSource/BoundedReader.

This should cover:
* background on beam's source/sink API design 
* design patterns
* evaluating different data sources (eg, what are the properties of a pub sub 
system that affect how you should write your UnboundedSource? What is the best 
design for reading from a NoSql style source?)
* testing - how to write unit, integration (and once we have them, performance 
tests)
* public API recommendations

This is related, but not strictly overlapping with: 
https://issues.apache.org/jira/browse/BEAM-193
- the Dataflow SDK documentation for "Custom Sources and Sinks"  contains some 
info about writing Sources/Sinks, but it is somewhat out of date, and doesn't 
reflect the things we've learned recently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1403: Remove microbenchmarks from pom.xml aggre...

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1403

Remove microbenchmarks from pom.xml aggregation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The microbenchmarks should not be released as part of the Beam
distribution, rather they exist for internal measurements and testing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam microbenchmarks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1403


commit 582152cd1b63026b11cd33e229a4d87914520acf
Author: bchambers 
Date:   2016-11-21T21:35:39Z

Remove microbenchmarks from pom.xml aggregation

The microbenchmarks should not be released as part of the Beam
distribution, rather they exist for internal measurements and testing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1121: Remove unnecessary rawtype cast

[bjchambers]

Github user bjchambers closed the pull request at:

https://github.com/apache/incubator-beam/pull/1121


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1396: Add fields retry

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/incubator-beam/pull/1396


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add missing fields to the retry decorator

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 8e88c7b03 -> 50afc87cd


Add missing fields to the retry decorator


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/88d4da43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/88d4da43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/88d4da43

Branch: refs/heads/python-sdk
Commit: 88d4da43b1acb5098cf40c453bc1ac4fb25f2ca7
Parents: 8e88c7b
Author: Sourabh Bajaj 
Authored: Mon Nov 21 11:28:56 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 13:40:41 2016 -0800

--
 sdks/python/apache_beam/utils/retry.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/88d4da43/sdks/python/apache_beam/utils/retry.py
--
diff --git a/sdks/python/apache_beam/utils/retry.py 
b/sdks/python/apache_beam/utils/retry.py
index b3016fd..3874c7a 100644
--- a/sdks/python/apache_beam/utils/retry.py
+++ b/sdks/python/apache_beam/utils/retry.py
@@ -117,7 +117,7 @@ def no_retries(fun):
 def with_exponential_backoff(
 num_retries=16, initial_delay_secs=5.0, logger=logging.warning,
 retry_filter=retry_on_server_errors_filter,
-clock=Clock(), fuzz=True):
+clock=Clock(), fuzz=True, factor=2, max_delay_secs=60 * 60 * 4):
   """Decorator with arguments that control the retry logic.
 
   Args:
@@ -134,6 +134,11 @@ def with_exponential_backoff(
   use time.sleep().
 fuzz: True if the delay should be fuzzed (default). During testing False
   can be used so that the delays are not randomized.
+factor: The exponential factor to use on subsequent retries.
+  Default is 2 (doubling).
+max_delay_sec: Maximum delay (in seconds). After this limit is reached,
+  further tries use max_delay_sec instead of exponentially increasing
+  the time. Defaults to 4 hours.
 
   Returns:
 As per Python decorators with arguments pattern returns a decorator
@@ -155,7 +160,8 @@ def with_exponential_backoff(
 def wrapper(*args, **kwargs):
   retry_intervals = iter(
   FuzzedExponentialIntervals(
-  initial_delay_secs, num_retries, fuzz=0.5 if fuzz else 0))
+  initial_delay_secs, num_retries, factor,
+  fuzz=0.5 if fuzz else 0, max_delay_secs=max_delay_secs))
   while True:
 try:
   return fun(*args, **kwargs)

[GitHub] incubator-beam-site pull request #92: Update download page to specify backwa...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam-site/pull/92

Update download page to specify backward-compatibility guarantees

R: @francesperry or @bchambers or @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site graduation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/92.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #92


commit 17f6dc351953eba051b78cb5353d4d01c6d11ba2
Author: Davor Bonaci 
Date:   2016-11-21T21:36:18Z

Update download page to specify backward-compatibility guarantees




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1402: Update README.md to include instructions ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1402


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1402

[davor]

This closes #1402


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cb5d216b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cb5d216b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cb5d216b

Branch: refs/heads/master
Commit: cb5d216b69ec53f3522621c0acc75c5b364b767f
Parents: c6b86f6 30899db
Author: Davor Bonaci 
Authored: Mon Nov 21 13:33:12 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 13:33:12 2016 -0800

--
 README.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #1402: Update README.md to include instructions ...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam/pull/1402

Update README.md to include instructions on building from source

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam instructions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1402


commit 8c120f775131f91ac0893d37140eafa8b674b544
Author: Davor Bonaci 
Date:   2016-11-21T19:40:03Z

Update README.md to include instructions on building from source




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1024) upgrade to protobuf-3.1.0

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684774#comment-15684774
 ] 

Luke Cwik commented on BEAM-1024:
-

There was an  upgrade to protobuf 3.0.0 in commit 
https://github.com/apache/incubator-beam/commit/f93ca9ce803a8847a7178ff0d7c5e1631bed8f2d
 for Apache Beam.

Upgrading to 3.1.0 would require either shading protobuf everywhere or making 
sure that all our dependencies use protobuf 3.1.0

> upgrade to protobuf-3.1.0
> -
>
> Key: BEAM-1024
> URL: https://issues.apache.org/jira/browse/BEAM-1024
> Project: Beam
>  Issue Type: Wish
>Reporter: Rafael Fernandez
>
> The SDK currently uses protobuf 3.0.0-beta-1. There are critical improvements 
> to the library since (such as JsonFormat.parser().ignoringUnknownFields()).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-998) Consider asking Apache to register Apache Beam trademark

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-998:
--
Component/s: (was: process)
 project-management

> Consider asking Apache to register Apache Beam trademark
> 
>
> Key: BEAM-998
> URL: https://issues.apache.org/jira/browse/BEAM-998
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
> Fix For: Not applicable
>
>
> "Registered Trademarks If a PMC would like to request legal registration of 
> their project's trademarks, please registering their marks, please follow the 
> REGREQUEST instructions."
> http://www.apache.org/foundation/marks/pmcs#other
> The link to REGREQUEST: 
> http://www.apache.org/foundation/marks/register#register



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1024) upgrade to protobuf-3.1.0

[Rafael Fernandez (JIRA)]

Rafael Fernandez created BEAM-1024:
--

 Summary: upgrade to protobuf-3.1.0
 Key: BEAM-1024
 URL: https://issues.apache.org/jira/browse/BEAM-1024
 Project: Beam
  Issue Type: Wish
Reporter: Rafael Fernandez


The SDK currently uses protobuf 3.0.0-beta-1. There are critical improvements 
to the library since (such as JsonFormat.parser().ignoringUnknownFields()).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1401: Remove dependency on findbugs annotations...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1401

Remove dependency on findbugs annotations artifact

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The findbugs annotations artifact has an incompatible license and cannot
be included within Beam. We were previously referencing it for the
@SuppressFBWarning annotation for inline FindBugs suppression.

This change moves inline suppressions out to the existing
findbugs-filter.xml file. While not ideal as it removes the suppressions
from the offending context, it allows us to drop our dependency on the
incompatible artifact.

We are also referencing the @Nullable attribute from findbugs. This is
now source from findbugs jsr305, which does have a compatible license.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam 
findbugs-convert-suppressions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1401


commit 850fa2c0010324447e8ebf9c31ebfec4a1974db5
Author: Scott Wegner 
Date:   2016-11-21T21:07:46Z

Remove dependency on findbugs annotations artifact

The findbugs annotations artifact has an incompatible license and cannot
be included within Beam. We were previously referencing it for the
@SuppressFBWarning annotation for inline FindBugs suppression.

This change moves inline suppressions out to the existing
findbugs-filter.xml file. While not ideal as it removes the suppressions
from the offending context, it allows us to drop our dependency on the
incompatible artifact.

We are also referencing the @Nullable attribute from findbugs. This is
now source from findbugs jsr305, which does have a compatible license.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1022) TableNamespace should not use Java object equality when comparing windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684660#comment-15684660
 ] 

ASF GitHub Bot commented on BEAM-1022:
--

GitHub user reuvenlax opened a pull request:

https://github.com/apache/incubator-beam/pull/1400

[BEAM-1022] Add test coverage for BigQuery.Write streaming

There was no unit test coverage for BigQuery.Write in the case of unbounded 
input data. Add such unit-test coverage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam bigquery_testing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1400


commit f1c5e5f5b9d842adc0774b3941adb56ca1605ab1
Author: Reuven Lax 
Date:   2016-11-17T18:57:41Z

Add dead-letter support to BigQueryIO

commit 2e44b3ae9298edecd8269ad52aab23644bf38673
Author: Reuven Lax 
Date:   2016-11-19T22:28:10Z

Add dead-letter support to BigQueryIO

commit fdc4794451fb0dcb8a13980fbdf5e64b58129045
Author: Reuven Lax 
Date:   2016-11-19T22:49:46Z

Strip out non-testing changes.

commit 37f33d4b910edbf87cd3b37a9a5efc8cf9775e39
Author: Reuven Lax 
Date:   2016-11-19T23:09:10Z

Remove dead code.

commit 835da28f8168b6d0d599bab3b55563f000313cb9
Author: Reuven Lax 
Date:   2016-11-21T20:30:29Z

Add testing coverage for BigQuery streaming writes.




> TableNamespace should not use Java object equality when comparing windows
> -
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.3.0-incubating
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1400: [BEAM-1022] Add test coverage for BigQuer...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/incubator-beam/pull/1400

[BEAM-1022] Add test coverage for BigQuery.Write streaming

There was no unit test coverage for BigQuery.Write in the case of unbounded 
input data. Add such unit-test coverage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam bigquery_testing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1400


commit f1c5e5f5b9d842adc0774b3941adb56ca1605ab1
Author: Reuven Lax 
Date:   2016-11-17T18:57:41Z

Add dead-letter support to BigQueryIO

commit 2e44b3ae9298edecd8269ad52aab23644bf38673
Author: Reuven Lax 
Date:   2016-11-19T22:28:10Z

Add dead-letter support to BigQueryIO

commit fdc4794451fb0dcb8a13980fbdf5e64b58129045
Author: Reuven Lax 
Date:   2016-11-19T22:49:46Z

Strip out non-testing changes.

commit 37f33d4b910edbf87cd3b37a9a5efc8cf9775e39
Author: Reuven Lax 
Date:   2016-11-19T23:09:10Z

Remove dead code.

commit 835da28f8168b6d0d599bab3b55563f000313cb9
Author: Reuven Lax 
Date:   2016-11-21T20:30:29Z

Add testing coverage for BigQuery streaming writes.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1022) TableNamespace should not use Java object equality when comparing windows

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-1022:


 Summary: TableNamespace should not use Java object equality when 
comparing windows
 Key: BEAM-1022
 URL: https://issues.apache.org/jira/browse/BEAM-1022
 Project: Beam
  Issue Type: Bug
  Components: beam-model
Affects Versions: 0.3.0-incubating
Reporter: Reuven Lax
Assignee: Thomas Groh






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1399: [BEAM-25] Direct runner state

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1399

[BEAM-25] Direct runner state

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DirectRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1399


commit 9d35e9f715f2c76ae54f3b304e41d1dbdf8c0f92
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:01Z

Reject stateful DoFn in ApexRunner

commit aab81549a70d1ac3f6580f66759566f673c243b8
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:13Z

Reject stateful DoFn in SparkRunner

commit 0c2aa666ecaf7def9fa28c6dc81b90c3ad0dc873
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:28Z

Reject stateful DoFn in FlinkRunner

commit ec12dd3102d0a76e5b4cf7ad0a503b746e165d01
Author: Kenneth Knowles 
Date:   2016-11-16T05:35:03Z

Reject stateful DoFn in DataflowRunner

commit a66e71fb92734f6d423ce9ee012812322992ba40
Author: Kenneth Knowles 
Date:   2016-11-10T05:10:51Z

Add State parameter support to SimpleDoFnRunner

commit 8a058b14ab4084ee122d396f96ef2b76d86d3368
Author: Kenneth Knowles 
Date:   2016-11-21T19:34:45Z

Remove overspecified type in ParDoEvaluator

commit 677155a6148ea383065c1792db49e283e466cef1
Author: Kenneth Knowles 
Date:   2016-11-10T05:59:15Z

Add support for Stateful ParDo in the Direct runner

This adds overrides and new evaluators to ensure that
state is accessed in a single-threaded manner per key
and is cleaned up when a window expires.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684574#comment-15684574
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1399

[BEAM-25] Direct runner state

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam DirectRunner-state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1399


commit 9d35e9f715f2c76ae54f3b304e41d1dbdf8c0f92
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:01Z

Reject stateful DoFn in ApexRunner

commit aab81549a70d1ac3f6580f66759566f673c243b8
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:13Z

Reject stateful DoFn in SparkRunner

commit 0c2aa666ecaf7def9fa28c6dc81b90c3ad0dc873
Author: Kenneth Knowles 
Date:   2016-11-16T05:33:28Z

Reject stateful DoFn in FlinkRunner

commit ec12dd3102d0a76e5b4cf7ad0a503b746e165d01
Author: Kenneth Knowles 
Date:   2016-11-16T05:35:03Z

Reject stateful DoFn in DataflowRunner

commit a66e71fb92734f6d423ce9ee012812322992ba40
Author: Kenneth Knowles 
Date:   2016-11-10T05:10:51Z

Add State parameter support to SimpleDoFnRunner

commit 8a058b14ab4084ee122d396f96ef2b76d86d3368
Author: Kenneth Knowles 
Date:   2016-11-21T19:34:45Z

Remove overspecified type in ParDoEvaluator

commit 677155a6148ea383065c1792db49e283e466cef1
Author: Kenneth Knowles 
Date:   2016-11-10T05:59:15Z

Add support for Stateful ParDo in the Direct runner

This adds overrides and new evaluators to ensure that
state is accessed in a single-threaded manner per key
and is cleaned up when a window expires.




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1398: [BEAM-1021] Datatoreio Read Ptransform in...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1398

[BEAM-1021] Datatoreio Read Ptransform in Python

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_datastoreio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1398


commit a53197d01aa1039eea1d13a541288ab1e251f13e
Author: Vikas Kedigehalli 
Date:   2016-11-16T00:41:24Z

datastoreio

commit 45adf557172c210305c62218504123b267eb44fa
Author: Vikas Kedigehalli 
Date:   2016-11-18T21:25:21Z

datatsoreio test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1021) DatastoreIO for python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684570#comment-15684570
 ] 

ASF GitHub Bot commented on BEAM-1021:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1398

[BEAM-1021] Datatoreio Read Ptransform in Python

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_datastoreio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1398


commit a53197d01aa1039eea1d13a541288ab1e251f13e
Author: Vikas Kedigehalli 
Date:   2016-11-16T00:41:24Z

datastoreio

commit 45adf557172c210305c62218504123b267eb44fa
Author: Vikas Kedigehalli 
Date:   2016-11-18T21:25:21Z

datatsoreio test




> DatastoreIO for python
> --
>
> Key: BEAM-1021
> URL: https://issues.apache.org/jira/browse/BEAM-1021
> Project: Beam
>  Issue Type: New Feature
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1021) DatastoreIO for python

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-1021:
---

 Summary: DatastoreIO for python
 Key: BEAM-1021
 URL: https://issues.apache.org/jira/browse/BEAM-1021
 Project: Beam
  Issue Type: New Feature
Reporter: Vikas Kedigehalli
Assignee: Vikas Kedigehalli






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-992) Python SDK build broken due to datastore dependency

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli resolved BEAM-992.

   Resolution: Fixed
Fix Version/s: Not applicable

> Python SDK build broken due to datastore dependency
> ---
>
> Key: BEAM-992
> URL: https://issues.apache.org/jira/browse/BEAM-992
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: Not applicable
>
>
> https://github.com/apache/incubator-beam/pull/1310 introduced the 
> googledatastore dependency, which then depends on pyOpenSSL. pyOpenSSL pip 
> install runs into issues of not having c modules like libffi. Working with 
> datastore team to get this fixed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-943) Implement Datastore query splitter for python

[Vikas Kedigehalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Kedigehalli resolved BEAM-943.

   Resolution: Fixed
Fix Version/s: Not applicable

> Implement Datastore query splitter for python
> -
>
> Key: BEAM-943
> URL: https://issues.apache.org/jira/browse/BEAM-943
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Tenghuan He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tenghuan He updated BEAM-1020:
--
External issue URL: https://github.com/apache/incubator-beam/pull/1397

> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary 
> directory, while Windows does not support wildcard * in the file path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1394: BEAM-1018: updated getEstimatedSizeBytes ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1394


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1018] Updated getEstimatedSizeBytes() to use Number.longValue() in MongoDbIO

[jbonofre]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 875631f07 -> c6b86f60f


[BEAM-1018] Updated getEstimatedSizeBytes() to use Number.longValue() in 
MongoDbIO


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a761b0c1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a761b0c1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a761b0c1

Branch: refs/heads/master
Commit: a761b0c154b67daada2968c24c7de9c4afe13884
Parents: 875631f
Author: Sandeep Parikh 
Authored: Mon Nov 21 10:05:36 2016 -0600
Committer: Jean-Baptiste Onofré 
Committed: Mon Nov 21 20:37:11 2016 +0100

--
 .../src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a761b0c1/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
--
diff --git 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
index 71c017d..cd5cdaf 100644
--- 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
+++ 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
@@ -224,7 +224,7 @@ public class MongoDbIO {
   BasicDBObject stat = new BasicDBObject();
   stat.append("collStats", spec.collection());
   Document stats = mongoDatabase.runCommand(stat);
-  return Long.parseLong(stats.get("size").toString());
+  return stats.get("size", Number.class).longValue();
 }
 
 @Override

[jira] [Updated] (BEAM-1018) getEstimatedSizeBytes fails with large MongoDB collection sizes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-1018:
---
Component/s: sdk-java-extensions

> getEstimatedSizeBytes fails with large MongoDB collection sizes
> ---
>
> Key: BEAM-1018
> URL: https://issues.apache.org/jira/browse/BEAM-1018
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.4.0-incubating
>Reporter: Sandeep Parikh
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> When running against large collections sizes (20M+ documents), MongoDbIO 
> fails to correctly parse the {{size}} element in the document returned by 
> {code:javascript}
> db.runCommand({'collStats', 'collectionName'})
> {code}
> As the collection sizes grow larger, the returned value is in scientific 
> notation which cannot be parsed as a Long. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1018) getEstimatedSizeBytes fails with large MongoDB collection sizes

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1018.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> getEstimatedSizeBytes fails with large MongoDB collection sizes
> ---
>
> Key: BEAM-1018
> URL: https://issues.apache.org/jira/browse/BEAM-1018
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.4.0-incubating
>Reporter: Sandeep Parikh
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> When running against large collections sizes (20M+ documents), MongoDbIO 
> fails to correctly parse the {{size}} element in the document returned by 
> {code:javascript}
> db.runCommand({'collStats', 'collectionName'})
> {code}
> As the collection sizes grow larger, the returned value is in scientific 
> notation which cannot be parsed as a Long. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-1018] This closes #1394

[jbonofre]

[BEAM-1018] This closes #1394


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c6b86f60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c6b86f60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c6b86f60

Branch: refs/heads/master
Commit: c6b86f60f7b159fd188bfb6e5bf742a9b65080a0
Parents: 875631f a761b0c
Author: Jean-Baptiste Onofré 
Authored: Mon Nov 21 20:52:10 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Mon Nov 21 20:52:10 2016 +0100

--
 .../src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684531#comment-15684531
 ] 

ASF GitHub Bot commented on BEAM-1020:
--

GitHub user tenghuanhe opened a pull request:

https://github.com/apache/incubator-beam/pull/1397

[BEAM-1020] Fix removing temporary directory bug on Windows operating system

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Failing to remove temporary directory is due to the use of wildcard * when 
matching all files in the temporary directory, while Windows does not support 
wildcard * in the file path.
`Collection matches = factory.match(factory.resolve(tempDirectory, 
"*"));`
I think It'is not very appropriate to use wildcards when removing all files 
in the temporary directory because there is no matching pattern need, which may 
cause incompatibility problems on Windows operating system.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tenghuanhe/incubator-beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1397


commit 26fcb3a962df9d55c8b98c510177b799d4b94cf4
Author: Tenghuan He 
Date:   2016-11-21T19:17:19Z

Fix removing temporary directory on Windows operating system




> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary 
> directory, while Windows does not support wildcard * in the file path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1397: [BEAM-1020] Fix removing temporary direct...

[tenghuanhe]

GitHub user tenghuanhe opened a pull request:

https://github.com/apache/incubator-beam/pull/1397

[BEAM-1020] Fix removing temporary directory bug on Windows operating system

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Failing to remove temporary directory is due to the use of wildcard * when 
matching all files in the temporary directory, while Windows does not support 
wildcard * in the file path.
`Collection matches = factory.match(factory.resolve(tempDirectory, 
"*"));`
I think It'is not very appropriate to use wildcards when removing all files 
in the temporary directory because there is no matching pattern need, which may 
cause incompatibility problems on Windows operating system.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tenghuanhe/incubator-beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1397


commit 26fcb3a962df9d55c8b98c510177b799d4b94cf4
Author: Tenghuan He 
Date:   2016-11-21T19:17:19Z

Fix removing temporary directory on Windows operating system




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1392: [BEAM-1010] A few improvements to Apache ...

[katsiapis]

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/1392


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1010) Custom FileSinks should respect AUTO compression

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684519#comment-15684519
 ] 

ASF GitHub Bot commented on BEAM-1010:
--

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/1392


> Custom FileSinks should respect AUTO compression
> 
>
> Key: BEAM-1010
> URL: https://issues.apache.org/jira/browse/BEAM-1010
> Project: Beam
>  Issue Type: Bug
>Reporter: Konstantinos Katsiapis
>Assignee: Konstantinos Katsiapis
>
> Currently AUTO compression is respected by Native FileSinks but not Custom 
> FileSinks.
> Also, it would be good for Beam's _CompresedFile to support usage with Python 
> "with" clause (ie implement __enter__ and __exit__).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/4] incubator-beam git commit: [BEAM-1010] A few improvements to Apache Beam Python's FileIO.

[lcwik]

[BEAM-1010] A few improvements to Apache Beam Python's FileIO.

This closes #1392


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8e88c7b0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8e88c7b0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8e88c7b0

Branch: refs/heads/python-sdk
Commit: 8e88c7b035e76c6e15d03a79f9751c6e53786859
Parents: c1440f7 6aa50c1
Author: Luke Cwik 
Authored: Mon Nov 21 11:31:42 2016 -0800
Committer: Luke Cwik 
Committed: Mon Nov 21 11:31:42 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py  | 14 +++-
 sdks/python/apache_beam/io/fileio_test.py | 48 +++---
 sdks/python/apache_beam/io/textio.py  |  6 +---
 sdks/python/apache_beam/io/textio_test.py | 26 ++
 4 files changed, 84 insertions(+), 10 deletions(-)
--

[2/4] incubator-beam git commit: Handling the 'collision' case for UIDs and also augmenting comments.

[lcwik]

Handling the 'collision' case for UIDs and also augmenting comments.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f9c9865
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9f9c9865
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9f9c9865

Branch: refs/heads/python-sdk
Commit: 9f9c986500f769ebeb0baaf32f1a86334a36eec5
Parents: e85f67a
Author: Gus Katsiapis 
Authored: Sat Nov 19 15:22:47 2016 -0800
Committer: Luke Cwik 
Committed: Mon Nov 21 11:29:07 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f9c9865/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 1dcd622..cb7f25c 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -862,7 +862,11 @@ class FileSink(iobase.Sink):
 
   def open_writer(self, init_result, uid):
 # A proper suffix is needed for AUTO compression detection.
-suffix = os.path.basename(self.file_path_prefix) + self.file_name_suffix
+# We also ensure there will be no collisions with uid and a
+# (possibly unsharded) file_path_prefix and a (possibly empty)
+# file_name_suffix.
+suffix = (
+  '.' + os.path.basename(self.file_path_prefix) + self.file_name_suffix)
 return FileSinkWriter(self, os.path.join(init_result, uid) + suffix)
 
   def finalize_write(self, init_result, writer_results):

[3/4] incubator-beam git commit: A few improvements to Apache Beam Python's FileIO.

[lcwik]

A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e85f67a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e85f67a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e85f67a1

Branch: refs/heads/python-sdk
Commit: e85f67a1a467a26259a849bd20c42e89f165828e
Parents: c1440f7
Author: Gus Katsiapis 
Authored: Fri Nov 18 18:31:20 2016 -0800
Committer: Luke Cwik 
Committed: Mon Nov 21 11:29:07 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py  | 10 +-
 sdks/python/apache_beam/io/fileio_test.py | 48 +++---
 sdks/python/apache_beam/io/textio.py  |  6 +---
 sdks/python/apache_beam/io/textio_test.py | 26 ++
 4 files changed, 80 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e85f67a1/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 4d0eea6..1dcd622 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -749,6 +749,12 @@ class _CompressedFile(object):
   def seekable(self):
 return False
 
+  def __enter__(self):
+return self
+
+  def __exit__(self, exception_type, exception_value, traceback):
+self.close()
+
 
 class FileSink(iobase.Sink):
   """A sink to a GCS or local files.
@@ -855,7 +861,9 @@ class FileSink(iobase.Sink):
 return tmp_dir
 
   def open_writer(self, init_result, uid):
-return FileSinkWriter(self, os.path.join(init_result, uid))
+# A proper suffix is needed for AUTO compression detection.
+suffix = os.path.basename(self.file_path_prefix) + self.file_name_suffix
+return FileSinkWriter(self, os.path.join(init_result, uid) + suffix)
 
   def finalize_write(self, init_result, writer_results):
 writer_results = sorted(writer_results)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e85f67a1/sdks/python/apache_beam/io/fileio_test.py
--
diff --git a/sdks/python/apache_beam/io/fileio_test.py 
b/sdks/python/apache_beam/io/fileio_test.py
index 9d1e424..098ace1 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -38,10 +38,7 @@ from apache_beam.transforms.display import DisplayData
 from apache_beam.transforms.display_test import DisplayDataItemMatcher
 
 # TODO: Add tests for file patterns (ie not just individual files) for both
-# uncompressed
-
-# TODO: Update code to not use NamedTemporaryFile (or to use it in a way that
-# doesn't violate its assumptions).
+# compressed and uncompressed files.
 
 
 class TestTextFileSource(unittest.TestCase):
@@ -721,6 +718,49 @@ class TestNativeTextFileSink(unittest.TestCase):
 with bz2.BZ2File(self.path, 'r') as f:
   self.assertEqual(f.read().splitlines(), [])
 
+  def test_write_dataflow(self):
+pipeline = beam.Pipeline('DirectPipelineRunner')
+pcoll = pipeline | beam.core.Create('Create', self.lines)
+pcoll | 'Write' >> beam.Write(fileio.NativeTextFileSink(self.path))  # 
pylint: disable=expression-not-assigned
+pipeline.run()
+
+read_result = []
+for file_name in glob.glob(self.path + '*'):
+  with open(file_name, 'r') as f:
+read_result.extend(f.read().splitlines())
+
+self.assertEqual(read_result, self.lines)
+
+  def test_write_dataflow_auto_compression(self):
+pipeline = beam.Pipeline('DirectPipelineRunner')
+pcoll = pipeline | beam.core.Create('Create', self.lines)
+pcoll | 'Write' >> beam.Write(  # pylint: disable=expression-not-assigned
+fileio.NativeTextFileSink(
+self.path, file_name_suffix='.gz'))
+pipeline.run()
+
+read_result = []
+for file_name in glob.glob(self.path + '*'):
+  with gzip.GzipFile(file_name, 'r') as f:
+read_result.extend(f.read().splitlines())
+
+self.assertEqual(read_result, self.lines)
+
+  def test_write_dataflow_auto_compression_unsharded(self):
+pipeline = beam.Pipeline('DirectPipelineRunner')
+pcoll = pipeline | beam.core.Create('Create', self.lines)
+pcoll | 'Write' >> beam.Write(  # pylint: disable=expression-not-assigned
+fileio.NativeTextFileSink(
+self.path + '.gz', shard_name_template=''))
+pipeline.run()
+
+read_result = []
+for file_name in glob.glob(self.path + '*'):
+  with 

[1/4] incubator-beam git commit: Fixing lynt warnings related to indentation.

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk c1440f7aa -> 8e88c7b03


Fixing lynt warnings related to indentation.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6aa50c1d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6aa50c1d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6aa50c1d

Branch: refs/heads/python-sdk
Commit: 6aa50c1d5249ac70c37196a9957874aaf26dd54d
Parents: 9f9c986
Author: Gus Katsiapis 
Authored: Sun Nov 20 12:33:16 2016 -0800
Committer: Luke Cwik 
Committed: Mon Nov 21 11:29:07 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6aa50c1d/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index cb7f25c..30044c3 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -866,7 +866,7 @@ class FileSink(iobase.Sink):
 # (possibly unsharded) file_path_prefix and a (possibly empty)
 # file_name_suffix.
 suffix = (
-  '.' + os.path.basename(self.file_path_prefix) + self.file_name_suffix)
+'.' + os.path.basename(self.file_path_prefix) + self.file_name_suffix)
 return FileSinkWriter(self, os.path.join(init_result, uid) + suffix)
 
   def finalize_write(self, init_result, writer_results):

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Tenghuan He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tenghuan He updated BEAM-1020:
--
Description: 
 Beam fails to remove temporary directory after completing pipelines and 
sinking output results on Windows operating system and throws the following 
exception
{noformat} 
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
at 
org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
at 
org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
{noformat}

This is due to the use of wildcard * when matching all files in the temporary 
directory, while Windows does not support wildcard * in the file path.

  was:
 Beam fails to remove temporary directory after completing pipelines and 
sinking output results on Windows operating system and throws the following 
exception
{noformat} 
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
at 
org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
at 
org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
{noformat}

This is due to the use of wildcards * when matching all files in the temporary 
directory, while Windows does not support wildcard * in the file path.


> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam 

[jira] [Commented] (BEAM-277) Add Transforms Section

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684502#comment-15684502
 ] 

ASF GitHub Bot commented on BEAM-277:
-

GitHub user melap opened a pull request:

https://github.com/apache/incubator-beam-site/pull/91

[BEAM-277] Add transforms section to the programming guide

Adds the remaining content for the transforms section: Combine, Flatten, 
Partition, Side inputs and side outputs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site progguide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/91.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #91


commit 512c30356ba0bdaca49625613ffe15002fbe2af1
Author: melissa 
Date:   2016-11-21T19:22:04Z

[BEAM-277] Add transforms section to the programming guide




> Add Transforms Section
> --
>
> Key: BEAM-277
> URL: https://issues.apache.org/jira/browse/BEAM-277
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Melissa Pashniak
>
> Document general transforms usage and ParDo usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1396: Add fields retry

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1396

Add fields retry

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Allow changing all the fields from FuzzedExponentialIntervals in the retry 
decorator

R: @chamikaramj PTAL 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam BEAM-add-fields-retry

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1396


commit 68dc37be6dc0f59f6c36a9bb52025d921fb01229
Author: Sourabh Bajaj 
Date:   2016-11-21T19:28:56Z

Add missing fields to the retry decorator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #91: [BEAM-277] Add transforms section to t...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/incubator-beam-site/pull/91

[BEAM-277] Add transforms section to the programming guide

Adds the remaining content for the transforms section: Combine, Flatten, 
Partition, Side inputs and side outputs


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site progguide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/91.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #91


commit 512c30356ba0bdaca49625613ffe15002fbe2af1
Author: melissa 
Date:   2016-11-21T19:22:04Z

[BEAM-277] Add transforms section to the programming guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Tenghuan He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tenghuan He updated BEAM-1020:
--
Remaining Estimate: 12h  (was: 0.5h)
 Original Estimate: 12h  (was: 0.5h)

> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcards * when matching all files in the 
> temporary directory, while Windows does not support wildcard * in the file 
> path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1395: Fixing error with PipelineOptions Display...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1395

Fixing error with PipelineOptions DisplayData of lists

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam dd-options-list

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1395


commit 942d0c1cb775876b1a6bd91959a60aa9ae576aa7
Author: Pablo 
Date:   2016-11-21T19:15:37Z

Fixing error with PipelineOptions DisplayData of lists




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Tenghuan He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tenghuan He updated BEAM-1020:
--
Description: 
 Beam fails to remove temporary directory after completing pipelines and 
sinking output results on Windows operating system and throws the following 
exception
{noformat} 
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
at 
org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
at 
org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
at 
org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
{noformat}

This is due to the use of wildcards * when matching all files in the temporary 
directory, while Windows does not support wildcard * in the file path.

  was: 


> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcards * when matching all files in the 
> temporary directory, while Windows does not 

[jira] [Created] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Tenghuan He (JIRA)]

Tenghuan He created BEAM-1020:
-

 Summary: FileBasedSink fails to remove temporary directory on 
Windows operating system
 Key: BEAM-1020
 URL: https://issues.apache.org/jira/browse/BEAM-1020
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: Not applicable
Reporter: Tenghuan He
Assignee: Davor Bonaci
 Fix For: Not applicable


 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Neelesh Srinivas Salian (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684253#comment-15684253
 ] 

Neelesh Srinivas Salian commented on BEAM-301:
--

This is helpful [~takidau]. Thank you. Allow me to go through them.
I'm travelling at the moment so I should have something up by next week.


> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Neelesh Srinivas Salian
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Tyler Akidau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684238#comment-15684238
 ] 

Tyler Akidau commented on BEAM-301:
---

You may already be aware of these, Neelesh, but I'd like to reference some docs 
here that are relevant to this effort here in case you aren't (or for others). 
I've been chatting on and off w/ the Calcite folks regarding SQL this year, in 
particular regarding adding robust streaming semantics into the Calcite 
algebra. We don't need any additions to get something basic, similar to what 
Apache Flink has (and there's plenty of work around just defining how schemas 
are mapped, how SQL is expressed in the pipeline, etc). But we will want them 
some day.

At any rate, the relevant docs so far that I'm aware of:

1. Streaming SQL - Julian Hyde’s initial manifesto on integrating streaming in 
Calcite SQL: https://calcite.apache.org/docs/stream.html

2. Triggers in Streaming SQL - My first attempt at tackling out-of-order 
processing semantics for streaming SQL: 
https://docs.google.com/document/d/1tSey4CeTrbb4VjWvtSA78OcU6BERXXDZ3t0HzSLij9Q/edit

3. Streams, joins and temporal tables - Julian’s proposal for tables which are 
parameterized on a timestamp value: 
https://docs.google.com/document/d/1RvnLEEQK92axdAaZ9XIU5szpkbGqFMBtzYiIY4dHe0Q/edit

4. Relational Queries on Data Streams - Apache Flink proposal for standard SQL 
based queries on data streams: 
https://docs.google.com/document/d/1qVVt_16kdaZQ8RTfA_f4konQPW4tnl8THw6rzGUdaqU/edit

I've been wanting to write up one more doc trying to reconcile #2 with #3, but 
haven't gotten around to finishing it yet.

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Neelesh Srinivas Salian
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-999) DirectRunner flake related to splitting

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-999.
--
Resolution: Fixed

> DirectRunner flake related to splitting
> ---
>
> Key: BEAM-999
> URL: https://issues.apache.org/jira/browse/BEAM-999
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1838/
> {code}
> Error Message
> Expected: a value greater than <1>
>  but: <1> was equal to <1>
> Stacktrace
> java.lang.AssertionError: 
> Expected: a value greater than <1>
>  but: <1> was equal to <1>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:865)
>   at org.junit.Assert.assertThat(Assert.java:832)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactoryTest.boundedSourceEvaluatorProducesDynamicSplits(BoundedReadEvaluatorFactoryTest.java:183)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:161)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)