Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #738

[Apache Jenkins Server]

See 


--
[...truncated 541.21 KB...]
2017-03-11T06:04:14.753 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar
2017-03-11T06:04:14.765 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
 (9 KB at 6.1 KB/sec)
2017-03-11T06:04:14.765 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.0/hadoop-auth-2.6.0.jar
2017-03-11T06:04:14.767 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar
 (528 KB at 373.3 KB/sec)
2017-03-11T06:04:14.767 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar
2017-03-11T06:04:14.793 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar
 (186 KB at 129.3 KB/sec)
2017-03-11T06:04:14.793 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar
2017-03-11T06:04:14.794 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar
 (292 KB at 202.8 KB/sec)
2017-03-11T06:04:14.794 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar
2017-03-11T06:04:14.798 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.6.0/hadoop-auth-2.6.0.jar
 (66 KB at 45.5 KB/sec)
2017-03-11T06:04:14.798 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar
2017-03-11T06:04:14.824 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar
 (17 KB at 11.0 KB/sec)
2017-03-11T06:04:14.824 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
2017-03-11T06:04:14.825 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar
 (44 KB at 29.8 KB/sec)
2017-03-11T06:04:14.825 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar
2017-03-11T06:04:14.831 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar
 (79 KB at 52.9 KB/sec)
2017-03-11T06:04:14.831 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
2017-03-11T06:04:14.876 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
 (68 KB at 44.2 KB/sec)
2017-03-11T06:04:14.876 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar
2017-03-11T06:04:14.877 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
 (181 KB at 118.9 KB/sec)
2017-03-11T06:04:14.877 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar 
(182 KB at 119.2 KB/sec)
2017-03-11T06:04:14.877 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar
2017-03-11T06:04:14.877 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar
2017-03-11T06:04:14.880 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar
 (676 KB at 442.8 KB/sec)
2017-03-11T06:04:14.880 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar
2017-03-11T06:04:14.916 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar
 (31 KB at 19.5 KB/sec)
2017-03-11T06:04:14.916 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
2017-03-11T06:04:14.937 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar
 (243 KB at 153.2 KB/sec)
2017-03-11T06:04:14.937 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
2017-03-11T06:04:14.960 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
 (129 KB at 79.9 KB/sec)
2017-03-11T06:04:14.960 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2879

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-789) Review python sdk dependencies

[Tibor Kiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906106#comment-15906106
 ] 

Tibor Kiss commented on BEAM-789:
-

[~altay]: This issue has been fixed. For some reason I'm not able to resolve 
this issue. Could you please close it? Thanks!

> Review python sdk dependencies
> --
>
> Key: BEAM-789
> URL: https://issues.apache.org/jira/browse/BEAM-789
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Tibor Kiss
>Priority: Minor
>
> Review the existing dependencies for the python sdk. Are they still all 
> required? (e.g. protorpc might not be a required dependency any more.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1691) Dynamic properties supported in PipelineOptions

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1691:


 Summary: Dynamic properties supported in PipelineOptions
 Key: BEAM-1691
 URL: https://issues.apache.org/jira/browse/BEAM-1691
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Xu Mingmin
Assignee: Davor Bonaci


Usually the two lines to create a new Beam pipeline are:
{code}
Options options = 
PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
Pipeline pipeline = Pipeline.create(options);
{code} 

As each runner has its own PipelineOptions, one piece of code is hardly to run 
on different runners without code change, --as least Options needs to be 
updated.

Dynamic property could be a choice, similar as
{code}
-D property1=value1 -D property2=value2 ...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1682) Enable all GCP unit tests in Python-SDK

[Tibor Kiss (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tibor Kiss resolved BEAM-1682.
--
   Resolution: Not A Bug
Fix Version/s: 0.6.0

Works as designed: I missed the GCP profile in tox. Thanks for [~sb2nov] for 
reviewing & pointing me to the profile.

> Enable all GCP unit tests in Python-SDK
> ---
>
> Key: BEAM-1682
> URL: https://issues.apache.org/jira/browse/BEAM-1682
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
> Fix For: 0.6.0
>
>
> Some of the GCP unit tests are not running due to missing 'apitools' 
> dependency in the test environment:
> {noformat}
> test_remove_objects_from_args_nothing_to_remove 
> (apache_beam.internal.util_test.UtilTest) ... ok
> test_DatastoreWriteFn_with_batch_size_exact_multiple 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_emtpy_batch 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_multiple_batches 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_one_batch 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_SplitQueryFn_with_exception 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1682) Enable all GCP unit tests in Python-SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906099#comment-15906099
 ] 

ASF GitHub Bot commented on BEAM-1682:
--

Github user tibkiss closed the pull request at:

https://github.com/apache/beam/pull/2218


> Enable all GCP unit tests in Python-SDK
> ---
>
> Key: BEAM-1682
> URL: https://issues.apache.org/jira/browse/BEAM-1682
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
>
> Some of the GCP unit tests are not running due to missing 'apitools' 
> dependency in the test environment:
> {noformat}
> test_remove_objects_from_args_nothing_to_remove 
> (apache_beam.internal.util_test.UtilTest) ... ok
> test_DatastoreWriteFn_with_batch_size_exact_multiple 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_emtpy_batch 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_multiple_batches 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_DatastoreWriteFn_with_one_batch 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest) ... SKIP: 
> GCP dependencies are not installed
> test_SplitQueryFn_with_exception 
> (apache_beam.io.gcp.datastore.v1.datastoreio_test.DatastoreioTest)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2218: [BEAM-1682] Enable all GCP unit tests in Python-SDK

[tibkiss]

Github user tibkiss closed the pull request at:

https://github.com/apache/beam/pull/2218


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1688) Getting listener timeout after waiting for [10000] ms from ElasticsearchIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906097#comment-15906097
 ] 

Jean-Baptiste Onofré commented on BEAM-1688:


Current {{ElasticsearchIO}} doesn't support Elasticsearch 5.x (it's 
Elasticsearch 2.x only). We are working on a new IO to fully support 
Elasticsearch 5 (see https://issues.apache.org/jira/browse/BEAM-1637).

> Getting listener timeout after waiting for [1] ms from ElasticsearchIO
> --
>
> Key: BEAM-1688
> URL: https://issues.apache.org/jira/browse/BEAM-1688
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.5.0
> Environment: Linux,  ElasticSearch 5.2.1
>Reporter: Sathish Jayaraman
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> I am trying to test a simple Apache Beam code with source as Elasticsearch. I 
> modified Beam's MinimalWordCount example to include source as Elasticsearch 
> instead of TextIO. 
> When I run the code,  I get below error, 
> An exception occured while executing the Java class. null: 
> InvocationTargetException: java.io.IOException: listener timeout after 
> waiting for [1] ms
> I debugged ElasticsearchIO.java and could see everything works fine, the 
> Elasicsearch client is built & data from index is being retrieved by the 
> code. But ParDo function after read transform does not get executed at all. 
> The Elasticsearch client keeps waiting and finally gets a timeout error.
> Any idea about the issue? 
> Have raised issue here too 
> http://stackoverflow.com/questions/42720857/getting-listener-timeout-after-waiting-for-1-ms-from-elasticsearchio



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1678) Create MemcachedIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906093#comment-15906093
 ] 

Ismaël Mejía commented on BEAM-1678:


Yes, I know all IOs are PTransforms, and this is even more explicitly with the 
SDF API ;) I clarified that this was an IO because for me it was so obvious 
that such transform made sense, that I thought that somehow the question you 
made was because the name 'cache' was deceiving and people could think this was 
a different transform.

The signature you propose is more or less what I had in mind, I was thinking 
about a more user friendly PTransform>, PDone> like 
KafkaIO does, but maybe I can support both I have to figure this out.

Now the open question after this discussion is authorization because 'vanilla' 
Memcached uses no auth by default, and you can configure eventually SASL, but I 
am pretty sure every other provider is a different beast. I will have to 
investigate more, for the moment maybe I will focus first on the basic 
Memcached store.


> Create MemcachedIO
> --
>
> Key: BEAM-1678
> URL: https://issues.apache.org/jira/browse/BEAM-1678
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #1209

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2877

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1208

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1690) BigQueryTornadoesIT failing

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906034#comment-15906034
 ] 

Kenneth Knowles commented on BEAM-1690:
---

The most troubling thing to me is that it was only caught by this example IT.

> BigQueryTornadoesIT failing
> ---
>
> Key: BEAM-1690
> URL: https://issues.apache.org/jira/browse/BEAM-1690
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-dataflow, testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> [Since build 
> 2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
> (with one aberration) BigQueryTornadoesIT has been failing reliably on the 
> Dataflow runner. The changes around that time do not seem related.
> Example build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1690) BigQueryTornadoesIT failing

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1690:
--
Description: 
[Since build 
2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
(with one aberration) BigQueryTornadoesIT has been failing reliably on the 
Dataflow runner. The changes around that time do not seem related.

Example build: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/


  was:
[Since build 
2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
(with one aberration) BigQueryTornadoesIT has been failing reliably on the 
Dataflow runner. The changes around that time do not seem related.

Example build: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/

Brief summary of failure:

{code}

{code}

And an extract of the log that might be relevant:


> BigQueryTornadoesIT failing
> ---
>
> Key: BEAM-1690
> URL: https://issues.apache.org/jira/browse/BEAM-1690
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-dataflow, testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> [Since build 
> 2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
> (with one aberration) BigQueryTornadoesIT has been failing reliably on the 
> Dataflow runner. The changes around that time do not seem related.
> Example build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2226: [BEAM-1690] Revert BigQueryIO bit of 'Make all uses...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2226

[BEAM-1690] Revert BigQueryIO bit of 'Make all uses of CountingOutputStream 
close their resources'

This reverts the portion of commit 3115dbdca1858511e98476b5c79e6cca98782b0b
that touches BigQueryIO, which caused a double close bug, seen here: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam BigQueryTornadoesIT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2226.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2226


commit 598c24e1fc2cf607d9fd01dd07ceca7ea97da84a
Author: Kenneth Knowles 
Date:   2017-03-11T03:01:23Z

Revert BigQueryIO bit of 'Make all uses of CountingOutputStream close their 
resources'

This reverts the portion of commit 3115dbdca1858511e98476b5c79e6cca98782b0b
that touches BigQueryIO, which caused a double close bug.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1690) BigQueryTornadoesIT failing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906033#comment-15906033
 ] 

ASF GitHub Bot commented on BEAM-1690:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2226

[BEAM-1690] Revert BigQueryIO bit of 'Make all uses of CountingOutputStream 
close their resources'

This reverts the portion of commit 3115dbdca1858511e98476b5c79e6cca98782b0b
that touches BigQueryIO, which caused a double close bug, seen here: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam BigQueryTornadoesIT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2226.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2226


commit 598c24e1fc2cf607d9fd01dd07ceca7ea97da84a
Author: Kenneth Knowles 
Date:   2017-03-11T03:01:23Z

Revert BigQueryIO bit of 'Make all uses of CountingOutputStream close their 
resources'

This reverts the portion of commit 3115dbdca1858511e98476b5c79e6cca98782b0b
that touches BigQueryIO, which caused a double close bug.




> BigQueryTornadoesIT failing
> ---
>
> Key: BEAM-1690
> URL: https://issues.apache.org/jira/browse/BEAM-1690
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-dataflow, testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> [Since build 
> 2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
> (with one aberration) BigQueryTornadoesIT has been failing reliably on the 
> Dataflow runner. The changes around that time do not seem related.
> Example build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/
> Brief summary of failure:
> {code}
> {code}
> And an extract of the log that might be relevant:



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1690) BigQueryTornadoesIT failing

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1690:
-

 Summary: BigQueryTornadoesIT failing
 Key: BEAM-1690
 URL: https://issues.apache.org/jira/browse/BEAM-1690
 Project: Beam
  Issue Type: Bug
  Components: examples-java, runner-dataflow, testing
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker


[Since build 
2857|https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/changes] 
(with one aberration) BigQueryTornadoesIT has been failing reliably on the 
Dataflow runner. The changes around that time do not seem related.

Example build: 
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/2857/testReport/junit/org.apache.beam.examples.cookbook/BigQueryTornadoesIT/testE2EBigQueryTornadoes/

Brief summary of failure:

{code}

{code}

And an extract of the log that might be relevant:



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

svn commit: r18679 - /dev/beam/0.6.0/

[altay]

Author: altay
Date: Sat Mar 11 02:59:47 2017
New Revision: 18679

Log:
Update dist to RC2 from RC1.


Added:
dev/beam/0.6.0/apache-beam-0.6.0-python.zip   (with props)
dev/beam/0.6.0/apache-beam-0.6.0-python.zip.asc
dev/beam/0.6.0/apache-beam-0.6.0-python.zip.md5
dev/beam/0.6.0/apache-beam-0.6.0-python.zip.sha1
Removed:
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.asc
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.md5
dev/beam/0.6.0/apache-beam-0.6.0.tar.gz.sha1
Modified:
dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip
dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.asc
dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.md5
dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.sha1

Added: dev/beam/0.6.0/apache-beam-0.6.0-python.zip
==
Binary file - no diff available.

Propchange: dev/beam/0.6.0/apache-beam-0.6.0-python.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/0.6.0/apache-beam-0.6.0-python.zip.asc
==
--- dev/beam/0.6.0/apache-beam-0.6.0-python.zip.asc (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0-python.zip.asc Sat Mar 11 02:59:47 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJYw2Z4AAoJENYucUFglvoA6EcP/14PohGfqKi6XYAhHgHiEnoG
+lYqZzVUICtziB0WVE896kcbaFYsK20VmVy8X1ZrUqy6Si/D3cQXl8axji3CIAaXG
+wS7NyaJoAcNUQvryj0j3Qa064OWmpzCxNPdETMn07JPPQBPyFpeuI4ePgHyJRCMd
+YhJ9otu27RltD+1ooSbqBM+7Xdk/ySEvo8M3H44XX74Cc45cBCwoCSX2xmepfWZ7
+Pq7D3uKI3dzpTCE595mmRTTL+mv8oV+HAxWje2XPBXSLYKCqJY8i933PX9hBuAL/
+G4I3YsFip0ex+x09Q0ToTMgJ3daj3lG6ozjVgC2/vaIB+0ryMPX5+zAM3ljrvjOI
+KTNp1RCKXcHYXo3PdDCp9phD3neShPMDL9Ow+pCWm7u04dYuT7XzGie+5Mv48y7Q
+HC/9mqcPkwMaVaF7NVez58WFs9vOujPZ6TK2KD2EUioXgtbHYyP8i5yVetL3ebBs
+0iBZTdxOqDy6w3vf5UIiJVIaw5KBQw7zKSTR/1RbpxvN2U4+SJ0LsoszXsYCHXG5
+3LrkzlWxfexSeulYJ62xxDHZwTqU5rgdfc+0X6YyExtbrBCK+OmqR232l3MNwIsH
+OSh8J2HFhFDILJ5lewUDtx+szwj7jm21dRwR7Y6FApNa8osl9XE/RYHUA1SUphUR
+sUCpzAkeMLZ+jQPAp56L
+=/Gid
+-END PGP SIGNATURE-

Added: dev/beam/0.6.0/apache-beam-0.6.0-python.zip.md5
==
--- dev/beam/0.6.0/apache-beam-0.6.0-python.zip.md5 (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0-python.zip.md5 Sat Mar 11 02:59:47 2017
@@ -0,0 +1 @@
+7d4170e381ce0e1aa8d11bee2e63d151  apache-beam-0.6.0.zip

Added: dev/beam/0.6.0/apache-beam-0.6.0-python.zip.sha1
==
--- dev/beam/0.6.0/apache-beam-0.6.0-python.zip.sha1 (added)
+++ dev/beam/0.6.0/apache-beam-0.6.0-python.zip.sha1 Sat Mar 11 02:59:47 2017
@@ -0,0 +1 @@
+ccece0ecca10c4c6019cba2ffb0963b187bb89d3  apache-beam-0.6.0.zip

Modified: dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip
==
Binary files - no diff available.

Modified: dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.asc
==
--- dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.asc (original)
+++ dev/beam/0.6.0/apache-beam-0.6.0-source-release.zip.asc Sat Mar 11 02:59:47 
2017
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJYwJ4CAAoJENYucUFglvoAt24P/AgktsCfPFBheHC2WhkpqKxK
-RNTCFtUxzK2CbcjSGfvdLildhToJsrzewXUS17DovZZWxkbankiacU47qjTr1cmj
-G9ONJxMrmKJWxIeR0/ffrNCl0Y9XnIGRGyWSmkE6ZuQR2zuAgD1TZ5ZURETfKkAg
-mCkTCwH81k6uBb58kW83KemWXBSracylTQ0ZYgdNAXwk+lpJjGYHOHdjlDFkmtnQ
-0z+Kc4YqnAHyQe/oCicxqtwy6nsVi/auS4rBoTjC81ef7nxBElWtGhFaX7/gVMZv
-/44XMKrdp19IlTm/y1rb40FUdbtdOomZlWyQjZkjKKlZEhJUpAqIghSISqYMdsCJ
-OYTViE434xcsVarFhzz1HIwznnIoW5NmVSGHhzZ1gI9+9HrJf3dGehGiK/NnHdda
-LkXOltIxFjVQ45lP0n1KbHwPMrX43qQ3LtXbsbR3AIj6xIJPyyxxl6VEJiOJiBFE
-R3oMNOgCqfjT0Y09vYaw7vQw4lppyNLfFUpnNHcAJj82SfStrwAOUmRivHffg+cT
-PfYDKNMLhM/sH4r3OS0WVXqurH/24FJcwQGJZAnd1GXToWhHF6KFdALJ3DVpD+yb
-Cm4zWUlA8sRv3qrkuKaAqYgRKnIOFCzhw794qyZYuWvfuXLFhA0rQxdJsQqm/k3G
-RUfV3Ou1KoIduNf/ZmnL
-=RbwC
+iQIcBAABAgAGBQJYw1elAAoJENYucUFglvoANXMP/2LeHIfeg25VhtGToAEcofAx
+1Tr2g/We+oNVkcrvsrt3x66ZbMIOzZJseW6M99bnQ9h8A0FoCV0Tc2MUrT+WtjVt
+weyV0/JjdSlgvvM1L8DQbpgktEEmTjKgQI+3ZvIoe4ySW+07wkMGte3rSmaL9PW7
+nxlwVhPG8G/IG5TxBPegK45ug1p2cP1vcKL86UD/QTahmaUFphHOEPwyokzIhuG0
+L+5XsM3tCcbv5KBEqb2NkQ5G0aYqg8I4/GjLSV5SuyFsZ6bY7ZDzqIyiHMrByFr4
+7SnZDakch1kwtnFCMTW4MMUCSRqKYRvZd+VGOimOVgvGodXtgRA/1s/HltAi68tn
+J4LpVbihVH9mPJdUUTK9PLgQbGyiYzqV9s+hRWr2jrJBqgJb2KnY62I03gl/n6FU
+ehDbn7JcyiKaO3IVuat1mQ5k3gEoPozyObW2Mp3hV+rzbaiUpWJDzDj9Tg5oqEpz
++s4Fyv6jlElVRZ+vQNJWl41+nOUVwKXXB82tkQ3TiAXZjpmE3YwoTbi54yBoPDCx
+gjMlGk+RDszmxC0QjJR31l4KtgXfrYchOeO1kf5mTQcf5Ho5IQHQxAUtSmL+8HUy
+RhQfVyzmbGpuHlnTJn1gXrxK+jjiI8bI+6l5PGH0ka5tHqywbaMdHJATnLW23gsy
+j9DVkRowyyUjGjcx6iEK
+=Z72Z

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2876

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1479

[Apache Jenkins Server]

See 

beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.6.0-RC2

[altay]

Repository: beam
Updated Branches:
  refs/heads/release-0.6.0 dc4acfdd1 -> 11d8a67c0


[maven-release-plugin] rollback changes from release preparation of v0.6.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/11d8a67c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/11d8a67c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/11d8a67c

Branch: refs/heads/release-0.6.0
Commit: 11d8a67c0348e72eab34616b962e686f61d6a4eb
Parents: dc4acfd
Author: Ahmet Altay 
Authored: Fri Mar 10 18:10:06 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 18:10:06 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-construction-java/pom.xml| 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/common/fn-api/pom.xml| 2 +-
 sdks/common/pom.xml   | 2 +-
 sdks/common/runner-api/pom.xml| 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/jackson/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/harness/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hadoop-common/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/javadoc/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 sdks/python/pom.xml   | 2 +-
 45 files changed, 46 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/11d8a67c/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 89d55b5..60e4fed 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/11d8a67c/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index d00dbc9..580a154 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/11d8a67c/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index f5f7175..550578b 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/11d8a67c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 109faf6..eded684 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.6.0
+  0.6.0-SNAPSHOT
 
 

[beam] Git Push Summary

[altay]

Repository: beam
Updated Tags:  refs/tags/v0.6.0-RC2 [created] d56fdbe28

beam git commit: [maven-release-plugin] prepare release v0.6.0-RC2

[altay]

Repository: beam
Updated Branches:
  refs/heads/release-0.6.0 ebc2ba5bf -> dc4acfdd1


[maven-release-plugin] prepare release v0.6.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dc4acfdd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dc4acfdd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dc4acfdd

Branch: refs/heads/release-0.6.0
Commit: dc4acfdd1bb30a07a9c48849f88a67f60bc8ff08
Parents: ebc2ba5
Author: Ahmet Altay 
Authored: Fri Mar 10 18:09:10 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 18:09:10 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-construction-java/pom.xml| 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/common/fn-api/pom.xml| 2 +-
 sdks/common/pom.xml   | 2 +-
 sdks/common/runner-api/pom.xml| 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/jackson/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/harness/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hadoop-common/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/javadoc/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 sdks/python/pom.xml   | 2 +-
 45 files changed, 46 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dc4acfdd/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 60e4fed..89d55b5 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/dc4acfdd/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 580a154..d00dbc9 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/dc4acfdd/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 550578b..f5f7175 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/dc4acfdd/pom.xml
--
diff --git a/pom.xml b/pom.xml
index eded684..109faf6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.6.0-SNAPSHOT
+  0.6.0
 
   
 
@@ -48,7 +48,7 @@
  

[9/9] beam git commit: Revert "[maven-release-plugin] prepare release v0.6.0-RC2"

[altay]

Revert "[maven-release-plugin] prepare release v0.6.0-RC2"

This reverts commit 7321c9afc5aeb3b786584bfe4b145cc3bf639830.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ebc2ba5b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ebc2ba5b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ebc2ba5b

Branch: refs/heads/release-0.6.0
Commit: ebc2ba5bf4cc368b25a9cd6131175bac3afffe13
Parents: 7321c9a
Author: Ahmet Altay 
Authored: Fri Mar 10 17:46:44 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:46:44 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-construction-java/pom.xml| 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/common/fn-api/pom.xml| 2 +-
 sdks/common/pom.xml   | 2 +-
 sdks/common/runner-api/pom.xml| 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/jackson/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/harness/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hadoop-common/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/javadoc/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 sdks/python/pom.xml   | 2 +-
 45 files changed, 46 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ebc2ba5b/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 89d55b5..60e4fed 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ebc2ba5b/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index d00dbc9..580a154 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ebc2ba5b/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index f5f7175..550578b 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.6.0
+0.6.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ebc2ba5b/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 109faf6..eded684 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.6.0
+  0.6.0-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:htt

[1/9] beam git commit: Move GC timer checking to StatefulDoFnRunner.CleanupTimer

[altay]

Repository: beam
Updated Branches:
  refs/heads/release-0.6.0 dc64c2fc0 -> ebc2ba5bf


Move GC timer checking to StatefulDoFnRunner.CleanupTimer


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a18b5b16
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a18b5b16
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a18b5b16

Branch: refs/heads/release-0.6.0
Commit: a18b5b1648489f14fd7a621f345e4d21c09b437f
Parents: dc64c2f
Author: Aljoscha Krettek 
Authored: Fri Mar 10 08:29:27 2017 +0100
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:13:40 2017 -0800

--
 .../beam/runners/core/StatefulDoFnRunner.java   | 29 
 1 file changed, 23 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a18b5b16/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index 154d8bc..926345e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -115,15 +115,12 @@ public class StatefulDoFnRunner
   @Override
   public void onTimer(
   String timerId, BoundedWindow window, Instant timestamp, TimeDomain 
timeDomain) {
-boolean isEventTimer = timeDomain.equals(TimeDomain.EVENT_TIME);
-Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
-if (isEventTimer && GC_TIMER_ID.equals(timerId) && 
gcTime.equals(timestamp)) {
+if (cleanupTimer.isForWindow(timerId, window, timestamp, timeDomain)) {
   stateCleaner.clearForWindow(window);
   // There should invoke the onWindowExpiration of DoFn
 } else {
-  if (isEventTimer || !dropLateData(window)) {
-doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
-  }
+  // a timer can never be late because we don't allow setting timers after 
GC time
+  doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
 }
   }
 
@@ -151,6 +148,16 @@ public class StatefulDoFnRunner
  * Set the garbage collect time of the window to timer.
  */
 void setForWindow(BoundedWindow window);
+
+/**
+ * Checks whether the given timer is a cleanup timer for the window.
+ */
+boolean isForWindow(
+String timerId,
+BoundedWindow window,
+Instant timestamp,
+TimeDomain timeDomain);
+
   }
 
   /**
@@ -191,6 +198,16 @@ public class StatefulDoFnRunner
   GC_TIMER_ID, gcTime, TimeDomain.EVENT_TIME);
 }
 
+@Override
+public boolean isForWindow(
+String timerId,
+BoundedWindow window,
+Instant timestamp,
+TimeDomain timeDomain) {
+  boolean isEventTimer = timeDomain.equals(TimeDomain.EVENT_TIME);
+  Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
+  return isEventTimer && GC_TIMER_ID.equals(timerId) && 
gcTime.equals(timestamp);
+}
   }
 
   /**

[2/9] beam git commit: Introduce Flink-specific state GC implementations

[altay]

Introduce Flink-specific state GC implementations

We now set the GC timer for window.maxTimestamp() + 1 to ensure that a
user timer set for window.maxTimestamp() still has all state.

This also adds tests for late data dropping and state GC specifically
for the Flink DoFnOperator.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8fa718db
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8fa718db
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8fa718db

Branch: refs/heads/release-0.6.0
Commit: 8fa718db5bc14efd1beefc2c757c331a5bdbf927
Parents: a18b5b1
Author: Aljoscha Krettek 
Authored: Fri Mar 10 11:07:00 2017 +0100
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:13:49 2017 -0800

--
 .../apache/beam/runners/core/DoFnRunners.java   |  15 +-
 .../beam/runners/core/StatefulDoFnRunner.java   |  87 ---
 .../runners/core/StatefulDoFnRunnerTest.java| 110 -
 .../wrappers/streaming/DoFnOperator.java| 111 -
 .../flink/streaming/DoFnOperatorTest.java   | 225 +++
 5 files changed, 439 insertions(+), 109 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8fa718db/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 9455eea..a1b7c8b 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -21,9 +21,6 @@ import java.util.List;
 import org.apache.beam.runners.core.ExecutionContext.StepContext;
 import org.apache.beam.runners.core.StatefulDoFnRunner.CleanupTimer;
 import org.apache.beam.runners.core.StatefulDoFnRunner.StateCleaner;
-import 
org.apache.beam.runners.core.StatefulDoFnRunner.StateInternalsStateCleaner;
-import 
org.apache.beam.runners.core.StatefulDoFnRunner.TimeInternalsCleanupTimer;
-import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -135,18 +132,13 @@ public class DoFnRunners {
   DoFnRunner doFnRunner,
   StepContext stepContext,
   AggregatorFactory aggregatorFactory,
-  WindowingStrategy windowingStrategy) {
+  WindowingStrategy windowingStrategy,
+  CleanupTimer cleanupTimer,
+  StateCleaner stateCleaner) {
 Aggregator droppedDueToLateness = 
aggregatorFactory.createAggregatorForDoFn(
 fn.getClass(), stepContext, 
StatefulDoFnRunner.DROPPED_DUE_TO_LATENESS_COUNTER,
 Sum.ofLongs());
 
-CleanupTimer cleanupTimer =
-new TimeInternalsCleanupTimer(stepContext.timerInternals(), 
windowingStrategy);
-
-Coder windowCoder = (Coder) 
windowingStrategy.getWindowFn().windowCoder();
-StateCleaner stateCleaner =
-new StateInternalsStateCleaner<>(fn, stepContext.stateInternals(), 
windowCoder);
-
 return new StatefulDoFnRunner<>(
 doFnRunner,
 windowingStrategy,
@@ -154,5 +146,4 @@ public class DoFnRunners {
 stateCleaner,
 droppedDueToLateness);
   }
-
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/8fa718db/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index 926345e..c672902 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -17,12 +17,8 @@
  */
 package org.apache.beam.runners.core;
 
-import java.util.Map;
-import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -30,8 +26,6 @@ import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.WindowTracing;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.state.State;
-import org.apache.beam.

[7/9] beam git commit: Generate zip distribution for pyhthon

[altay]

Generate zip distribution for pyhthon


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bce94c4e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bce94c4e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bce94c4e

Branch: refs/heads/release-0.6.0
Commit: bce94c4e06bf2ee2428e7b6fa71d0d9144b7ee61
Parents: ef47c9f
Author: Ahmet Altay 
Authored: Fri Mar 10 16:40:34 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:16:09 2017 -0800

--
 sdks/python/pom.xml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bce94c4e/sdks/python/pom.xml
--
diff --git a/sdks/python/pom.xml b/sdks/python/pom.xml
index 98b7fa3..2fcbaad 100644
--- a/sdks/python/pom.xml
+++ b/sdks/python/pom.xml
@@ -136,6 +136,8 @@
 sdist
 --dist-dir
 ${project.build.directory}
+--formats
+zip,gztar
   
   
 ${python.user.base}

[3/9] beam git commit: Properly deal with late processing-time timers

[altay]

Properly deal with late processing-time timers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/86522157
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/86522157
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/86522157

Branch: refs/heads/release-0.6.0
Commit: 86522157a79fd9a753436312ff8b746cb5740135
Parents: 8fa718d
Author: Aljoscha Krettek 
Authored: Fri Mar 10 15:25:26 2017 +0100
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:13:57 2017 -0800

--
 .../beam/runners/core/StatefulDoFnRunner.java   | 40 
 1 file changed, 24 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/86522157/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index c672902..d27193c 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -76,33 +76,31 @@ public class StatefulDoFnRunner
   }
 
   @Override
-  public void processElement(WindowedValue compressedElem) {
+  public void processElement(WindowedValue input) {
 
 // StatefulDoFnRunner always observes windows, so we need to explode
-for (WindowedValue value : compressedElem.explodeWindows()) {
+for (WindowedValue value : input.explodeWindows()) {
 
   BoundedWindow window = value.getWindows().iterator().next();
 
-  if (!dropLateData(window)) {
+  if (isLate(window)) {
+// The element is too late for this window.
+droppedDueToLateness.addValue(1L);
+WindowTracing.debug(
+"StatefulDoFnRunner.processElement: Dropping element at {}; 
window:{} "
++ "since too far behind inputWatermark:{}",
+input.getTimestamp(), window, 
cleanupTimer.currentInputWatermarkTime());
+  } else {
 cleanupTimer.setForWindow(window);
 doFnRunner.processElement(value);
   }
 }
   }
 
-  private boolean dropLateData(BoundedWindow window) {
+  private boolean isLate(BoundedWindow window) {
 Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
 Instant inputWM = cleanupTimer.currentInputWatermarkTime();
-if (gcTime.isBefore(inputWM)) {
-  // The element is too late for this window.
-  droppedDueToLateness.addValue(1L);
-  WindowTracing.debug(
-  "StatefulDoFnRunner.processElement/onTimer: Dropping element for 
window:{} "
-  + "since too far behind inputWatermark:{}", window, inputWM);
-  return true;
-} else {
-  return false;
-}
+return gcTime.isBefore(inputWM);
   }
 
   @Override
@@ -112,8 +110,18 @@ public class StatefulDoFnRunner
   stateCleaner.clearForWindow(window);
   // There should invoke the onWindowExpiration of DoFn
 } else {
-  // a timer can never be late because we don't allow setting timers after 
GC time
-  doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
+  // An event-time timer can never be late because we don't allow setting 
timers after GC time.
+  // Ot can happen that a processing-time time fires for a late window, we 
need to ignore
+  // this.
+  if (!timeDomain.equals(TimeDomain.EVENT_TIME) && isLate(window)) {
+// don't increment the dropped counter, only do that for elements
+WindowTracing.debug(
+"StatefulDoFnRunner.onTimer: Ignoring processing-time timer at {}; 
window:{} "
++ "since window is too far behind inputWatermark:{}",
+timestamp, window, cleanupTimer.currentInputWatermarkTime());
+  } else {
+doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
+  }
 }
   }
 

[5/9] beam git commit: Remove duplicated dependency from Dataflow runner pom.xml

[altay]

Remove duplicated dependency from Dataflow runner pom.xml


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c7c4da28
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c7c4da28
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c7c4da28

Branch: refs/heads/release-0.6.0
Commit: c7c4da28b7925de38e7c10fcc4e9ef52a5ea76fc
Parents: 2b92b0d
Author: Kenneth Knowles 
Authored: Fri Mar 10 13:14:58 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:15:32 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 4 
 1 file changed, 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c7c4da28/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index d305e15..a2bca0d 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -357,9 +357,5 @@
   test-jar
   test
 
-  
-  org.apache.beam
-  beam-runners-core-construction-java
-  
   
 

[6/9] beam git commit: Ignore results from the tox clean up phase

[altay]

Ignore results from the tox clean up phase

Some temporary files are generated only under certain conditions and
this should not fail tox.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ef47c9f5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ef47c9f5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ef47c9f5

Branch: refs/heads/release-0.6.0
Commit: ef47c9f511b0f5730b0dc417aefa703fd6f974c5
Parents: c7c4da2
Author: Ahmet Altay 
Authored: Fri Mar 10 16:21:17 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:15:45 2017 -0800

--
 sdks/python/tox.ini | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ef47c9f5/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 807fe3f..2ed21c6 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -51,11 +51,11 @@ commands =
   pip install -e .[test]
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-  # Clean up all cython generated files.
-  find apache_beam -type f -name '*.c' -delete
-  find apache_beam -type f -name '*.so' -delete
-  find target/build -type f -name '*.c' -delete
-  find target/build -type f -name '*.so' -delete
+  # Clean up all cython generated files. Ignore if deletion fails.
+  - find apache_beam -type f -name '*.c' -delete
+  - find apache_beam -type f -name '*.so' -delete
+  - find target/build -type f -name '*.c' -delete
+  - find target/build -type f -name '*.so' -delete
 passenv = TRAVIS*
 
 [testenv:py27gcp]

[8/9] beam git commit: [maven-release-plugin] prepare release v0.6.0-RC2

[altay]

[maven-release-plugin] prepare release v0.6.0-RC2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7321c9af
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7321c9af
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7321c9af

Branch: refs/heads/release-0.6.0
Commit: 7321c9afc5aeb3b786584bfe4b145cc3bf639830
Parents: bce94c4
Author: Ahmet Altay 
Authored: Fri Mar 10 17:41:21 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:41:21 2017 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-construction-java/pom.xml| 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/common/fn-api/pom.xml| 2 +-
 sdks/common/pom.xml   | 2 +-
 sdks/common/runner-api/pom.xml| 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/jackson/pom.xml  | 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/harness/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hadoop-common/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/mqtt/pom.xml | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/javadoc/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 sdks/python/pom.xml   | 2 +-
 45 files changed, 46 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7321c9af/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 60e4fed..89d55b5 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7321c9af/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 580a154..d00dbc9 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7321c9af/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 550578b..f5f7175 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.6.0-SNAPSHOT
+0.6.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7321c9af/pom.xml
--
diff --git a/pom.xml b/pom.xml
index eded684..109faf6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.apache.org/
   2016
 
-  0.6.0-SNAPSHOT
+  0.6.0
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/beam.git
 
scm:git:https://git-

[4/9] beam git commit: Add README to python tarball.

[altay]

Add README to python tarball.

And, delete test created files, to avoid them being included in the tarball.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2b92b0d8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2b92b0d8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2b92b0d8

Branch: refs/heads/release-0.6.0
Commit: 2b92b0d851bcc5aedcc40ebf02ad4f39f3d67514
Parents: 8652215
Author: Ahmet Altay 
Authored: Fri Mar 10 13:42:17 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 17:15:03 2017 -0800

--
 sdks/python/MANIFEST.in | 2 ++
 sdks/python/tox.ini | 2 ++
 2 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2b92b0d8/sdks/python/MANIFEST.in
--
diff --git a/sdks/python/MANIFEST.in b/sdks/python/MANIFEST.in
index baa2fda..57f684e 100644
--- a/sdks/python/MANIFEST.in
+++ b/sdks/python/MANIFEST.in
@@ -17,3 +17,5 @@
 
 # This file is used from Python to sync versions
 include pom.xml
+
+include README.md

http://git-wip-us.apache.org/repos/asf/beam/blob/2b92b0d8/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 8d8acfa..807fe3f 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -54,6 +54,8 @@ commands =
   # Clean up all cython generated files.
   find apache_beam -type f -name '*.c' -delete
   find apache_beam -type f -name '*.so' -delete
+  find target/build -type f -name '*.c' -delete
+  find target/build -type f -name '*.so' -delete
 passenv = TRAVIS*
 
 [testenv:py27gcp]

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2875

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2874

[Apache Jenkins Server]

See 

[1/2] beam git commit: Generate zip distribution for pyhthon

[altay]

Repository: beam
Updated Branches:
  refs/heads/master f29bf89c1 -> 7c78480cf


Generate zip distribution for pyhthon


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/466e83fd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/466e83fd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/466e83fd

Branch: refs/heads/master
Commit: 466e83fdd88c16f096f842a2ca941f49d3701658
Parents: f29bf89
Author: Ahmet Altay 
Authored: Fri Mar 10 16:40:34 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 16:54:02 2017 -0800

--
 sdks/python/pom.xml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/466e83fd/sdks/python/pom.xml
--
diff --git a/sdks/python/pom.xml b/sdks/python/pom.xml
index 0d1fdb4..cb8307a 100644
--- a/sdks/python/pom.xml
+++ b/sdks/python/pom.xml
@@ -136,6 +136,8 @@
 sdist
 --dist-dir
 ${project.build.directory}
+--formats
+zip,gztar
   
   
 ${python.user.base}

[GitHub] beam pull request #2225: Generate zip distribution for pyhthon

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2225


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2225

[altay]

This closes #2225


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c78480c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c78480c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c78480c

Branch: refs/heads/master
Commit: 7c78480cfe82c9f3bae2da4ed72eda11e036e99a
Parents: f29bf89 466e83f
Author: Ahmet Altay 
Authored: Fri Mar 10 16:54:10 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 16:54:10 2017 -0800

--
 sdks/python/pom.xml | 2 ++
 1 file changed, 2 insertions(+)
--

[GitHub] beam pull request #2224: Ignore results from the tox clean up phase

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2224


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Ignore results from the tox clean up phase

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 39688d8d4 -> f29bf89c1


Ignore results from the tox clean up phase

Some temporary files are generated only under certain conditions and
this should not fail tox.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/12016e59
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/12016e59
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/12016e59

Branch: refs/heads/master
Commit: 12016e59e84a841afcfcd55402daf7701460dcbc
Parents: 39688d8
Author: Ahmet Altay 
Authored: Fri Mar 10 16:21:17 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 16:21:17 2017 -0800

--
 sdks/python/tox.ini | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/12016e59/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 807fe3f..2ed21c6 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -51,11 +51,11 @@ commands =
   pip install -e .[test]
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-  # Clean up all cython generated files.
-  find apache_beam -type f -name '*.c' -delete
-  find apache_beam -type f -name '*.so' -delete
-  find target/build -type f -name '*.c' -delete
-  find target/build -type f -name '*.so' -delete
+  # Clean up all cython generated files. Ignore if deletion fails.
+  - find apache_beam -type f -name '*.c' -delete
+  - find apache_beam -type f -name '*.so' -delete
+  - find target/build -type f -name '*.c' -delete
+  - find target/build -type f -name '*.so' -delete
 passenv = TRAVIS*
 
 [testenv:py27gcp]

[2/2] beam git commit: This closes #2224

[altay]

This closes #2224


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f29bf89c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f29bf89c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f29bf89c

Branch: refs/heads/master
Commit: f29bf89c1df6a81c77e60ef8aac1edd1db37124e
Parents: 39688d8 12016e5
Author: Ahmet Altay 
Authored: Fri Mar 10 16:32:08 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 16:32:08 2017 -0800

--
 sdks/python/tox.ini | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--

[GitHub] beam pull request #2225: Generate zip distribution for pyhthon

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2225

Generate zip distribution for pyhthon

R: @sb2nov 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam zipfile

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2225.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2225


commit ed1658b22c966e3121e80cda15ab9c703625e3ba
Author: Ahmet Altay 
Date:   2017-03-11T00:40:34Z

Generate zip distribution for pyhthon




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2873

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2224: Ignore results from the tox clean up phase

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2224

Ignore results from the tox clean up phase

Some temporary files are generated only under certain conditions and
this should not fail tox.

R: @sb2nov 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam err

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2224.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2224


commit 12016e59e84a841afcfcd55402daf7701460dcbc
Author: Ahmet Altay 
Date:   2017-03-11T00:21:17Z

Ignore results from the tox clean up phase

Some temporary files are generated only under certain conditions and
this should not fail tox.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-772) Implement Metrics support for Dataflow Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905919#comment-15905919
 ] 

ASF GitHub Bot commented on BEAM-772:
-

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2223

[BEAM-772] Adding support for metrics querying in Dataflow Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam query-metrics-dataflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2223.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2223


commit 45d3a17ea9ab1ee8e4d8febd02e0f6c49d939877
Author: Pablo 
Date:   2017-03-11T00:10:31Z

Adding support for metrics querying in Dataflow Runner




> Implement Metrics support for Dataflow Runner
> -
>
> Key: BEAM-772
> URL: https://issues.apache.org/jira/browse/BEAM-772
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Ben Chambers
>Assignee: Ben Chambers
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2223: [BEAM-772] Adding support for metrics querying in D...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/2223

[BEAM-772] Adding support for metrics querying in Dataflow Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam query-metrics-dataflow

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2223.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2223


commit 45d3a17ea9ab1ee8e4d8febd02e0f6c49d939877
Author: Pablo 
Date:   2017-03-11T00:10:31Z

Adding support for metrics querying in Dataflow Runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1688) Getting listener timeout after waiting for [10000] ms from ElasticsearchIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1688:
--

Assignee: Jean-Baptiste Onofré  (was: Davor Bonaci)

> Getting listener timeout after waiting for [1] ms from ElasticsearchIO
> --
>
> Key: BEAM-1688
> URL: https://issues.apache.org/jira/browse/BEAM-1688
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.5.0
> Environment: Linux,  ElasticSearch 5.2.1
>Reporter: Sathish Jayaraman
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> I am trying to test a simple Apache Beam code with source as Elasticsearch. I 
> modified Beam's MinimalWordCount example to include source as Elasticsearch 
> instead of TextIO. 
> When I run the code,  I get below error, 
> An exception occured while executing the Java class. null: 
> InvocationTargetException: java.io.IOException: listener timeout after 
> waiting for [1] ms
> I debugged ElasticsearchIO.java and could see everything works fine, the 
> Elasicsearch client is built & data from index is being retrieved by the 
> code. But ParDo function after read transform does not get executed at all. 
> The Elasticsearch client keeps waiting and finally gets a timeout error.
> Any idea about the issue? 
> Have raised issue here too 
> http://stackoverflow.com/questions/42720857/getting-listener-timeout-after-waiting-for-1-ms-from-elasticsearchio



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1478

[Apache Jenkins Server]

See 


Changes:

[altay] Add README to python tarball.

--
[...truncated 318.47 KB...]
testBuiltins 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testCall (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testClosure (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testDict (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testDictComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testGenerator 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testGetAttr (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testGlobals (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testIdentity 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testMethod (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testTupleListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testTuples (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testUnpack (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
test_custom_transform 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_flat_type_hint 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_keyword_type_hints 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_bad_main_input (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_loose_bounds (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_non_function (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_typed_dofn_class (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_typed_dofn_instance 
(apache_beam.typehints.typed_pipeline_test.MainInputTest) ... ok
test_basic_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_typed_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_untyped_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_input_iterable 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_inputs 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_keyword_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_any_compatibility 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... 
ok
test_type_check 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_composite_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_returns 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_takes 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_simple_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_valid_mix_pos_and_keyword_with_both_orders 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_getcallargs_forhints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_hint_helper (apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_positional_arg_hints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_compatibility (apache_beam.typehints.typehints_test.DictHintTestCase) ... 
ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... 
:497:
 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  e.exception.message)
ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_sim

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1477

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2872

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Gearpump #163

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905810#comment-15905810
 ] 

Mark Liu commented on BEAM-1676:


I removed environment parameters that starts with "-Dghprb", which seems 
redundant params appended by Jenkins, and run on Linux machine but failed to 
reproduce the failure.

command I used:
{code}
mvn -f pom.xml -Dlabel=beam -Dsha1=origin/pr/2204/merge "-Djdk=JDK 1.8 
(latest)" -DGIT_BRANCH=jenkins_java_sdk_version_test 
"-Dmaven.repo.local=/tmp/beam_PostCommit_Java_Version_Test/jdk/JDK 1.8 
(latest)/label/beam/.repository"  -P release,dataflow-runner clean install 
coveralls:report -DrepoToken= -DskipITs=false 
'-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'
{code}

However, test on Jenkins failed consistently: 
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1612) Support real Bundle in Flink runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905798#comment-15905798
 ] 

Kenneth Knowles commented on BEAM-1612:
---

I just noticed this, and I agree strongly that it needs to be solved. It is a 
major concern for all runners to get this right in order to have realistic 
performance.

There are some questions surrounding when and how to output data from 
{{@FinishBundle}} since it generally doesn't work with windowing. Bundles are 
not related to a window and can have data from lots of windows. I filed 
BEAM-1283 because I think the spec is very bad, and [~tgroh] filed BEAM-1312 
with an even stronger viewpoint that there should be no output, only flush-like 
operations. But sometimes there may be data that comes back from a flush that 
you need to output - it should be mostly deterministic and independent of 
bundling.

For the relationship with snapshotting, I do want to make sure the ordering is 
clear: The runner is required to call {{FinishBundle}} before durably 
committing otherwise it might not be actually committed, but the commit can 
still fail so the runner is not required to commit right away. So it would be 
fine to just call it every once in a while, and even less often take a 
snapshot, but the important thing is that you can't have other method calls in 
between the {{FinishBundle}} and the commit because they might set up new 
transient state that needs to be flushed.


> Support real Bundle in Flink runner
> ---
>
> Key: BEAM-1612
> URL: https://issues.apache.org/jira/browse/BEAM-1612
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>
> The Bundle is very important in the beam model. Users can use the bundle to 
> flush buffer, can reuse many heavyweight resources in a bundle. Most IO 
> plugins use the bundle to flush. 
> Moreover, FlinkRunner can also use Bundle to reduce access to the FlinkState, 
> such as first placed in JavaHeap, flush into RocksDbState when invoke 
> finishBundle , this can reduce the number of serialization.
> But now FlinkRunner calls the finishBundle every processElement. We need 
> support real Bundle.
> I think we can have the following implementations:
> 1.Invoke finishBundle and next startBundle in {{snapshot}} of Flink. But 
> sometimes this "Bundle" maybe too big. This depends on the user's checkpoint 
> configuration.
> 2.Manually control the size of the bundle. The half-bundle will be flushed to 
> a full-bundle by count or eventTime or processTime or {{snapshot}}. We do not 
> need to wait, just call the startBundle and finishBundle at the right time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2871

[Apache Jenkins Server]

See 

[1/2] beam git commit: Add README to python tarball.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 9299e2632 -> 39688d8d4


Add README to python tarball.

And, delete test created files, to avoid them being included in the tarball.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ec6da893
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ec6da893
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ec6da893

Branch: refs/heads/master
Commit: ec6da893bf36e7780728d0c08d47e1e4824a66c7
Parents: 9299e26
Author: Ahmet Altay 
Authored: Fri Mar 10 13:42:17 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 14:22:37 2017 -0800

--
 sdks/python/MANIFEST.in | 2 ++
 sdks/python/tox.ini | 2 ++
 2 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ec6da893/sdks/python/MANIFEST.in
--
diff --git a/sdks/python/MANIFEST.in b/sdks/python/MANIFEST.in
index baa2fda..57f684e 100644
--- a/sdks/python/MANIFEST.in
+++ b/sdks/python/MANIFEST.in
@@ -17,3 +17,5 @@
 
 # This file is used from Python to sync versions
 include pom.xml
+
+include README.md

http://git-wip-us.apache.org/repos/asf/beam/blob/ec6da893/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 8d8acfa..807fe3f 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -54,6 +54,8 @@ commands =
   # Clean up all cython generated files.
   find apache_beam -type f -name '*.c' -delete
   find apache_beam -type f -name '*.so' -delete
+  find target/build -type f -name '*.c' -delete
+  find target/build -type f -name '*.so' -delete
 passenv = TRAVIS*
 
 [testenv:py27gcp]

[GitHub] beam pull request #2222: Add README to python tarball

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2222

[altay]

This closes #


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/39688d8d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/39688d8d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/39688d8d

Branch: refs/heads/master
Commit: 39688d8d48b92702b7b3a0b8744ba0f6f018f967
Parents: 9299e26 ec6da89
Author: Ahmet Altay 
Authored: Fri Mar 10 14:23:09 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 14:23:09 2017 -0800

--
 sdks/python/MANIFEST.in | 2 ++
 sdks/python/tox.ini | 2 ++
 2 files changed, 4 insertions(+)
--

[jira] [Resolved] (BEAM-1593) Remove access to the Pipeline Runner during Pipeline Construction

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1593.
---
   Resolution: Fixed
Fix Version/s: 0.6.0

> Remove access to the Pipeline Runner during Pipeline Construction
> -
>
> Key: BEAM-1593
> URL: https://issues.apache.org/jira/browse/BEAM-1593
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: 0.6.0
>
>
> Pipelines prior to the call to run are runner-independent, so PTransforms 
> shouldn't have access to any knowledge about the runner.
> This consists of removing Pipeline.getRunner



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905763#comment-15905763
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2150


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/4] beam git commit: enable ParDoTest

[kenn]

enable ParDoTest


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3dc8fc81
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3dc8fc81
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3dc8fc81

Branch: refs/heads/gearpump-runner
Commit: 3dc8fc81d14d67318e26b99691f1583aacbd3509
Parents: 4eb50d1
Author: manuzhang 
Authored: Fri Mar 10 16:50:01 2017 +0800
Committer: manuzhang 
Committed: Fri Mar 10 16:50:01 2017 +0800

--
 runners/gearpump/pom.xml | 5 -
 1 file changed, 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3dc8fc81/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 6a41dc0..3efb1f6 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -75,11 +75,6 @@
   
 
org.apache.beam:beam-sdks-java-core
   
-  
-
-  org.apache.beam.sdk.transforms.ParDoTest
-
-  
   
 
   [

[1/4] beam git commit: [BEAM-79] Add SideInput support for GearpumpRunner

[kenn]

Repository: beam
Updated Branches:
  refs/heads/gearpump-runner 2d0aed922 -> 15a8ad62a


http://git-wip-us.apache.org/repos/asf/beam/blob/4eb50d15/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
index 656fc6a..b8a5233 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
@@ -18,14 +18,28 @@
 
 package org.apache.beam.runners.gearpump.translators.utils;
 
+import com.google.common.collect.Lists;
+
 import java.time.Instant;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
 
+import org.apache.beam.runners.gearpump.translators.TranslationContext;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PCollectionView;
+
+import org.apache.gearpump.streaming.dsl.api.functions.FoldFunction;
+import org.apache.gearpump.streaming.dsl.api.functions.MapFunction;
+import org.apache.gearpump.streaming.dsl.javaapi.JavaStream;
 import org.apache.gearpump.streaming.dsl.window.impl.Window;
 
 
+
 /**
  * Utility methods for translators.
  */
@@ -52,4 +66,137 @@ public class TranslatorUtils {
   throw new RuntimeException("unknown window " + 
window.getClass().getName());
 }
   }
+
+  public static  JavaStream withSideInputStream(
+  TranslationContext context,
+  JavaStream> inputStream,
+  Map> tagsToSideInputs) {
+JavaStream mainStream =
+inputStream.map(new ToRawUnionValue("0"), 
"map_to_RawUnionValue");
+
+for (Map.Entry> tagToSideInput: 
tagsToSideInputs.entrySet()) {
+  // actually JavaStream>>
+  // check CreatePCollectionViewTranslator
+  JavaStream> sideInputStream = 
context.getInputStream(
+  tagToSideInput.getValue());
+  mainStream = mainStream.merge(sideInputStream.map(new ToRawUnionValue<>(
+  tagToSideInput.getKey()), "map_to_RawUnionValue"), 
"merge_to_MainStream");
+}
+return mainStream;
+  }
+
+  public static Map> getTagsToSideInputs(
+  Collection> sideInputs) {
+Map> tagsToSideInputs = new HashMap<>();
+// tag 0 is reserved for main input
+int tag = 1;
+for (PCollectionView sideInput: sideInputs) {
+  tagsToSideInputs.put(tag + "", sideInput);
+  tag++;
+}
+return tagsToSideInputs;
+  }
+
+  public static JavaStream> 
toList(JavaStream stream) {
+return stream.fold(new FoldFunction>() {
+
+  @Override
+  public List init() {
+return Lists.newArrayList();
+  }
+
+  @Override
+  public List fold(List accumulator,
+  RawUnionValue rawUnionValue) {
+accumulator.add(rawUnionValue);
+return accumulator;
+  }
+}, "fold_to_iterable");
+  }
+
+  /**
+   * Converts @link{RawUnionValue} to @link{WindowedValue}.
+   */
+  public static class FromRawUnionValue extends
+  MapFunction> {
+
+private static final long serialVersionUID = -4764968219713478955L;
+
+@Override
+public WindowedValue map(RawUnionValue value) {
+  return (WindowedValue) value.getValue();
+}
+  }
+
+  private static class ToRawUnionValue extends
+  MapFunction, RawUnionValue> {
+
+private static final long serialVersionUID = 8648852871014813583L;
+private final String tag;
+
+ToRawUnionValue(String tag) {
+  this.tag = tag;
+}
+
+@Override
+public RawUnionValue map(WindowedValue windowedValue) {
+  return new RawUnionValue(tag, windowedValue);
+}
+  }
+
+
+  /**
+   * This is copied from org.apache.beam.sdk.transforms.join.RawUnionValue.
+   */
+  public static class RawUnionValue {
+private final String unionTag;
+private final Object value;
+
+/**
+ * Constructs a partial union from the given union tag and value.
+ */
+public RawUnionValue(String unionTag, Object value) {
+  this.unionTag = unionTag;
+  this.value = value;
+}
+
+public String getUnionTag() {
+  return unionTag;
+}
+
+public Object getValue() {
+  return value;
+}
+
+@Override
+public String toString() {
+  return unionTag + ":" + value;
+}
+
+@Override
+public boolean equals(Object o) {
+  if (this == o) {
+return true;
+  }
+  if (o == null || getClass() != o.getClass()) {
+return false;
+  }
+
+  RawUnionValue that = (RawUnionValue) o;

[GitHub] beam pull request #2150: [BEAM-79] Add SideInput support for GearpumpRunner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2150


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] beam git commit: This closes #2150: Add SideInput support for GearpumpRunner

[kenn]

This closes #2150: Add SideInput support for GearpumpRunner

  enable ParDoTest
  [BEAM-79] Add SideInput support for GearpumpRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/15a8ad62
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/15a8ad62
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/15a8ad62

Branch: refs/heads/gearpump-runner
Commit: 15a8ad62afdfaef20d87a9384dd8417979bee750
Parents: 2d0aed9 3dc8fc8
Author: Kenneth Knowles 
Authored: Fri Mar 10 13:56:37 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Mar 10 13:56:37 2017 -0800

--
 runners/gearpump/pom.xml|  19 -
 .../gearpump/GearpumpPipelineResult.java|  47 +--
 .../gearpump/GearpumpPipelineTranslator.java|   8 +
 .../beam/runners/gearpump/GearpumpRunner.java   | 373 ++-
 .../runners/gearpump/TestGearpumpRunner.java|  39 +-
 ...CreateGearpumpPCollectionViewTranslator.java |  44 +++
 .../CreatePCollectionViewTranslator.java|  43 +++
 .../translators/CreateValuesTranslator.java |   2 +
 .../FlattenPCollectionTranslator.java   |  38 ++
 .../translators/GroupByKeyTranslator.java   |  82 ++--
 .../translators/ParDoBoundMultiTranslator.java  | 165 +++-
 .../translators/ParDoBoundTranslator.java   |  32 +-
 .../translators/WindowBoundTranslator.java  |  21 +-
 .../translators/functions/DoFnFunction.java | 158 ++--
 .../translators/io/BoundedSourceWrapper.java|   1 +
 .../gearpump/translators/io/GearpumpSource.java |  23 +-
 .../gearpump/translators/io/ValuesSource.java   |  14 +-
 .../translators/utils/DoFnRunnerFactory.java|  20 +-
 .../translators/utils/NoOpSideInputReader.java  |  48 ---
 .../translators/utils/TranslatorUtils.java  | 147 
 20 files changed, 1001 insertions(+), 323 deletions(-)
--

[2/4] beam git commit: [BEAM-79] Add SideInput support for GearpumpRunner

[kenn]

[BEAM-79] Add SideInput support for GearpumpRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4eb50d15
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4eb50d15
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4eb50d15

Branch: refs/heads/gearpump-runner
Commit: 4eb50d152b91df46bd7f0478650cb4abac3808c6
Parents: 2d0aed9
Author: manuzhang 
Authored: Tue Feb 14 12:33:31 2017 +0800
Committer: manuzhang 
Committed: Tue Mar 7 22:15:26 2017 +0800

--
 runners/gearpump/pom.xml|  16 +-
 .../gearpump/GearpumpPipelineResult.java|  47 +--
 .../gearpump/GearpumpPipelineTranslator.java|   8 +
 .../beam/runners/gearpump/GearpumpRunner.java   | 373 ++-
 .../runners/gearpump/TestGearpumpRunner.java|  39 +-
 ...CreateGearpumpPCollectionViewTranslator.java |  44 +++
 .../CreatePCollectionViewTranslator.java|  43 +++
 .../translators/CreateValuesTranslator.java |   2 +
 .../FlattenPCollectionTranslator.java   |  38 ++
 .../translators/GroupByKeyTranslator.java   |  82 ++--
 .../translators/ParDoBoundMultiTranslator.java  | 165 +++-
 .../translators/ParDoBoundTranslator.java   |  32 +-
 .../translators/WindowBoundTranslator.java  |  21 +-
 .../translators/functions/DoFnFunction.java | 158 ++--
 .../translators/io/BoundedSourceWrapper.java|   1 +
 .../gearpump/translators/io/GearpumpSource.java |  23 +-
 .../gearpump/translators/io/ValuesSource.java   |  14 +-
 .../translators/utils/DoFnRunnerFactory.java|  20 +-
 .../translators/utils/NoOpSideInputReader.java  |  48 ---
 .../translators/utils/TranslatorUtils.java  | 147 
 20 files changed, 1002 insertions(+), 319 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4eb50d15/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 6f91c50..6a41dc0 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -75,23 +75,9 @@
   
 
org.apache.beam:beam-sdks-java-core
   
-  -noverify
   
-
 
-  org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOTest,
-  org.apache.beam.sdk.io.CountingInputTest,
-  org.apache.beam.sdk.io.CountingSourceTest,
-  org.apache.beam.sdk.testing.PAssertTest,
-  org.apache.beam.sdk.transforms.ApproximateUniqueTest,
-  org.apache.beam.sdk.transforms.CombineTest,
-  org.apache.beam.sdk.transforms.CombineFnsTest,
-  org.apache.beam.sdk.transforms.CountTest,
-  org.apache.beam.sdk.transforms.FlattenTest,
-  org.apache.beam.sdk.transforms.ParDoTest,
-  org.apache.beam.sdk.transforms.SampleTest,
-  org.apache.beam.sdk.transforms.ViewTest,
-  org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
+  org.apache.beam.sdk.transforms.ParDoTest
 
   
   

http://git-wip-us.apache.org/repos/asf/beam/blob/4eb50d15/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
index a3740b7..8f90898 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineResult.java
@@ -30,6 +30,7 @@ import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.gearpump.cluster.ApplicationStatus;
 import org.apache.gearpump.cluster.MasterToAppMaster.AppMasterData;
 import org.apache.gearpump.cluster.client.ClientContext;
+import org.apache.gearpump.cluster.client.RunningApplication;
 import org.joda.time.Duration;
 
 import scala.collection.JavaConverters;
@@ -41,13 +42,11 @@ import scala.collection.Seq;
 public class GearpumpPipelineResult implements PipelineResult {
 
   private final ClientContext client;
-  private final int appId;
-  private final Duration defaultWaitDuration = Duration.standardSeconds(30);
-  private final Duration defaultWaitInterval = Duration.standardSeconds(5);
+  private final RunningApplication app;
 
-  public GearpumpPipelineResult(ClientContext client, int appId) {
+  public GearpumpPipelineResult(ClientContex

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2870

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905748#comment-15905748
 ] 

Mark Liu edited comment on BEAM-1676 at 3/10/17 9:47 PM:
-

Thank you Kenn, let me see if I can reproduce the error on local machine.


was (Author: markflyhigh):
Thank you Kenn, let me see if I can reproduce the error from local machine.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905748#comment-15905748
 ] 

Mark Liu commented on BEAM-1676:


Thank you Kenn, let me see if I can reproduce the error from local machine.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2221: Remove duplicated dependency from Dataflow runner p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2221


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2221

[davor]

This closes #2221


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9299e263
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9299e263
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9299e263

Branch: refs/heads/master
Commit: 9299e2632da4c368e29c6a38df5e0b0293faf5e5
Parents: e31cb2b ed5cb8a
Author: Davor Bonaci 
Authored: Fri Mar 10 13:46:11 2017 -0800
Committer: Davor Bonaci 
Committed: Fri Mar 10 13:46:11 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 4 
 1 file changed, 4 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2869

[Apache Jenkins Server]

See 

[1/2] beam git commit: Remove duplicated dependency from Dataflow runner pom.xml

[davor]

Repository: beam
Updated Branches:
  refs/heads/master e31cb2b45 -> 9299e2632


Remove duplicated dependency from Dataflow runner pom.xml


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed5cb8a6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed5cb8a6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ed5cb8a6

Branch: refs/heads/master
Commit: ed5cb8a685c1afeb7c894c247c2cf6c12e5515a9
Parents: e31cb2b
Author: Kenneth Knowles 
Authored: Fri Mar 10 13:14:58 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Mar 10 13:14:58 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 4 
 1 file changed, 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ed5cb8a6/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index f541394..ff79681 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -357,9 +357,5 @@
   test-jar
   test
 
-  
-  org.apache.beam
-  beam-runners-core-construction-java
-  
   
 

[GitHub] beam pull request #2222: Add README to python tarball

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/

Add README to python tarball

And, delete test created files, to avoid them being included in the tarball.

R: @sb2nov 
(cc: @davorbonaci)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam readme

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #


commit b0bac4972a9cbb776cdeb98c704a0b154d26e67b
Author: Ahmet Altay 
Date:   2017-03-10T21:42:17Z

Add README to python tarball.

And, delete test created files, to avoid them being included in the tarball.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #1200

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1476

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Move GC timer checking to StatefulDoFnRunner.CleanupTimer

[aljoscha.krettek] Introduce Flink-specific state GC implementations

[aljoscha.krettek] Properly deal with late processing-time timers

[tgroh] Remove Pipeline.getRunner

--
[...truncated 329.50 KB...]
testGenerator 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testGetAttr (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testGlobals (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testIdentity 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testMethod (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testTupleListComprehension 
(apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) ... ok
testTuples (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
testUnpack (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) 
... ok
test_custom_transform 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_flat_type_hint 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_keyword_type_hints 
(apache_beam.typehints.typed_pipeline_test.CustomTransformTest) ... ok
test_bad_main_input (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_loose_bounds (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_non_function (apache_beam.typehints.typed_pipeline_test.MainInputTest) ... 
ok
test_typed_dofn_class (apache_beam.typehints.typed_pipeline_test.MainInputTest) 
... ok
test_typed_dofn_instance 
(apache_beam.typehints.typed_pipeline_test.MainInputTest) ... ok
test_basic_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_typed_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_default_untyped_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_input_iterable 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_deferred_side_inputs 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_keyword_side_input_hint 
(apache_beam.typehints.typed_pipeline_test.SideInputTest) ... ok
test_any_compatibility 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... 
ok
test_type_check 
(apache_beam.typehints.typehints_test.AnyTypeConstraintTestCase) ... ok
test_composite_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_returns 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_enable_and_disable_type_checking_takes 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_simple_takes_and_returns_hints 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_valid_mix_pos_and_keyword_with_both_orders 
(apache_beam.typehints.typehints_test.CombinedReturnsAndTakesTestCase) ... ok
test_getcallargs_forhints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_hint_helper (apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_positional_arg_hints 
(apache_beam.typehints.typehints_test.DecoratorHelpers) ... ok
test_compatibility (apache_beam.typehints.typehints_test.DictHintTestCase) ... 
ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... 
:497:
 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  e.exception.message)
ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.t

[jira] [Commented] (BEAM-1678) Create MemcachedIO

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905722#comment-15905722
 ] 

Eugene Kirpichov commented on BEAM-1678:


Note that all IO's are families of PTransforms (usually one or two - 
read/write, sometimes more), which is why I referred to this as "transforms".

I agree that support of the memcached protocol by many non-cache systems for 
write makes it reasonable to have Beam include a library for writing things to 
it - basically something like a PTransform>, 
PDone> that writes them to a given memcached-compatible endpoint. Is this all, 
or do you have something more in mind?

> Create MemcachedIO
> --
>
> Key: BEAM-1678
> URL: https://issues.apache.org/jira/browse/BEAM-1678
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1199

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2221: Remove duplicated dependency from Dataflow runner p...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2221

Remove duplicated dependency from Dataflow runner pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @davorbonaci

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam core-construction

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2221.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2221


commit ed5cb8a685c1afeb7c894c247c2cf6c12e5515a9
Author: Kenneth Knowles 
Date:   2017-03-10T21:14:58Z

Remove duplicated dependency from Dataflow runner pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Flink #1887

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1593) Remove access to the Pipeline Runner during Pipeline Construction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905690#comment-15905690
 ] 

ASF GitHub Bot commented on BEAM-1593:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2144


> Remove access to the Pipeline Runner during Pipeline Construction
> -
>
> Key: BEAM-1593
> URL: https://issues.apache.org/jira/browse/BEAM-1593
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Pipelines prior to the call to run are runner-independent, so PTransforms 
> shouldn't have access to any knowledge about the runner.
> This consists of removing Pipeline.getRunner



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2144: [BEAM-1593] Remove Pipeline.getRunner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2144


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2144: Remove Pipeline.getRunner

[kenn]

This closes #2144: Remove Pipeline.getRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e31cb2b4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e31cb2b4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e31cb2b4

Branch: refs/heads/master
Commit: e31cb2b451763608b0e0cd2c1c80634284ccaaa2
Parents: 75fe559 d41fe1d
Author: Kenneth Knowles 
Authored: Fri Mar 10 13:05:48 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Mar 10 13:05:48 2017 -0800

--
 .../beam/runners/direct/DirectRunner.java   | 95 ++--
 .../direct/TestStreamEvaluatorFactory.java  | 22 +++--
 .../direct/TestStreamEvaluatorFactoryTest.java  |  6 +-
 .../BatchStatefulParDoOverridesTest.java|  4 +-
 .../DataflowPipelineTranslatorTest.java | 39 
 .../runners/dataflow/DataflowRunnerTest.java|  4 +-
 .../testing/TestDataflowRunnerTest.java | 50 +--
 .../main/java/org/apache/beam/sdk/Pipeline.java |  7 --
 8 files changed, 110 insertions(+), 117 deletions(-)
--

[1/2] beam git commit: Remove Pipeline.getRunner

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 75fe559a4 -> e31cb2b45


Remove Pipeline.getRunner

Runners need not be instantiated until after pipeline construction, so
they should not be exposed by the Pipeline class.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d41fe1df
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d41fe1df
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d41fe1df

Branch: refs/heads/master
Commit: d41fe1df26329479b82cc59d260998f2b88b4799
Parents: 2c2424c
Author: Thomas Groh 
Authored: Thu Mar 2 10:54:29 2017 -0800
Committer: Thomas Groh 
Committed: Fri Mar 10 09:40:50 2017 -0800

--
 .../beam/runners/direct/DirectRunner.java   | 95 ++--
 .../direct/TestStreamEvaluatorFactory.java  | 22 +++--
 .../direct/TestStreamEvaluatorFactoryTest.java  |  6 +-
 .../BatchStatefulParDoOverridesTest.java|  4 +-
 .../DataflowPipelineTranslatorTest.java | 39 
 .../runners/dataflow/DataflowRunnerTest.java|  4 +-
 .../testing/TestDataflowRunnerTest.java | 50 +--
 .../main/java/org/apache/beam/sdk/Pipeline.java |  7 --
 8 files changed, 110 insertions(+), 117 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d41fe1df/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index f56d225..ce8dbc0 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -70,53 +70,6 @@ import org.joda.time.Instant;
  */
 public class DirectRunner extends PipelineRunner {
   /**
-   * The default set of transform overrides to use in the {@link DirectRunner}.
-   *
-   * The order in which overrides is applied is important, as some 
overrides are expanded into a
-   * composite. If the composite contains {@link PTransform PTransforms} which 
are also overridden,
-   * these PTransforms must occur later in the iteration order. {@link 
ImmutableMap} has an
-   * iteration order based on the order at which elements are added to it.
-   */
-  @SuppressWarnings("rawtypes")
-  private static Map 
defaultTransformOverrides =
-  ImmutableMap.builder()
-  .put(
-  PTransformMatchers.writeWithRunnerDeterminedSharding(),
-  new WriteWithShardingFactory()) /* Uses a view internally. */
-  .put(
-  PTransformMatchers.classEqualTo(CreatePCollectionView.class),
-  new ViewOverrideFactory()) /* Uses pardos and GBKs */
-  .put(
-  PTransformMatchers.classEqualTo(TestStream.class),
-  new DirectTestStreamFactory()) /* primitive */
-  /* Single-output ParDos are implemented in terms of Multi-output 
ParDos. Any override
-  that is applied to a multi-output ParDo must first have all matching 
Single-output ParDos
-  converted to match.
-   */
-  .put(PTransformMatchers.splittableParDoSingle(), new 
ParDoSingleViaMultiOverrideFactory())
-  .put(
-  PTransformMatchers.stateOrTimerParDoSingle(),
-  new ParDoSingleViaMultiOverrideFactory())
-  // SplittableParMultiDo is implemented in terms of nonsplittable 
single ParDos
-  .put(PTransformMatchers.splittableParDoMulti(), new 
ParDoMultiOverrideFactory())
-  // state and timer pardos are implemented in terms of nonsplittable 
single ParDos
-  .put(PTransformMatchers.stateOrTimerParDoMulti(), new 
ParDoMultiOverrideFactory())
-  .put(
-  PTransformMatchers.classEqualTo(ParDo.Bound.class),
-  new ParDoSingleViaMultiOverrideFactory()) /* returns a 
BoundMulti */
-  .put(
-  PTransformMatchers.classEqualTo(BoundMulti.class),
-  /* returns one of two primitives; SplittableParDos are replaced 
above. */
-  new ParDoMultiOverrideFactory())
-  .put(
-  PTransformMatchers.classEqualTo(GBKIntoKeyedWorkItems.class),
-  new DirectGBKIntoKeyedWorkItemsOverrideFactory()) /* Returns a 
GBKO */
-  .put(
-  PTransformMatchers.classEqualTo(GroupByKey.class),
-  new DirectGroupByKeyOverrideFactory()) /* returns two chained 
primitives. */
-  .build();
-
-  /**
* Part of a {@link PCollection}. Elements are output to a bundle, which 
will cause them to be
* executed by {@link PTransform PTransforms} that consume the {@link 
PCollection} this bundle i

[jira] [Commented] (BEAM-1689) Apply changes for Flink's StatefulDoFnRunner to the primary StatefulDoFnRunner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905689#comment-15905689
 ] 

Kenneth Knowles commented on BEAM-1689:
---

There was also some javadoc copy/paste leftovers, but once the cleanup is done 
the javadoc will match again anyhow.

> Apply changes for Flink's StatefulDoFnRunner to the primary StatefulDoFnRunner
> --
>
> Key: BEAM-1689
> URL: https://issues.apache.org/jira/browse/BEAM-1689
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Aljoscha Krettek
>Priority: Trivial
>
> This is a follow-up on the comments on 
> https://github.com/apache/beam/pull/2217 which we left unaddressed in order 
> to unblock the release. The contents of the PR were good for users, and the 
> only issue was a bit of code health and correctness of support code for other 
> runner authors moving forwards.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1689) Apply changes for Flink's StatefulDoFnRunner to the primary StatefulDoFnRunner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1689:
-

 Summary: Apply changes for Flink's StatefulDoFnRunner to the 
primary StatefulDoFnRunner
 Key: BEAM-1689
 URL: https://issues.apache.org/jira/browse/BEAM-1689
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Kenneth Knowles
Assignee: Aljoscha Krettek
Priority: Trivial


This is a follow-up on the comments on https://github.com/apache/beam/pull/2217 
which we left unaddressed in order to unblock the release. The contents of the 
PR were good for users, and the only issue was a bit of code health and 
correctness of support code for other runner authors moving forwards.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1674) Flink user state GC depends on order of timer firing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905688#comment-15905688
 ] 

ASF GitHub Bot commented on BEAM-1674:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2217


> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2217: [BEAM-1674] Fix Flink State GC

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2217


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/4] beam git commit: Move GC timer checking to StatefulDoFnRunner.CleanupTimer

[kenn]

Move GC timer checking to StatefulDoFnRunner.CleanupTimer


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bf6d2748
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bf6d2748
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bf6d2748

Branch: refs/heads/master
Commit: bf6d2748c8876a7415290069163625598928f02f
Parents: 2c2424c
Author: Aljoscha Krettek 
Authored: Fri Mar 10 08:29:27 2017 +0100
Committer: Aljoscha Krettek 
Committed: Fri Mar 10 11:09:04 2017 +0100

--
 .../beam/runners/core/StatefulDoFnRunner.java   | 29 
 1 file changed, 23 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bf6d2748/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index 154d8bc..926345e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -115,15 +115,12 @@ public class StatefulDoFnRunner
   @Override
   public void onTimer(
   String timerId, BoundedWindow window, Instant timestamp, TimeDomain 
timeDomain) {
-boolean isEventTimer = timeDomain.equals(TimeDomain.EVENT_TIME);
-Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
-if (isEventTimer && GC_TIMER_ID.equals(timerId) && 
gcTime.equals(timestamp)) {
+if (cleanupTimer.isForWindow(timerId, window, timestamp, timeDomain)) {
   stateCleaner.clearForWindow(window);
   // There should invoke the onWindowExpiration of DoFn
 } else {
-  if (isEventTimer || !dropLateData(window)) {
-doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
-  }
+  // a timer can never be late because we don't allow setting timers after 
GC time
+  doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
 }
   }
 
@@ -151,6 +148,16 @@ public class StatefulDoFnRunner
  * Set the garbage collect time of the window to timer.
  */
 void setForWindow(BoundedWindow window);
+
+/**
+ * Checks whether the given timer is a cleanup timer for the window.
+ */
+boolean isForWindow(
+String timerId,
+BoundedWindow window,
+Instant timestamp,
+TimeDomain timeDomain);
+
   }
 
   /**
@@ -191,6 +198,16 @@ public class StatefulDoFnRunner
   GC_TIMER_ID, gcTime, TimeDomain.EVENT_TIME);
 }
 
+@Override
+public boolean isForWindow(
+String timerId,
+BoundedWindow window,
+Instant timestamp,
+TimeDomain timeDomain) {
+  boolean isEventTimer = timeDomain.equals(TimeDomain.EVENT_TIME);
+  Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
+  return isEventTimer && GC_TIMER_ID.equals(timerId) && 
gcTime.equals(timestamp);
+}
   }
 
   /**

[4/4] beam git commit: This closes #2217: Fix Flink State GC

[kenn]

This closes #2217: Fix Flink State GC

  Properly deal with late processing-time timers
  Introduce Flink-specific state GC implementations
  Move GC timer checking to StatefulDoFnRunner.CleanupTimer


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/75fe559a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/75fe559a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/75fe559a

Branch: refs/heads/master
Commit: 75fe559a4115661c9a138099e58a73c6f42a038f
Parents: 54390a3 dbfcf4b
Author: Kenneth Knowles 
Authored: Fri Mar 10 13:01:00 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Mar 10 13:01:00 2017 -0800

--
 .../apache/beam/runners/core/DoFnRunners.java   |  15 +-
 .../beam/runners/core/StatefulDoFnRunner.java   | 128 +++
 .../runners/core/StatefulDoFnRunnerTest.java| 110 -
 .../wrappers/streaming/DoFnOperator.java| 111 -
 .../flink/streaming/DoFnOperatorTest.java   | 225 +++
 5 files changed, 472 insertions(+), 117 deletions(-)
--

[3/4] beam git commit: Properly deal with late processing-time timers

[kenn]

Properly deal with late processing-time timers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dbfcf4b4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dbfcf4b4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dbfcf4b4

Branch: refs/heads/master
Commit: dbfcf4b4a63b38653adc21d1cf37d6c4cfd955ad
Parents: 1a8e1f7
Author: Aljoscha Krettek 
Authored: Fri Mar 10 15:25:26 2017 +0100
Committer: Aljoscha Krettek 
Committed: Fri Mar 10 15:25:26 2017 +0100

--
 .../beam/runners/core/StatefulDoFnRunner.java   | 40 
 1 file changed, 24 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dbfcf4b4/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index c672902..d27193c 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -76,33 +76,31 @@ public class StatefulDoFnRunner
   }
 
   @Override
-  public void processElement(WindowedValue compressedElem) {
+  public void processElement(WindowedValue input) {
 
 // StatefulDoFnRunner always observes windows, so we need to explode
-for (WindowedValue value : compressedElem.explodeWindows()) {
+for (WindowedValue value : input.explodeWindows()) {
 
   BoundedWindow window = value.getWindows().iterator().next();
 
-  if (!dropLateData(window)) {
+  if (isLate(window)) {
+// The element is too late for this window.
+droppedDueToLateness.addValue(1L);
+WindowTracing.debug(
+"StatefulDoFnRunner.processElement: Dropping element at {}; 
window:{} "
++ "since too far behind inputWatermark:{}",
+input.getTimestamp(), window, 
cleanupTimer.currentInputWatermarkTime());
+  } else {
 cleanupTimer.setForWindow(window);
 doFnRunner.processElement(value);
   }
 }
   }
 
-  private boolean dropLateData(BoundedWindow window) {
+  private boolean isLate(BoundedWindow window) {
 Instant gcTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
 Instant inputWM = cleanupTimer.currentInputWatermarkTime();
-if (gcTime.isBefore(inputWM)) {
-  // The element is too late for this window.
-  droppedDueToLateness.addValue(1L);
-  WindowTracing.debug(
-  "StatefulDoFnRunner.processElement/onTimer: Dropping element for 
window:{} "
-  + "since too far behind inputWatermark:{}", window, inputWM);
-  return true;
-} else {
-  return false;
-}
+return gcTime.isBefore(inputWM);
   }
 
   @Override
@@ -112,8 +110,18 @@ public class StatefulDoFnRunner
   stateCleaner.clearForWindow(window);
   // There should invoke the onWindowExpiration of DoFn
 } else {
-  // a timer can never be late because we don't allow setting timers after 
GC time
-  doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
+  // An event-time timer can never be late because we don't allow setting 
timers after GC time.
+  // Ot can happen that a processing-time time fires for a late window, we 
need to ignore
+  // this.
+  if (!timeDomain.equals(TimeDomain.EVENT_TIME) && isLate(window)) {
+// don't increment the dropped counter, only do that for elements
+WindowTracing.debug(
+"StatefulDoFnRunner.onTimer: Ignoring processing-time timer at {}; 
window:{} "
++ "since window is too far behind inputWatermark:{}",
+timestamp, window, cleanupTimer.currentInputWatermarkTime());
+  } else {
+doFnRunner.onTimer(timerId, window, timestamp, timeDomain);
+  }
 }
   }
 

[1/4] beam git commit: Introduce Flink-specific state GC implementations

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 54390a333 -> 75fe559a4


Introduce Flink-specific state GC implementations

We now set the GC timer for window.maxTimestamp() + 1 to ensure that a
user timer set for window.maxTimestamp() still has all state.

This also adds tests for late data dropping and state GC specifically
for the Flink DoFnOperator.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1a8e1f74
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1a8e1f74
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1a8e1f74

Branch: refs/heads/master
Commit: 1a8e1f7463cbc7c6b5edfe1dbbc98502e5612511
Parents: bf6d274
Author: Aljoscha Krettek 
Authored: Fri Mar 10 11:07:00 2017 +0100
Committer: Aljoscha Krettek 
Committed: Fri Mar 10 11:09:04 2017 +0100

--
 .../apache/beam/runners/core/DoFnRunners.java   |  15 +-
 .../beam/runners/core/StatefulDoFnRunner.java   |  87 ---
 .../runners/core/StatefulDoFnRunnerTest.java| 110 -
 .../wrappers/streaming/DoFnOperator.java| 111 -
 .../flink/streaming/DoFnOperatorTest.java   | 225 +++
 5 files changed, 439 insertions(+), 109 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1a8e1f74/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 9455eea..a1b7c8b 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -21,9 +21,6 @@ import java.util.List;
 import org.apache.beam.runners.core.ExecutionContext.StepContext;
 import org.apache.beam.runners.core.StatefulDoFnRunner.CleanupTimer;
 import org.apache.beam.runners.core.StatefulDoFnRunner.StateCleaner;
-import 
org.apache.beam.runners.core.StatefulDoFnRunner.StateInternalsStateCleaner;
-import 
org.apache.beam.runners.core.StatefulDoFnRunner.TimeInternalsCleanupTimer;
-import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -135,18 +132,13 @@ public class DoFnRunners {
   DoFnRunner doFnRunner,
   StepContext stepContext,
   AggregatorFactory aggregatorFactory,
-  WindowingStrategy windowingStrategy) {
+  WindowingStrategy windowingStrategy,
+  CleanupTimer cleanupTimer,
+  StateCleaner stateCleaner) {
 Aggregator droppedDueToLateness = 
aggregatorFactory.createAggregatorForDoFn(
 fn.getClass(), stepContext, 
StatefulDoFnRunner.DROPPED_DUE_TO_LATENESS_COUNTER,
 Sum.ofLongs());
 
-CleanupTimer cleanupTimer =
-new TimeInternalsCleanupTimer(stepContext.timerInternals(), 
windowingStrategy);
-
-Coder windowCoder = (Coder) 
windowingStrategy.getWindowFn().windowCoder();
-StateCleaner stateCleaner =
-new StateInternalsStateCleaner<>(fn, stepContext.stateInternals(), 
windowCoder);
-
 return new StatefulDoFnRunner<>(
 doFnRunner,
 windowingStrategy,
@@ -154,5 +146,4 @@ public class DoFnRunners {
 stateCleaner,
 droppedDueToLateness);
   }
-
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/1a8e1f74/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index 926345e..c672902 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -17,12 +17,8 @@
  */
 package org.apache.beam.runners.core;
 
-import java.util.Map;
-import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -30,8 +26,6 @@ import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.WindowTracing;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingSt

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2868

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #1198

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1474

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1674) Flink user state GC depends on order of timer firing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905616#comment-15905616
 ] 

ASF GitHub Bot commented on BEAM-1674:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2210


> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Flink #1886

[Apache Jenkins Server]

See 


--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/beam.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:806)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/beam.git 
+refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*" 
returned status code 128:
stdout: 
stderr: error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1793)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1519)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to beam1(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:822)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy97.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:804)
... 11 more
ERROR: null
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/beam.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:806)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at hudson.scm.SCM

[GitHub] beam pull request #2210: [BEAM-1674] Add unicode type to the typeDict attrib...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2210


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: add unicode type to the typeDict attribute in Python SDK

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 818fc9412 -> 54390a333


add unicode type to the typeDict attribute in Python SDK


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7daf9abd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7daf9abd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7daf9abd

Branch: refs/heads/master
Commit: 7daf9abd52d4b89c563d370396506d2db71b7700
Parents: 818fc94
Author: tajima 
Authored: Thu Mar 9 18:08:25 2017 +0900
Committer: Ahmet Altay 
Committed: Fri Mar 10 11:56:06 2017 -0800

--
 sdks/python/apache_beam/transforms/display.py  |  1 +
 sdks/python/apache_beam/transforms/display_test.py | 11 +++
 2 files changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7daf9abd/sdks/python/apache_beam/transforms/display.py
--
diff --git a/sdks/python/apache_beam/transforms/display.py 
b/sdks/python/apache_beam/transforms/display.py
index 5e25060..2ced1af 100644
--- a/sdks/python/apache_beam/transforms/display.py
+++ b/sdks/python/apache_beam/transforms/display.py
@@ -160,6 +160,7 @@ class DisplayDataItem(object):
   display item belongs to.
   """
   typeDict = {str:'STRING',
+  unicode:'STRING',
   int:'INTEGER',
   float:'FLOAT',
   bool: 'BOOLEAN',

http://git-wip-us.apache.org/repos/asf/beam/blob/7daf9abd/sdks/python/apache_beam/transforms/display_test.py
--
diff --git a/sdks/python/apache_beam/transforms/display_test.py 
b/sdks/python/apache_beam/transforms/display_test.py
index 5a95c42..5e106e5 100644
--- a/sdks/python/apache_beam/transforms/display_test.py
+++ b/sdks/python/apache_beam/transforms/display_test.py
@@ -122,6 +122,17 @@ class DisplayDataTest(unittest.TestCase):
 DisplayDataItemMatcher('extra_packages',
str(['package1', 'package2']
 
+  def test_unicode_type_display_data(self):
+class MyDoFn(beam.DoFn):
+  def display_data(self):
+return {'unicode_string': unicode('my string'),
+'unicode_literal_string': u'my literal string'}
+
+fn = MyDoFn()
+dd = DisplayData.create_from(fn)
+for item in dd.items:
+  self.assertEqual(item.type, 'STRING')
+
   def test_base_cases(self):
 """ Tests basic display data cases (key:value, key:dict)
 It does not test subcomponent inclusion

[2/2] beam git commit: This closes #2210

[altay]

This closes #2210


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/54390a33
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/54390a33
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/54390a33

Branch: refs/heads/master
Commit: 54390a3338b2216ca82cb9c8e8acd81711068691
Parents: 818fc94 7daf9ab
Author: Ahmet Altay 
Authored: Fri Mar 10 11:56:40 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Mar 10 11:56:40 2017 -0800

--
 sdks/python/apache_beam/transforms/display.py  |  1 +
 sdks/python/apache_beam/transforms/display_test.py | 11 +++
 2 files changed, 12 insertions(+)
--

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #1197

[Apache Jenkins Server]

See 


--
[...truncated 635.00 KB...]
2017-03-10T19:46:20.852 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/typesafe/akka/akka-actor_2.10/2.3.11/akka-actor_2.10-2.3.11.jar
 (2596 KB at 2460.5 KB/sec)
2017-03-10T19:46:20.852 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar
2017-03-10T19:46:20.865 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scalap/2.10.0/scalap-2.10.0.jar
 (835 KB at 781.8 KB/sec)
2017-03-10T19:46:20.865 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar
2017-03-10T19:46:20.936 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar
 (171 KB at 149.5 KB/sec)
2017-03-10T19:46:20.936 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar
2017-03-10T19:46:20.941 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/mesos/mesos/0.21.1/mesos-0.21.1-shaded-protobuf.jar
 (1248 KB at 1090.9 KB/sec)
2017-03-10T19:46:20.941 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-json/3.1.2/metrics-json-3.1.2.jar
2017-03-10T19:46:20.968 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar
 (39 KB at 32.8 KB/sec)
2017-03-10T19:46:20.968 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-graphite/3.1.2/metrics-graphite-3.1.2.jar
2017-03-10T19:46:20.969 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-json/3.1.2/metrics-json-3.1.2.jar
 (16 KB at 13.2 KB/sec)
2017-03-10T19:46:20.969 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.10/2.4.4/jackson-module-scala_2.10-2.4.4.jar
2017-03-10T19:46:20.998 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/dropwizard/metrics/metrics-graphite/3.1.2/metrics-graphite-3.1.2.jar
 (21 KB at 17.0 KB/sec)
2017-03-10T19:46:20.998 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.4/scala-reflect-2.10.4.jar
2017-03-10T19:46:21.069 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.10/2.4.4/jackson-module-scala_2.10-2.4.4.jar
 (537 KB at 421.8 KB/sec)
2017-03-10T19:46:21.069 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar
2017-03-10T19:46:21.120 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar
 (2007 KB at 1516.8 KB/sec)
2017-03-10T19:46:21.120 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-client/0.8.2/tachyon-client-0.8.2.jar
2017-03-10T19:46:21.262 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.10/1.6.3/spark-core_2.10-1.6.3.jar
 (10968 KB at 7460.6 KB/sec)
2017-03-10T19:46:21.263 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-hdfs/0.8.2/tachyon-underfs-hdfs-0.8.2.jar
2017-03-10T19:46:21.263 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar 
(1253 KB at 854.3 KB/sec)
2017-03-10T19:46:21.264 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-s3/0.8.2/tachyon-underfs-s3-0.8.2.jar
2017-03-10T19:46:21.298 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-hdfs/0.8.2/tachyon-underfs-hdfs-0.8.2.jar
 (11 KB at 7.2 KB/sec)
2017-03-10T19:46:21.298 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-local/0.8.2/tachyon-underfs-local-0.8.2.jar
2017-03-10T19:46:21.327 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-local/0.8.2/tachyon-underfs-local-0.8.2.jar
 (8 KB at 4.6 KB/sec)
2017-03-10T19:46:21.327 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
2017-03-10T19:46:21.368 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
 (92 KB at 58.1 KB/sec)
2017-03-10T19:46:21.368 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-s3/0.8.2/tachyon-underfs-s3-0.8.2.jar
 (494 KB at 314.2 KB/sec)
2017-03-10T19:46:21.368 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
2017-03-10T19:46:21.369 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/sf/py4j/py4j/0.9/py4j-0.9.jar
2017-03-10T19:46:21.395 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/spark-p

[jira] [Created] (BEAM-1688) Getting listener timeout after waiting for [10000] ms from ElasticsearchIO

[Sathish Jayaraman (JIRA)]

Sathish Jayaraman created BEAM-1688:
---

 Summary: Getting listener timeout after waiting for [1] ms 
from ElasticsearchIO
 Key: BEAM-1688
 URL: https://issues.apache.org/jira/browse/BEAM-1688
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 0.5.0
 Environment: Linux,  ElasticSearch 5.2.1
Reporter: Sathish Jayaraman
Assignee: Davor Bonaci
Priority: Minor


I am trying to test a simple Apache Beam code with source as Elasticsearch. I 
modified Beam's MinimalWordCount example to include source as Elasticsearch 
instead of TextIO. 

When I run the code,  I get below error, 
An exception occured while executing the Java class. null: 
InvocationTargetException: java.io.IOException: listener timeout after waiting 
for [1] ms

I debugged ElasticsearchIO.java and could see everything works fine, the 
Elasicsearch client is built & data from index is being retrieved by the code. 
But ParDo function after read transform does not get executed at all. The 
Elasticsearch client keeps waiting and finally gets a timeout error.

Any idea about the issue? 

Have raised issue here too 
http://stackoverflow.com/questions/42720857/getting-listener-timeout-after-waiting-for-1-ms-from-elasticsearchio



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1473

[Apache Jenkins Server]

See 


--
[...truncated 1001.25 KB...]
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/ToVoidKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s5"
}, 
"serialized_fn": "", 
"user_name": "assert_that/ToVoidKey"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s7", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference",