date:20181004

[jira] [Commented] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639308#comment-16639308
 ] 

Valentyn Tymofieiev commented on BEAM-5620:
---

It's possible although in this case some of our test classes will inherit from 
unittest.TestCase, while others will inherit from something like 
apache_beam.testing.util.TestCase, which may cause confusion. Everytime someone 
reading the code would see the latter, they may be wondering why do we have two 
classes, and which one needs to be used. 

While if we have a patch in an individual class, the goal is self-explanatory, 
but I agree that this is not a perfect solution.

Another possibility is to use six.assertCountEqual, but according to the 
approach we have chosen in Beam [1] , we'd like to use native idioms of Python 
3 when possible. This is why we eliminated the usages of six, and removed this 
dependency. 

[1] 
https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #238

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[iemejia] [BEAM-5532] Update Spark runner to Spark version 2.3.2

--
[...truncated 51.02 MB...]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16) 
(5ba3f24b178f1cd83979303afcbafa49).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16) 
(893c541c67cb9f1e64a1e6bfad10b3b8) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16) 
(893c541c67cb9f1e64a1e6bfad10b3b8).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16) 
(5ba3f24b178f1cd83979303afcbafa49) [FINISHED]
[ToKeyedWorkItem (8/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (8/16) 
(2d873127ae7b96fe42f163c6cfecc54f) [FINISHED]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (8/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (8/16) 
(e2907667f9d9d0163bb57871fa1b8fc4).
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
d969550c6a8fa4a3d0b706c26c74eed8.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (8/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (8/16) 
(e2907667f9d9d0163bb57871fa1b8fc4) [FINISHED]
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (6/16) 
(16b8460040a6b309be19e38969b506f2) [FINISHED]
[ToKeyedWorkItem (3/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (3/16) 
(277192f3a88fc85ff66f4b8bda307ac9).
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (10/16) 
(6bd8364f9427fef0ee8508533da70219) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
0eacf31c7b381a0ec713816851edd65d.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (4/16) (f7c45152c1f9b56f772dc40dce6fd5f9) switched from RUNNING 
to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
da167749534fe669eac51cc3bf0a27ef.
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (2/16) 
(6d7acadae6c09acf3c126f0670c6059b).
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (5/16) 
(d85e701224d9fb14f47d67b80b5cbe7f) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (4/16) 
(f7c45152c1f9b56f772dc40dce6fd5f9).
[ToKeyedWorkItem (3/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (3/16) 
(277192f3a88fc85ff66f4b8bda307ac9) [FINISHED]
[ToKeyedWorkItem (9/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (9/16) (04a91a02e463be27f0cb81d48a91d13b) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (9/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (9/16) 
(04a91a02e463be27f0cb81d48a91d13b).
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (4/16) 
(f7c45152c1f9b56f772dc40dce6fd5f9) [FINISHED]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16)] 
INFO

[beam] branch asf-site updated: Publishing website 2018/10/05 05:38:08 at commit fed2d1d

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new ae0e876  Publishing website 2018/10/05 05:38:08 at commit fed2d1d
ae0e876 is described below

commit ae0e87611676b1ed98ca2a7578efdae16358d1e6
Author: jenkins 
AuthorDate: Fri Oct 5 05:38:09 2018 +

Publishing website 2018/10/05 05:38:08 at commit fed2d1d

[jira] [Resolved] (BEAM-5532) Update Spark runner to Spark version 2.3.2

2018-10-04 Thread JIRA



 [ 
https://issues.apache.org/jira/browse/BEAM-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-5532.

Resolution: Fixed

> Update Spark runner to Spark version 2.3.2
> --
>
> Key: BEAM-5532
> URL: https://issues.apache.org/jira/browse/BEAM-5532
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Latest release of Spark runner seems to fix some data corruption bugs 
> including SPARK-23243, So worth the upgrade even if the dependency is mostly 
> provided for our use case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6568 from iemejia/BEAM-5532-spark-runner

2018-10-04 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fed2d1d85d8077c59fbff52d207d32453e06f5bf
Merge: dae9a52 c75d06a
Author: Jean-Baptiste Onofré 
AuthorDate: Fri Oct 5 07:36:34 2018 +0200

Merge pull request #6568 from iemejia/BEAM-5532-spark-runner

[BEAM-5532] Update Spark runner to Spark version 2.3.2

 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5532) Update Spark runner to Spark version 2.3.2

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5532?focusedWorklogId=151499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151499
 ]

ASF GitHub Bot logged work on BEAM-5532:


Author: ASF GitHub Bot
Created on: 05/Oct/18 05:36
Start Date: 05/Oct/18 05:36
Worklog Time Spent: 10m 
  Work Description: jbonofre closed pull request #6568: [BEAM-5532] Update 
Spark runner to Spark version 2.3.2
URL: https://github.com/apache/beam/pull/6568
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 454eb025c05..648424d5cc9 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -312,7 +312,7 @@ class BeamModulePlugin implements Plugin {
 def hamcrest_version = "1.3"
 def hadoop_version = "2.7.3"
 def jackson_version = "2.9.5"
-def spark_version = "2.3.1"
+def spark_version = "2.3.2"
 def apex_core_version = "3.7.0"
 def apex_malhar_version = "3.4.0"
 def postgres_version = "42.2.2"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151499)
Time Spent: 50m  (was: 40m)

> Update Spark runner to Spark version 2.3.2
> --
>
> Key: BEAM-5532
> URL: https://issues.apache.org/jira/browse/BEAM-5532
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Latest release of Spark runner seems to fix some data corruption bugs 
> including SPARK-23243, So worth the upgrade even if the dependency is mostly 
> provided for our use case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (dae9a52 -> fed2d1d)

2018-10-04 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from dae9a52  Merge pull request #6579: [BEAM-2687] Skip test_pardo_timers 
for Python Flink VR
 add c75d06a  [BEAM-5532] Update Spark runner to Spark version 2.3.2
 new fed2d1d  Merge pull request #6568 from iemejia/BEAM-5532-spark-runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Assigned] (BEAM-5663) Add tox suites for various Python 3 versions

2018-10-04 Thread Manu Zhang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manu Zhang reassigned BEAM-5663:


Assignee: (was: Ahmet Altay)

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5663) Add tox suites for various Python 3 versions

2018-10-04 Thread Manu Zhang (JIRA)

Manu Zhang created BEAM-5663:


 Summary: Add tox suites for various Python 3 versions
 Key: BEAM-5663
 URL: https://issues.apache.org/jira/browse/BEAM-5663
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Manu Zhang
Assignee: Ahmet Altay


Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
failings across various Python 3 versions. It will be valuable to add tox 
suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2687) Python SDK support for Stateful Processing

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2687?focusedWorklogId=151444=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151444
 ]

ASF GitHub Bot logged work on BEAM-2687:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:05
Start Date: 05/Oct/18 00:05
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #6579: [BEAM-2687] 
Skip test_pardo_timers for Python Flink VR
URL: https://github.com/apache/beam/pull/6579
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151444)
Time Spent: 6h 40m  (was: 6.5h)

> Python SDK support for Stateful Processing
> --
>
> Key: BEAM-2687
> URL: https://issues.apache.org/jira/browse/BEAM-2687
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee:

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1259

2018-10-04 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-2953) Create more advanced Timeseries processing examples using state API

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2953?focusedWorklogId=151486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151486
 ]

ASF GitHub Bot logged work on BEAM-2953:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:45
Start Date: 05/Oct/18 01:45
Worklog Time Spent: 10m 
  Work Description: rezarokni commented on a change in pull request #6540: 
[BEAM-2953] Advanced Timeseries examples.
URL: https://github.com/apache/beam/pull/6540#discussion_r222873431
 
 

 ##
 File path: 
examples/java/src/main/java/org/apache/beam/examples/timeseries/TimeSeriesExampleToBigTable.java
 ##
 @@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.examples.timeseries;
+
+import com.google.cloud.bigtable.beam.CloudBigtableIO;
+import com.google.cloud.bigtable.beam.CloudBigtableTableConfiguration;
+import com.google.protobuf.util.Timestamps;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.examples.timeseries.configuration.TSConfiguration;
+import org.apache.beam.examples.timeseries.protos.TimeSeriesData;
+import org.apache.beam.examples.timeseries.transforms.*;
+import org.apache.beam.examples.timeseries.utils.TSAccumSequences;
+import org.apache.beam.examples.timeseries.utils.TSAccums;
+import org.apache.beam.examples.timeseries.utils.TSMultiVariateDataPoints;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This example pipeline is used to illustrate an advanced use of Keyed state 
and timers. The
+ * pipeline extracts interesting information from timeseries data. One of the 
key elements, is the
+ * transfer of data between fixed windows for a given key, as well as backfill 
when a key does not
+ * have any new data within a time boundary. This sample should not be used in 
production.
+ *
+ * The output of this pipeline is to Google Cloud Bigtable.
+ */
+@Experimental
 
 Review comment:
   As discussed, I will be moving the whole library to extensions. Shall I keep 
@Experimental in that context?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151486)
Time Spent: 2h 20m  (was: 2h 10m)

> Create more advanced Timeseries processing examples using state API
> ---
>
> Key: BEAM-2953
> URL: https://issues.apache.org/jira/browse/BEAM-2953
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Affects Versions: 2.1.0
>Reporter: Reza ardeshir rokni
>Assignee: Reuven Lax
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> As described in the phase 1 portion of this solution outline:
> https://cloud.google.com/solutions/correlating-time-series-dataflow
> BEAM can be used to build out some very interesting pre-processing stages for 
> time series data. Some examples that will be useful:
> - Downsampling time series based on simple AVG, MIN, MAX
> - Creating a value for each time window using generatesequence as a seed 
> - Loading the value of a downsample with the previous value (used in FX with 
> previous close being brought into current open value) 
> This will show some concrete examples of keyed state as well as the use of 
> combiners. 
> The samples can also be used to show how you can create a ordered list of

[jira] [Work logged] (BEAM-2953) Create more advanced Timeseries processing examples using state API

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2953?focusedWorklogId=151489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151489
 ]

ASF GitHub Bot logged work on BEAM-2953:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:50
Start Date: 05/Oct/18 01:50
Worklog Time Spent: 10m 
  Work Description: rezarokni commented on a change in pull request #6540: 
[BEAM-2953] Advanced Timeseries examples.
URL: https://github.com/apache/beam/pull/6540#discussion_r222874005
 
 

 ##
 File path: examples/java/src/main/proto/TimeSeriesData.proto
 ##
 @@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
KIND, either express
+ * or implied. See the License for the specific language governing permissions 
and limitations under
+ * the License.
+ */
+syntax = "proto3";
+
+package protos;
+
+import "google/protobuf/duration.proto";
+import "google/protobuf/timestamp.proto";
+option java_outer_classname = "TimeSeriesData";
+option java_multiple_files = false;
+option java_package = "org.apache.beam.examples.timeseries.protos";
+
+// The TimeSeriesData proto type will be used as the internal 
representation of all timeseries within the the TS
 
 Review comment:
   The expectation is that downstream the data can be used in other languages ( 
mainly python). 
   
   Utility class for each of the main protos. In similar style to Timestamp 
with protobuff which has utils in Timestamps. Each of the protos have a (s) 
util class.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151489)
Time Spent: 2h 40m  (was: 2.5h)

> Create more advanced Timeseries processing examples using state API
> ---
>
> Key: BEAM-2953
> URL: https://issues.apache.org/jira/browse/BEAM-2953
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Affects Versions: 2.1.0
>Reporter: Reza ardeshir rokni
>Assignee: Reuven Lax
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> As described in the phase 1 portion of this solution outline:
> https://cloud.google.com/solutions/correlating-time-series-dataflow
> BEAM can be used to build out some very interesting pre-processing stages for 
> time series data. Some examples that will be useful:
> - Downsampling time series based on simple AVG, MIN, MAX
> - Creating a value for each time window using generatesequence as a seed 
> - Loading the value of a downsample with the previous value (used in FX with 
> previous close being brought into current open value) 
> This will show some concrete examples of keyed state as well as the use of 
> combiners. 
> The samples can also be used to show how you can create a ordered list of 
> values per key from a unbounded topic which has multiple time series keys. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2953) Create more advanced Timeseries processing examples using state API

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2953?focusedWorklogId=151488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151488
 ]

ASF GitHub Bot logged work on BEAM-2953:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:48
Start Date: 05/Oct/18 01:48
Worklog Time Spent: 10m 
  Work Description: rezarokni commented on a change in pull request #6540: 
[BEAM-2953] Advanced Timeseries examples.
URL: https://github.com/apache/beam/pull/6540#discussion_r222873827
 
 

 ##
 File path: 
examples/java/src/main/java/org/apache/beam/examples/timeseries/TimeSeriesExampleToBigTable.java
 ##
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.examples.timeseries;
+
+import com.google.cloud.bigtable.beam.CloudBigtableIO;
+import com.google.cloud.bigtable.beam.CloudBigtableTableConfiguration;
+import com.google.protobuf.util.Timestamps;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.examples.timeseries.configuration.TSConfiguration;
+import org.apache.beam.examples.timeseries.protos.TimeSeriesData;
+import org.apache.beam.examples.timeseries.transforms.*;
+import org.apache.beam.examples.timeseries.utils.TSAccumSequences;
+import org.apache.beam.examples.timeseries.utils.TSAccums;
+import org.apache.beam.examples.timeseries.utils.TSMultiVariateDataPoints;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This example pipeline is used to illustrate an advanced use of Keyed state 
and timers. The
+ * pipeline extracts interesting information from timeseries data. One of the 
key elements, is the
+ * transfer of data between fixed windows for a given key, as well as backfill 
when a key does not
+ * have any new data within a time boundary. This sample should not be used in 
production.
+ *
+ * The output of this pipeline is to Google Cloud Bigtable.
+ */
+@Experimental
+public class TimeSeriesExampleToBigTable {
+
+  static final Logger LOG = 
LoggerFactory.getLogger(TimeSeriesExampleToBigTable.class);
+
+  static final String FILE_LOCATION = "/tmp/tf/";
+
+  /** Push Timeseries data into BigTable */
+  public static void main(String[] args) {
+
+// Create pipeline
+TimeSeriesOptions options =
+
PipelineOptionsFactory.fromArgs(args).withValidation().as(TimeSeriesOptions.class);
+
+TSConfiguration configuration =
+TSConfiguration.builder()
+.downSampleDuration(Duration.standardSeconds(5))
+.timeToLive(Duration.standardMinutes(1))
+.fillOption(TSConfiguration.BFillOptions.LAST_KNOWN_VALUE);
+
+Pipeline p = Pipeline.create(options);
+
+// [START bigtable_dataflow_connector_config]
+
+CloudBigtableTableConfiguration config =
+new CloudBigtableTableConfiguration.Builder()
+.withProjectId(options.getProjectId())
+.withInstanceId(options.getBigTableInstanceId())
+.withTableId(options.getBigTableTableId())
+.build();
+
+// [END bigtable_dataflow_connector_config]
+
+//  READ DATA 
+
+// Read some dummy timeseries data
+PCollection> readData 
=
+p.apply(Create.of(SinWaveSample.generateSinWave()))
+.apply(ParDo.of(new TSMultiVariateDataPoints.ExtractTimeStamp()))
+.apply(ParDo.of(new 
TSMultiVariateDataPoints.ConvertMultiToUniDataPoint()));
+
+//  Create perfect rectangles of data
+
+PCollection> downSampled =
+readData.apply(new ExtractAggregates(configuration)).apply(new 
GetWindowData());
+
+PCollection> weHaveOrder =
+downSampled.apply(new OrderOutput(configuration));
+
+//  OutPut Data as

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1258

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[tweise] [BEAM-5528] Re-add CREATE_VIEW in portable Flink translators (#6523)

--
[...truncated 79.19 KB...]
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r

[jira] [Work logged] (BEAM-5634) Bring Dataflow Java Worker Code into Beam

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5634?focusedWorklogId=151462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151462
 ]

ASF GitHub Bot logged work on BEAM-5634:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:46
Start Date: 05/Oct/18 00:46
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #6561: [BEAM-5634] Bring 
dataflow java worker code into beam
URL: https://github.com/apache/beam/pull/6561#issuecomment-427213084
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151462)
Time Spent: 40m  (was: 0.5h)

> Bring Dataflow Java Worker Code into Beam
> -
>
> Key: BEAM-5634
> URL: https://issues.apache.org/jira/browse/BEAM-5634
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Discussion: 
> https://lists.apache.org/thread.html/89efd3bc1d30f3d43d4b361a5ee05bd52778c9dc3f43ac72354c2bd9@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151472
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:23
Start Date: 05/Oct/18 01:23
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6563: [BEAM-4176] Cleanup 
SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#issuecomment-427218323
 
 
   @angoenka for future PRs, please tag the JIRA in the commit messages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151472)
Time Spent: 29h 10m  (was: 29h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #237

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[thw] [BEAM-2687] Skip test_pardo_timers for Python Flink VR

--
[...truncated 487.25 KB...]
ERROR: test_pardo (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 67, in 
test_pardo
with self.create_pipeline() as p:
  File "apache_beam/runners/portability/portable_runner_test.py", line 170, in 
create_pipeline
return beam.Pipeline(self.get_runner(), self.create_options())
  File 
"
 line 66, in create_options
options = super(FlinkRunnerTest, self).create_options()
  File "apache_beam/runners/portability/portable_runner_test.py", line 166, in 
create_options
options.view_as(PortableOptions).job_endpoint = self._get_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 125, in 
_get_job_endpoint
cls._job_endpoint = cls._create_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 131, in 
_create_job_endpoint
return cls._start_local_runner_subprocess_job_service()
  File "apache_beam/runners/portability/portable_runner_test.py", line 106, in 
_start_local_runner_subprocess_job_service
cls._subprocess.returncode)
RuntimeError: Subprocess terminated unexpectedly with exit code 0.

==
ERROR: test_pardo_metrics (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 94, in 
test_pardo_metrics
with self.create_pipeline() as p:
  File "apache_beam/runners/portability/portable_runner_test.py", line 170, in 
create_pipeline
return beam.Pipeline(self.get_runner(), self.create_options())
  File 
"
 line 66, in create_options
options = super(FlinkRunnerTest, self).create_options()
  File "apache_beam/runners/portability/portable_runner_test.py", line 166, in 
create_options
options.view_as(PortableOptions).job_endpoint = self._get_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 125, in 
_get_job_endpoint
cls._job_endpoint = cls._create_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 131, in 
_create_job_endpoint
return cls._start_local_runner_subprocess_job_service()
  File "apache_beam/runners/portability/portable_runner_test.py", line 106, in 
_start_local_runner_subprocess_job_service
cls._subprocess.returncode)
RuntimeError: Subprocess terminated unexpectedly with exit code 0.

==
ERROR: test_pardo_side_and_main_outputs (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 132, in 
test_pardo_side_and_main_outputs
with self.create_pipeline() as p:
  File "apache_beam/runners/portability/portable_runner_test.py", line 170, in 
create_pipeline
return beam.Pipeline(self.get_runner(), self.create_options())
  File 
"
 line 66, in create_options
options = super(FlinkRunnerTest, self).create_options()
  File "apache_beam/runners/portability/portable_runner_test.py", line 166, in 
create_options
options.view_as(PortableOptions).job_endpoint = self._get_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 125, in 
_get_job_endpoint
cls._job_endpoint = cls._create_job_endpoint()
  File "apache_beam/runners/portability/portable_runner_test.py", line 131, in 
_create_job_endpoint
return cls._start_local_runner_subprocess_job_service()
  File "apache_beam/runners/portability/portable_runner_test.py", line 106, in 
_start_local_runner_subprocess_job_service
cls._subprocess.returncode)
RuntimeError: Subprocess terminated unexpectedly with exit code 0.

==
ERROR: test_pardo_side_inputs (__main__.FlinkRunnerTest)
--
Traceback (most recent call last):
  File "apache_beam/runners/portability/fn_api_runner_test.py", line 152, in 
test_pardo_side_inputs
with self.create_pipeline() as p:
  File

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=151469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151469
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:11
Start Date: 05/Oct/18 01:11
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6557: [BEAM-5442] Pass 
unknown SDK pipeline options to Runner
URL: https://github.com/apache/beam/pull/6557#issuecomment-427216617
 
 
   Run Python Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151469)
Time Spent: 3.5h  (was: 3h 20m)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=151471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151471
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:16
Start Date: 05/Oct/18 01:16
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6557: [BEAM-5442] Pass 
unknown SDK pipeline options to Runner
URL: https://github.com/apache/beam/pull/6557#issuecomment-427217396
 
 
   @mxm FYI the VR job and trigger phrase: 
https://github.com/apache/beam/blob/6b8f65df78d9b68c3c09e06bd54bd70fe362b897/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Flink.groovy#L24


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151471)
Time Spent: 3h 40m  (was: 3.5h)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/05 01:15:27 at commit dae9a52

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 7a89eea  Publishing website 2018/10/05 01:15:27 at commit dae9a52
7a89eea is described below

commit 7a89eea72d09e6d89a608cb657d0908112a07e97
Author: jenkins 
AuthorDate: Fri Oct 5 01:15:27 2018 +

Publishing website 2018/10/05 01:15:27 at commit dae9a52

[beam] 01/01: Merge pull request #6579: [BEAM-2687] Skip test_pardo_timers for Python Flink VR

2018-10-04 Thread thw

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dae9a522ca5aaea962da2a25ffc2da2220964a41
Merge: b57283b ed01d65
Author: Thomas Weise 
AuthorDate: Thu Oct 4 18:13:29 2018 -0700

Merge pull request #6579: [BEAM-2687] Skip test_pardo_timers for Python 
Flink VR

 sdks/python/apache_beam/runners/portability/flink_runner_test.py | 3 +++
 1 file changed, 3 insertions(+)

[jira] [Work logged] (BEAM-2687) Python SDK support for Stateful Processing

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2687?focusedWorklogId=151470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151470
 ]

ASF GitHub Bot logged work on BEAM-2687:


Author: ASF GitHub Bot
Created on: 05/Oct/18 01:13
Start Date: 05/Oct/18 01:13
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6579: [BEAM-2687] Skip 
test_pardo_timers for Python Flink VR
URL: https://github.com/apache/beam/pull/6579
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/portability/flink_runner_test.py 
b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
index 09261c9030b..044e0e9a913 100644
--- a/sdks/python/apache_beam/runners/portability/flink_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
@@ -81,6 +81,9 @@ def test_no_subtransform_composite(self):
 def test_pardo_state_only(self):
   raise unittest.SkipTest("BEAM-2918 - User state not yet supported.")
 
+def test_pardo_timers(self):
+  raise unittest.SkipTest("BEAM-4681 - User timers not yet supported.")
+
 # Inherits all other tests.
 
   # Run the tests.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151470)
Time Spent: 7h  (was: 6h 50m)

> Python SDK support for Stateful Processing
> --
>
> Key: BEAM-2687
> URL: https://issues.apache.org/jira/browse/BEAM-2687
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Python SDK should support stateful processing 
> (https://beam.apache.org/blog/2017/02/13/stateful-processing.html)
> In the meantime, runner capability matrix should be updated to show the lack 
> of this feature 
> (https://beam.apache.org/documentation/runners/capability-matrix/)
> Use this as an umbrella issue for all related issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (b57283b -> dae9a52)

2018-10-04 Thread thw

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b57283b  [BEAM-5528] Re-add CREATE_VIEW in portable Flink translators 
(#6523)
 add ed01d65  [BEAM-2687] Skip test_pardo_timers for Python Flink VR
 new dae9a52  Merge pull request #6579: [BEAM-2687] Skip test_pardo_timers 
for Python Flink VR

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/portability/flink_runner_test.py | 3 +++
 1 file changed, 3 insertions(+)

[jira] [Work logged] (BEAM-5634) Bring Dataflow Java Worker Code into Beam

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5634?focusedWorklogId=151463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151463
 ]

ASF GitHub Bot logged work on BEAM-5634:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:47
Start Date: 05/Oct/18 00:47
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #6561: [BEAM-5634] Bring 
dataflow java worker code into beam
URL: https://github.com/apache/beam/pull/6561#issuecomment-427213270
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151463)
Time Spent: 50m  (was: 40m)

> Bring Dataflow Java Worker Code into Beam
> -
>
> Key: BEAM-5634
> URL: https://issues.apache.org/jira/browse/BEAM-5634
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Discussion: 
> https://lists.apache.org/thread.html/89efd3bc1d30f3d43d4b361a5ee05bd52778c9dc3f43ac72354c2bd9@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151453
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222857134
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -49,7 +49,7 @@ task createDockerContainer(type: Exec) {
 return standardOutput.toString().trim()
   }
   commandLine '/bin/bash', '-c',
-"docker create -v $project.rootDir:$dockerWorkDir -u \$(id -u):\$(id -g) 
$dockerImageTag"
+"docker create -v $project.rootDir:$dockerWorkDir -u \$(id -u):\$(id -g) 
-p 127.0.0.1:4000:4000 $dockerImageTag"
 
 Review comment:
   It will fail if two of these tasks try to run concurrently (could happen on 
Jenkins). I added a property to enable this port publishing, which should also 
reduce any possible attack vectors on Jenkins machines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151453)
Time Spent: 3h 40m  (was: 3.5h)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4502) Add redirect notice on apache/beam-site README.md

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4502?focusedWorklogId=151464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151464
 ]

ASF GitHub Bot logged work on BEAM-4502:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:53
Start Date: 05/Oct/18 00:53
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #567: [BEAM-4502] Add 
deprecation notices
URL: https://github.com/apache/beam-site/pull/567#issuecomment-427214022
 
 
   @swegner PTAL. Let's merge this (and delete the branch when ready).
   I'll create another PR against master that directs to apache/beam.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151464)
Time Spent: 1h 10m  (was: 1h)

> Add redirect notice on apache/beam-site README.md
> -
>
> Key: BEAM-4502
> URL: https://issues.apache.org/jira/browse/BEAM-4502
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151455=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151455
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222860478
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
-Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+* Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+  * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into 
some temporary location. Call this `${JAVADOC_TMP}`.
+  * Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} content/documentation/sdks/javadoc/${RELEASE}`.
+* Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+  * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} 
content/documentation/sdks/pydoc/${RELEASE}`.
+  * Remove `.doctrees` directory.
+* Set up the necessary git commands to account for new and deleted files.
 
 Review comment:
   Yeah, I believe these should be brand new directories, so no deletions are 
necessary.
   (I copied this text from elsewhere.)
   
   https://stackoverflow.com/a/26039014/63302


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151455)
Time Spent: 3h 50m  (was: 3h 40m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151448
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222857674
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -167,8 +167,23 @@ please submit your GPG public key into [MIT PGP Public 
Key Server](http://pgp.mi
 
  Website development setup
 
-Prepare to update the Beam website by following the [website development
-instructions]({{ site.baseurl }}/contribute/website-contributions/).
+Updating the Beam website requires access to both the man `apache/beam` repo 
and
+the `apache/beam-site` repo. The first contains most of the website sources,
+while the second contains reference manual generated from SDK code.
+
+You should already have setup access to `apache/beam`. Setting up access to
 
 Review comment:
   Clarified access to be 'submitting a PR'.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151448)
Time Spent: 2h 50m  (was: 2h 40m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151450
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222856901
 
 

 ##
 File path: website/README.md
 ##
 @@ -68,6 +80,12 @@ Before sending the PR for review, please run:
 
 $ git checkout -- content
 
+## Website generation
+
+After a PR is merged, a background Jenkins job will run and update the 
website's
+[generated-content
+directory](https://github.com/apache/beam/tree/asf-site/website/generated-content).
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151450)
Time Spent: 3h 10m  (was: 3h)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151449
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222860561
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
-Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+* Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+  * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into 
some temporary location. Call this `${JAVADOC_TMP}`.
+  * Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} content/documentation/sdks/javadoc/${RELEASE}`.
+* Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+  * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} 
content/documentation/sdks/pydoc/${RELEASE}`.
+  * Remove `.doctrees` directory.
+* Set up the necessary git commands to account for new and deleted files.
 
-* Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into some 
temporary location. Call this `${JAVADOC_TMP}`.
-* Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} src/documentation/sdks/javadoc/${RELEASE}`.
-* Set up the necessary git commands to account for the new and deleted files 
from the javadoc.
-* Update the Javadoc link on this page to point to the new version (in 
`src/documentation/sdks/javadoc/current.md`).
+**PR 2: apache/beam**
 
-# Create Pydoc
-Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+The first pull request is against the `apache/beam` repo, on the `master`
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151449)
Time Spent: 3h  (was: 2h 50m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151452
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222859301
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
-Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+* Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+  * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into 
some temporary location. Call this `${JAVADOC_TMP}`.
+  * Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} content/documentation/sdks/javadoc/${RELEASE}`.
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151452)
Time Spent: 3.5h  (was: 3h 20m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151458
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222861136
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
-Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+* Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+  * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into 
some temporary location. Call this `${JAVADOC_TMP}`.
+  * Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} content/documentation/sdks/javadoc/${RELEASE}`.
+* Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+  * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} 
content/documentation/sdks/pydoc/${RELEASE}`.
+  * Remove `.doctrees` directory.
+* Set up the necessary git commands to account for new and deleted files.
 
 
 Review comment:
   Added above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151458)
Time Spent: 4h  (was: 3h 50m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151457
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222863319
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
-Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+* Add the new Javadoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/javadoc/) page, as follows:
+  * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into 
some temporary location. Call this `${JAVADOC_TMP}`.
+  * Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} content/documentation/sdks/javadoc/${RELEASE}`.
+* Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+  * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} 
content/documentation/sdks/pydoc/${RELEASE}`.
+  * Remove `.doctrees` directory.
+* Set up the necessary git commands to account for new and deleted files.
 
-* Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into some 
temporary location. Call this `${JAVADOC_TMP}`.
-* Copy the generated Javadoc into the website repository: `cp -r 
${JAVADOC_TMP} src/documentation/sdks/javadoc/${RELEASE}`.
-* Set up the necessary git commands to account for the new and deleted files 
from the javadoc.
-* Update the Javadoc link on this page to point to the new version (in 
`src/documentation/sdks/javadoc/current.md`).
+**PR 2: apache/beam**
 
-# Create Pydoc
-Add the new Pydoc to [SDK API Reference page]({{ site.baseurl 
}}/documentation/sdks/pydoc/) page, as follows:
+The first pull request is against the `apache/beam` repo, on the `master`
+branch.
 
-* Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} 
src/documentation/sdks/pydoc/${RELEASE}`.
-* Remove `.doctrees` directory.
-* Update the Pydoc link on this page to point to the new version (in 
`src/documentation/sdks/pydoc/current.md`).
+* Update the `release_latest` version flag in `/website/_config.yml`, and list
+  the new release in `/website/src/get-started/downloads.md`, linking to the
+  source code download and the Release Notes in JIRA.
+* Update `redirect_to` in `/website/src/documentation/sdks/*/current.md` to 
point to the
 
 Review comment:
   Changed to update .htaccess.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151457)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151454
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222857243
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -167,8 +167,23 @@ please submit your GPG public key into [MIT PGP Public 
Key Server](http://pgp.mi
 
  Website development setup
 
-Prepare to update the Beam website by following the [website development
-instructions]({{ site.baseurl }}/contribute/website-contributions/).
+Updating the Beam website requires access to both the man `apache/beam` repo 
and
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151454)
Time Spent: 3h 40m  (was: 3.5h)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151456
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222857845
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151456)
Time Spent: 3h 50m  (was: 3h 40m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151451
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:29
Start Date: 05/Oct/18 00:29
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222859211
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -555,33 +570,40 @@ cd sdks/python && tox -e docs
 ```
 By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let 
`${PYDOC_ROOT}` be the absolute path to `_build`.
 
- Propose a pull request for website updates
+ Propose pull requests for website updates
 
-The final step of building the candidate is to propose a website pull request.
+Beam publishes API reference manuals for each release on the website. For Java
+and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
+building the candidate is to propose website pull requests that update these
+manuals.
 
-Start by updating `release_latest` version flag in the top-level 
`_config.yml`, and list the new release in the [Apache Beam Downloads]({{ 
site.baseurl }}/get-started/downloads/), linking to the source code download 
and the Release Notes in JIRA.
+Merge the pull requests only after finalizing the release. To avoid invalid
+redirects in `current.md` files, merge these PRs in the order listed.
 
-Beam publishes API reference manual for each release on the website. For Java 
SDK, that’s Javadoc.
+**PR 1: apache/beam-site**
 
-One of the artifacts created in the release contains the Javadoc for the
-website. To update the website, you must unpack this jar file from the release
-candidate into the source tree of the website.
+The second pull request is against the `apache/beam-site` repo, on the
+`asf-site` branch.
 
 Review comment:
   udpated


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151451)
Time Spent: 3h 20m  (was: 3h 10m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1257

2018-10-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #236

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[github] [BEAM-5529] Before starting the final countdown, check that job is not

[tweise] [BEAM-5528] Re-add CREATE_VIEW in portable Flink translators (#6523)

--
[...truncated 51.05 MB...]
[ToKeyedWorkItem (14/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (14/16) 
(60b46a331a63cdccb0efb82b3e073f4f) [FINISHED]
[ToKeyedWorkItem (12/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (12/16) 
(ec75792243509d1b95923a4a55c7a430) [FINISHED]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (2/16) 
(aed5691b4ecc85d8d3b0134e7e7d44f4) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
ce60b57f8c68591beed37319ae492e76.
[ToKeyedWorkItem (11/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (11/16) 
(ec40c7cf99015ee98500df7b958e9f48).
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (16/16) (4004725018b983cd259fa4bd43b0322a) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (16/16) 
(4004725018b983cd259fa4bd43b0322a).
[ToKeyedWorkItem (11/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (11/16) 
(ec40c7cf99015ee98500df7b958e9f48) [FINISHED]
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (16/16) 
(4004725018b983cd259fa4bd43b0322a) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
81323887847973c4180b35604c0b2ec2.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
b726c578a68db632d4c6f8b2070bca59.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
621fe712a5fcea7299783a80067feed0.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16) 
(e0c4e1a7fac412eade91ad54e1043520) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
72415a3ecce145e942dd8c2515cddbeb.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
e7f70b876d558d262ee2229bc4beb22a.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (6/16) (058618001f3abe3e57eb24fb6a72583a) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (6/16) 
(058618001f3abe3e57eb24fb6a72583a).
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
0deb0bd4d5a0a2a91ac450056383e5af.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (6/16) 
(058618001f3abe3e57eb24fb6a72583a) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
f541d5e28009ea2914f5dbb09d6ebb96.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(011da10465ec454547611291b723e645) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151446
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:20
Start Date: 05/Oct/18 00:20
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #6559: 
[BEAM-5630] add more tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#discussion_r222863957
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java
 ##
 @@ -83,13 +94,31 @@ private void runBigQueryIOReadPipeline() {
 
   @Test
   public void testBigQueryReadEmpty() throws Exception {
-setupTestEnvironment("empty");
+setupTestEnvironment("empty", false);
 runBigQueryIOReadPipeline();
   }
 
   @Test
-  public void testBigQueryRead1K() throws Exception {
-setupTestEnvironment("1K");
+  public void testBigQueryRead1G() throws Exception {
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151446)
Time Spent: 2.5h  (was: 2h 20m)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5630) supplement Bigquery Read IT test cases and blacklist them in post-commit

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5630?focusedWorklogId=151447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151447
 ]

ASF GitHub Bot logged work on BEAM-5630:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:20
Start Date: 05/Oct/18 00:20
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6559: [BEAM-5630] add more 
tests into BigQueryIOReadIT
URL: https://github.com/apache/beam/pull/6559#issuecomment-427209430
 
 
   @jasonkuster PTAL, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151447)
Time Spent: 2h 40m  (was: 2.5h)

> supplement Bigquery Read IT test cases and blacklist them in post-commit
> 
>
> Key: BEAM-5630
> URL: https://issues.apache.org/jira/browse/BEAM-5630
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-5528] Re-add CREATE_VIEW in portable Flink translators (#6523)

2018-10-04 Thread thw

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new b57283b  [BEAM-5528] Re-add CREATE_VIEW in portable Flink translators 
(#6523)
b57283b is described below

commit b57283b6ab6706ecb523bf2e496021dc5b5cfad8
Author: Maximilian Michels 
AuthorDate: Fri Oct 5 02:03:12 2018 +0200

[BEAM-5528] Re-add CREATE_VIEW in portable Flink translators (#6523)
---
 .../beam/runners/flink/FlinkBatchPortablePipelineTranslator.java   | 7 +++
 .../runners/flink/FlinkStreamingPortablePipelineTranslator.java| 6 ++
 2 files changed, 13 insertions(+)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index bb3a890..e67020d 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -140,6 +140,13 @@ public class FlinkBatchPortablePipelineTranslator
 translatorMap.put(
 PTransformTranslation.RESHUFFLE_URN,
 FlinkBatchPortablePipelineTranslator::translateReshuffle);
+translatorMap.put(
+PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
+// https://issues.apache.org/jira/browse/BEAM-5649
+// Need to support this via a NOOP until the primitive is removed
+(PTransformNode transform,
+RunnerApi.Pipeline pipeline,
+BatchTranslationContext context) -> {});
 return new FlinkBatchPortablePipelineTranslator(translatorMap.build());
   }
 
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
index 3f4f7c8..9fe2b2b 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
@@ -169,6 +169,12 @@ public class FlinkStreamingPortablePipelineTranslator
 translatorMap.put(ExecutableStage.URN, this::translateExecutableStage);
 translatorMap.put(PTransformTranslation.RESHUFFLE_URN, 
this::translateReshuffle);
 
+translatorMap.put(
+// https://issues.apache.org/jira/browse/BEAM-5649
+// Need to support this via a NOOP until the primitive is removed
+PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
+(String id, RunnerApi.Pipeline pipeline, StreamingTranslationContext 
context) -> {});
+
 this.urnToTransformTranslator = translatorMap.build();
   }

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151432
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:40
Start Date: 04/Oct/18 23:40
Worklog Time Spent: 10m 
  Work Description: swegner commented on a change in pull request #6574: 
[BEAM-4494] Migrate recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#discussion_r222858595
 
 

 ##
 File path: website/src/documentation/dsls/sql/create-external-table.md
 ##
 @@ -19,20 +21,20 @@ See the License for the specific language governing 
permissions and
 limitations under the License.
 -->
 
-# CREATE TABLE
+# CREATE EXTERNAL TABLE
 
-Beam SQL's `CREATE TABLE` statement registers a virtual table that maps to an
+Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that 
maps to an
 [external storage system](https://beam.apache.org/documentation/io/built-in/).
 
 Review comment:
   Thanks; I will fix these upstream in `beam-site` and then re-import


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151432)
Time Spent: 4.5h  (was: 4h 20m)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5528) Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5528?focusedWorklogId=151438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151438
 ]

ASF GitHub Bot logged work on BEAM-5528:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:03
Start Date: 05/Oct/18 00:03
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6523: [BEAM-5528] Re-add 
CREATE_VIEW in portable Flink translators
URL: https://github.com/apache/beam/pull/6523
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index bb3a8903100..e67020d42f0 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -140,6 +140,13 @@ public static FlinkBatchPortablePipelineTranslator 
createTranslator() {
 translatorMap.put(
 PTransformTranslation.RESHUFFLE_URN,
 FlinkBatchPortablePipelineTranslator::translateReshuffle);
+translatorMap.put(
+PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
+// https://issues.apache.org/jira/browse/BEAM-5649
+// Need to support this via a NOOP until the primitive is removed
+(PTransformNode transform,
+RunnerApi.Pipeline pipeline,
+BatchTranslationContext context) -> {});
 return new FlinkBatchPortablePipelineTranslator(translatorMap.build());
   }
 
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
index 3f4f7c8e34c..9fe2b2b2de2 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
@@ -169,6 +169,12 @@ public StreamExecutionEnvironment 
getExecutionEnvironment() {
 translatorMap.put(ExecutableStage.URN, this::translateExecutableStage);
 translatorMap.put(PTransformTranslation.RESHUFFLE_URN, 
this::translateReshuffle);
 
+translatorMap.put(
+// https://issues.apache.org/jira/browse/BEAM-5649
+// Need to support this via a NOOP until the primitive is removed
+PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
+(String id, RunnerApi.Pipeline pipeline, StreamingTranslationContext 
context) -> {});
+
 this.urnToTransformTranslator = translatorMap.build();
   }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151438)
Time Spent: 2h 10m  (was: 2h)

> Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW
> 
>
> Key: BEAM-5528
> URL: https://issues.apache.org/jira/browse/BEAM-5528
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Critical
>  Labels: portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We recently removed the CREATE_VIEW translation which was a NOOP. Turns out 
> we still need to support it until it is removed. 
> {noformat}
> java.lang.IllegalArgumentException: Unknown type of URN 
> beam:transform:create_view:v1 for PTransform with id 
> TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/View.CreatePCollectionView.
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.urnNotFound(FlinkBatchPortablePipelineTranslator.java:578)
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:233)
> at 
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:112)
> at 
>

[jira] [Work logged] (BEAM-5528) Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5528?focusedWorklogId=151443=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151443
 ]

ASF GitHub Bot logged work on BEAM-5528:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:03
Start Date: 05/Oct/18 00:03
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6523: [BEAM-5528] Re-add 
CREATE_VIEW in portable Flink translators
URL: https://github.com/apache/beam/pull/6523#issuecomment-427206835
 
 
   @mxm please update https://issues.apache.org/jira/browse/BEAM-5649 with your 
findings


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151443)
Time Spent: 2h 20m  (was: 2h 10m)

> Java PortableRunner pipeline fails on FlinkRunner due to CREATE_VIEW
> 
>
> Key: BEAM-5528
> URL: https://issues.apache.org/jira/browse/BEAM-5528
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Critical
>  Labels: portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We recently removed the CREATE_VIEW translation which was a NOOP. Turns out 
> we still need to support it until it is removed. 
> {noformat}
> java.lang.IllegalArgumentException: Unknown type of URN 
> beam:transform:create_view:v1 for PTransform with id 
> TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/View.CreatePCollectionView.
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.urnNotFound(FlinkBatchPortablePipelineTranslator.java:578)
> at 
> org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:233)
> at 
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:112)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
> at 
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2687) Python SDK support for Stateful Processing

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2687?focusedWorklogId=151445=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151445
 ]

ASF GitHub Bot logged work on BEAM-2687:


Author: ASF GitHub Bot
Created on: 05/Oct/18 00:05
Start Date: 05/Oct/18 00:05
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6579: [BEAM-2687] Skip 
test_pardo_timers for Python Flink VR
URL: https://github.com/apache/beam/pull/6579#issuecomment-427207104
 
 
   R: @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151445)
Time Spent: 6h 50m  (was: 6h 40m)

> Python SDK support for Stateful Processing
> --
>
> Key: BEAM-2687
> URL: https://issues.apache.org/jira/browse/BEAM-2687
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Python SDK should support stateful processing 
> (https://beam.apache.org/blog/2017/02/13/stateful-processing.html)
> In the meantime, runner capability matrix should be updated to show the lack 
> of this feature 
> (https://beam.apache.org/documentation/runners/capability-matrix/)
> Use this as an umbrella issue for all related issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/05 00:04:26 at commit b57283b

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 55a9321  Publishing website 2018/10/05 00:04:26 at commit b57283b
55a9321 is described below

commit 55a93216bd382fa7c722aa39f1a1842a6ee2f887
Author: jenkins 
AuthorDate: Fri Oct 5 00:04:26 2018 +

Publishing website 2018/10/05 00:04:26 at commit b57283b

[beam] branch asf-site updated: Publishing website 2018/10/05 00:01:05 at commit 845f8d0

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 815d88b  Publishing website 2018/10/05 00:01:05 at commit 845f8d0
815d88b is described below

commit 815d88bed46adfae507bd5c8eb843ee38f2598a3
Author: jenkins 
AuthorDate: Fri Oct 5 00:01:06 2018 +

Publishing website 2018/10/05 00:01:05 at commit 845f8d0

Build failed in Jenkins: beam_PreCommit_Website_Cron #140

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[ankurgoenka] Remove default worker id in python

[valentyn] Update dependency versions and switch gradle build to use /target

[stijn.decubber] Partially port testing

[github] [BEAM-5651] Add --update-snapshots flag to archetype tests

[scott] Add .htaccess file to redirect javadoc/pydoc URLs to the new location

[scott] Update existing links to point to new javadoc/pydoc location

[scott] Remove outdated IDE pages and point to wiki instead

[github] Fix redirect for current release docs

[github] [BEAM-5529] Before starting the final countdown, check that job is not

--
[...truncated 167.11 KB...]
  *  External link http://images/logos/runners/spark.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/go.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/java.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/python.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/logos/sdks/scala.png failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/downloads/index.html
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/index.html
  *  External link http://get-started/beam-overview failed: response code 0 
means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://get-started/mobile-gaming-example failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  http:// is an invalid URL (line 77)
 

  
- ./generated-content/get-started/mobile-gaming-example/index.html
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://documentation/programming-guide/ failed: response 
code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link http://images/gaming-example-basic.png failed: response code 
0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: 
Couldn't resolve host name
  *  External link

Build failed in Jenkins: beam_PreCommit_Website_Stage_GCS_Cron #2

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[joar] [BEAM-5417] Parity between GCS and local match

[joar] Give test a more descriptive name

[joar] [BEAM-5417] ** matches anything, * matches [^/\]*

[joar] Make isort happy

[joar] Remove invalid sphinx references

[joar] Tidy up logging

[joar] Add test cases for {only,no}-root-level

[joar] Use str.join

[joar] Update FileSystems.match docstring

[joar] Remove reference to excluded method

[migryz] Add beamsyncjenkins container to docker-compose config

[joar] Parameterize test_match_glob

[huangry] Expose flag min_cpu_platform in PipelineOption.

[boyuanz] Fix bugs found in 2.8.0 release

[valentyn] Move SDK dependencies from Dockerfile to base_image_requirements.txt

[valentyn] Sort the dependencies alphabetically.

[valentyn] Update the versions of SDK dependencies to be installed into the

[valentyn] Add a Python container based on Python 3 interpreter.

[huangry] Update doc and log more info.

[pablo] Revert "Merge pull request #6411 from huygaa11/revert-6311-twsit"

[boyuanz] Address Charles's comments

[github] Avoid KeyError on Python 3 when BEAM_EXPERIMENTAL_PY3 is not defined.

[joar] Include CPython's license for translate_pattern

[pablo] Fixing odd Options bug

[migryz] Update README.md with instructions to spinup service locally.

[scott] [BEAM-5611] Fix failure in testWebsite due to 508 resource limit error

[sekikn] [BEAM-5619] Fix minor bug in JdbcIO example code

[ankurgoenka] Remove default worker id in python

[thw] [BEAM-5625] Add missing CancelJobRequest parameter.

[joar] isort: Reorder imports

[ankurgoenka] Store and server termination state after portable job termination

[ankurgoenka] Correctly deserialize pipeline options on Fn harness

[aaltay] [BEAM-5322] Partially port typehints (#6541)

[apilloud] [BEAM-5112] Don't use BeamSqlInputRefExpression to pass Integer

[scott] [BEAM-5427] Fix and update sample code for CombineFn. (#6439)

[valentyn] Update dependency versions and switch gradle build to use /target

[amyrvold] [BEAM-4495] Create precommit to stage website on GCS

[github] Use standard format for python dependencies (#6560)

[yifanzou] [BEAM-5643] fix broken python it test

[robertwb] [BEAM-4660] Add well known timer coder for Python SDK.

[robertwb] [BEAM-2687] Implement timers in Python SDK harness.

[stijn.decubber] Partially port testing

[github] [BEAM-5651] Add --update-snapshots flag to archetype tests

[scott] Add .htaccess file to redirect javadoc/pydoc URLs to the new location

[scott] Update existing links to point to new javadoc/pydoc location

[scott] Remove outdated IDE pages and point to wiki instead

[github] Fix redirect for current release docs

[github] [BEAM-5529] Before starting the final countdown, check that job is not

--
[...truncated 7.26 KB...]
Skipping task ':buildSrc:classes' as it has no actions.
:buildSrc:classes (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:jar (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:jar
Build cache key for task ':buildSrc:jar' is 60c54a62be893247a90eb1dbbfdbb239
Caching disabled for task ':buildSrc:jar': Caching has not been enabled for the 
task
Task ':buildSrc:jar' is not up-to-date because:
  No history is available.
:buildSrc:jar (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.11 
secs.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 1.19 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task

[jira] [Work logged] (BEAM-5653) Dataflow FnApi Worker overrides some of Coders due to coder ID generation collision.

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5653?focusedWorklogId=151437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151437
 ]

ASF GitHub Bot logged work on BEAM-5653:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:53
Start Date: 04/Oct/18 23:53
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6578:  [BEAM-5653] Allow 
initialize SdkComponents with existing Components list.
URL: https://github.com/apache/beam/pull/6578#issuecomment-427205177
 
 
   R: @kennknowles


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151437)
Time Spent: 20m  (was: 10m)
Remaining Estimate: 71h 40m  (was: 71h 50m)

> Dataflow FnApi Worker overrides some of Coders due to coder ID generation 
> collision.
> 
>
> Key: BEAM-5653
> URL: https://issues.apache.org/jira/browse/BEAM-5653
> Project: Beam
>  Issue Type: Test
>  Components: java-fn-execution
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Blocker
> Fix For: 2.8.0
>
>   Original Estimate: 72h
>  Time Spent: 20m
>  Remaining Estimate: 71h 40m
>
> Due to one of latest refactorings, we got a bug in Java FnApi Worker that it 
> overrides Coders in ProcessBundleDescriptor sent to SDK Harness that causes 
> jobs to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5653) Dataflow FnApi Worker overrides some of Coders due to coder ID generation collision.

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5653?focusedWorklogId=151435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151435
 ]

ASF GitHub Bot logged work on BEAM-5653:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:50
Start Date: 04/Oct/18 23:50
Worklog Time Spent: 10m 
  Work Description: Ardagan opened a new pull request #6578:  [BEAM-5653] 
Allow initialize SdkComponents with existing Components list.
URL: https://github.com/apache/beam/pull/6578
 
 
   The idea of the fix is to initialize SdkComponents with initial pipeline 
Components and later extend it.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151435)
Time Spent: 10m
Remaining Estimate: 71h 50m  (was: 72h)

> Dataflow FnApi Worker overrides some of Coders due to coder ID generation 
> collision.
> 
>
> Key: BEAM-5653
> URL:

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1256

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[github] [BEAM-5529] Before starting the final countdown, check that job is not

--
[...truncated 75.81 KB...]
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ERROR
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six

[jira] [Work logged] (BEAM-5662) [beam_PostCommit_Website_Publish] [:testWebsite] External link http://wiki.apache.org/incubator/BeamProposal failed: got a time out

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5662?focusedWorklogId=151430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151430
 ]

ASF GitHub Bot logged work on BEAM-5662:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:39
Start Date: 04/Oct/18 23:39
Worklog Time Spent: 10m 
  Work Description: swegner opened a new pull request #6577: [BEAM-5662] 
Increase timeouts and limit parallelism of website link check
URL: https://github.com/apache/beam/pull/6577
 
 
   The website validation checks external links resolve correctly. However
   this has been a source of flakiness in Jenkins. These updates seek to
   reduce the current flakiness by:
   
   * Increasing timeout limits
   * Limiting parallelism. Current config uses parallelism = numCores, and
 Jenkins machines are very beefy.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151430)
Time Spent: 10m
Remaining Estimate: 0h

> [beam_PostCommit_Website_Publish]

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151429
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:36
Start Date: 04/Oct/18 23:36
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6574: [BEAM-4494] 
Migrate recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#issuecomment-427202508
 
 
   Rename of create-external-table.md file causes issues getting external link 
https://beam.apache.org/documentation/dsls/sql/create-external-table ... this 
link should use {{ site.baseurl }} to be testable when it is new


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151429)
Time Spent: 4h 20m  (was: 4h 10m)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5662) [beam_PostCommit_Website_Publish] [:testWebsite] External link http://wiki.apache.org/incubator/BeamProposal failed: got a time out

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5662?focusedWorklogId=151431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151431
 ]

ASF GitHub Bot logged work on BEAM-5662:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:39
Start Date: 04/Oct/18 23:39
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6577: [BEAM-5662] Increase 
timeouts and limit parallelism of website link check
URL: https://github.com/apache/beam/pull/6577#issuecomment-427203023
 
 
   R: @alanmyrvold 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151431)
Time Spent: 20m  (was: 10m)

> [beam_PostCommit_Website_Publish] [:testWebsite] External link 
> http://wiki.apache.org/incubator/BeamProposal failed: got a time out
> ---
>
> Key: BEAM-5662
> URL: https://issues.apache.org/jira/browse/BEAM-5662
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Website_Publish/94/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/h4mayefon7v7q/console-log?task=:beam-website:testWebsite#L12]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/website/Rakefile#L6]
> Initial investigation:
> The failed link is http://wiki.apache.org/incubator/BeamProposal 
> When I visit this link, it works for me. This is likely a flake.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5622) Several tests fail on Python 3 with: Runtime type violation detected

2018-10-04 Thread Ruoyun Huang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruoyun Huang reassigned BEAM-5622:
--

Assignee: (was: Ruoyun Huang)

> Several tests fail on Python 3 with: Runtime type violation detected
> 
>
> Key: BEAM-5622
> URL: https://issues.apache.org/jira/browse/BEAM-5622
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ==
> FAIL: test_combine_runtime_type_check_violation_using_decorators 
> (apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 1543, in test_combine_runtime_type_check_violation_using_decorators
> "Runtime type violation detected within "
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 911, in assertStartswith
> '"%s" does not start with "%s"' % (msg, prefix))
> AssertionError: False is not true : "Runtime type violation detected within 
> Mul/CombinePerKey: Type-hint for return type violated. Expected an instance 
> of , instead found 
> 25252525252525252525252525252525252525252525252525, an instance of  'str'>." does not start with "Runtime type violation detected within 
> Mul/CombinePerKey: Type-hint for return type violated. Expected an instance 
> of , instead found"
> ==
> FAIL: test_combine_runtime_type_check_violation_using_methods 
> (apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 1597, in test_combine_runtime_type_check_violation_using_methods
> "Runtime type violation detected within "
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 911, in assertStartswith
> '"%s" does not start with "%s"' % (msg, prefix))
> AssertionError: False is not true : "Runtime type violation detected within 
> ParDo(SortJoin/KeyWithVoid): Type-hint for argument: 'v' violated. Expected 
> an instance of , instead found 0, an instance of . 
> [while running 'SortJoin/KeyWithVoid']" does not start with "Runtime type 
> violation detected within ParDo(SortJoin/KeyWithVoid): Type-hint for 
> argument: 'v' violated. Expected an instance of , instead found 
> 0, an instance of ."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

2018-10-04 Thread Ruoyun Huang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruoyun Huang reassigned BEAM-5626:
--

Assignee: Ruoyun Huang

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151428
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:32
Start Date: 04/Oct/18 23:32
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6574: [BEAM-4494] Migrate recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#discussion_r222857575
 
 

 ##
 File path: website/src/_posts/2018-10-03-beam-2.7.0.md
 ##
 @@ -0,0 +1,76 @@
+---
+layout: post
+title:  "Apache Beam 2.7.0"
+date:   2018-10-03 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+- ccy
+
+---
+
+
+We are happy to present the new 2.7.0 release of Beam. This release includes 
both improvements and new functionality.
+See the [download 
page](https://beam.apache.org/get-started/downloads/#270-2018-10-02) for this 
release.
 
 Review comment:
   Should be {{ site.baseurl }}


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151428)
Time Spent: 4h 10m  (was: 4h)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151427
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:32
Start Date: 04/Oct/18 23:32
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6574: [BEAM-4494] Migrate recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#discussion_r222857529
 
 

 ##
 File path: website/src/documentation/dsls/sql/create-external-table.md
 ##
 @@ -19,20 +21,20 @@ See the License for the specific language governing 
permissions and
 limitations under the License.
 -->
 
-# CREATE TABLE
+# CREATE EXTERNAL TABLE
 
-Beam SQL's `CREATE TABLE` statement registers a virtual table that maps to an
+Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that 
maps to an
 [external storage system](https://beam.apache.org/documentation/io/built-in/).
 
 Review comment:
   Should be {{ site.baseurl }}


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151427)
Time Spent: 4h  (was: 3h 50m)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5622) Several tests fail on Python 3 with: Runtime type violation detected

2018-10-04 Thread Ruoyun Huang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruoyun Huang reassigned BEAM-5622:
--

Assignee: Ruoyun Huang

> Several tests fail on Python 3 with: Runtime type violation detected
> 
>
> Key: BEAM-5622
> URL: https://issues.apache.org/jira/browse/BEAM-5622
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>
> ==
> FAIL: test_combine_runtime_type_check_violation_using_decorators 
> (apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 1543, in test_combine_runtime_type_check_violation_using_decorators
> "Runtime type violation detected within "
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 911, in assertStartswith
> '"%s" does not start with "%s"' % (msg, prefix))
> AssertionError: False is not true : "Runtime type violation detected within 
> Mul/CombinePerKey: Type-hint for return type violated. Expected an instance 
> of , instead found 
> 25252525252525252525252525252525252525252525252525, an instance of  'str'>." does not start with "Runtime type violation detected within 
> Mul/CombinePerKey: Type-hint for return type violated. Expected an instance 
> of , instead found"
> ==
> FAIL: test_combine_runtime_type_check_violation_using_methods 
> (apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 1597, in test_combine_runtime_type_check_violation_using_methods
> "Runtime type violation detected within "
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 911, in assertStartswith
> '"%s" does not start with "%s"' % (msg, prefix))
> AssertionError: False is not true : "Runtime type violation detected within 
> ParDo(SortJoin/KeyWithVoid): Type-hint for argument: 'v' violated. Expected 
> an instance of , instead found 0, an instance of . 
> [while running 'SortJoin/KeyWithVoid']" does not start with "Runtime type 
> violation detected within ParDo(SortJoin/KeyWithVoid): Type-hint for 
> argument: 'v' violated. Expected an instance of , instead found 
> 0, an instance of ."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5662) [beam_PostCommit_Website_Publish] [:testWebsite] External link http://wiki.apache.org/incubator/BeamProposal failed: got a time out

2018-10-04 Thread Scott Wegner (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639036#comment-16639036
 ] 

Scott Wegner commented on BEAM-5662:


We've seen a few flakes like this; where the link fails during the test and 
then works later. I wonder if there's a way to configure some retry+backoff on 
the test.

> [beam_PostCommit_Website_Publish] [:testWebsite] External link 
> http://wiki.apache.org/incubator/BeamProposal failed: got a time out
> ---
>
> Key: BEAM-5662
> URL: https://issues.apache.org/jira/browse/BEAM-5662
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostCommit_Website_Publish/94/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/h4mayefon7v7q/console-log?task=:beam-website:testWebsite#L12]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/website/Rakefile#L6]
> Initial investigation:
> The failed link is http://wiki.apache.org/incubator/BeamProposal 
> When I visit this link, it works for me. This is likely a flake.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151426
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:26
Start Date: 04/Oct/18 23:26
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222856609
 
 

 ##
 File path: website/README.md
 ##
 @@ -18,14 +18,26 @@ The Beam website is built using 
[Jekyll](http://jekyllrb.com/). Additionally,
 for additional formatting capabilities, this website uses
 [Twitter Bootstrap](http://getbootstrap.com/).
 
-### Repository Structure
+Documentation generated from source code, such as Javadoc and Pydoc, is stored
+separately on the [beam-site
+repository](https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks).
 
-This repository contains:
+## Development Workflow with Docker
 
-1. `src/`: the source of the site, including markdown files containing the 
bulk of the content
-1. `content/`: html generated from the markdown (which is what is actually 
hosted on the website)
+### Active development
+
+If you have Docker configured on your machine, the following command may be 
used
+to build and serve the website locally.
+
+$ ./gradlew :beam-website:serveWebsite
+
+Any changes made locally will trigger a rebuild of the website.
 
-## Development Workflow
+You can also run website tests using this command:
+
+$ ./gradlew :beam-website:testWebsite
+
+## Development Workflow without Docker
 
 ### Setup
 
 Review comment:
   I'll go ahead and remove the more complicated non-docker steps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151426)
Time Spent: 2h 40m  (was: 2.5h)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5662) [beam_PostCommit_Website_Publish] [:testWebsite] External link http://wiki.apache.org/incubator/BeamProposal failed: got a time out

2018-10-04 Thread Scott Wegner (JIRA)

Scott Wegner created BEAM-5662:
--

 Summary: [beam_PostCommit_Website_Publish] [:testWebsite] External 
link http://wiki.apache.org/incubator/BeamProposal failed: got a time out
 Key: BEAM-5662
 URL: https://issues.apache.org/jira/browse/BEAM-5662
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Scott Wegner
Assignee: Scott Wegner


_Use this form to file an issue for test failure:_
 * [Jenkins 
Job|https://builds.apache.org/job/beam_PostCommit_Website_Publish/94/]
 * [Gradle Build 
Scan|https://scans.gradle.com/s/h4mayefon7v7q/console-log?task=:beam-website:testWebsite#L12]
 * [Test source 
code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/website/Rakefile#L6]

Initial investigation:

The failed link is http://wiki.apache.org/incubator/BeamProposal 
When I visit this link, it works for me. This is likely a flake.


_After you've filled out the above details, please [assign the issue to an 
individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
 Assignee should [treat test failures as 
high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
 helping to fix the issue or find a more appropriate owner. See [Apache Beam 
Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639009#comment-16639009
 ] 

Valentyn Tymofieiev commented on BEAM-5661:
---

I don't think so. It sounds like this is a flake of Docker itself. [~herohde] 
may have some advice here.

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151424=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151424
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:13
Start Date: 04/Oct/18 23:13
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6574: [BEAM-4494] Migrate 
recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#issuecomment-427198578
 
 
   I believe pre-commits are failing because current_release is updated from 
2.6.0 to 2.7.0, but we haven't published the javadocs/pydocs to the new 
`release-docs` branch yet. Sent a PR here: 
https://github.com/apache/beam-site/pull/575


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151424)
Time Spent: 3h 50m  (was: 3h 40m)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5613) Snapshot Python dependencies and add to Python_NightlySnapshot

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5613?focusedWorklogId=151416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151416
 ]

ASF GitHub Bot logged work on BEAM-5613:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:51
Start Date: 04/Oct/18 22:51
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6551: [BEAM-5613] Snapshot of Python depedency and add it to nightly snapshot 
job
URL: https://github.com/apache/beam/pull/6551#discussion_r222850531
 
 

 ##
 File path: .test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
 ##
 @@ -41,11 +41,19 @@ job('beam_Release_Python_NightlySnapshot') {
 'Run Python Publish')
 
 steps {
+  // Cleanup Python directory.
   gradle {
 rootBuildScriptDir(commonJobProperties.checkoutDir)
-tasks(':beam-sdks-python:sdist')
+tasks(':beam-sdks-python:clean')
 
 Review comment:
   looks like `clean` and `buildSnapshot` can't run from single context since 
virtualenv directory will be cleanup right after `clean` but following 
`buildSnapshot` depends on it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151416)
Time Spent: 2h 10m  (was: 2h)

> Snapshot Python dependencies and add to Python_NightlySnapshot
> --
>
> Key: BEAM-5613
> URL: https://issues.apache.org/jira/browse/BEAM-5613
> Project: Beam
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Python SDK depends on a list of libraries without specific versions. 
> Unexpected or unaware version change could happen if there is a new release 
> or range change in setup.py. This can be detected by dependency checking 
> tools which require to track dependency list and their versions periodically. 
> We could snapshot dependencies periodically using pip freeze and python 
> setup.py egg_info and upload them to a public accessible location to benefit 
> people who want to check dependency info or building related tools.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5661:
-

Assignee: Henning Rohde  (was: Valentyn Tymofieiev)

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Henning Rohde
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4501) Update contribution guide for new website contribution process

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4501?focusedWorklogId=151422=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151422
 ]

ASF GitHub Bot logged work on BEAM-4501:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:05
Start Date: 04/Oct/18 23:05
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #6533: 
[BEAM-4501] Update website contribution and release docs
URL: https://github.com/apache/beam/pull/6533#discussion_r222853217
 
 

 ##
 File path: website/README.md
 ##
 @@ -68,6 +80,12 @@ Before sending the PR for review, please run:
 
 $ git checkout -- content
 
 Review comment:
   If you run Jekyll manually, it'll still create a `./generated-content` 
directory in the CWD. The Gradle task takes care to run under `build/website`.
   I'll go ahead and remove the Jekyll commands to make this guide shorter and 
less  TLDR-ish.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151422)
Time Spent: 2.5h  (was: 2h 20m)

> Update contribution guide for new website contribution process
> --
>
> Key: BEAM-4501
> URL: https://issues.apache.org/jira/browse/BEAM-4501
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Udi Meiri
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5355) Create GroupByKey load test for Java SDK

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5355?focusedWorklogId=151420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151420
 ]

ASF GitHub Bot logged work on BEAM-5355:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:02
Start Date: 04/Oct/18 23:02
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #6571: [BEAM-5355] Add 
GroupByKeyLoadTest
URL: https://github.com/apache/beam/pull/6571#issuecomment-427196494
 
 
   This change looks good to me. We had already reviewed many parts of it. 
Please go ahead an self merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151420)
Time Spent: 2h  (was: 1h 50m)

> Create GroupByKey load test for Java SDK
> 
>
> Key: BEAM-5355
> URL: https://issues.apache.org/jira/browse/BEAM-5355
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> This is more thoroughly described in this proposal: 
> [https://docs.google.com/document/d/1PuIQv4v06eosKKwT76u7S6IP88AnXhTf870Rcj1AHt4/edit?usp=sharing]
>  
> In short: this ticket is about implementing the GroupByKeyLoadIT that uses 
> SyntheticStep and Synthetic source to create load on the pipeline. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151421
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 23:02
Start Date: 04/Oct/18 23:02
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6574: [BEAM-4494] Migrate 
recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#issuecomment-427196603
 
 
   Run Website PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151421)
Time Spent: 3h 40m  (was: 3.5h)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Scott Wegner (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639003#comment-16639003
 ] 

Scott Wegner commented on BEAM-5661:


The only recent change I see is https://github.com/apache/beam/pull/6565

[~tvalentyn] do you think the PR could've caused this failure?

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Scott Wegner (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner reassigned BEAM-5661:
--

Assignee: Scott Wegner

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Scott Wegner (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner reassigned BEAM-5661:
--

Assignee: Valentyn Tymofieiev  (was: Scott Wegner)

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

2018-10-04 Thread Scott Wegner (JIRA)

Scott Wegner created BEAM-5661:
--

 Summary: [beam_PostCommit_Py_ValCont] 
[:beam-sdks-python-container:docker] no such file or directory
 Key: BEAM-5661
 URL: https://issues.apache.org/jira/browse/BEAM-5661
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Scott Wegner


_Use this form to file an issue for test failure:_
 * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
 * [Gradle Build 
Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
 * [Test source 
code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]

Initial investigation:

{{failed to get digest 
sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
/var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
 no such file or directory}}


_After you've filled out the above details, please [assign the issue to an 
individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
 Assignee should [treat test failures as 
high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
 helping to fix the issue or find a more appropriate owner. See [Apache Beam 
Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638998#comment-16638998
 ] 

Valentyn Tymofieiev edited comment on BEAM-5243 at 10/4/18 10:55 PM:
-

[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally, in a virtual environment, that has nose 
package and Beam SDK installed via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.




was (Author: tvalentyn):
[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally  via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.



> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/04 22:55:07 at commit 845f8d0

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 0ba00ad  Publishing website 2018/10/04 22:55:07 at commit 845f8d0
0ba00ad is described below

commit 0ba00ad4d88e1d3cefa2fd831185967832231914
Author: jenkins 
AuthorDate: Thu Oct 4 22:55:07 2018 +

Publishing website 2018/10/04 22:55:07 at commit 845f8d0

[beam] branch master updated (0bef2fc -> 845f8d0)

2018-10-04 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0bef2fc  Merge pull request #6575: Remove outdated IDE pages and point 
to wiki instead
 add 845f8d0  [BEAM-5529] Before starting the final countdown, check that 
job is not in pre-running state (#6535)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)

[jira] [Commented] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638998#comment-16638998
 ] 

Valentyn Tymofieiev commented on BEAM-5243:
---

[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally  via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.



> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5529) Dataflow runner raises AssertionError if job takes > 50 seconds to go from PENDING to RUNNING

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5529?focusedWorklogId=151419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151419
 ]

ASF GitHub Bot logged work on BEAM-5529:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:53
Start Date: 04/Oct/18 22:53
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6535: [BEAM-5529] Before 
starting the final countdown, check that job is not in pre-running state
URL: https://github.com/apache/beam/pull/6535
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 868d264a9d9..1acd3488524 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -155,9 +155,15 @@ def rank_error(msg):
   or str(response.currentState) == 'JOB_STATE_UPDATED'
   or str(response.currentState) == 'JOB_STATE_DRAINED'):
 break
-  # The job has failed; ensure we see any final error messages.
-  sleep_secs = 1.0  # poll faster during the final countdown
-  final_countdown_timer_secs -= sleep_secs
+
+  # Check that job is in a post-preparation state before starting the
+  # final countdown.
+  if (str(response.currentState) not in (
+  'JOB_STATE_PENDING', 'JOB_STATE_QUEUED')):
+# The job has failed; ensure we see any final error messages.
+sleep_secs = 1.0  # poll faster during the final countdown
+final_countdown_timer_secs -= sleep_secs
+
   time.sleep(sleep_secs)
 
   # Get all messages since beginning of the job run or since last message.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151419)
Time Spent: 3h 10m  (was: 3h)

> Dataflow runner raises AssertionError if job takes > 50 seconds to go from 
> PENDING to RUNNING
> -
>
> Key: BEAM-5529
> URL: https://issues.apache.org/jira/browse/BEAM-5529
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> To reproduce:
>  - Try to run a Pipeline that takes longer than 50 seconds to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:" 
> (automatically executes {{pipeline.run().wait_until_finish()}} via 
> {{Pipeline.__exit__}}) on Dataflow.
>  - Your script will raise "AssertionError: Job did not reach to a terminal 
> state after waiting indefinitely." (from 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
> I think the issue is that the block at 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
>  unintentionally exits if it takes > 50 seconds for a job to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5529) Dataflow runner raises AssertionError if job takes > 50 seconds to go from PENDING to RUNNING

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5529?focusedWorklogId=151418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151418
 ]

ASF GitHub Bot logged work on BEAM-5529:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:53
Start Date: 04/Oct/18 22:53
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6535: [BEAM-5529] Before 
starting the final countdown, check that job is not in pre-running state
URL: https://github.com/apache/beam/pull/6535#issuecomment-427194920
 
 
   Merging, all tests passed and PR was approved.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151418)
Time Spent: 3h  (was: 2h 50m)

> Dataflow runner raises AssertionError if job takes > 50 seconds to go from 
> PENDING to RUNNING
> -
>
> Key: BEAM-5529
> URL: https://issues.apache.org/jira/browse/BEAM-5529
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Joar Wandborg
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> To reproduce:
>  - Try to run a Pipeline that takes longer than 50 seconds to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING using "with Pipeline() as pipeline:" 
> (automatically executes {{pipeline.run().wait_until_finish()}} via 
> {{Pipeline.__exit__}}) on Dataflow.
>  - Your script will raise "AssertionError: Job did not reach to a terminal 
> state after waiting indefinitely." (from 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L1152-L1153])
> I think the issue is that the block at 
> [https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L147-L160]
>  unintentionally exits if it takes > 50 seconds for a job to go from 
> JOB_STATE_PENDING to JOB_STATE_RUNNING.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5613) Snapshot Python dependencies and add to Python_NightlySnapshot

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5613?focusedWorklogId=151415=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151415
 ]

ASF GitHub Bot logged work on BEAM-5613:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:50
Start Date: 04/Oct/18 22:50
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6551: [BEAM-5613] Snapshot of Python depedency and add it to nightly snapshot 
job
URL: https://github.com/apache/beam/pull/6551#discussion_r222850531
 
 

 ##
 File path: .test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
 ##
 @@ -41,11 +41,19 @@ job('beam_Release_Python_NightlySnapshot') {
 'Run Python Publish')
 
 steps {
+  // Cleanup Python directory.
   gradle {
 rootBuildScriptDir(commonJobProperties.checkoutDir)
-tasks(':beam-sdks-python:sdist')
+tasks(':beam-sdks-python:clean')
 
 Review comment:
   looks like `clean` and `buildSnapshot` can't run from single context since 
virtualenv directory will be cleanup right after `clean` and `buildSnapshot` 
depends on it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151415)
Time Spent: 2h  (was: 1h 50m)

> Snapshot Python dependencies and add to Python_NightlySnapshot
> --
>
> Key: BEAM-5613
> URL: https://issues.apache.org/jira/browse/BEAM-5613
> Project: Beam
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Python SDK depends on a list of libraries without specific versions. 
> Unexpected or unaware version change could happen if there is a new release 
> or range change in setup.py. This can be detected by dependency checking 
> tools which require to track dependency list and their versions periodically. 
> We could snapshot dependencies periodically using pip freeze and python 
> setup.py egg_info and upload them to a public accessible location to benefit 
> people who want to check dependency info or building related tools.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5613) Snapshot Python dependencies and add to Python_NightlySnapshot

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5613?focusedWorklogId=151414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151414
 ]

ASF GitHub Bot logged work on BEAM-5613:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:50
Start Date: 04/Oct/18 22:50
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6551: [BEAM-5613] Snapshot of Python depedency and add it to nightly snapshot 
job
URL: https://github.com/apache/beam/pull/6551#discussion_r222850531
 
 

 ##
 File path: .test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
 ##
 @@ -41,11 +41,19 @@ job('beam_Release_Python_NightlySnapshot') {
 'Run Python Publish')
 
 steps {
+  // Cleanup Python directory.
   gradle {
 rootBuildScriptDir(commonJobProperties.checkoutDir)
-tasks(':beam-sdks-python:sdist')
+tasks(':beam-sdks-python:clean')
 
 Review comment:
   looks like `clean` and `buildSnapshot` shouldn't run from single context 
since virtualenv directory will be cleanup right after `clean` and 
`buildSnapshot` depends on it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151414)
Time Spent: 1h 50m  (was: 1h 40m)

> Snapshot Python dependencies and add to Python_NightlySnapshot
> --
>
> Key: BEAM-5613
> URL: https://issues.apache.org/jira/browse/BEAM-5613
> Project: Beam
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Python SDK depends on a list of libraries without specific versions. 
> Unexpected or unaware version change could happen if there is a new release 
> or range change in setup.py. This can be detected by dependency checking 
> tools which require to track dependency list and their versions periodically. 
> We could snapshot dependencies periodically using pip freeze and python 
> setup.py egg_info and upload them to a public accessible location to benefit 
> people who want to check dependency info or building related tools.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5634) Bring Dataflow Java Worker Code into Beam

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5634?focusedWorklogId=151412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151412
 ]

ASF GitHub Bot logged work on BEAM-5634:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:40
Start Date: 04/Oct/18 22:40
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #6561: [BEAM-5634] Bring 
dataflow java worker code into beam
URL: https://github.com/apache/beam/pull/6561#issuecomment-427192397
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151412)
Time Spent: 0.5h  (was: 20m)

> Bring Dataflow Java Worker Code into Beam
> -
>
> Key: BEAM-5634
> URL: https://issues.apache.org/jira/browse/BEAM-5634
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Discussion: 
> https://lists.apache.org/thread.html/89efd3bc1d30f3d43d4b361a5ee05bd52778c9dc3f43ac72354c2bd9@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_ValCont #903

2018-10-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #234

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[github] Fix redirect for current release docs

--
[...truncated 51.01 MB...]
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (10/16) (115fbc4fc1cd09ccd9c338ac511f52de) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (10/16) 
(115fbc4fc1cd09ccd9c338ac511f52de).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
ce13f21dcc23a840317a79e6f3c1e2b3.
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (10/16) 
(115fbc4fc1cd09ccd9c338ac511f52de) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16) 
(2f3761ac013403ea801d0fa7385a7d5d) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
1442ff9d8c1f0f4dddabda57eb8452e1.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
80be65ff4561935d8a7a524c52233cb9.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
04b036527b052384131177933bf24553.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (12/16) 
(31fd5477088939f57634bc2999e08c82) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
5f5f6ac1eb815fcbe17b0d6d75c661a1.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
c5f831f93d075e20ac5d78082c11ebf8.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16) 
(9bff693b712368505b928d4b738ecdf1) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (16/16) (e09773e419cd6bc0ebd04650207c145d) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (16/16) 
(e09773e419cd6bc0ebd04650207c145d).
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (16/16) 
(e09773e419cd6bc0ebd04650207c145d) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
a8af1c30541fa77ab072701f9818f405.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
2723e0cd933894bf49bdb2309d8b443b.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
203b7305016dad8ff34eefaf133b749e.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16) 
(ce13f21dcc23a840317a79e6f3c1e2b3) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
34b4f6dc81447f0b17ae24d6810b07e7.
[flink-akka.actor.default-dispatcher-3] INFO

[jira] [Work logged] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4494?focusedWorklogId=151406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151406
 ]

ASF GitHub Bot logged work on BEAM-4494:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:33
Start Date: 04/Oct/18 22:33
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6574: [BEAM-4494] Migrate 
recent website changes from beam-site to beam
URL: https://github.com/apache/beam/pull/6574#issuecomment-427191083
 
 
   Looks like the `.gradle` files were added in 
https://github.com/apache/beam-site/pull/554 . @yifanzou do you know anything 
about these? I assume it was a mistake; we can follow-up with a subsequent PR 
either here or on the `beam-site` repository.
   
   Looking into precommit failure now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151406)
Time Spent: 3.5h  (was: 3h 20m)

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
> Fix For: 2.5.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/10/04 22:34:23 at commit 0bef2fc

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new 5d79e0e  Publishing website 2018/10/04 22:34:23 at commit 0bef2fc
5d79e0e is described below

commit 5d79e0e44caed47bc6e6de779588331f2c2add0d
Author: jenkins 
AuthorDate: Thu Oct 4 22:34:23 2018 +

Publishing website 2018/10/04 22:34:23 at commit 0bef2fc
---
 .../contribute/become-a-committer/index.html   |   4 +-
 .../contribute/committer-guide/index.html  |   4 +-
 .../contribute/dependencies/index.html |   4 +-
 .../contribute/design-documents/index.html |   4 +-
 .../contribute/docker-images/index.html|   4 +-
 .../contribute/eclipse/index.html  | 387 
 website/generated-content/contribute/index.html|   4 +-
 .../contribute/intellij/index.html | 389 -
 .../contribute/portability/index.html  |   4 +-
 .../contribute/postcommits-guides/index.html   |   4 +-
 .../postcommits-policies-details/index.html|   4 +-
 .../contribute/postcommits-policies/index.html |   4 +-
 .../contribute/precommit-policies/index.html   |   4 +-
 .../contribute/precommit-triage-guide/index.html   |   4 +-
 .../contribute/ptransform-style-guide/index.html   |   4 +-
 .../contribute/release-guide/index.html|   4 +-
 .../contribute/runner-guide/index.html |   4 +-
 .../contribute/testing/index.html  |   4 +-
 .../contribute/website-contributions/index.html|   4 +-
 19 files changed, 34 insertions(+), 810 deletions(-)

diff --git a/website/generated-content/contribute/become-a-committer/index.html 
b/website/generated-content/contribute/become-a-committer/index.html
index 0c910c5..20ebfac 100644
--- a/website/generated-content/contribute/become-a-committer/index.html
+++ b/website/generated-content/contribute/become-a-committer/index.html
@@ -142,8 +142,8 @@
   IDE tips
 
   
-IntelliJ Tips
-Eclipse Tips
+https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips;>IntelliJ 
Tips
+https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips;>Eclipse 
Tips
   
 
 
diff --git a/website/generated-content/contribute/committer-guide/index.html 
b/website/generated-content/contribute/committer-guide/index.html
index 3ad43d6..fa29d8d 100644
--- a/website/generated-content/contribute/committer-guide/index.html
+++ b/website/generated-content/contribute/committer-guide/index.html
@@ -142,8 +142,8 @@
   IDE tips
 
   
-IntelliJ Tips
-Eclipse Tips
+https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips;>IntelliJ 
Tips
+https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips;>Eclipse 
Tips
   
 
 
diff --git a/website/generated-content/contribute/dependencies/index.html 
b/website/generated-content/contribute/dependencies/index.html
index dd97540..a1f750c 100644
--- a/website/generated-content/contribute/dependencies/index.html
+++ b/website/generated-content/contribute/dependencies/index.html
@@ -142,8 +142,8 @@
   IDE tips
 
   
-IntelliJ Tips
-Eclipse Tips
+https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips;>IntelliJ 
Tips
+https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips;>Eclipse 
Tips
   
 
 
diff --git a/website/generated-content/contribute/design-documents/index.html 
b/website/generated-content/contribute/design-documents/index.html
index fb2559e..4448a5e 100644
--- a/website/generated-content/contribute/design-documents/index.html
+++ b/website/generated-content/contribute/design-documents/index.html
@@ -142,8 +142,8 @@
   IDE tips
 
   
-IntelliJ Tips
-Eclipse Tips
+https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips;>IntelliJ 
Tips
+https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips;>Eclipse 
Tips
   
 
 
diff --git a/website/generated-content/contribute/docker-images/index.html 
b/website/generated-content/contribute/docker-images/index.html
index b19985f..6311a18 100644
--- a/website/generated-content/contribute/docker-images/index.html
+++ b/website/generated-content/contribute/docker-images/index.html
@@ -142,8 +142,8 @@
   IDE tips
 
   
-IntelliJ Tips
-Eclipse Tips
+https://cwiki.apache.org/confluence/display/BEAM/IntelliJ+Tips;>IntelliJ 
Tips
+https://cwiki.apache.org/confluence/display/BEAM/Eclipse+Tips;>Eclipse 
Tips
   
 
 
diff --git a/website/generated-content/contribute/eclipse/index.html 
b/website/generated-content/contribute/eclipse/index.html
deleted file mode 100644
index 480e7f2..000
--- a/website/generated-content/contribute/eclipse/index.html
+++ /dev/null
@@ -1,387 +0,0 @@
-
-
-
-
-  
-
-
-  
-  
-  
-  Beam Eclipse Tips
-  
-  https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
-

Jenkins build is back to normal : beam_PostCommit_Website_Publish #95

2018-10-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Website_Publish #94

2018-10-04 Thread Apache Jenkins Server

See 


Changes:

[github] [BEAM-5651] Add --update-snapshots flag to archetype tests

--
[...truncated 8.70 KB...]

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 1.394 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) completed. Took 0.022 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
2,5,main]) completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.002 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:buildSrc:build (Thread[Task worker for ':buildSrc'

[beam] branch master updated (bf22926 -> 0bef2fc)

2018-10-04 Thread scott

This is an automated email from the ASF dual-hosted git repository.

scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bf22926  Merge pull request #6573: [BEAM-5651] Add --update-snapshots 
flag to archetype tests
 add 60d7de9  Remove outdated IDE pages and point to wiki instead
 add 0bef2fc  Merge pull request #6575: Remove outdated IDE pages and point 
to wiki instead

No new revisions were added by this update.

Summary of changes:
 website/src/_includes/section-menu/contribute.html |  4 +-
 website/src/contribute/eclipse.md  | 96 --
 website/src/contribute/intellij.md | 94 -
 3 files changed, 2 insertions(+), 192 deletions(-)
 delete mode 100644 website/src/contribute/eclipse.md
 delete mode 100644 website/src/contribute/intellij.md

[jira] [Commented] (BEAM-5652) [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py nosetests' (exited with code -11)

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638980#comment-16638980
 ] 

Valentyn Tymofieiev commented on BEAM-5652:
---

This is a duplicate of https://issues.apache.org/jira/browse/BEAM-5243?, I'll 
add some comments there.

> [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py 
> nosetests' (exited with code -11)
> --
>
> Key: BEAM-5652
> URL: https://issues.apache.org/jira/browse/BEAM-5652
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
> Fix For: Not applicable
>
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/ysct2bmi66p5c/console-log?task=:beam-sdks-python:testPython2#L4784]
>  * [Test source 
> code|https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/build.gradle#L130]
> Initial investigation:
> It seems that all tests passed, but the overall command failed. I'm not sure 
> why.
> Ran 1726 tests in 261.611s
> OK (SKIP=324)
> ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
>  setup.py nosetests' (exited with code -11)
> ___ summary 
> 
> ERROR:   py27: commands failed
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5613) Snapshot Python dependencies and add to Python_NightlySnapshot

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5613?focusedWorklogId=151403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151403
 ]

ASF GitHub Bot logged work on BEAM-5613:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:30
Start Date: 04/Oct/18 22:30
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6551: [BEAM-5613] Snapshot of Python depedency and add it to nightly snapshot 
job
URL: https://github.com/apache/beam/pull/6551#discussion_r222847032
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -84,6 +84,18 @@ task buildPython(dependsOn: 'setupVirtualenv') {
 }
 build.dependsOn buildPython
 
+// Snapshot of dependency requirements defined in setup.py.
+// Results will be stored in files under Gradle build directory.
+task depSnapshot(dependsOn: 'installGcpTest') {
+  doLast {
+println 'Snapshoting full dependencies requirements with versions info to 
requirements.txt.'
+exec {
+  executable 'sh'
+  args '-c', ". ${envdir}/bin/activate && pip freeze --local > 
${project.buildDir}/requirements.txt"
 
 Review comment:
   sg.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151403)
Time Spent: 1h 40m  (was: 1.5h)

> Snapshot Python dependencies and add to Python_NightlySnapshot
> --
>
> Key: BEAM-5613
> URL: https://issues.apache.org/jira/browse/BEAM-5613
> Project: Beam
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Python SDK depends on a list of libraries without specific versions. 
> Unexpected or unaware version change could happen if there is a new release 
> or range change in setup.py. This can be detected by dependency checking 
> tools which require to track dependency list and their versions periodically. 
> We could snapshot dependencies periodically using pip freeze and python 
> setup.py egg_info and upload them to a public accessible location to benefit 
> people who want to check dependency info or building related tools.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5652) [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py nosetests' (exited with code -11)

2018-10-04 Thread Valentyn Tymofieiev (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev resolved BEAM-5652.
---
   Resolution: Duplicate
Fix Version/s: Not applicable

> [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py 
> nosetests' (exited with code -11)
> --
>
> Key: BEAM-5652
> URL: https://issues.apache.org/jira/browse/BEAM-5652
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
> Fix For: Not applicable
>
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/ysct2bmi66p5c/console-log?task=:beam-sdks-python:testPython2#L4784]
>  * [Test source 
> code|https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/build.gradle#L130]
> Initial investigation:
> It seems that all tests passed, but the overall command failed. I'm not sure 
> why.
> Ran 1726 tests in 261.611s
> OK (SKIP=324)
> ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
>  setup.py nosetests' (exited with code -11)
> ___ summary 
> 
> ERROR:   py27: commands failed
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (3de49ec -> bf22926)

2018-10-04 Thread scott

This is an automated email from the ASF dual-hosted git repository.

scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3de49ec  Merge pull request #6576: Fix redirect for current release 
docs
 add e68b5ff  [BEAM-5651] Add --update-snapshots flag to archetype tests
 new bf22926  Merge pull request #6573: [BEAM-5651] Add --update-snapshots 
flag to archetype tests

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 release/src/main/groovy/QuickstartArchetype.groovy | 1 +
 1 file changed, 1 insertion(+)

[jira] [Work logged] (BEAM-5651) [beam_PostRelease_NightlySnapshot] [:beam-runners-spark:runQuickstartJavaSpark] Could not find artifact org.apache.beam:beam-model-fn-execution:jar:2.8.0-SNAPSHOT

2018-10-04 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5651?focusedWorklogId=151402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151402
 ]

ASF GitHub Bot logged work on BEAM-5651:


Author: ASF GitHub Bot
Created on: 04/Oct/18 22:30
Start Date: 04/Oct/18 22:30
Worklog Time Spent: 10m 
  Work Description: swegner closed pull request #6573: [BEAM-5651] Add 
--update-snapshots flag to archetype tests
URL: https://github.com/apache/beam/pull/6573
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 151402)
Time Spent: 1h  (was: 50m)

> [beam_PostRelease_NightlySnapshot] 
> [:beam-runners-spark:runQuickstartJavaSpark] Could not find artifact 
> org.apache.beam:beam-model-fn-execution:jar:2.8.0-SNAPSHOT
> --
>
> Key: BEAM-5651
> URL: https://issues.apache.org/jira/browse/BEAM-5651
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/389/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/bfqknnt245izk/console-log?task=:beam-runners-spark:runQuickstartJavaSpark#L479]
>  * [Test source 
> code|https://github.com/apache/beam/blob/master/release/src/main/groovy/quickstart-java-spark.groovy]
> Initial investigation:
> Maven failed to download the 2.8.0-SNAPSHOT artifact for 
> beam-model-fn-execution. However, I can see that snapshot binaries exist: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-model-fn-execution/2.8.0-SNAPSHOT/
> {{[ERROR] Failed to execute goal on project word-count-beam: Could not 
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could 
> not find artifact org.apache.beam:beam-model-fn-execution:jar:2.8.0-SNAPSHOT 
> in test.release 
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6573: [BEAM-5651] Add --update-snapshots flag to archetype tests

2018-10-04 Thread scott

This is an automated email from the ASF dual-hosted git repository.

scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit bf22926061521db5b00498d70ee35cda8303a05a
Merge: 3de49ec e68b5ff
Author: Scott Wegner 
AuthorDate: Thu Oct 4 18:29:57 2018 -0400

Merge pull request #6573: [BEAM-5651] Add --update-snapshots flag to 
archetype tests

 release/src/main/groovy/QuickstartArchetype.groovy | 1 +
 1 file changed, 1 insertion(+)

[beam] branch asf-site updated: Publishing website 2018/10/04 22:23:47 at commit 3de49ec

2018-10-04 Thread git-site-role

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new a949eab  Publishing website 2018/10/04 22:23:47 at commit 3de49ec
a949eab is described below

commit a949eab10d1f87dd0358951fa7e3568bfaa34119
Author: jenkins 
AuthorDate: Thu Oct 4 22:23:48 2018 +

Publishing website 2018/10/04 22:23:47 at commit 3de49ec
---
 website/generated-content/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/website/generated-content/.htaccess 
b/website/generated-content/.htaccess
index 77dabf4..b36da9e 100644
--- a/website/generated-content/.htaccess
+++ b/website/generated-content/.htaccess
@@ -21,4 +21,4 @@ RewriteRule ^(.*)$ https://beam.apache.org/$1 [L,R=301]
 # The following redirect maintains the previously supported URLs.
 RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" 
"https://beam.apache.org/documentation/releases/$1$2;
 # Keep this updated to point to the current release.
-RedirectMatch "/releases/([^/]+)/current(.*)" 
"https://beam.apache.org/documentation/releases/$1/2.6.0$2;
+RedirectMatch "/releases/([^/]+)/current(.*)" 
"https://beam.apache.org/releases/$1/2.6.0$2;

[beam] branch master updated (87dae1e -> 3de49ec)

2018-10-04 Thread melap

This is an automated email from the ASF dual-hosted git repository.

melap pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 87dae1e  Merge pull request #6552 from angoenka/python_worker_uuid
 add 0817dbd  Fix redirect for current release docs
 new 3de49ec  Merge pull request #6576: Fix redirect for current release 
docs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 website/src/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] 01/01: Merge pull request #6576: Fix redirect for current release docs

2018-10-04 Thread melap

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3de49ec8adb581c25b7f252df9d893abcbc31e8c
Merge: 87dae1e 0817dbd
Author: Melissa Pashniak 
AuthorDate: Thu Oct 4 15:22:23 2018 -0700

Merge pull request #6576: Fix redirect for current release docs

 website/src/.htaccess | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

1 2 3 4 >

1 - 100 of 374 matches

Mail list logo