date:20180319


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82167
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:48
Start Date: 20/Mar/18 05:48
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4853: 
[BEAM-3840] Get python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#discussion_r175660458
 
 

 ##
 File path: release/src/main/groovy/run_release_candidate_python_quickstart.sh
 ##
 @@ -179,12 +111,13 @@ gpg --verify $ASC_FILE_NAME $BEAM_PYTHON_SDK
 
 print_separator "Creating new virtualenv and installing the SDK"
 virtualenv temp_virtualenv
-. temp_virtualenv/bin/activate && pip install $BEAM_PYTHON_SDK[gcp]
+. temp_virtualenv/bin/activate
 
 Review comment:
   Thank you for the explanation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82167)
Time Spent: 19h 50m  (was: 19h 40m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=82166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82166
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:45
Start Date: 20/Mar/18 05:45
Worklog Time Spent: 10m 
  Work Description: cclauss commented on issue #4820: [BEAM-1251] Upgrade 
from buffer to memoryview for Python 3
URL: https://github.com/apache/beam/pull/4820#issuecomment-374483374
 
 
   Are we using the current [python-snappy 
0.52](https://github.com/andrix/python-snappy/releases)?  Perhaps @martindurant 
has some ideas for us.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82166)
Time Spent: 8.5h  (was: 8h 20m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82163
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:25
Start Date: 20/Mar/18 05:25
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374480227
 
 
   @aaltay Changes are made base on comments. Please review.
   1. Add new function verify_user_score();
   2. Re-formatting code follows google3 shell style.



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82163)
Time Spent: 19h 40m  (was: 19.5h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82160
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374439404
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82160)
Time Spent: 19h 10m  (was: 19h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82158
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374428164
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82158)
Time Spent: 18h 50m  (was: 18h 40m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82161
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374440071
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82161)
Time Spent: 19h 20m  (was: 19h 10m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82159
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374432619
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82159)
Time Spent: 19h  (was: 18h 50m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82162
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4853: 
[BEAM-3840] Get python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#discussion_r175658181
 
 

 ##
 File path: 
release/src/main/groovy/run_release_candidate_python_mobile_gaming.sh
 ##
 @@ -0,0 +1,167 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+#  This file will verify Apache/Beam release candidate python by following 
steps:
+#
+#  1. Create a new virtualenv and install the SDK
+#  2. Run UserScore examples with DirectRunner
+#  3. Run UserScore examples with DataflowRunner
+#  4. Run HourlyTeamScore on DirectRunner
+#  5. Run HourlyTeamScore on DataflowRunner
+#
+
+set -e
+set -v
+
+source release/src/main/groovy/python_release_automation_utils.sh
+
+complete() {
+print_separator "Validation $1"
+rm -rf $TMPDIR
+}
+
+verify_houly_team_score() {
+# $1 runner type
+retry=3
+should_see='AntiqueBrassPlatypus'
+while(( $retry >= 0 ))
+do
+if [[ $retry > 0 ]]
+then
+bq_pull_result=$(bq head -n 100 
$DATASET.hourly_team_score_python_$1)
+if [[ $bq_pull_result = *"$should_see"* ]]
+then
+echo "SUCCEED: HourlyTeamScore example successful run on 
$1-runner"
+break
+else
+retry=$(($retry-1))
+echo "Did not find team scores, retry left: $retry"
+sleep 15
+fi
+else
+echo "FAILED: HourlyTeamScore example failed running on $1-runner. 
Did not found scores of team $should_see in $DATASET.leader_board"
+complete "FAILED"
+exit 1
+fi
+done
+}
+
+
+print_separator "Start Mobile Gaming Examples"
+echo "SDK version: $VERSION"
+
+TMPDIR=$(mktemp -d)
+echo $TMPDIR
+pushd $TMPDIR
+
+#
+# 1. Download files from RC staging location, install python sdk
+#
+
+wget $CANDIDATE_URL$BEAM_PYTHON_SDK
+print_separator "Creating new virtualenv and installing the SDK"
+virtualenv temp_virtualenv
+. temp_virtualenv/bin/activate
+gcloud_version=$(gcloud --version | head -1 | awk '{print $4}')
+if [[ "$gcloud_version" < "189" ]]
+then
+  update_gcloud
+fi
+pip install $BEAM_PYTHON_SDK[gcp]
+
+
+#
+# 2. Run UserScore with DirectRunner
+#
+
+print_separator "Running userscore example with DirectRunner"
+output_file_name="$USERSCORE_OUTPUT_PREFIX-direct-runner.txt"
+python -m apache_beam.examples.complete.game.user_score \
+--output=$output_file_name \
+--project=$PROJECT_ID \
+--dataset=$DATASET \
+--input=gs://$BUCKET_NAME/5000_gaming_data.csv
+if ls $output_file_name* 1> /dev/null 2>&1;
+then
+   echo "Found output file(s):"
+   ls $output_file_name*
+else
+   echo "ERROR: output file not found."
+   complete "failed when running userscore example with DirectRunner."
+   exit 1
+fi
+echo "SUCCEED: UserScore successfully run on DirectRunner."
+
+
+#
+# 3. Run UserScore with DataflowRunner
+#
+
+print_separator "Running userscore example with DataflowRunner"
+output_file_name="$USERSCORE_OUTPUT_PREFIX-dataflow-runner.txt"
+python -m apache_beam.examples.complete.game.user_score \
+--project=$PROJECT_ID \
+--runner=DataflowRunner \
+--temp_location=gs://$BUCKET_NAME/temp/ \
+--sdk_location=$BEAM_PYTHON_SDK \
+--input=gs://$BUCKET_NAME/5000_gaming_data.csv \
+--output=gs://$BUCKET_NAME/$output_file_name
+# verify results.
+userscore_output_in_gcs="gs://$BUCKET_NAME/$output_file_name"
+gcs_pull_result=$(gsutil ls gs://$BUCKET_NAME)
+if [[ $gcs_pull_result != *$userscore_output_in_gcs* ]]
 
 Review comment:
   Done. extracted those sections of code to a common function.


This is an automated message from the

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82155
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374392304
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82155)
Time Spent: 18h 20m  (was: 18h 10m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82146
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374456758
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82146)
Time Spent: 16h 50m  (was: 16h 40m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82151
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374342521
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82151)
Time Spent: 17h 40m  (was: 17.5h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82157
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:23
Start Date: 20/Mar/18 05:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374413097
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82157)
Time Spent: 18h 40m  (was: 18.5h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82153
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374350300
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82153)
Time Spent: 18h  (was: 17h 50m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82149
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374303143
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82149)
Time Spent: 17h 20m  (was: 17h 10m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82147
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374455458
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82147)
Time Spent: 17h  (was: 16h 50m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82154
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374382626
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82154)
Time Spent: 18h 10m  (was: 18h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82152
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374344271
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82152)
Time Spent: 17h 50m  (was: 17h 40m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82156
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374406439
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82156)
Time Spent: 18.5h  (was: 18h 20m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82148
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374301984
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82148)
Time Spent: 17h 10m  (was: 17h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82150
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 05:22
Start Date: 20/Mar/18 05:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374312697
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82150)
Time Spent: 17.5h  (was: 17h 20m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3889) Revise python streaming mobile gaming examples.


[ 
https://issues.apache.org/jira/browse/BEAM-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405801#comment-16405801
 ] 

Valentyn Tymofieiev commented on BEAM-3889:
---

Leader_board example also did not work for me. 

> Revise python streaming mobile gaming examples.
> ---
>
> Key: BEAM-3889
> URL: https://issues.apache.org/jira/browse/BEAM-3889
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Major
>
> I spent some time trying to run streaming examples [1] on Direct runner and 
> ran into the following issues:
>  * Examples use deprecated code as per warnings 
>  ** DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  project = pipeline.options.view_as(GoogleCloudOptions).project
> /tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
>  DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  pipeline.replace_all(_get_transform_overrides(pipeline.options))
>  * Path to Injector has a typo (java8 instead of java)
>  * I was not able to use the injector to produce an output on a pubsub topic, 
> but perhaps I missed some instructions. Injector would exit after:
>  ** Warning! You are not using service account credentials to authenticate.
> You need to use service account credentials for this example,
> since user-level credentials do not have enough pubsub quota,
> and so you will run out of PubSub quota very quickly.
>  ** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS 
> and also tried passing 
> -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
>  * Examples say they are not applicable for Dataflow runner, is it still the 
> case?
>  * Running injector requires mvn, which is being replaced with gradle and 
> also requires familiarity with java toolchain, even though the example is for 
> Python SDK.
>  
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3851) Support element timestamps while publishing to Kafka.


 [ 
https://issues.apache.org/jira/browse/BEAM-3851?focusedWorklogId=82144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82144
 ]

ASF GitHub Bot logged work on BEAM-3851:


Author: ASF GitHub Bot
Created on: 20/Mar/18 04:28
Start Date: 20/Mar/18 04:28
Worklog Time Spent: 10m 
  Work Description: rangadi commented on issue #4868: [BEAM-3851] Option to 
preserve element timestamp while publishing to Kafka.
URL: https://github.com/apache/beam/pull/4868#issuecomment-374470637
 
 
   Added `.withPublishTimestampFunction()` (this is lambda friendly). Also 
retained `withInputTimestamp()` which makes it simpler for users to use element 
timestamp. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82144)
Time Spent: 40m  (was: 0.5h)

> Support element timestamps while publishing to Kafka.
> -
>
> Key: BEAM-3851
> URL: https://issues.apache.org/jira/browse/BEAM-3851
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.3.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> KafkaIO sink should support using input element timestamp for the message 
> published to Kafka. Otherwise there is no way for user to influence the 
> timestamp of the messages in Kafka sink.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3889) Revise python streaming mobile gaming examples.


[ 
https://issues.apache.org/jira/browse/BEAM-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405789#comment-16405789
 ] 

Valentyn Tymofieiev commented on BEAM-3889:
---

Correction: I was able to run the injector to populate a pubsub topic after 
fixing credential file, but was not able to produce an output in BigQuery with 
game_stats pipeline running on DirectRunner.

> Revise python streaming mobile gaming examples.
> ---
>
> Key: BEAM-3889
> URL: https://issues.apache.org/jira/browse/BEAM-3889
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Major
>
> I spent some time trying to run streaming examples [1] on Direct runner and 
> ran into the following issues:
>  * Examples use deprecated code as per warnings 
>  ** DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  project = pipeline.options.view_as(GoogleCloudOptions).project
> /tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
>  DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  pipeline.replace_all(_get_transform_overrides(pipeline.options))
>  * Path to Injector has a typo (java8 instead of java)
>  * I was not able to use the injector to produce an output on a pubsub topic, 
> but perhaps I missed some instructions. Injector would exit after:
>  ** Warning! You are not using service account credentials to authenticate.
> You need to use service account credentials for this example,
> since user-level credentials do not have enough pubsub quota,
> and so you will run out of PubSub quota very quickly.
>  ** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS 
> and also tried passing 
> -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
>  * Examples say they are not applicable for Dataflow runner, is it still the 
> case?
>  * Running injector requires mvn, which is being replaced with gradle and 
> also requires familiarity with java toolchain, even though the example is for 
> Python SDK.
>  
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #1146

See

[jira] [Created] (BEAM-3889) Revise pythonstreaming mobile gaming examples.

Valentyn Tymofieiev created BEAM-3889:
-

Summary: Revise pythonstreaming mobile gaming examples.
Key: BEAM-3889
URL: https://issues.apache.org/jira/browse/BEAM-3889
Project: Beam
Issue Type: Bug
Components: examples-python
Reporter: Valentyn Tymofieiev
Assignee: David Cavazos

I spent some time trying to run streaming examples [1] on Direct runner and ran
into the following issues:
* Examples use deprecated code as per warnings
** DeprecationWarning: options is deprecated since First stable release..
References to .options will not be supported
project = pipeline.options.view_as(GoogleCloudOptions).project
/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
DeprecationWarning: options is deprecated since First stable release..
References to .options will not be supported
pipeline.replace_all(_get_transform_overrides(pipeline.options))
* Path to Injector has a typo (java8 instead of java)
* I was not able to use the injector to produce an output on a pubsub topic,
but perhaps I missed some instructions. Injector would exit after:
** Warning! You are not using service account credentials to authenticate.
You need to use service account credentials for this example,
since user-level credentials do not have enough pubsub quota,
and so you will run out of PubSub quota very quickly.
** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS and
also tried passing -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
* Examples say they are not applicable for Dataflow runner, is it still the
case?
* Running injector requires mvn, which is being replaced with gradle and also
requires familiarity with java toolchain, even though the example is for Python
SDK.

[1]
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3889) Revise python streaming mobile gaming examples.


 [ 
https://issues.apache.org/jira/browse/BEAM-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3889:
--
Summary: Revise python streaming mobile gaming examples.  (was: Revise 
pythonstreaming mobile gaming examples.)

> Revise python streaming mobile gaming examples.
> ---
>
> Key: BEAM-3889
> URL: https://issues.apache.org/jira/browse/BEAM-3889
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Major
>
> I spent some time trying to run streaming examples [1] on Direct runner and 
> ran into the following issues:
>  * Examples use deprecated code as per warnings 
>  ** DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  project = pipeline.options.view_as(GoogleCloudOptions).project
> /tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py:337:
>  DeprecationWarning: options is deprecated since First stable release.. 
> References to .options will not be supported
>  pipeline.replace_all(_get_transform_overrides(pipeline.options))
>  * Path to Injector has a typo (java8 instead of java)
>  * I was not able to use the injector to produce an output on a pubsub topic, 
> but perhaps I missed some instructions. Injector would exit after:
>  ** Warning! You are not using service account credentials to authenticate.
> You need to use service account credentials for this example,
> since user-level credentials do not have enough pubsub quota,
> and so you will run out of PubSub quota very quickly.
>  ** Note that I set an environment variable GOOGLE_APPLICATION_CREDENTIALS 
> and also tried passing 
> -DGOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json.
>  * Examples say they are not applicable for Dataflow runner, is it still the 
> case?
>  * Running injector requires mvn, which is being replaced with gradle and 
> also requires familiarity with java toolchain, even though the example is for 
> Python SDK.
>  
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82138
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 03:02
Start Date: 20/Mar/18 03:02
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374456758
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82138)
Time Spent: 16h 40m  (was: 16.5h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82137
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 02:54
Start Date: 20/Mar/18 02:54
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374455458
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82137)
Time Spent: 16.5h  (was: 16h 20m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.


 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3824:
--
Summary: Use  WriteToBigQuery in Python mobile gaming examples.   (was: BQ 
sink fails on Direct Runner for 2.4.0 RC2)

> Use  WriteToBigQuery in Python mobile gaming examples. 
> ---
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File

[jira] [Updated] (BEAM-3824) Use WriteToBigQuery in Python mobile gaming examples.


 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3824:
--
Component/s: (was: runner-direct)
 examples-python

> Use  WriteToBigQuery in Python mobile gaming examples. 
> ---
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
>

[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2


 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3824:
--
Affects Version/s: (was: 2.4.0)

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>

[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2


 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3824:
--
Priority: Minor  (was: Blocker)

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82129
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 02:08
Start Date: 20/Mar/18 02:08
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go 
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374448020
 
 
   Now I get:
   
   ```
   [flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkRunner - 
Artifact registration failed
   java.lang.RuntimeException: Unexpected exception while writing artifact
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:358)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
at 
org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at 
org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
   Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusException: 
INVALID_ARGUMENT: No such artifact ARTIFACT_default_worker
at 
org.apache.beam.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
org.apache.beam.util.concurrent.AbstractFuture.get(AbstractFuture.java:459)
at 
org.apache.beam.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76)
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:352)
... 8 more
   Caused by: io.grpc.StatusException: INVALID_ARGUMENT: No such artifact 
ARTIFACT_default_worker
at io.grpc.Status.asException(Status.java:548)
at 
org.apache.beam.artifact.local.LocalArtifactSource.getArtifact(LocalArtifactSource.java:54)
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
... 8 more
   Caused by: java.io.FileNotFoundException: 
/tmp/beam-artifact-staging/go-job-1521511453974075000_1141769330/artifacts/ARTIFACT_default_worker
 (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at 
org.apache.beam.artifact.local.LocalArtifactSource.getArtifact(LocalArtifactSource.java:42)
... 9 more
   [flink-runner-job-server] ERROR 
org.apache.beam.runners.flink.FlinkJobInvocation - Error during job invocation 
go-job-1521511453974075000_1141769330_1342644097.
   java.lang.RuntimeException: Artifact registration failed
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:124)
at 
org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at 
org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.RuntimeException: Unexpected exception while writing 
artifact
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:358)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
... 7 more
   Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusException: 
INVALID_ARGUMENT: No such artifact ARTIFACT_default_worker
at 
org.apache.beam.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
org.apache.beam.util.concurrent.AbstractFuture.get(AbstractFuture.java:459)
at 
org.apache.beam.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76)
at

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5185

See

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82128
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:41
Start Date: 20/Mar/18 01:41
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go 
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374443415
 
 
   Thanks @aljoscha. I figured it was something simple.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82128)
Time Spent: 2h 50m  (was: 2h 40m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82127
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:40
Start Date: 20/Mar/18 01:40
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go 
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374443211
 
 
   @lostluck Thanks! PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82127)
Time Spent: 2h 40m  (was: 2.5h)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82125=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82125
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:37
Start Date: 20/Mar/18 01:37
Worklog Time Spent: 10m 
  Work Description: aljoscha commented on issue #4888: [BEAM-3287] Add Go 
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374442695
 
 
   @herohde We figured out what the problem is: this line 
https://github.com/apache/beam/blob/f490cdf912f9614ebcc94b8df7eaa0614d57c737/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java#L352
 calls `wait()` while it should call `get()` on the future.
   
   @axelmagn Is fixing it on his `hacking-job-server` branch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82125)
Time Spent: 2h 20m  (was: 2h 10m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82126
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:37
Start Date: 20/Mar/18 01:37
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175634571
 
 

 ##
 File path: sdks/go/pkg/beam/runners/universal/runnerlib/stage.go
 ##
 @@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package runnerlib
+
+import (
+   "context"
+   "fmt"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/artifact"
+   jobpb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
+   "github.com/apache/beam/sdks/go/pkg/beam/util/grpcx"
+)
+
+// Stage stages the worker binary and any additional content to the given 
endpoint.
 
 Review comment:
   Change it to files, which is what this helper function accepts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82126)
Time Spent: 2.5h  (was: 2h 20m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82124
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:36
Start Date: 20/Mar/18 01:36
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175634408
 
 

 ##
 File path: sdks/go/pkg/beam/runners/universal/runnerlib/job.go
 ##
 @@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package runnerlib
+
+import (
+   "context"
+   "fmt"
+   "io"
+
+   "github.com/apache/beam/sdks/go/pkg/beam"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   jobpb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+   "github.com/apache/beam/sdks/go/pkg/beam/provision"
+   "github.com/golang/protobuf/proto"
+)
+
+// JobOptions capture the various options for submitting jobs
+// to universal runners.
+type JobOptions struct {
+   // Name is the job name.
+   Name string
+   // Experiments are additional experiments.
+   Experiments []string
+
+   // TODO(herohde) 3/17/2018: add further parametrization as needed
+
+   // InternalJavaRunner is the class of the receiving Java runner. To be 
removed.
+   InternalJavaRunner string
+}
+
+// Prepare prepares a job to the given endpoint. It returns an id and 
endpoint, if successful.
+func Prepare(ctx context.Context, client jobpb.JobServiceClient, p 
*pb.Pipeline, opt *JobOptions) (string, string, error) {
+   raw := runtime.RawOptionsWrapper{
+   Options: beam.PipelineOptions.Export(),
+   Runner:  opt.InternalJavaRunner,
+   AppName: opt.Name,
+   Experiments: append(opt.Experiments, "beam_fn_api"),
+   }
+
+   options, err := provision.OptionsToProto(raw)
+   if err != nil {
+   return "", "", fmt.Errorf("failed to produce pipeline options: 
%v", err)
+   }
+   req := {
+   Pipeline:p,
+   PipelineOptions: options,
+   JobName: opt.Name,
+   }
+   resp, err := client.Prepare(ctx, req)
+   if err != nil {
+   return "", "", fmt.Errorf("failed to connect to job service: 
%v", err)
+   }
+   return resp.GetPreparationId(), 
resp.GetArtifactStagingEndpoint().GetUrl(), nil
+}
+
+// Submit submits a job to the given endpoint. It returns a jobID, if 
successful.
 
 Review comment:
   Made them consistent.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82124)
Time Spent: 2h 10m  (was: 2h)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82122
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:35
Start Date: 20/Mar/18 01:35
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175634310
 
 

 ##
 File path: sdks/go/pkg/beam/runners/universal/runnerlib/job.go
 ##
 @@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package runnerlib
+
+import (
+   "context"
+   "fmt"
+   "io"
+
+   "github.com/apache/beam/sdks/go/pkg/beam"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   jobpb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+   "github.com/apache/beam/sdks/go/pkg/beam/provision"
+   "github.com/golang/protobuf/proto"
+)
+
+// JobOptions capture the various options for submitting jobs
+// to universal runners.
+type JobOptions struct {
+   // Name is the job name.
+   Name string
+   // Experiments are additional experiments.
+   Experiments []string
+
+   // TODO(herohde) 3/17/2018: add further parametrization as needed
+
+   // InternalJavaRunner is the class of the receiving Java runner. To be 
removed.
+   InternalJavaRunner string
+}
+
+// Prepare prepares a job to the given endpoint. It returns an id and 
endpoint, if successful.
 
 Review comment:
   Yes. Rephrased the description.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82122)
Time Spent: 2h  (was: 1h 50m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82123
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:35
Start Date: 20/Mar/18 01:35
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374442495
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82123)
Time Spent: 16h 20m  (was: 16h 10m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages


 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=82121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82121
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:33
Start Date: 20/Mar/18 01:33
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #4901: [BEAM-3744] Expand 
Pubsub read API for Python.
URL: https://github.com/apache/beam/pull/4901#issuecomment-374442161
 
 
   R: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82121)
Time Spent: 3h 50m  (was: 3h 40m)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82120
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:33
Start Date: 20/Mar/18 01:33
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374442032
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82120)
Time Spent: 16h 10m  (was: 16h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82119
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:33
Start Date: 20/Mar/18 01:33
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175634031
 
 

 ##
 File path: sdks/go/pkg/beam/runners/universal/runnerlib/job.go
 ##
 @@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package runnerlib
+
+import (
+   "context"
+   "fmt"
+   "io"
+
+   "github.com/apache/beam/sdks/go/pkg/beam"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   jobpb "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
+   pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
+   "github.com/apache/beam/sdks/go/pkg/beam/provision"
+   "github.com/golang/protobuf/proto"
+)
+
+// JobOptions capture the various options for submitting jobs
+// to universal runners.
+type JobOptions struct {
+   // Name is the job name.
+   Name string
+   // Experiments are additional experiments.
+   Experiments []string
+
+   // TODO(herohde) 3/17/2018: add further parametrization as needed
+
+   // InternalJavaRunner is the class of the receiving Java runner. To be 
removed.
+   InternalJavaRunner string
+}
+
+// Prepare prepares a job to the given endpoint. It returns an id and 
endpoint, if successful.
+func Prepare(ctx context.Context, client jobpb.JobServiceClient, p 
*pb.Pipeline, opt *JobOptions) (string, string, error) {
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82119)
Time Spent: 1h 50m  (was: 1h 40m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82118
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:28
Start Date: 20/Mar/18 01:28
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175633577
 
 

 ##
 File path: sdks/go/pkg/beam/runners/universal/runnerlib/compile.go
 ##
 @@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package runnerlib contains utilities for submitting Go pipelines
+// to a Beam model runner.
 
 Review comment:
   I don't think there is an agreed upon name long-term. The code package is 
"universal", so I'm not even internally consistent in this PR :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82118)
Time Spent: 1h 40m  (was: 1.5h)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82117
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:25
Start Date: 20/Mar/18 01:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4888: 
[BEAM-3287] Add Go support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#discussion_r175633251
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/translate.go
 ##
 @@ -28,12 +28,12 @@ import (
 const (
// Model constants
 
-   URNImpulse = "urn:beam:transform:impulse:v1"
+   URNImpulse = "beam:transform:impulse:v1"
URNParDo   = "urn:beam:transform:pardo:v1"
 
 Review comment:
   Yes, that one is still the old way.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82117)
Time Spent: 1.5h  (was: 1h 20m)

> Go SDK support for portable pipelines
> -
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job 
> submission w/ a docker container image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3287) Go SDK support for portable pipelines


 [ 
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82116
 ]

ASF GitHub Bot logged work on BEAM-3287:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:24
Start Date: 20/Mar/18 01:24
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go 
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374440447
 
 
   @aljoscha Thanks! Yes! That is how I'm testing this stuff. I'm currently 
hitting an issue with the artifacts on the Flink side:
   
   ```
   $ go run examples/wordcount/wordcount.go --runner=flink --output=/tmp/foo 
--endpoint=localhost:3000
   [{6: KV/GW/KV}]
   [{10: KV/GW/KV}]
   2018/03/19 18:19:09 No container image specified. Using dev image: 
'herohde-docker-apache.bintray.io/beam/go:latest'
   2018/03/19 18:19:09 Prepared job with id: 
go-job-1521508749032988000_2100233503
   2018/03/19 18:19:09 Cross-compiling 
/Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go
 as /var/folders/s2/97strbs55_353t_t7r24yf94009w4s/T/beam-go-1521508749041446000
   2018/03/19 18:19:25 Staged binary artifact with token: 
/private/tmp/beam-artifact-staging/go-job-1521508749032988000_2100233503
   2018/03/19 18:19:25 Submitted job: 
go-job-1521508749032988000_2100233503_-2090543569
   2018/03/19 18:19:25 Job state: RUNNING
   2018/03/19 18:19:25 Job state: FAILED
   2018/03/19 18:19:25 Failed to execute job: job 
go-job-1521508749032988000_2100233503_-2090543569 failed
   exit status 1
   
   ```
   
   Flink runner logs (info):
   ```
   [...]
   [grpc-default-executor-1] INFO org.apache.beam.runners.flink.FlinkRunner - 
PipelineOptions.filesToStage was not specified. Defaulting to files from the 
classpath: will stage 1 files. Enable logging at DEBUG level to see which files 
will be staged.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - 
Executing pipeline using FlinkRunner.
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - 
Translating pipeline to Flink program.
   [grpc-default-executor-2] WARN 
org.apache.beam.runners.flink.FlinkJobInvocation - addMessageObserver() not yet 
implemented.
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment - Using 
portability layer
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment - Creating the 
required Batch Execution Environment.
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator -  
enterCompositeTransform- 
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- Impulse
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- 
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- CoGBK
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- 
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- CoGBK
   [flink-runner-job-server] INFO 
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator -  
leaveCompositeTransform- 
   [flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner - 
Registering pipeline artifacts in Flink program.
   [flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkRunner - 
Artifact registration failed
   java.lang.IllegalMonitorStateException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
at 
org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at 
org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at 
org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
   [flink-runner-job-server] ERROR

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82115
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:22
Start Date: 20/Mar/18 01:22
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374440071
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82115)
Time Spent: 16h  (was: 15h 50m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82113
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:18
Start Date: 20/Mar/18 01:18
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374439404
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82113)
Time Spent: 15h 50m  (was: 15h 40m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 15h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3817) Incompatible input encoding running Tornadoes example on dataflow


 [ 
https://issues.apache.org/jira/browse/BEAM-3817?focusedWorklogId=82111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82111
 ]

ASF GitHub Bot logged work on BEAM-3817:


Author: ASF GitHub Bot
Created on: 20/Mar/18 01:15
Start Date: 20/Mar/18 01:15
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4840: [BEAM-3817] Switch Go 
SDK BQ write to not use side input
URL: https://github.com/apache/beam/pull/4840#issuecomment-374438977
 
 
   @robertwb Any concerns about merging?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82111)
Time Spent: 1h  (was: 50m)

> Incompatible input encoding running Tornadoes example on dataflow
> -
>
> Key: BEAM-3817
> URL: https://issues.apache.org/jira/browse/BEAM-3817
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Braden Bassingthwaite
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Trying to run:
> go run tornadoes.go --output=:bbass.tornadoes --project  
> --runner dataflow --staging_location=gs://bbass/tornadoes 
> --worker_harness_container_image=gcr.io//beam/go
> Found here:
> [https://github.com/apache/beam/blob/master/sdks/go/examples/cookbook/tornadoes/tornadoes.go]
> I can run it locally but I get the error on Dataflow:
> (8fa522c2bb03a769): Workflow failed. Causes: (8fa522c2bb03ab04): Incompatible 
> input encoding. 
>  
> I built the worker_harness_container_image using:
> mvn clean install -DskipTests -Pbuild-containers 
> -Ddocker-repository-root=gcr.io//beam
>  
> Thanks!
>  
> Very excited to start using the golang beam sdk! great work!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

2018-03-19 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405654#comment-16405654
 ] 

Ahmet Altay commented on BEAM-3824:
---

I was able to run hourly team score after the rollback. I used an invalid 
argument to run. You can ignore the previous comment.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82110
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 00:36
Start Date: 20/Mar/18 00:36
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374432619
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82110)
Time Spent: 15h 40m  (was: 15.5h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

2018-03-19 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405644#comment-16405644
 ] 

Ahmet Altay commented on BEAM-3824:
---

I think the title should be hourly team score example is broken. Even after the 
rollback I got the following error when I ran:

ValueError: Expected a table reference (PROJECT:DATASET.TABLE or DATASET.TABLE) 
instead of google.com:clouddfe:.leader_board.

 

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
>

Build failed in Jenkins: beam_PerformanceTests_Spark #1487

See 


Changes:

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[axelmagn] Add a generic interface for the state service.

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

--
[...truncated 90.23 KB...]
'apache-beam-testing:bqjob_r54bd684cc60676f_016240c750b9_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-20 00:20:29,013 7b3b9770 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-20 00:20:58,370 7b3b9770 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-20 00:21:00,723 7b3b9770 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r62db7538f2e02ca1_016240c7cc81_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r62db7538f2e02ca1_016240c7cc81_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r62db7538f2e02ca1_016240c7cc81_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-20 00:21:00,723 7b3b9770 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-20 00:21:29,473 7b3b9770 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-20 00:21:31,958 7b3b9770 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r4e55419ec741cd3b_016240c84682_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4e55419ec741cd3b_016240c84682_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4e55419ec741cd3b_016240c84682_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-20 00:21:31,958 7b3b9770 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-20 00:21:56,436 7b3b9770 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-20 00:21:58,913 7b3b9770 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r174a49105eb00f64_016240c8af5d_1 ... (0s) Current status: 
RUNNING

Jenkins build is back to normal : beam_PerformanceTests_JDBC #350

See

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #39

See

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages


 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=82101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82101
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 20/Mar/18 00:12
Start Date: 20/Mar/18 00:12
Worklog Time Spent: 10m 
  Work Description: udim opened a new pull request #4901: [BEAM-3744] 
Expand Pubsub read API for Python.
URL: https://github.com/apache/beam/pull/4901
 
 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82101)
Time Spent: 3h 40m  (was: 3.5h)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82100
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 20/Mar/18 00:10
Start Date: 20/Mar/18 00:10
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374428164
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82100)
Time Spent: 15.5h  (was: 15h 20m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1043

See 


Changes:

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[axelmagn] Add a generic interface for the state service.

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

--
[...truncated 1.31 KB...]
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3587328440009568570.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2047636121631671986.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1739272547545385836.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins115803349295491.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5189194080758032229.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6787764694078833860.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #6246

See 


--
[...truncated 35.48 KB...]
2018-03-20T00:05:09.522 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2018-03-20T00:05:09.522 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] 
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2018-03-20T00:05:09.522 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] 
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2018-03-20T00:05:09.522 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] 
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2018-03-20T00:05:09.522 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.522 [INFO] 

2018-03-20T00:05:09.522 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: Examples :: Java
2018-03-20T00:05:09.523 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2018-03-20T00:05:09.523 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Jackson
2018-03-20T00:05:09.523 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Join library
2018-03-20T00:05:09.523 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sketching
2018-03-20T00:05:09.523 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] 
2018-03-20T00:05:09.523 [INFO] 

2018-03-20T00:05:09.523 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sorter
2018-03-20T00:05:09.524 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.524 [INFO] 

2018-03-20T00:05:09.524 [INFO] 
2018-03-20T00:05:09.524 [INFO] 

2018-03-20T00:05:09.524 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: SQL
2018-03-20T00:05:09.524 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-20T00:05:09.524 [INFO] 

2018-03-20T00:05:09.524 [INFO] 
2018-03-20T00:05:09.524 [INFO] 

2018-03-20T00:05:09.524 [INFO] Skipping Apache Beam ::

[jira] [Work logged] (BEAM-1251) Python 3 Support


 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=82096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82096
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:44
Start Date: 19/Mar/18 23:44
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4820: [BEAM-1251] Upgrade 
from buffer to memoryview for Python 3
URL: https://github.com/apache/beam/pull/4820#issuecomment-374423323
 
 
   No, the changed version also does not work. This 
`six.binary_type(memoryview(data)[:-4])` results in the literal string of the 
form `` and fails with `snappy.UncompressError: Error 
while decompressing: invalid input`
   
   Besides binary_type is just str, even if it worked as expected in this case 
it would have created a copy of data, which beats the purpose.
   
   The real solution here would be to upgrade snappy to accept memoryview as an 
argument. If we cannot do that, we can remove the optimization and settle for 
`snappy.decompress(data[:-4])`. Or perhaps better we can conditionally keep the 
`buffer` for python2 only.
   
   CC'ing a few people who might have an idea of the impact of copying data 
here:
   cc: @chamikaramj @katsiapis
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82096)
Time Spent: 8h 20m  (was: 8h 10m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Priority: Trivial
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3888) Python SDK propagates pipeline options correctly when talking to Job API

Ben Sidhom created BEAM-3888:


 Summary: Python SDK propagates pipeline options correctly when 
talking to Job API
 Key: BEAM-3888
 URL: https://issues.apache.org/jira/browse/BEAM-3888
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


Pipeline options are not propagated correctly from the Python SDK to the 
runner. At least part of the problem is due to option key casing (e.g., between 
Java and Python). Additionally, some runners require special options for 
configuration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3882) Make StateRequestHandler::accept interface idiomatic.


 [ 
https://issues.apache.org/jira/browse/BEAM-3882?focusedWorklogId=82094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82094
 ]

ASF GitHub Bot logged work on BEAM-3882:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:39
Start Date: 19/Mar/18 23:39
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4900: 
[BEAM-3882] Fix StateRequestHandler interface to be idiomatic
URL: https://github.com/apache/beam/pull/4900#discussion_r175616304
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,7 +34,6 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  void accept(
-  BeamFnApi.StateRequest request, 
CompletionStage result)
+  CompletionStage 
accept(BeamFnApi.StateRequest request)
 
 Review comment:
   s/accept/handle


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82094)
Time Spent: 0.5h  (was: 20m)

> Make StateRequestHandler::accept interface idiomatic.
> -
>
> Key: BEAM-3882
> URL: https://issues.apache.org/jira/browse/BEAM-3882
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> StateRequestHandler was based on some Dataflow SDK source code that does not 
> conform to Beam coding conventions.  In particular, the accept method takes 
> its return value as a parameter, which is a code smell.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3882) Make StateRequestHandler::accept interface idiomatic.


 [ 
https://issues.apache.org/jira/browse/BEAM-3882?focusedWorklogId=82093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82093
 ]

ASF GitHub Bot logged work on BEAM-3882:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:39
Start Date: 19/Mar/18 23:39
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4900: 
[BEAM-3882] Fix StateRequestHandler interface to be idiomatic
URL: https://github.com/apache/beam/pull/4900#discussion_r175616304
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
 ##
 @@ -34,7 +34,6 @@
* Throwing an error during handling will complete the handler result 
{@link CompletionStage}
* exceptionally.
*/
-  void accept(
-  BeamFnApi.StateRequest request, 
CompletionStage result)
+  CompletionStage 
accept(BeamFnApi.StateRequest request)
 
 Review comment:
   s/accept/handle


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82093)
Time Spent: 20m  (was: 10m)

> Make StateRequestHandler::accept interface idiomatic.
> -
>
> Key: BEAM-3882
> URL: https://issues.apache.org/jira/browse/BEAM-3882
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> StateRequestHandler was based on some Dataflow SDK source code that does not 
> conform to Beam coding conventions.  In particular, the accept method takes 
> its return value as a parameter, which is a code smell.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82091
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175616259
 
 

 ##
 File path: sdks/python/apache_beam/testing/test_utils.py
 ##
 @@ -129,3 +131,47 @@ def delete_files(file_paths):
 raise RuntimeError('Clean up failed. Invalid file path: %s.' %
file_paths)
   FileSystems.delete(file_paths)
+
+
+def wait_for_subscriptions_created(subs, timeout=60):
+  """Wait for all PubSub subscriptions are created."""
+  return _wait_until_all_exist(subs, timeout)
+
+
+def wait_for_topics_created(topics, timeout=60):
+  """Wait for all PubSub topics are created."""
+  return _wait_until_all_exist(topics, timeout)
+
+
+def _wait_until_all_exist(components, timeout):
+  need_wait = set(components)
+  start_time = time.time()
+  while time.time() - start_time <= timeout:
+for c in components:
 
 Review comment:
   I think you can do `for c in needs_wait` instead to iterate over a shorter 
list and remove `c in need_wait` below.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82091)
Time Spent: 1h 20m  (was: 1h 10m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82085
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175615483
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
 ##
 @@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for the streaming wordcount example.
+
+Important: End-to-end test infrastructure for streaming pipeine in Python SDK
+is in development and is not yet available for use.
+
+Currently, this test is blocked until manually terminate the pipeline job.
 
 Review comment:
   How about:
   
   Currently, this test blocks until the job is manually terminated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82085)
Time Spent: 0.5h  (was: 20m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82088
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175616557
 
 

 ##
 File path: sdks/python/apache_beam/testing/test_utils.py
 ##
 @@ -129,3 +131,47 @@ def delete_files(file_paths):
 raise RuntimeError('Clean up failed. Invalid file path: %s.' %
file_paths)
   FileSystems.delete(file_paths)
+
+
+def wait_for_subscriptions_created(subs, timeout=60):
+  """Wait for all PubSub subscriptions are created."""
+  return _wait_until_all_exist(subs, timeout)
+
+
+def wait_for_topics_created(topics, timeout=60):
+  """Wait for all PubSub topics are created."""
+  return _wait_until_all_exist(topics, timeout)
+
+
+def _wait_until_all_exist(components, timeout):
+  need_wait = set(components)
+  start_time = time.time()
+  while time.time() - start_time <= timeout:
+for c in components:
+  if c in need_wait and c.exists():
+need_wait.remove(c)
+if len(need_wait) == 0:
+  return True
+time.sleep(2)
+
+  raise RuntimeError('Timeout after %d seconds. %d of %d topics/subscriptions '
 
 Review comment:
   Would it be helpful if you log what does not exist?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82088)
Time Spent: 50m  (was: 40m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82087
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175590831
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
 ##
 @@ -46,10 +52,39 @@ def run_pipeline(self, pipeline):
   print (
   'Found: https://console.cloud.google.com/dataflow/jobsDetail'
   '/locations/%s/jobs/%s?project=%s' % (region_id, job_id, project))
-self.result.wait_until_finish()
+
+if not options.view_as(StandardOptions).streaming:
+  self.result.wait_until_finish()
+else:
+  # TODO: Ideally, we want to wait until workers start successfully.
+  self.wait_until_running()
 
 if on_success_matcher:
   from hamcrest import assert_that as hc_assert_that
   hc_assert_that(self.result, pickler.loads(on_success_matcher))
 
 return self.result
+
+  def _is_in_terminate_state(self, job_state):
 
 Review comment:
   Name this `_is_in_terminal_state`.
   
   Can you use `_is_in_terminal_state` from `DataflowPipelineResult`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82087)
Time Spent: 40m  (was: 0.5h)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3887) Python SDK harness retrieves element codes from gRPC read/write nodes

Ben Sidhom created BEAM-3887:


 Summary: Python SDK harness retrieves element codes from gRPC 
read/write nodes
 Key: BEAM-3887
 URL: https://issues.apache.org/jira/browse/BEAM-3887
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


The python harness does not correctly get coders from the gRPC read/write 
transforms but rather from PCollection inputs/outputs. This prevents the 
harness from correctly receiving/transmitting prefix-encoded elements (seen as 
prefix-encoded byte arrays by the runner).

It should instead retrieve input and output coders from the gRPC nodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82089
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175615705
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
 ##
 @@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for the streaming wordcount example.
+
+Important: End-to-end test infrastructure for streaming pipeine in Python SDK
+is in development and is not yet available for use.
+
+Currently, this test is blocked until manually terminate the pipeline job.
+"""
+
+import logging
+import unittest
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.examples import streaming_wordcount
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing import test_utils
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+INPUT_TOPIC = 'wc_topic_input'
+OUTPUT_TOPIC = 'wc_topic_output'
+INPUT_SUB = 'wc_subscription_input'
+OUTPUT_SUB = 'wc_subscription_output'
+
+DEFAULT_INPUT_NUMBERS = 500
+
+
+class StreamingWordCountIT(unittest.TestCase):
+
+  def setUp(self):
+self.test_pipeline = TestPipeline(is_integration_test=True)
+
+# Set up PubSub environment.
+from google.cloud import pubsub
+self.pubsub_client = pubsub.Client(
+project=self.test_pipeline.get_option('project'))
+self.input_topic = self.pubsub_client.topic(INPUT_TOPIC)
+self.output_topic = self.pubsub_client.topic(OUTPUT_TOPIC)
+self.input_sub = self.input_topic.subscription(INPUT_SUB)
+self.output_sub = self.output_topic.subscription(OUTPUT_SUB)
+
+self._cleanup_pubsub()
+
+self.input_topic.create()
+self.output_topic.create()
+test_utils.wait_for_topics_created([self.input_topic, self.output_topic])
+self.input_sub.create()
+self.output_sub.create()
+
+  def _inject_numbers(self, topic, num_messages):
+"""Inject numbers as test data to PubSub."""
+logging.debug('Injecting %d numbers to topic %s',
+  num_messages, topic.full_name)
+for n in range(num_messages):
+  topic.publish(str(n))
+
+  def _cleanup_pubsub(self):
+test_utils.cleanup_subscription([self.input_sub, self.output_sub])
 
 Review comment:
   Rename to `cleanup_subscriptions`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82089)
Time Spent: 1h  (was: 50m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82090
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175591205
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
 ##
 @@ -46,10 +52,39 @@ def run_pipeline(self, pipeline):
   print (
   'Found: https://console.cloud.google.com/dataflow/jobsDetail'
   '/locations/%s/jobs/%s?project=%s' % (region_id, job_id, project))
-self.result.wait_until_finish()
+
+if not options.view_as(StandardOptions).streaming:
+  self.result.wait_until_finish()
+else:
+  # TODO: Ideally, we want to wait until workers start successfully.
+  self.wait_until_running()
 
 if on_success_matcher:
   from hamcrest import assert_that as hc_assert_that
   hc_assert_that(self.result, pickler.loads(on_success_matcher))
 
 return self.result
+
+  def _is_in_terminate_state(self, job_state):
+return job_state in [
+PipelineState.STOPPED, PipelineState.DONE,
+PipelineState.FAILED, PipelineState.CANCELLED,
+PipelineState.UPDATED, PipelineState.DRAINED,
+]
+
+  def wait_until_running(self):
 
 Review comment:
   Maybe you can generalize this as wait until state with timeout. (Pass 
desired state and timeout as arguments.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82090)
Time Spent: 1h 10m  (was: 1h)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82092
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175615878
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
 ##
 @@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for the streaming wordcount example.
+
+Important: End-to-end test infrastructure for streaming pipeine in Python SDK
+is in development and is not yet available for use.
+
+Currently, this test is blocked until manually terminate the pipeline job.
+"""
+
+import logging
+import unittest
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.examples import streaming_wordcount
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing import test_utils
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+INPUT_TOPIC = 'wc_topic_input'
+OUTPUT_TOPIC = 'wc_topic_output'
+INPUT_SUB = 'wc_subscription_input'
+OUTPUT_SUB = 'wc_subscription_output'
+
+DEFAULT_INPUT_NUMBERS = 500
+
+
+class StreamingWordCountIT(unittest.TestCase):
+
+  def setUp(self):
+self.test_pipeline = TestPipeline(is_integration_test=True)
+
+# Set up PubSub environment.
+from google.cloud import pubsub
+self.pubsub_client = pubsub.Client(
+project=self.test_pipeline.get_option('project'))
+self.input_topic = self.pubsub_client.topic(INPUT_TOPIC)
+self.output_topic = self.pubsub_client.topic(OUTPUT_TOPIC)
+self.input_sub = self.input_topic.subscription(INPUT_SUB)
+self.output_sub = self.output_topic.subscription(OUTPUT_SUB)
+
+self._cleanup_pubsub()
+
+self.input_topic.create()
+self.output_topic.create()
+test_utils.wait_for_topics_created([self.input_topic, self.output_topic])
+self.input_sub.create()
+self.output_sub.create()
+
+  def _inject_numbers(self, topic, num_messages):
+"""Inject numbers as test data to PubSub."""
+logging.debug('Injecting %d numbers to topic %s',
+  num_messages, topic.full_name)
+for n in range(num_messages):
+  topic.publish(str(n))
+
+  def _cleanup_pubsub(self):
+test_utils.cleanup_subscription([self.input_sub, self.output_sub])
+test_utils.cleanup_topics([self.input_topic, self.output_topic])
+
+  def tearDown(self):
+self._cleanup_pubsub()
+
+  @attr('developing_test')
+  def test_streaming_wordcount_it(self):
+# Set extra options to the pipeline for test purpose
+pipeline_verifiers = [PipelineStateMatcher(PipelineState.RUNNING)]
+extra_opts = {'input_sub': self.input_sub.full_name,
+  'output_topic': self.output_topic.full_name,
+  'on_success_matcher': all_of(*pipeline_verifiers)}
+
+# Generate input data and inject to PubSub.
+test_utils.wait_for_subscriptions_created([self.input_sub])
 
 Review comment:
   You already created subscriptions in `setUp`. Do you need to check it here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82092)
Time Spent: 1.5h  (was: 1h 20m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK


 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=82086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82086
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:27
Start Date: 19/Mar/18 23:27
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4874: 
[BEAM-3861] Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#discussion_r175591382
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
 ##
 @@ -46,10 +52,39 @@ def run_pipeline(self, pipeline):
   print (
   'Found: https://console.cloud.google.com/dataflow/jobsDetail'
   '/locations/%s/jobs/%s?project=%s' % (region_id, job_id, project))
-self.result.wait_until_finish()
+
+if not options.view_as(StandardOptions).streaming:
+  self.result.wait_until_finish()
+else:
+  # TODO: Ideally, we want to wait until workers start successfully.
+  self.wait_until_running()
 
 Review comment:
   What is the point of waiting until running? (i.e. What are we verifying by 
doing this?) Is this going to be addressed in the follow up PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82086)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3886) Python SDK harness does not contact State API if not needed

Ben Sidhom created BEAM-3886:


 Summary: Python SDK harness does not contact State API if not 
needed
 Key: BEAM-3886
 URL: https://issues.apache.org/jira/browse/BEAM-3886
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


The Python harness always talks to the State API, even if it is never used by 
the current process bundle. As a minor optimization and to make implementing 
new runners easier, the harness should not talk to the State server unless it's 
actually needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5184

See

[jira] [Work logged] (BEAM-3882) Make StateRequestHandler::accept interface idiomatic.


 [ 
https://issues.apache.org/jira/browse/BEAM-3882?focusedWorklogId=82084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82084
 ]

ASF GitHub Bot logged work on BEAM-3882:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:22
Start Date: 19/Mar/18 23:22
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #4900: [BEAM-3882] 
Fix StateRequestHandler interface to be idiomatic
URL: https://github.com/apache/beam/pull/4900
 
 
   The previous implementation of StateRequestHandler did not conform
   to Beam coding conventions.  It should not have accepted a return
   value as a parameter.  This fix updates the interface to instead
   return a CompletableStage.
   
   R: @tgroh 
   CC: @aljoscha, @bsidhom, @lukecwik 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82084)
Time Spent: 10m
Remaining Estimate: 0h

> Make StateRequestHandler::accept interface idiomatic.
> -
>
> Key: BEAM-3882
> URL: https://issues.apache.org/jira/browse/BEAM-3882
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> StateRequestHandler was based on some Dataflow SDK source code that does not 
> conform to Beam coding conventions.  In particular, the accept method takes 
> its return value as a parameter, which is a code smell.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=82083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82083
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:01
Start Date: 19/Mar/18 23:01
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #4899: [BEAM-3545] Go SDK 
UserCounters
URL: https://github.com/apache/beam/pull/4899#issuecomment-374414489
 
 
   R: @herohde @wcn3


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82083)
Time Spent: 20m  (was: 10m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=82082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82082
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 19/Mar/18 23:00
Start Date: 19/Mar/18 23:00
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #4899: [BEAM-3545] 
Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899
 
 
   Add basic usercounters to the Go SDK.
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82082)
Time Spent: 10m
Remaining Estimate: 0h

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82081
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:56
Start Date: 19/Mar/18 22:56
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374413097
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82081)
Time Spent: 15h 20m  (was: 15h 10m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 15h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3885) Python SDK Read IO is expressed as Impulse -> ParDo

Ben Sidhom created BEAM-3885:


 Summary: Python SDK Read IO is expressed as Impulse -> ParDo
 Key: BEAM-3885
 URL: https://issues.apache.org/jira/browse/BEAM-3885
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


Portable runners cannot understand SDK-specific Read transforms. The Python SDK 
will need to rewrite Read as Impulse followed by a ParDo that actually does the 
IO.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3884) Python SDK supports Impulse as a primitive transform

Ben Sidhom created BEAM-3884:


 Summary: Python SDK supports Impulse as a primitive transform
 Key: BEAM-3884
 URL: https://issues.apache.org/jira/browse/BEAM-3884
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


Portable runners require Impulse to be the only root nodes of pipelines. The 
Python SDK should provide this for pipeline construction.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3883) Python SDK stages artifacts when talking to job server

Ben Sidhom created BEAM-3883:


 Summary: Python SDK stages artifacts when talking to job server
 Key: BEAM-3883
 URL: https://issues.apache.org/jira/browse/BEAM-3883
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Ben Sidhom
Assignee: Ahmet Altay


The Python SDK does not currently stage its user-defined functions or 
dependencies when talking to the job API. Artifacts that need to be staged 
include the user code itself, any SDK components not included in the container 
image, and the list of Python packages that must be installed at runtime.

 

Artifacts that are currently expected can be found in the harness boot code: 
[https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82075
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4844: 
[BEAM-3565] Add ExecutableStagePayload to simplify runner stage reconstruction
URL: https://github.com/apache/beam/pull/4844#discussion_r175603054
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -84,64 +81,72 @@
* follows:
*
* 
-   *   The {@link PTransform#getSubtransformsList()} contains the result 
of {@link
-   *   #getTransforms()}.
+   *   The {@link PTransform#getSubtransformsList()} contains no 
subtransforms. This ensures
+   *   that executable stages are treated as primitive transforms.
*   The only {@link PCollection} in the {@link 
PTransform#getInputsMap()} is the result of
*   {@link #getInputPCollection()}.
*   The output {@link PCollection PCollections} in the values of {@link
*   PTransform#getOutputsMap()} are the {@link PCollectionNode 
PCollections} returned by
*   {@link #getOutputPCollections()}.
+   *   The {@link FunctionSpec} contains an {@link ExecutableStagePayload} 
which has its input
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82075)
Time Spent: 13.5h  (was: 13h 20m)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82079
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4844: 
[BEAM-3565] Add ExecutableStagePayload to simplify runner stage reconstruction
URL: https://github.com/apache/beam/pull/4844#discussion_r175595281
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -84,64 +81,72 @@
* follows:
*
* 
-   *   The {@link PTransform#getSubtransformsList()} contains the result 
of {@link
-   *   #getTransforms()}.
+   *   The {@link PTransform#getSubtransformsList()} contains no 
subtransforms. This ensures
+   *   that executable stages are treated as primitive transforms.
*   The only {@link PCollection} in the {@link 
PTransform#getInputsMap()} is the result of
*   {@link #getInputPCollection()}.
*   The output {@link PCollection PCollections} in the values of {@link
*   PTransform#getOutputsMap()} are the {@link PCollectionNode 
PCollections} returned by
*   {@link #getOutputPCollections()}.
+   *   The {@link FunctionSpec} contains an {@link ExecutableStagePayload} 
which has its input
+   *   and output PCollections set to the same values as the outer 
PTransform itself. It further
+   *   contains the environment set of transforms for this stage.
* 
+   *
+   * The executable stage can be reconstructed from the resulting {@link 
ExecutableStagePayload}
+   * and components alone via {@link #fromPayload(ExecutableStagePayload, 
Components)}.
*/
   default PTransform toPTransform() {
+ExecutableStagePayload.Builder payload = 
ExecutableStagePayload.newBuilder();
+
+payload.setEnvironment(getEnvironment());
+
+PCollectionNode input = getInputPCollection();
+payload.setInput(input.getId());
+
+for (PTransformNode transform : getTransforms()) {
+  payload.addTransforms(transform.getId());
+}
+
+for (PCollectionNode output : getOutputPCollections()) {
+  payload.addOutputs(output.getId());
+}
+
 PTransform.Builder pt = PTransform.newBuilder();
+pt.setSpec(FunctionSpec.newBuilder()
+.setUrn(ExecutableStage.URN)
+.setPayload(payload.build().toByteString())
+.build());
 pt.putInputs("input", getInputPCollection().getId());
-int i = 0;
-for (PCollectionNode materializedPCollection : getOutputPCollections()) {
-  pt.putOutputs(String.format("materialized_%s", i), 
materializedPCollection.getId());
-  i++;
-}
-for (PTransformNode fusedTransform : getTransforms()) {
-  pt.addSubtransforms(fusedTransform.getId());
+int outputIndex = 0;
+for (PCollectionNode pcNode : getOutputPCollections()) {
+  // Do something
 
 Review comment:
   Meaningless.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82079)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82077
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4844: 
[BEAM-3565] Add ExecutableStagePayload to simplify runner stage reconstruction
URL: https://github.com/apache/beam/pull/4844#discussion_r175594386
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -84,64 +81,72 @@
* follows:
*
* 
-   *   The {@link PTransform#getSubtransformsList()} contains the result 
of {@link
-   *   #getTransforms()}.
+   *   The {@link PTransform#getSubtransformsList()} contains no 
subtransforms. This ensures
+   *   that executable stages are treated as primitive transforms.
*   The only {@link PCollection} in the {@link 
PTransform#getInputsMap()} is the result of
*   {@link #getInputPCollection()}.
*   The output {@link PCollection PCollections} in the values of {@link
*   PTransform#getOutputsMap()} are the {@link PCollectionNode 
PCollections} returned by
*   {@link #getOutputPCollections()}.
+   *   The {@link FunctionSpec} contains an {@link ExecutableStagePayload} 
which has its input
+   *   and output PCollections set to the same values as the outer 
PTransform itself. It further
+   *   contains the environment set of transforms for this stage.
* 
+   *
+   * The executable stage can be reconstructed from the resulting {@link 
ExecutableStagePayload}
+   * and components alone via {@link #fromPayload(ExecutableStagePayload, 
Components)}.
*/
   default PTransform toPTransform() {
+ExecutableStagePayload.Builder payload = 
ExecutableStagePayload.newBuilder();
+
+payload.setEnvironment(getEnvironment());
+
+PCollectionNode input = getInputPCollection();
+payload.setInput(input.getId());
+
+for (PTransformNode transform : getTransforms()) {
+  payload.addTransforms(transform.getId());
+}
+
+for (PCollectionNode output : getOutputPCollections()) {
+  payload.addOutputs(output.getId());
+}
+
 PTransform.Builder pt = PTransform.newBuilder();
+pt.setSpec(FunctionSpec.newBuilder()
+.setUrn(ExecutableStage.URN)
+.setPayload(payload.build().toByteString())
+.build());
 pt.putInputs("input", getInputPCollection().getId());
-int i = 0;
-for (PCollectionNode materializedPCollection : getOutputPCollections()) {
-  pt.putOutputs(String.format("materialized_%s", i), 
materializedPCollection.getId());
-  i++;
-}
-for (PTransformNode fusedTransform : getTransforms()) {
-  pt.addSubtransforms(fusedTransform.getId());
+int outputIndex = 0;
+for (PCollectionNode pcNode : getOutputPCollections()) {
+  // Do something
+  pt.putOutputs(String.format("materialized_%d", outputIndex), 
pcNode.getId());
+  outputIndex++;
 }
-pt.setSpec(FunctionSpec.newBuilder().setUrn(ExecutableStage.URN));
 return pt.build();
   }
 
+  // TODO: Should this live under ExecutableStageTranslation?
 
 Review comment:
   Ack. I meant this to be a question to the reviewer rather than an actual 
checked-in TODO.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82077)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82080
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #4898: [BEAM-3565] Clean up 
ExecutableStage
URL: https://github.com/apache/beam/pull/4898#issuecomment-374408951
 
 
   R: @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82080)
Time Spent: 13h 50m  (was: 13h 40m)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13h 50m
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82078
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4844: 
[BEAM-3565] Add ExecutableStagePayload to simplify runner stage reconstruction
URL: https://github.com/apache/beam/pull/4844#discussion_r175602123
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -203,6 +203,23 @@ message PCollection {
   DisplayData display_data = 5;
 }
 
+// The payload for an executable stage. This will eventually be passed to an 
SDK in the form of a
+// ProcessBundleDescriptor.
+message ExecutableStagePayload {
+
+  Environment environment = 1;
 
 Review comment:
   Added a comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82078)
Time Spent: 13h 40m  (was: 13.5h)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82076
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:41
Start Date: 19/Mar/18 22:41
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #4844: 
[BEAM-3565] Add ExecutableStagePayload to simplify runner stage reconstruction
URL: https://github.com/apache/beam/pull/4844#discussion_r175602637
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -84,64 +81,72 @@
* follows:
*
* 
-   *   The {@link PTransform#getSubtransformsList()} contains the result 
of {@link
-   *   #getTransforms()}.
+   *   The {@link PTransform#getSubtransformsList()} contains no 
subtransforms. This ensures
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82076)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness


 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=82074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82074
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:40
Start Date: 19/Mar/18 22:40
Worklog Time Spent: 10m 
  Work Description: bsidhom opened a new pull request #4898: [BEAM-3565] 
Clean up ExecutableStage
URL: https://github.com/apache/beam/pull/4898
 
 
   Addressing minor issues from #4844.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82074)
Time Spent: 13h 20m  (was: 13h 10m)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3882) Make StateRequestHandler::accept interface idiomatic.

2018-03-19 Thread Axel Magnuson (JIRA)

Axel Magnuson created BEAM-3882:
---

 Summary: Make StateRequestHandler::accept interface idiomatic.
 Key: BEAM-3882
 URL: https://issues.apache.org/jira/browse/BEAM-3882
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Axel Magnuson
Assignee: Axel Magnuson


StateRequestHandler was based on some Dataflow SDK source code that does not 
conform to Beam coding conventions.  In particular, the accept method takes its 
return value as a parameter, which is a code smell.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners


 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=82073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82073
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:32
Start Date: 19/Mar/18 22:32
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-374406439
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82073)
Time Spent: 15h 10m  (was: 15h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 15h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1145

See 


Changes:

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[ccy] Add support for PaneInfo in WindowedValues

[XuMingmin] Bump calcite and avatica versions (#4887)

--
[...truncated 418.34 KB...]
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s30"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s32", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s31"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id:

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window


 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=82067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82067
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:15
Start Date: 19/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4793: 
[BEAM-3776] Fix issue with merging late windows where a watermark hold could be 
added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#discussion_r175587347
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
 ##
 @@ -533,28 +534,33 @@ public void advanceSynchronizedProcessingTime(
*/
   @SafeVarargs
   public final void injectElements(TimestampedValue... values) throws 
Exception {
+injectElements(Arrays.asList(values));
+  }
+
+  public final void injectElements(Iterable values) 
throws Exception {
 for (TimestampedValue value : values) {
   WindowTracing.trace("TriggerTester.injectElements: {}", value);
 }
 
 Iterable inputs =
-Arrays.asList(values)
-.stream()
-.map(
-input -> {
-  try {
-InputT value = input.getValue();
-Instant timestamp = input.getTimestamp();
-Collection windows =
-windowFn.assignWindows(
-new TestAssignContext(
-windowFn, value, timestamp, 
GlobalWindow.INSTANCE));
-return WindowedValue.of(value, timestamp, windows, 
PaneInfo.NO_FIRING);
-  } catch (Exception e) {
-throw new RuntimeException(e);
-  }
-})
-.collect(Collectors.toList());
+Iterables.transform(
 
 Review comment:
   this seems like the wrong direction for this change - certainly the 
lambda-to-function.
   
   You can use `StreamSupport` to keep using streams


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82067)
Time Spent: 3h 10m  (was: 3h)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging 
> windows that depend only on the window
> ---
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.1.0, 2.2.0, 2.3.0
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>Priority: Critical
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take 
> to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the 
> window depends only on window, sets a hold for the end of the window 
> regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark 
> until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window


 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=82065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82065
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:15
Start Date: 19/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4793: 
[BEAM-3776] Fix issue with merging late windows where a watermark hold could be 
added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#discussion_r175581212
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 ##
 @@ -110,6 +119,24 @@
 return Mockito.any();
   }
 
+  static class Action {
+
+static Action times(Integer... times) {
+  Action a = new Action();
+  a.times = Arrays.asList(times);
+  return a;
+}
+
+static Action inputWatermark(long watermark) {
+  Action a = new Action();
+  a.inputWatermark = watermark;
+  return a;
+}
+
+List times;
+long inputWatermark = 0;
 
 Review comment:
   `Instant`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82065)
Time Spent: 3h  (was: 2h 50m)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging 
> windows that depend only on the window
> ---
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.1.0, 2.2.0, 2.3.0
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>Priority: Critical
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take 
> to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the 
> window depends only on window, sets a hold for the end of the window 
> regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark 
> until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window


 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=82063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82063
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:15
Start Date: 19/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4793: 
[BEAM-3776] Fix issue with merging late windows where a watermark hold could be 
added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#discussion_r175585280
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 ##
 @@ -873,6 +911,288 @@ public void testWatermarkHoldAndLateData() throws 
Exception {
 tester.assertHasOnlyGlobalAndFinishedSetsFor();
   }
 
+  // Performs the specified actions and verifies that the watermark hold is 
set correctly.
+  public void mergingWatermarkHoldTestHelper(List configuration) 
throws Exception {
+LOG.info("Running config {}",  configuration);
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
+// Test handling of late data. Specifically, ensure the watermark hold is 
correct.
+Duration allowedLateness = Duration.standardMinutes(1);
+Duration gapDuration = Duration.millis(10);
+LOG.info("Gap duration {}", gapDuration);
+ReduceFnTester tester =
+ReduceFnTester.nonCombining(
+WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
+.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
+.withTrigger(
+Repeatedly.forever(
+AfterWatermark.pastEndOfWindow()
+
.withLateFirings(AfterPane.elementCountAtLeast(1
+.withAllowedLateness(allowedLateness));
+tester.setAutoAdvanceOutputWatermark(false);
+
+// Input watermark -> null
+assertEquals(null, tester.getWatermarkHold());
+assertEquals(null, tester.getOutputWatermark());
+
+int maxTs = 0;
+long watermark = 0;
+for (Action action : configuration) {
+  if (action.times != null) {
+LOG.info("Injecting {}", action.times);
+injectElements(tester, action.times);
+int maxLocalTs = Ordering.natural().max(action.times);
+if (maxLocalTs > maxTs) {
+  maxTs = maxLocalTs;
+}
+  }
+  if (action.inputWatermark > watermark) {
+watermark = action.inputWatermark;
+LOG.info("Advancing watermark to {}", new Instant(watermark));
+tester.advanceInputWatermark(new Instant(watermark));
+  }
+  Instant hold = tester.getWatermarkHold();
+  if (hold != null) {
+assertThat(hold, greaterThanOrEqualTo(new Instant(watermark)));
+assertThat(watermark, lessThan(maxTs + gapDuration.getMillis()));
+  }
+}
+if (gapDuration.getMillis() + maxTs > watermark) {
+  watermark = gapDuration.getMillis() + maxTs;
+  tester.advanceInputWatermark(new Instant(watermark));
+}
+LOG.info("Output {}", tester.extractOutput());
+assertThat(tester.getWatermarkHold(), nullValue());
+tester.advanceInputWatermark(new Instant(watermark).plus(allowedLateness));
+assertThat(tester.getWatermarkHold(), nullValue());
+
+// Nothing dropped.
+long droppedElements =
+container
+.getCounter(
+MetricName.named(ReduceFnRunner.class,
+ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
+.getCumulative()
+.longValue();
+assertEquals(0, droppedElements);
+  }
+
+  @Test
+  public void testMergingWatermarkHoldLateNewWindow() throws Exception {
+LinkedList actions = new LinkedList<>();
+actions.add(Action.inputWatermark(40));
+actions.add(Action.times(1));
+mergingWatermarkHoldTestHelper(actions);
 
 Review comment:
   I'd love to separate out some of the assertions about what actually happened.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82063)
Time Spent: 2h 50m  (was: 2h 40m)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging 
> windows that depend only on the window
> ---
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
>  Issue

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window


 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=82064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82064
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:15
Start Date: 19/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4793: 
[BEAM-3776] Fix issue with merging late windows where a watermark hold could be 
added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#discussion_r175586984
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 ##
 @@ -873,6 +911,288 @@ public void testWatermarkHoldAndLateData() throws 
Exception {
 tester.assertHasOnlyGlobalAndFinishedSetsFor();
   }
 
+  // Performs the specified actions and verifies that the watermark hold is 
set correctly.
+  public void mergingWatermarkHoldTestHelper(List configuration) 
throws Exception {
+LOG.info("Running config {}",  configuration);
+MetricsContainerImpl container = new MetricsContainerImpl("any");
+MetricsEnvironment.setCurrentContainer(container);
+// Test handling of late data. Specifically, ensure the watermark hold is 
correct.
+Duration allowedLateness = Duration.standardMinutes(1);
+Duration gapDuration = Duration.millis(10);
+LOG.info("Gap duration {}", gapDuration);
+ReduceFnTester tester =
+ReduceFnTester.nonCombining(
+WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
+.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
+.withTrigger(
+Repeatedly.forever(
+AfterWatermark.pastEndOfWindow()
+
.withLateFirings(AfterPane.elementCountAtLeast(1
+.withAllowedLateness(allowedLateness));
+tester.setAutoAdvanceOutputWatermark(false);
+
+// Input watermark -> null
+assertEquals(null, tester.getWatermarkHold());
+assertEquals(null, tester.getOutputWatermark());
+
+int maxTs = 0;
+long watermark = 0;
+for (Action action : configuration) {
+  if (action.times != null) {
+LOG.info("Injecting {}", action.times);
+injectElements(tester, action.times);
+int maxLocalTs = Ordering.natural().max(action.times);
+if (maxLocalTs > maxTs) {
+  maxTs = maxLocalTs;
+}
+  }
+  if (action.inputWatermark > watermark) {
+watermark = action.inputWatermark;
+LOG.info("Advancing watermark to {}", new Instant(watermark));
+tester.advanceInputWatermark(new Instant(watermark));
+  }
+  Instant hold = tester.getWatermarkHold();
+  if (hold != null) {
+assertThat(hold, greaterThanOrEqualTo(new Instant(watermark)));
+assertThat(watermark, lessThan(maxTs + gapDuration.getMillis()));
+  }
+}
+if (gapDuration.getMillis() + maxTs > watermark) {
+  watermark = gapDuration.getMillis() + maxTs;
+  tester.advanceInputWatermark(new Instant(watermark));
+}
+LOG.info("Output {}", tester.extractOutput());
+assertThat(tester.getWatermarkHold(), nullValue());
+tester.advanceInputWatermark(new Instant(watermark).plus(allowedLateness));
+assertThat(tester.getWatermarkHold(), nullValue());
+
+// Nothing dropped.
+long droppedElements =
+container
+.getCounter(
+MetricName.named(ReduceFnRunner.class,
+ReduceFnRunner.DROPPED_DUE_TO_CLOSED_WINDOW))
+.getCumulative()
+.longValue();
+assertEquals(0, droppedElements);
+  }
+
+  @Test
+  public void testMergingWatermarkHoldLateNewWindow() throws Exception {
+LinkedList actions = new LinkedList<>();
+actions.add(Action.inputWatermark(40));
+actions.add(Action.times(1));
+mergingWatermarkHoldTestHelper(actions);
+  }
+
+  @Test
+  public void testMergingWatermarkHoldLateNewWindowExtended() throws Exception 
{
+LinkedList actions = new LinkedList<>();
+  actions.add(Action.inputWatermark(40));
+  actions.add(Action.times(1));
+  actions.add(Action.times(10));
+mergingWatermarkHoldTestHelper(actions);
+  }
+
+  @Test
+  public void testMergingWatermarkHoldLateNewWindowMerged() throws Exception {
+LinkedList actions = new LinkedList<>();
+actions.add(Action.inputWatermark(40));
+actions.add(Action.times(1));
+actions.add(Action.times(14));
+actions.add(Action.times(6));
+mergingWatermarkHoldTestHelper(actions);
+  }
+
+  @Test
+  public void 
testMergingWatermarkHoldLateNewWindowExtendedPastInputWatermark() throws 
Exception {
+LinkedList actions = new LinkedList<>();
+actions.add(Action.inputWatermark(40));
+actions.add(Action.times(25));
+

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window


 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=82066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82066
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 19/Mar/18 22:15
Start Date: 19/Mar/18 22:15
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4793: 
[BEAM-3776] Fix issue with merging late windows where a watermark hold could be 
added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#discussion_r175579336
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 ##
 @@ -110,6 +119,24 @@
 return Mockito.any();
   }
 
+  static class Action {
 
 Review comment:
   `AutoValue` with `@Nullable` fields, and a comment about what the field 
means in terms of the action that should be taken (I don't, for example, really 
understand what `times` means - multiply something? add an element with a 
timestamp? add something x times?)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 82066)
Time Spent: 3h  (was: 2h 50m)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging 
> windows that depend only on the window
> ---
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.1.0, 2.2.0, 2.3.0
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>Priority: Critical
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take 
> to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the 
> window depends only on window, sets a hold for the end of the window 
> regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark 
> until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3803) Dataflow runner should handle metrics per the spec

2018-03-19 Thread Andrew Pilloud (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud resolved BEAM-3803.
--
   Resolution: Fixed
Fix Version/s: 2.5.0

I should remember to close these when I'm done.

> Dataflow runner should handle metrics per the spec
> --
>
> Key: BEAM-3803
> URL: https://issues.apache.org/jira/browse/BEAM-3803
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Labels: nexmark
> Fix For: 2.5.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> The dataflow runner only supports committed metrics for batch jobs and 
> attempted metrics for streaming jobs. It should always support attempted 
> metrics and throw an UnsupportedOperationException when the metrics are 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #6245

See 


Changes:

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

--
[...truncated 35.50 KB...]
2018-03-19T21:51:30.829 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2018-03-19T21:51:30.829 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] 
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2018-03-19T21:51:30.829 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] 
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2018-03-19T21:51:30.829 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] 
2018-03-19T21:51:30.829 [INFO] 

2018-03-19T21:51:30.829 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: Examples :: Java
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Jackson
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Join library
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sketching
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sorter
2018-03-19T21:51:30.830 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.830 [INFO] 

2018-03-19T21:51:30.830 [INFO] 
2018-03-19T21:51:30.831 [INFO] 

2018-03-19T21:51:30.831 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: SQL
2018-03-19T21:51:30.831 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-19T21:51:30.831 [INFO] 

2018-03-19T21:51:30.831 [INFO] 
2018-03-19T21:51:30.831 [INFO]

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners