[jira] [Commented] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle

[Romain Manni-Bucau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441899#comment-16441899
 ] 

Romain Manni-Bucau commented on BEAM-4088:
--

Scott, this test illustrate that gradle test setup is wrong in respect of the 
tests isolation. Must be fixed to ensure build is deterministic ans reliable. 
See the list thread about it.

> ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 
> does not pass in gradle
> -
>
> Key: BEAM-4088
> URL: https://issues.apache.org/jira/browse/BEAM-4088
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Etienne Chauchot
>Assignee: Scott Wegner
>Priority: Major
>
> This test always fail using gradle but always pass using maven



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle

[Romain Manni-Bucau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Romain Manni-Bucau reassigned BEAM-4088:


Assignee: Scott Wegner  (was: Romain Manni-Bucau)

> ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 
> does not pass in gradle
> -
>
> Key: BEAM-4088
> URL: https://issues.apache.org/jira/browse/BEAM-4088
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Etienne Chauchot
>Assignee: Scott Wegner
>Priority: Major
>
> This test always fail using gradle but always pass using maven



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #107

[Apache Jenkins Server]

See 


Changes:

[apilloud] [SQL] Remove PRIMARY KEY it does nothing

[apilloud] [SQL] Plumb through column nullable field

[apilloud] [SQL] Copy in DDL code from Calicite 1.16

[apilloud] [SQL] Patch ddl code for beam

[herohde] Add prototype of Go streaming on Dataflow with PubSub

[ekirpichov] Fixes run_pylint to run on Mac

[swegner] Exclude UsesImpluse from Apex postcommit tests

[swegner] Exclude UsesImpluse from Spark postcommit tests

[swegner] Increase Dataflow max parallelism.

[axelmagn] Rename EnvironmentManager to EnvironmentFactory

[alan.myrvold] Fix faling spark quickstart

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[swegner] fixup! Improve comment

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[alan.myrvold] Fix spark validates runner test by skipping ImpulseTest

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

[aaltay] Support staging binary distributions (wheel files) of Beam SDK. (#5110)

--
[...truncated 64.42 KB...]
44246ca595f6: Waiting
20c527f217db: Waiting
580a720622b3: Layer already exists
44246ca595f6: Layer already exists
bc0d5482d762: Layer already exists
20c527f217db: Layer already exists
bcbe43405751: Layer already exists
61c06e07759a: Waiting
61c06e07759a: Layer already exists
e1df5dc88d2c: Layer already exists
b7a7cad6e83e: Pushed
a9efe24d0cad: Pushed
e3a528017bae: Pushed
20180418-033042: digest: 
sha256:680a0290183818d039bc1b8e5e3a1c8a2bc5619b6f865aead4fcc3c72d96a0d4 size: 
2857

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Virtualenv for the rest of the script to run setup & e2e test
${LOCAL_PATH}/virtualenv sdks/python/container
New python executable in 

Installing setuptools, pip, wheel...done.
. sdks/python/container/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining 

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92038
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 18/Apr/18 03:25
Start Date: 18/Apr/18 03:25
Worklog Time Spent: 10m 
  Work Description: rangadi commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182298038
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
 ##
 @@ -66,9 +77,32 @@ public void encode(KafkaRecord value, OutputStream 
outStream) throws IOExc
 longCoder.decode(inStream),
 longCoder.decode(inStream),
 KafkaTimestampType.forOrdinal(intCoder.decode(inStream)),
+getHeaders(headerCoder.decode(inStream)),
 kvCoder.decode(inStream));
   }
 
+  private Headers getHeaders(Iterable> records) {
 
 Review comment:
   minor: call this `toHeaders()`? same for `getIterable()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92038)
Time Spent: 6h  (was: 5h 50m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92037
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 18/Apr/18 03:25
Start Date: 18/Apr/18 03:25
Worklog Time Spent: 10m 
  Work Description: rangadi commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182297723
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -76,6 +80,10 @@ public long getOffset() {
 return offset;
   }
 
+  public Headers getHeaders() {
 
 Review comment:
   I think this should throw an exception if 'hasHeaders' is false. We don't 
need the user to worry about null values. `ConsumerRecord.headers` is not 
expected to be null either. We can do this once we know the current solution 
works as expected with older kafka-clients versions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92037)
Time Spent: 5h 50m  (was: 5h 40m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92039
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 18/Apr/18 03:25
Start Date: 18/Apr/18 03:25
Worklog Time Spent: 10m 
  Work Description: rangadi commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182299393
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -33,6 +34,7 @@
   private final String topic;
   private final int partition;
   private final long offset;
+  private final Headers headers;
 
 Review comment:
   Thanks for the update. I missed later part of your comment above.
   
   > and I package this class and use it as a dependency in another application 
which simply instantiates this class. Even though the downstream application 
did not call getHeaders, it will still throw a runtime exception.
   
   Is that true? Can you try it with a small simple java class? I tested with a 
small test app and it worked ok . 
   Run it with different versions of kafka, and with '-Dis.present=True|False':
   
   
   package org.example;
   
   import org.apache.kafka.clients.consumer.ConsumerRecord;
   import org.apache.kafka.common.header.Headers;
   
   public class TestRuntimeLoading {
   
 public static class A {
   
   Headers kafkaHeaders = null;
   
   Headers getHeaders(boolean isPresent) {
 if (isPresent && kafkaHeaders == null) {
   return new ConsumerRecord<>("test", 0, 0L, "key", "value").headers();
 } else {
   throw new RuntimeException("no headers");
 }
   }
   
 }
   
 public static void main(String[] args) {
   
   boolean isPresent = Boolean.valueOf(System.getProperty("is.present", 
"True"));
   Object headers = new A().getHeaders(isPresent);
   System.out.println("Headers : " + headers.toString());
 }
   }
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92039)
Time Spent: 6h  (was: 5h 50m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #107

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1389

[Apache Jenkins Server]

See 


--
[...truncated 1.53 KB...]
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8934322548642801585.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in 

Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0)
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.5.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.5.0.dev0)
Collecting grpcio<2,>=1.8 (from apache-beam==2.5.0.dev0)
  Using cached 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4729

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision de6ea7e878913acd87ec2f9b3cca47ad41668250 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f de6ea7e878913acd87ec2f9b3cca47ad41668250
Commit message: "Support staging binary distributions (wheel files) of Beam 
SDK. (#5110)"
 > git rev-list --no-walk de6ea7e878913acd87ec2f9b3cca47ad41668250 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins3647891383859747053.sh
+ cd src
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Remove any tox cache from previous workspace
# TODO(udim): Remove this line and add '-r' to tox invocation instead.
rm -rf sdks/python/target/.tox

# INFRA does not install these packages
pip install --user --upgrade virtualenv tox
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:339: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:137: 
InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Requirement already up-to-date: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
Requirement already up-to-date: tox in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.5.3)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (0.6.0)
Requirement not upgraded as not directly required: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.11.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: 

ERROR: invocation failed (exit code 1), logfile: 

ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', 
local('
 'sdist', '--formats=zip', '--dist-dir', 
local('

/usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
Normalizing '2.5.0.dev' to '2.5.0.dev0'
  normalized_version,
Regenerating common_urns module.
running sdist
:51:
 UserWarning: Installing grpcio-tools is recommended for development.
  warnings.warn('Installing grpcio-tools is recommended for development.')

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #129

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4728

[Apache Jenkins Server]

See 


Changes:

[aaltay] Support staging binary distributions (wheel files) of Beam SDK. (#5110)

--
Started by GitHub push by aaltay
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision de6ea7e878913acd87ec2f9b3cca47ad41668250 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f de6ea7e878913acd87ec2f9b3cca47ad41668250
Commit message: "Support staging binary distributions (wheel files) of Beam 
SDK. (#5110)"
 > git rev-list --no-walk 15da6412fef89dc08de59828a5df9ba11e944a79 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins6323688159252352697.sh
+ cd src
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Remove any tox cache from previous workspace
# TODO(udim): Remove this line and add '-r' to tox invocation instead.
rm -rf sdks/python/target/.tox

# INFRA does not install these packages
pip install --user --upgrade virtualenv tox
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:339: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:137: 
InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Requirement already up-to-date: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
Requirement already up-to-date: tox in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.5.3)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (0.6.0)
Requirement not upgraded as not directly required: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.11.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: 

ERROR: invocation failed (exit code 1), logfile: 

ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', 
local('
 'sdist', '--formats=zip', '--dist-dir', 
local('

/usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
Normalizing '2.5.0.dev' to '2.5.0.dev0'
  normalized_version,
Regenerating common_urns module.
running sdist
:51:
 UserWarning: Installing grpcio-tools is 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1388

[Apache Jenkins Server]

See 


Changes:

[aaltay] Support staging binary distributions (wheel files) of Beam SDK. (#5110)

--
[...truncated 1.54 KB...]
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6353260206710890286.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in 

Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0)
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.5.0.dev0)
Collecting dill==0.2.6 

[beam] branch master updated (15da641 -> de6ea7e)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 15da641  Merge pull request #5003: [BEAM-3942] Update performance 
testing framework to use Gradle
 add de6ea7e  Support staging binary distributions (wheel files) of Beam 
SDK. (#5110)

No new revisions were added by this update.

Summary of changes:
 .../runners/dataflow/internal/dependency.py| 159 ++
 .../runners/dataflow/internal/dependency_test.py   | 232 -
 2 files changed, 300 insertions(+), 91 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-4093) Support Python ValidatesRunner test against TestDataflowRunner in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4093?focusedWorklogId=92027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92027
 ]

ASF GitHub Bot logged work on BEAM-4093:


Author: ASF GitHub Bot
Created on: 18/Apr/18 02:00
Start Date: 18/Apr/18 02:00
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5147: 
[BEAM-4093] Support Python ValidatesRunner test in streaming
URL: https://github.com/apache/beam/pull/5147#discussion_r182290272
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
 ##
 @@ -37,10 +38,13 @@ def run_pipeline(self, pipeline):
 """Execute test pipeline and verify test matcher"""
 options = pipeline._options.view_as(TestOptions)
 on_success_matcher = options.on_success_matcher
+wait_duration = options.wait_until_finish_duration
+is_streaming = options.view_as(StandardOptions).streaming
 
 # [BEAM-1889] Do not send this to remote workers also, there is no need to
 # send this option to remote executors.
 options.on_success_matcher = None
+options.wait_until_finish_duration = None
 
 Review comment:
   Do you need this here? I do not think this this is necessary based on 
BEAM-1889?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92027)
Time Spent: 0.5h  (was: 20m)

> Support Python ValidatesRunner test against TestDataflowRunner in streaming
> ---
>
> Key: BEAM-4093
> URL: https://issues.apache.org/jira/browse/BEAM-4093
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4113) Periodic cleanup and setup of Python environment on Jenkins machines

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441756#comment-16441756
 ] 

Udi Meiri commented on BEAM-4113:
-

Note that I'm adding automatic upgrades of pip and virtualenv in 
[https://github.com/apache/beam/pull/5162,] and that code should be removed in 
favor of the script described here.

> Periodic cleanup and setup of Python environment on Jenkins machines
> 
>
> Key: BEAM-4113
> URL: https://issues.apache.org/jira/browse/BEAM-4113
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Jason Kuster
>Priority: Major
>
> Instead of having Python testing scripts tidy up the environment every time 
> they run, let's have a single script that runs once per day and does:
>  # Check if pip is installed under ~/.local and is the expected version. If 
> not, install the correct version.
>  # Same as above, but for virtualenv.
>  # Print a list of packages installed under ~/.local.
>  # Optional: Uninstall any unexpected packages under ~/.local. For example, 
> apache-beam shouldn't be installed under ~/.local, since 2 concurrent tests 
> on the same machine will attempt to install to the same location.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4113) Periodic cleanup and setup of Python environment on Jenkins machines

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-4113:
---

 Summary: Periodic cleanup and setup of Python environment on 
Jenkins machines
 Key: BEAM-4113
 URL: https://issues.apache.org/jira/browse/BEAM-4113
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Udi Meiri
Assignee: Jason Kuster


Instead of having Python testing scripts tidy up the environment every time 
they run, let's have a single script that runs once per day and does:
 # Check if pip is installed under ~/.local and is the expected version. If 
not, install the correct version.
 # Same as above, but for virtualenv.
 # Print a list of packages installed under ~/.local.
 # Optional: Uninstall any unexpected packages under ~/.local. For example, 
apache-beam shouldn't be installed under ~/.local, since 2 concurrent tests on 
the same machine will attempt to install to the same location.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4727

[Apache Jenkins Server]

See 


--
[...truncated 1.19 MB...]
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_invalid_ellipsis_type_param 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_params_must_be_type_or_constraint 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_raw_tuple (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_be_tuple 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_have_same_arity 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_types 

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92023
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 18/Apr/18 01:36
Start Date: 18/Apr/18 01:36
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize 
coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-382215896
 
 
   @tvalentyn if the test issues are unrelated to this PR and related to the 
current ongoing Jenkins issues should we move forward with it? (I suggest that 
we run a few manual tests locally to check whether this PR introduces issues or 
not.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92023)
Time Spent: 13h 10m  (was: 13h)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=92020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92020
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 18/Apr/18 01:29
Start Date: 18/Apr/18 01:29
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5053: [BEAM-3981] Futurize 
coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-382214475
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92020)
Time Spent: 13h  (was: 12h 50m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4112) PipelineOptions get_all_options should use flag names as well.

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-4112:
-

 Summary: PipelineOptions get_all_options should use flag names as 
well.
 Key: BEAM-4112
 URL: https://issues.apache.org/jira/browse/BEAM-4112
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay


Context: 
https://stackoverflow.com/questions/49790580/worker-machine-type-tag-not-working-in-google-cloud-dataflow-with-python



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3911) Do we want to encourage release pinning?

[Robert Bradshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441725#comment-16441725
 ] 

Robert Bradshaw commented on BEAM-3911:
---

No, we don't want to encourage this. 

> Do we want to encourage release pinning?
> 
>
> Key: BEAM-3911
> URL: https://issues.apache.org/jira/browse/BEAM-3911
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Robert Bradshaw
>Assignee: Melissa Pashniak
>Priority: Major
>
> The Python instructions recommend adding apache-beam==2.3.0 to one's 
> setup.py. Instead, we should probably recommend apache-beam<3. It may also be 
> worth calling out apache-beam can be installed with pip install apache-beam. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3697) Add errorprone to gradle builds

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3697?focusedWorklogId=92017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92017
 ]

ASF GitHub Bot logged work on BEAM-3697:


Author: ASF GitHub Bot
Created on: 18/Apr/18 01:01
Start Date: 18/Apr/18 01:01
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5161: 
[BEAM-3697] Add errorprone to Gradle build
URL: https://github.com/apache/beam/pull/5161#discussion_r182283356
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ResourceIdTester.java
 ##
 @@ -108,15 +108,15 @@ private static void 
validateFailureResolvingIds(ResourceId baseDirectory) {
 try {
   ResourceId badFile = baseDirectory.resolve("file/", RESOLVE_FILE);
   fail(String.format("Resolving badFile %s should have failed", badFile));
-} catch (Throwable t) {
+} catch (Exception e) {
 
 Review comment:
   Should we narrow the exception to the one which resolve throws?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92017)
Time Spent: 0.5h  (was: 20m)

> Add errorprone to gradle builds
> ---
>
> Key: BEAM-3697
> URL: https://issues.apache.org/jira/browse/BEAM-3697
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [http://errorprone.info/] is a good static checker that covers a number of 
> bugs not covered by FindBugs or Checkstyle. We use it internally at Google 
> and, when run on the Beam codebase, it occasionally uncovers issues missed 
> during PR review process.
>  
> It has Maven and Gradle plugins:
> [http://errorprone.info/docs/installation]
> [https://github.com/tbroyer/gradle-errorprone-plugin]
>  
> It would be good to integrate it into our Maven and Gradle builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3268?focusedWorklogId=92016=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92016
 ]

ASF GitHub Bot logged work on BEAM-3268:


Author: ASF GitHub Bot
Created on: 18/Apr/18 01:00
Start Date: 18/Apr/18 01:00
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5159: [BEAM-3268] Reshuffle 
filenames before returning them from WriteFilesResult
URL: https://github.com/apache/beam/pull/5159#issuecomment-382155025
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92016)
Time Spent: 40m  (was: 0.5h)

> getPerDestinationOutputFilenames() is getting processed before write is 
> finished on dataflow runner
> ---
>
> Key: BEAM-3268
> URL: https://issues.apache.org/jira/browse/BEAM-3268
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Kamil Szewczyk
>Assignee: Eugene Kirpichov
>Priority: Major
> Attachments: comparison.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> While running filebased-io-test we found dataflow-runnner misbehaving. We run 
> tests using single pipeline and without using Reshuffling between writing and 
> reading dataflow jobs are unsuccessful because the runner tries to access the 
> files that were not created yet. 
> On the picture the difference between execution of writting is presented. On 
> the left there is working example with Reshuffling added and on the right 
> without it.
> !comparison.png|thumbnail!
> Steps to reproduce: substitute your-bucket-name wit your valid bucket.
> {code:java}
> mvn -e -Pio-it verify -pl sdks/java/io/file-based-io-tests 
> -DintegrationTestPipelineOptions='["--runner=dataflow", 
> "--filenamePrefix=gs://your-bucket-name/TEXTIO_IT"]' -Pdataflow-runner
> {code}
> Then look on the cloud console and job should fail.
> Now add Reshuffling to 
> sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
>  as in the example.
> {code:java}
> .getPerDestinationOutputFilenames().apply(Values.create())
> .apply(Reshuffle.viaRandomKey());
> PCollection consolidatedHashcode = testFilenames
> {code}
> and trigger previously used maven command to see it working in the console 
> right now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3268) getPerDestinationOutputFilenames() is getting processed before write is finished on dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3268?focusedWorklogId=92015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92015
 ]

ASF GitHub Bot logged work on BEAM-3268:


Author: ASF GitHub Bot
Created on: 18/Apr/18 01:00
Start Date: 18/Apr/18 01:00
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5159: [BEAM-3268] Reshuffle 
filenames before returning them from WriteFilesResult
URL: https://github.com/apache/beam/pull/5159#issuecomment-382208505
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92015)
Time Spent: 0.5h  (was: 20m)

> getPerDestinationOutputFilenames() is getting processed before write is 
> finished on dataflow runner
> ---
>
> Key: BEAM-3268
> URL: https://issues.apache.org/jira/browse/BEAM-3268
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.3.0
>Reporter: Kamil Szewczyk
>Assignee: Eugene Kirpichov
>Priority: Major
> Attachments: comparison.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> While running filebased-io-test we found dataflow-runnner misbehaving. We run 
> tests using single pipeline and without using Reshuffling between writing and 
> reading dataflow jobs are unsuccessful because the runner tries to access the 
> files that were not created yet. 
> On the picture the difference between execution of writting is presented. On 
> the left there is working example with Reshuffling added and on the right 
> without it.
> !comparison.png|thumbnail!
> Steps to reproduce: substitute your-bucket-name wit your valid bucket.
> {code:java}
> mvn -e -Pio-it verify -pl sdks/java/io/file-based-io-tests 
> -DintegrationTestPipelineOptions='["--runner=dataflow", 
> "--filenamePrefix=gs://your-bucket-name/TEXTIO_IT"]' -Pdataflow-runner
> {code}
> Then look on the cloud console and job should fail.
> Now add Reshuffling to 
> sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
>  as in the example.
> {code:java}
> .getPerDestinationOutputFilenames().apply(Values.create())
> .apply(Reshuffle.viaRandomKey());
> PCollection consolidatedHashcode = testFilenames
> {code}
> and trigger previously used maven command to see it working in the console 
> right now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4726

[Apache Jenkins Server]

See 


--
GitHub pull request #5155 of commit c760d037123e64b237ccc0e6e79bbeec070ca7ab, 
no merge conflicts.
Setting status of c760d037123e64b237ccc0e6e79bbeec070ca7ab to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/4726/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Python SDK PostCommit Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5155/*:refs/remotes/origin/pr/5155/*
 > git rev-parse refs/remotes/origin/pr/5155/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5155/merge^{commit} # timeout=10
Checking out Revision 31ee4ee2cd344e7dd7fc707c18e04a930d7d8b46 
(refs/remotes/origin/pr/5155/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 31ee4ee2cd344e7dd7fc707c18e04a930d7d8b46
Commit message: "Merge c760d037123e64b237ccc0e6e79bbeec070ca7ab into 
15da6412fef89dc08de59828a5df9ba11e944a79"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins5591659310874476814.sh
+ cd src
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Remove any tox cache from previous workspace
# TODO(udim): Remove this line and add '-r' to tox invocation instead.
rm -rf sdks/python/target/.tox

# INFRA does not install these packages
pip install --user --upgrade virtualenv tox
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:339: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:137: 
InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Requirement already up-to-date: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
Requirement already up-to-date: tox in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.5.3)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (0.6.0)
Requirement not upgraded as not directly required: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.11.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: 

ERROR: invocation failed (exit code 1), logfile: 

ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', 
local('
 'sdist', '--formats=zip', '--dist-dir', 
local('

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #131

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #144

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1603

[Apache Jenkins Server]

See 


Changes:

[herohde] Add prototype of Go streaming on Dataflow with PubSub

[ekirpichov] Fixes run_pylint to run on Mac

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[alan.myrvold] Fix spark validates runner test by skipping ImpulseTest

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 91.94 KB...]
'apache-beam-testing:bqjob_r51c041f7a787c4d3_0162d6314135_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-18 00:39:37,144 d640ce27 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-18 00:39:55,993 d640ce27 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-18 00:39:58,476 d640ce27 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r73acc86cbf45794b_0162d63194e8_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r73acc86cbf45794b_0162d63194e8_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r73acc86cbf45794b_0162d63194e8_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-18 00:39:58,476 d640ce27 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-18 00:40:28,419 d640ce27 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-18 00:40:30,766 d640ce27 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r7d2bda087dbd509c_0162d6321370_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7d2bda087dbd509c_0162d6321370_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r7d2bda087dbd509c_0162d6321370_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-18 00:40:30,767 d640ce27 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-18 00:40:48,830 d640ce27 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-18 00:40:51,166 d640ce27 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #61

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #106

[Apache Jenkins Server]

See 


--
[...truncated 13.57 KB...]
Applying build_rules.gradle to beam-runners-core-construction-java
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-runners-core-construction-java
Evaluating project ':beam-runners-direct-java' using build file 
'
Applying build_rules.gradle to beam-runners-direct-java
Offline dependencies root configured at 'offline-repository'
applyJavaNature with 
[shadowClosure:org.codehaus.groovy.runtime.ComposedClosure@70efdd18] for 
project beam-runners-direct-java
Generating :runQuickstartJavaDirect
Evaluating project ':beam-runners-flink_2.11' using build file 
'
Applying build_rules.gradle to beam-runners-flink_2.11
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project beam-runners-flink_2.11
Generating :runQuickstartJavaFlinkLocal
Evaluating project ':beam-runners-gcp-gcemd' using build file 
'
Applying build_rules.gradle to beam-runners-gcp-gcemd
Offline dependencies root configured at 'offline-repository'
applyGoNature with default configuration for project beam-runners-gcp-gcemd
applyDockerNature with default configuration for project beam-runners-gcp-gcemd
containerImageName with [name:gcemd] for project beam-runners-gcp-gcemd
Ignore packages globally: 
appengine,a,gen,shared,common,thrifttest,appengine_internal,thrift,tutorial
Evaluating project ':beam-runners-gcp-gcsproxy' using build file 
'
Applying build_rules.gradle to beam-runners-gcp-gcsproxy
Offline dependencies root configured at 'offline-repository'
applyGoNature with default configuration for project beam-runners-gcp-gcsproxy
applyDockerNature with default configuration for project 
beam-runners-gcp-gcsproxy
containerImageName with [name:gcsproxy] for project beam-runners-gcp-gcsproxy
Ignore packages globally: 
appengine,a,gen,shared,common,thrifttest,appengine_internal,thrift,tutorial
Evaluating project ':beam-runners-gearpump' using build file 
'
Applying build_rules.gradle to beam-runners-gearpump
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project beam-runners-gearpump
Evaluating project ':beam-runners-java-fn-execution' using build file 
'
Applying build_rules.gradle to beam-runners-java-fn-execution
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-runners-java-fn-execution
Evaluating project ':beam-sdks-java-fn-execution' using build file 
'
Applying build_rules.gradle to beam-sdks-java-fn-execution
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-sdks-java-fn-execution
Evaluating project ':beam-runners-local-java-core' using build file 
'
Applying build_rules.gradle to beam-runners-local-java-core
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-runners-local-java-core
Evaluating project ':beam-runners-reference-java' using build file 
'
Applying build_rules.gradle to beam-runners-reference-java
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-runners-reference-java
Evaluating project ':beam-runners-reference-job-server' using build file 
'
Applying build_rules.gradle to beam-runners-reference-job-server
Offline dependencies root configured at 'offline-repository'
applyJavaNature with default configuration for project 
beam-runners-reference-job-server
Evaluating project ':beam-runners-spark' using build file 
'
Applying build_rules.gradle to 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #404

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #465

[Apache Jenkins Server]

See 


Changes:

[herohde] Add prototype of Go streaming on Dataflow with PubSub

[ekirpichov] Fixes run_pylint to run on Mac

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[alan.myrvold] Fix spark validates runner test by skipping ImpulseTest

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 37.02 KB...]
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the 

Build failed in Jenkins: beam_PerformanceTests_Python #1161

[Apache Jenkins Server]

See 


Changes:

[herohde] Add prototype of Go streaming on Dataflow with PubSub

[ekirpichov] Fixes run_pylint to run on Mac

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[alan.myrvold] Fix spark validates runner test by skipping ImpulseTest

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 62.77 KB...]
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.7:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get -u google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.7:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.7:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.7:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] 
/home/jenkins/.mvnGoLang/.go_path/src/cloud.google.com/go/pubsub/subscription.go:30:2:
 cannot find package "golang.org/x/sync/errgroup" in any of:
[ERROR] 
/home/jenkins/.mvnGoLang/go1.9.linux-amd64/src/golang.org/x/sync/errgroup (from 
$GOROOT)
[ERROR] 
/home/jenkins/.mvnGoLang/.go_path/src/golang.org/x/sync/errgroup (from $GOPATH)
[ERROR] 

[ERROR] 
/home/jenkins/.mvnGoLang/.go_path/src/cloud.google.com/go/pubsub/flow_controller.go:19:2:
 cannot find package "golang.org/x/sync/semaphore" in any of:
[ERROR] 
/home/jenkins/.mvnGoLang/go1.9.linux-amd64/src/golang.org/x/sync/semaphore 
(from $GOROOT)
[ERROR] 
/home/jenkins/.mvnGoLang/.go_path/src/golang.org/x/sync/semaphore (from $GOPATH)
[ERROR] 

[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  3.595 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.677 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.153 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [ 10.837 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.760 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.667 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.154 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 43.232 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #154

[Apache Jenkins Server]

See 


Changes:

[herohde] Add prototype of Go streaming on Dataflow with PubSub

[ekirpichov] Fixes run_pylint to run on Mac

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[alan.myrvold] Fix spark validates runner test by skipping ImpulseTest

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 151.30 KB...]
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding 

[jira] [Comment Edited] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441681#comment-16441681
 ] 

Valentyn Tymofieiev edited comment on BEAM-4105 at 4/17/18 11:52 PM:
-

[~robertwb],

Seems like pip is too new :)

Verified that pip.pep425tags.get_supported() fails in pip 10.0.0 but passes in 
9.0.3.

If we pick up the latest pip once we start a new virtual environment, it 
explains the issue.


was (Author: tvalentyn):
Seems like too new :)

Verified that pip.pep425tags.get_supported() fails in pip 10.0.0 but passes in 
9.0.3.

If we pick up the latest pip once we start a new virtual environment, it 
explains the issue.

> Python Postcommit suite does not start on Jenkins - proto generation failed
> ---
>
> Key: BEAM-4105
> URL: https://issues.apache.org/jira/browse/BEAM-4105
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
>
> The suite has been steadily failing for last 3 days. Sample logs:  
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
> ERROR: invocation failed (exit code 1), logfile: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log
> ERROR: actionid: tox
> msg: packaging
> cmdargs: ['/usr/bin/python', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py'),
>  'sdist', '--formats=zip', '--dist-dir', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist')]
> /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
> Normalizing '2.5.0.dev' to '2.5.0.dev0'
>   normalized_version,
> Regenerating common_urns module.
> running sdist
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py:51:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> WARNING:root:Installing grpcio-tools into 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels
> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
> _bootstrap
> self.run()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 149, in _install_grpcio_tools_and_generate_proto_files
> shutil.rmtree(build_path)
>   File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
> onerror(os.listdir, path, sys.exc_info())
>   File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
> names = os.listdir(path)
> OSError: [Errno 2] No such file or directory: 
> '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels-build'
> Traceback (most recent call last):
>   File "setup.py", line 235, in 
> 'test': generate_protos_first(test),
>   File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 
> 129, in setup
> return distutils.core.setup(**attrs)
>   File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
>   File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
>   File "setup.py", line 141, in run
> gen_protos.generate_proto_files()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 97, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
>  sdist --formats=zip --dist-dir 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist
>  (see 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log)',
>  1)
> Build step 'Execute shell' marked build as failure
> Sending e-mails to: commits@beam.apache.org
> Finished: FAILURE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441681#comment-16441681
 ] 

Valentyn Tymofieiev commented on BEAM-4105:
---

Seems like too new :)

Verified that pip.pep425tags.get_supported() fails in pip 10.0.0 but passes in 
9.0.3.

If we pick up the latest pip once we start a new virtual environment, it 
explains the issue.

> Python Postcommit suite does not start on Jenkins - proto generation failed
> ---
>
> Key: BEAM-4105
> URL: https://issues.apache.org/jira/browse/BEAM-4105
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
>
> The suite has been steadily failing for last 3 days. Sample logs:  
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
> ERROR: invocation failed (exit code 1), logfile: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log
> ERROR: actionid: tox
> msg: packaging
> cmdargs: ['/usr/bin/python', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py'),
>  'sdist', '--formats=zip', '--dist-dir', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist')]
> /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
> Normalizing '2.5.0.dev' to '2.5.0.dev0'
>   normalized_version,
> Regenerating common_urns module.
> running sdist
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py:51:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> WARNING:root:Installing grpcio-tools into 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels
> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
> _bootstrap
> self.run()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 149, in _install_grpcio_tools_and_generate_proto_files
> shutil.rmtree(build_path)
>   File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
> onerror(os.listdir, path, sys.exc_info())
>   File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
> names = os.listdir(path)
> OSError: [Errno 2] No such file or directory: 
> '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels-build'
> Traceback (most recent call last):
>   File "setup.py", line 235, in 
> 'test': generate_protos_first(test),
>   File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 
> 129, in setup
> return distutils.core.setup(**attrs)
>   File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
>   File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
>   File "setup.py", line 141, in run
> gen_protos.generate_proto_files()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 97, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
>  sdist --formats=zip --dist-dir 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist
>  (see 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log)',
>  1)
> Build step 'Execute shell' marked build as failure
> Sending e-mails to: commits@beam.apache.org
> Finished: FAILURE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441678#comment-16441678
 ] 

Pablo Estrada commented on BEAM-4105:
-

I am not 100% sure, but I think this happens because pip install -e .[gcp,test] 
installs dependencies from within /tmp directories, and the gen_protos call 
expects the directory structure to be like so:
{code:java}
beam/
-model/
-sdks/
--python/
{code}

In https://github.com/apache/beam/pull/5155/files I'm trying to run python 
setup.py install before running pip install, so that gen_protos will run before 
the pip install. I need to refactor the code a bit to be able to run this and 
change to a gradle build gradually.

> Python Postcommit suite does not start on Jenkins - proto generation failed
> ---
>
> Key: BEAM-4105
> URL: https://issues.apache.org/jira/browse/BEAM-4105
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
>
> The suite has been steadily failing for last 3 days. Sample logs:  
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
> ERROR: invocation failed (exit code 1), logfile: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log
> ERROR: actionid: tox
> msg: packaging
> cmdargs: ['/usr/bin/python', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py'),
>  'sdist', '--formats=zip', '--dist-dir', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist')]
> /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
> Normalizing '2.5.0.dev' to '2.5.0.dev0'
>   normalized_version,
> Regenerating common_urns module.
> running sdist
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py:51:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> WARNING:root:Installing grpcio-tools into 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels
> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
> _bootstrap
> self.run()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 149, in _install_grpcio_tools_and_generate_proto_files
> shutil.rmtree(build_path)
>   File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
> onerror(os.listdir, path, sys.exc_info())
>   File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
> names = os.listdir(path)
> OSError: [Errno 2] No such file or directory: 
> '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels-build'
> Traceback (most recent call last):
>   File "setup.py", line 235, in 
> 'test': generate_protos_first(test),
>   File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 
> 129, in setup
> return distutils.core.setup(**attrs)
>   File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
>   File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
>   File "setup.py", line 141, in run
> gen_protos.generate_proto_files()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 97, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
>  sdist --formats=zip --dist-dir 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist
>  (see 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log)',
>  1)
> Build step 'Execute shell' marked build as failure
> Sending e-mails to: commits@beam.apache.org
> Finished: FAILURE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92004
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182273236
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
 ##
 @@ -94,7 +96,7 @@ public void testCustomTimestampPolicyWithLimitedDelay() {
  -150_000L,
  -120_000L,
  -140_000L,
-  100_000L,  // <<< timestamp is in future
+ 100_000L, // <<< timestamp is in future
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92004)
Time Spent: 5h 20m  (was: 5h 10m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92007
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182272907
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
 ##
 @@ -66,9 +76,24 @@ public void encode(KafkaRecord value, OutputStream 
outStream) throws IOExc
 longCoder.decode(inStream),
 longCoder.decode(inStream),
 KafkaTimestampType.forOrdinal(intCoder.decode(inStream)),
+getHeaders(headerCoder.decode(inStream)),
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92007)
Time Spent: 5h 40m  (was: 5.5h)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=91999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91999
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182272666
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
 ##
 @@ -139,4 +158,14 @@ public long offsetForTime(Consumer consumer, 
TopicPartition topicPartition
   return offsetAndTimestamp.offset();
 }
   }
+
+  public Headers getHeaders(ConsumerRecord rawRecord) {
+Headers recordHeaders = new RecordHeaders();
 
 Review comment:
   Updated, instead of throwing RuntimeException, for now simply returning null.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91999)
Time Spent: 4h 50m  (was: 4h 40m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92006
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182273145
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
 ##
 @@ -46,13 +47,14 @@
   .map(ts -> {
 Instant result = policy.getTimestampForRecord(
   null, new KafkaRecord<>("topic", 0, 0, now.getMillis() + ts,
-  KafkaTimestampType.CREATE_TIME, "key", 
"value"));
+  KafkaTimestampType.CREATE_TIME,
+  new RecordHeaders(),
 
 Review comment:
   Kept headers null for this test, added tests to KafkaRecordCoderTest for 
null and non-null cases.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92006)
Time Spent: 5.5h  (was: 5h 20m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92005
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182273223
 
 

 ##
 File path: 
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
 ##
 @@ -79,7 +81,7 @@ public void testCustomTimestampPolicyWithLimitedDelay() {
 -150_000L,
 -120_000L,
 -140_000L,
--100_000L,  // <<< Max timestamp
+-100_000L, // <<< Max timestamp
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92005)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92003
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182273301
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -33,6 +34,7 @@
   private final String topic;
   private final int partition;
   private final long offset;
+  private final Headers headers;
 
 Review comment:
   Sure will do. See updated comment above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92003)
Time Spent: 5h 10m  (was: 5h)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92001
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182272599
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
 ##
 @@ -131,6 +149,7 @@ public long offsetForTime(Consumer consumer, 
TopicPartition topicPartition
 .offsetsForTimes(ImmutableMap.of(topicPartition, time.getMillis()))
 .values());
 
+
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92001)
Time Spent: 5h 10m  (was: 5h)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92002
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182273112
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
 ##
 @@ -66,9 +76,24 @@ public void encode(KafkaRecord value, OutputStream 
outStream) throws IOExc
 longCoder.decode(inStream),
 longCoder.decode(inStream),
 KafkaTimestampType.forOrdinal(intCoder.decode(inStream)),
+getHeaders(headerCoder.decode(inStream)),
 kvCoder.decode(inStream));
   }
 
+  private Headers getHeaders(Iterable> records) {
+Headers headers = new RecordHeaders();
 
 Review comment:
   Instantiating via ```ConsumerRecord``` is a bit hacky. It should be fine if 
we use ```RecordHeaders``` directly. If it was not meant to be used, it should 
be encapsulated more thoroughly (package-private).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92002)
Time Spent: 5h 10m  (was: 5h)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=92000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92000
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182272769
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -90,7 +98,7 @@ public KafkaTimestampType getTimestampType() {
 
   @Override
   public int hashCode() {
-return Arrays.deepHashCode(new Object[]{topic, partition, offset, 
timestamp, kv});
+return Arrays.deepHashCode(new Object[] {topic, partition, offset, 
timestamp, headers, kv});
 
 Review comment:
   Verified, null works as expected.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 92000)
Time Spent: 5h  (was: 4h 50m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=91998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91998
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:43
Start Date: 17/Apr/18 23:43
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182272786
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java
 ##
 @@ -55,6 +64,7 @@ public void encode(KafkaRecord value, OutputStream 
outStream) throws IOExc
 longCoder.encode(value.getOffset(), outStream);
 longCoder.encode(value.getTimestamp(), outStream);
 intCoder.encode(value.getTimestampType().ordinal(), outStream);
+headerCoder.encode(getIterable(value.getHeaders()), outStream);
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91998)
Time Spent: 4h 40m  (was: 4.5h)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3792) Python submits portable pipelines to the Flink-served endpoint.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3792?focusedWorklogId=91997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91997
 ]

ASF GitHub Bot logged work on BEAM-3792:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:41
Start Date: 17/Apr/18 23:41
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4811: [BEAM-3792] Allow 
manual specification of external address for ULR.
URL: https://github.com/apache/beam/pull/4811#issuecomment-382192749
 
 
   Jenkins: retest this please.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91997)
Time Spent: 50m  (was: 40m)

> Python submits portable pipelines to the Flink-served endpoint.
> ---
>
> Key: BEAM-3792
> URL: https://issues.apache.org/jira/browse/BEAM-3792
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

[Robert Bradshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441669#comment-16441669
 ] 

Robert Bradshaw commented on BEAM-4105:
---

Perhaps the pip is too old? 

> Python Postcommit suite does not start on Jenkins - proto generation failed
> ---
>
> Key: BEAM-4105
> URL: https://issues.apache.org/jira/browse/BEAM-4105
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
>
> The suite has been steadily failing for last 3 days. Sample logs:  
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
> ERROR: invocation failed (exit code 1), logfile: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log
> ERROR: actionid: tox
> msg: packaging
> cmdargs: ['/usr/bin/python', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py'),
>  'sdist', '--formats=zip', '--dist-dir', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist')]
> /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
> Normalizing '2.5.0.dev' to '2.5.0.dev0'
>   normalized_version,
> Regenerating common_urns module.
> running sdist
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py:51:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> WARNING:root:Installing grpcio-tools into 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels
> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
> _bootstrap
> self.run()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 149, in _install_grpcio_tools_and_generate_proto_files
> shutil.rmtree(build_path)
>   File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
> onerror(os.listdir, path, sys.exc_info())
>   File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
> names = os.listdir(path)
> OSError: [Errno 2] No such file or directory: 
> '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels-build'
> Traceback (most recent call last):
>   File "setup.py", line 235, in 
> 'test': generate_protos_first(test),
>   File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 
> 129, in setup
> return distutils.core.setup(**attrs)
>   File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
>   File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
>   File "setup.py", line 141, in run
> gen_protos.generate_proto_files()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 97, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
>  sdist --formats=zip --dist-dir 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist
>  (see 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log)',
>  1)
> Build step 'Execute shell' marked build as failure
> Sending e-mails to: commits@beam.apache.org
> Finished: FAILURE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #105

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=91994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91994
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:24
Start Date: 17/Apr/18 23:24
Worklog Time Spent: 10m 
  Work Description: gkumar7 commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182243015
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -33,6 +34,7 @@
   private final String topic;
   private final int partition;
   private final long offset;
+  private final Headers headers;
 
 Review comment:
   Could you explain how this would work? Marking this as ```@Nullable``` would 
not help circumvent the issue that ```Headers``` interface would still need to 
be on the classpath. To verify, I made the requested changes and I receive the 
following runtime error:
   
   ```
   Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/kafka/common/header/Headers
at 
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getOutputCoder(KafkaUnboundedSource.java:142)
   ```
   
   When using kafka-clients 0.10.1.0. The ```Headers``` interface is only 
available post 0.11.0.0.
   
   Did a few more tests, here is a simple example:
   
   Suppose I have a class with a single *unused* method:
   ```
   public void getHeaders() {
   if (!hasHeaders()){ // reflection logic using ConsumerRecord
 return;
   }
   Headers headers = new RecordHeaders();
   headers.add(new RecordHeader("hi", "hi".getBytes()));
}
   ```
   and I package this class and use it as a dependency in another application 
which simply instantiates this class. Even though the downstream application 
did not call ```getHeaders```, it will still throw a runtime exception.
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91994)
Time Spent: 4h 20m  (was: 4h 10m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4038) Support Kafka Headers in KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=91995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91995
 ]

ASF GitHub Bot logged work on BEAM-4038:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:24
Start Date: 17/Apr/18 23:24
Worklog Time Spent: 10m 
  Work Description: rangadi commented on a change in pull request #5111: 
[BEAM-4038] Support Kafka Headers in KafkaIO
URL: https://github.com/apache/beam/pull/5111#discussion_r182270754
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
 ##
 @@ -33,6 +34,7 @@
   private final String topic;
   private final int partition;
   private final long offset;
+  private final Headers headers;
 
 Review comment:
   can you push your changes? `KafkaRecordCoder` constructor does not access 
Headers at all. It is accessed only in encode and decode (where we should 
handle null values for headers).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91995)
Time Spent: 4.5h  (was: 4h 20m)

> Support Kafka Headers in KafkaIO
> 
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka
>Reporter: Geet Kumar
>Assignee: Raghu Angadi
>Priority: Minor
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The 
> purpose of this JIRA is to support this feature in KafkaIO.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4105) Python Postcommit suite does not start on Jenkins - proto generation failed

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441658#comment-16441658
 ] 

Valentyn Tymofieiev commented on BEAM-4105:
---

Following error that happens before OSError (in `try` clause):

Traceback (most recent call last):
 File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
 self.run()
 File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
 self._target(*self._args, **self._kwargs)
 File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
 line 140, in _install_grpcio_tools_and_generate_proto_files
 pprint.pprint(pip.pep425tags.get_supported())
 AttributeError: 'module' object has no attribute 'pep425tags'

> Python Postcommit suite does not start on Jenkins - proto generation failed
> ---
>
> Key: BEAM-4105
> URL: https://issues.apache.org/jira/browse/BEAM-4105
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
>
> The suite has been steadily failing for last 3 days. Sample logs:  
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
> ERROR: invocation failed (exit code 1), logfile: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log
> ERROR: actionid: tox
> msg: packaging
> cmdargs: ['/usr/bin/python', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py'),
>  'sdist', '--formats=zip', '--dist-dir', 
> local('/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist')]
> /usr/local/lib/python2.7/dist-packages/setuptools/dist.py:397: UserWarning: 
> Normalizing '2.5.0.dev' to '2.5.0.dev0'
>   normalized_version,
> Regenerating common_urns module.
> running sdist
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py:51:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> WARNING:root:Installing grpcio-tools into 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels
> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
> _bootstrap
> self.run()
>   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 149, in _install_grpcio_tools_and_generate_proto_files
> shutil.rmtree(build_path)
>   File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
> onerror(os.listdir, path, sys.exc_info())
>   File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
> names = os.listdir(path)
> OSError: [Errno 2] No such file or directory: 
> '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/grpcio-wheels-build'
> Traceback (most recent call last):
>   File "setup.py", line 235, in 
> 'test': generate_protos_first(test),
>   File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 
> 129, in setup
> return distutils.core.setup(**attrs)
>   File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
> dist.run_commands()
>   File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
> cmd_obj.run()
>   File "setup.py", line 141, in run
> gen_protos.generate_proto_files()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/gen_protos.py",
>  line 97, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/setup.py
>  sdist --formats=zip --dist-dir 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/dist
>  (see 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/target/.tox/log/tox-0.log)',
>  1)
> Build step 'Execute shell' marked build as failure
> Sending e-mails to: commits@beam.apache.org
> Finished: 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91985
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182267064
 
 

 ##
 File path: 
runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
 ##
 @@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.junit.Assert.assertTrue;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.beam.runners.reference.CloseableResource.CloseException;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link CloseableResource}. */
+@RunWith(JUnit4.class)
+public class CloseableResourceTest {
+
+  @Test
+  public void alwaysReturnsSameResource() {
+Foo foo = new Foo();
+CloseableResource resource = CloseableResource.of(foo, (ignored) -> 
{});
+assertThat(resource.get(), is(foo));
+assertThat(resource.get(), is(foo));
+  }
+
+  @Test
+  public void callsCloser() throws Exception {
+AtomicBoolean closed = new AtomicBoolean(false);
+try (CloseableResource ignored =
+CloseableResource.of(
+new Foo(),
+(foo) -> {
+  closed.set(true);
+})) {
+  // Do nothing.
+}
+assertTrue(closed.get());
+  }
+
+  @Test
+  public void wrapsExceptionsInCloseException() {
+Exception wrapped = new Exception();
+CloseException closeException = null;
+try (CloseableResource ignored =
+CloseableResource.of(
+new Foo(),
+(foo) -> {
+  throw wrapped;
+})) {
+  // Do nothing.
+} catch (CloseException e) {
+  closeException = e;
 
 Review comment:
   Yes, I would much prefer assertThrows. I dislike `ExpectedException` because 
it limits the assertions that can be made and requires the exception to be 
thrown as the very last statement of the method. Otherwise, you do not know 
from which line the exception is actually being thrown. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91985)
Time Spent: 7h 20m  (was: 7h 10m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91992
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182264068
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91976
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182260262
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
 
 Review comment:
   I would have gladly done that, but that doesn't fit with the 
`PipelineRunner` construction. As mentioned above, the fit between a Java 
runner and a portable runner service isn't great.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91976)
Time Spent: 6h 20m  (was: 6h 10m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91973
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182257618
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/DirectoryZipper.java
 ##
 @@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import org.apache.beam.sdk.util.ZipFiles;
+
+/** A utility that replaces directories with a zip file containing its 
contents. */
 
 Review comment:
   Actually, I've gone ahead and removed this. We're doing a lot of I/O in the 
end-to-end test anyway, so this doesn't buy us much. This code has been inlined 
and is no longer pluggable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91973)
Time Spent: 6h 10m  (was: 6h)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91965
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182232673
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
 ##
 @@ -87,26 +87,27 @@ private ArtifactServiceStager(Channel channel, int 
bufferSize) {
 this.bufferSize = bufferSize;
   }
 
-  public void stage(Iterable files) throws IOException, 
InterruptedException {
-final Map futures = new 
HashMap<>();
-for (File file : files) {
+  public String stage(Iterable files) throws IOException, 
InterruptedException {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91965)
Time Spent: 5h 10m  (was: 5h)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91982
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182261921
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
 ##
 @@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import com.google.common.util.concurrent.Uninterruptibles;
+import com.google.protobuf.ByteString;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class JobServicePipelineResult implements PipelineResult {
+
+  private static final long POLL_INTERVAL_SEC = 10;
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(JobServicePipelineResult.class);
+
+  private final ByteString jobId;
+  private final CloseableResource jobService;
+
+  JobServicePipelineResult(ByteString jobId, 
CloseableResource jobService) {
+this.jobId = jobId;
+this.jobService = jobService;
+  }
+
+  @Override
+  public State getState() {
+JobServiceBlockingStub stub = jobService.get();
+GetJobStateResponse response =
+
stub.getState(GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build());
+return getJavaState(response.getState());
+  }
+
+  @Override
+  public State cancel() {
+JobServiceBlockingStub stub = jobService.get();
+CancelJobResponse response =
+
stub.cancel(CancelJobRequest.newBuilder().setJobIdBytes(jobId).build());
+return getJavaState(response.getState());
+  }
+
+  @Override
+  public State waitUntilFinish(Duration duration) {
+if (duration.compareTo(Duration.millis(1)) < 1) {
+  // Equivalent to infinite timeout.
+  return waitUntilFinish();
+} else {
+  CompletableFuture result = 
CompletableFuture.supplyAsync(this::waitUntilFinish);
+  try {
+return Uninterruptibles.getUninterruptibly(
+result, duration.getMillis(), TimeUnit.MILLISECONDS);
+  } catch (TimeoutException e) {
+// Null result indicates a timeout.
+return null;
+  } catch (ExecutionException e) {
+throw new RuntimeException(e);
+  }
+}
+  }
+
+  @Override
+  public State waitUntilFinish() {
+JobServiceBlockingStub stub = jobService.get();
+GetJobStateRequest request = 
GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build();
+State lastState;
+do {
+  Uninterruptibles.sleepUninterruptibly(POLL_INTERVAL_SEC, 
TimeUnit.SECONDS);
 
 Review comment:
   Conciseness. Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91982)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
>   

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91980
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182260838
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91981
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182264134
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91979
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182260505
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91963
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182247718
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/DirectoryZipper.java
 ##
 @@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import org.apache.beam.sdk.util.ZipFiles;
+
+/** A utility that replaces directories with a zip file containing its 
contents. */
+class DirectoryZipper {
+
+  /**
+   * If the given path is a regular file, returns the file itself. If the 
given file is a directory,
+   * zips the file and returns the path to this new zip file.
+   */
+  public String replaceDirectoryWithZipFile(String path) throws IOException {
+return zipDirectoryInternal(path);
 
 Review comment:
   Done. This was an artifact of how I refactored the original code. ;)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91963)
Time Spent: 4h 50m  (was: 4h 40m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91983
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182264845
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InMemoryArtifactService.java
 ##
 @@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.io.BaseEncoding;
+import com.google.protobuf.ByteString;
+import io.grpc.Status;
+import io.grpc.stub.StreamObserver;
+import java.security.MessageDigest;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A StagingService for tests. */
+public class InMemoryArtifactService extends ArtifactStagingServiceImplBase {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(InMemoryArtifactService.class);
+
+  private final Object artifactLock = new Object();
+
+  @GuardedBy("artifactLock")
+  private final Map artifacts = 
Maps.newHashMap();
+
+  private final boolean keepArtifacts;
+
+  @GuardedBy("artifactLock")
+  private boolean committed = false;
+
+  public InMemoryArtifactService(boolean keepArtifacts) {
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91983)
Time Spent: 7h  (was: 6h 50m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91987
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182267916
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import java.util.List;
+import org.apache.beam.sdk.options.Validation.Required;
+
+/** Pipeline options common to all portable runners. */
+public interface PortablePipelineOptions extends PipelineOptions {
+
+  /**
+   * List of local files to make available to workers.
+   *
+   * Files are placed on the worker's classpath.
 
 Review comment:
   The worker will access all of the files though the manifest. The files are 
expected to be jars or directories that should be placed on the classpath and 
nothing else. This matches the current Dataflow behavior and is the only thing 
supported.
   
   To really drive this home, we could remove the option to override stage 
files and just always scan for URLClassLoaders and add elements to the stage 
file list. However, this prevents users from using custom ClassLoaders. Perhaps 
the way to solve this for now is to rename this to something like 
`getClasspathArtifacts` or similar.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91987)
Time Spent: 7.5h  (was: 7h 20m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91977
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182255696
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/DirectoryZipper.java
 ##
 @@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import org.apache.beam.sdk.util.ZipFiles;
+
+/** A utility that replaces directories with a zip file containing its 
contents. */
 
 Review comment:
   This was an artifact of how the functionality was originally written. I've 
updated the class to more closely reflect what we needed it to do in the first 
place. By doing it this way, we can also avoid writing out empty files in tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91977)
Time Spent: 6.5h  (was: 6h 20m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91967
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182243728
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
 ##
 @@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+/**
+ * An {@link AutoCloseable} that wraps a resource that needs to be cleaned up 
but does not implement
+ * {@link AutoCloseable} itself. Recipients of a {@link CloseableResource} are 
in general
+ * responsible for cleanup. Not thread-safe.
+ */
+public class CloseableResource implements AutoCloseable {
+
+  private final T resource;
+  private final Closer closer;
+
+  private boolean isClosed = false;
+
+  private CloseableResource(T resource, Closer closer) {
+this.resource = resource;
+this.closer = closer;
+  }
+
+  /** Creates a {@link CloseableResource} with the given resource and closer. 
*/
+  public static  CloseableResource of(T resource, Closer closer) {
+checkArgument(resource != null, "Resource must be non-null");
+checkArgument(closer != null, "%s must be non-null", 
Closer.class.getName());
+return new CloseableResource<>(resource, closer);
+  }
+
+  /** Gets the underlying resource. */
+  public T get() {
+checkState(!isClosed, "% is closed", CloseableResource.class.getName());
+return resource;
+  }
+
+  /**
+   * Close the underlying resource. Must only be called once.
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91967)
Time Spent: 5.5h  (was: 5h 20m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91964
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182234035
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
 ##
 @@ -222,13 +225,26 @@ public StagingResult call() {
 }
   }
 
+  /** A file along with a staging name. */
+  @AutoValue
+  public abstract static class FileToStage {
+public static FileToStage of(File file, String stageName) {
+  return new AutoValue_ArtifactServiceStager_FileToStage(file, stageName);
+}
+
+/** The file to stage. */
+public abstract File getFile();
+/** Staging handle to this file. */
 
 Review comment:
   We were using filenames and strings everywhere else, so I continued using 
strings here to minimize the change size (which is already large). If I update 
this to a `Path`, I think it makes sense to change all `File` references to 
match this. Let me know if that's a change worth making here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91964)
Time Spent: 5h  (was: 4h 50m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91969
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182235637
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91975
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182266551
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import io.grpc.stub.StreamObserver;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+
+/** A JobService for tests. */
 
 Review comment:
   Added javadoc.
   
   This _could_ be replaced by a mock if we decided to have the PortableRunner 
be handed a blocking stub rather than constructing one itself. In that case, we 
would mock the blocking stub directly. Of course, that becomes difficult once 
we want to start testing the streaming RPCs (job status, etc).
   
   I originally decided it was not reasonable to have the stub and underlying 
channel constructed externally due to ownership confusion. However, now that we 
have `CloseableResource`, this may be reasonable as long as we aren't using 
streaming RPCs. What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91975)
Time Spent: 6h 10m  (was: 6h)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91968
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182260015
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
 ##
 @@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import com.google.common.util.concurrent.Uninterruptibles;
+import com.google.protobuf.ByteString;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class JobServicePipelineResult implements PipelineResult {
+
+  private static final long POLL_INTERVAL_SEC = 10;
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(JobServicePipelineResult.class);
+
+  private final ByteString jobId;
+  private final CloseableResource jobService;
+
+  JobServicePipelineResult(ByteString jobId, 
CloseableResource jobService) {
+this.jobId = jobId;
+this.jobService = jobService;
+  }
+
+  @Override
+  public State getState() {
+JobServiceBlockingStub stub = jobService.get();
+GetJobStateResponse response =
+
stub.getState(GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build());
+return getJavaState(response.getState());
+  }
+
+  @Override
+  public State cancel() {
+JobServiceBlockingStub stub = jobService.get();
+CancelJobResponse response =
+
stub.cancel(CancelJobRequest.newBuilder().setJobIdBytes(jobId).build());
+return getJavaState(response.getState());
+  }
+
+  @Override
+  public State waitUntilFinish(Duration duration) {
+if (duration.compareTo(Duration.millis(1)) < 1) {
+  // Equivalent to infinite timeout.
+  return waitUntilFinish();
+} else {
+  CompletableFuture result = 
CompletableFuture.supplyAsync(this::waitUntilFinish);
+  try {
+return Uninterruptibles.getUninterruptibly(
 
 Review comment:
   The whole thing is funky. The fact that we have to wrap a long-running 
server connection in a pipeline result that is completely unaware of this makes 
it difficult to use correctly. This was really just a stop-gap to get something 
that we could eventually run in a test runner.
   
   I think that doing this the correct way will require more discussion.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91968)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91990
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182268058
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import java.util.List;
+import org.apache.beam.sdk.options.Validation.Required;
+
+/** Pipeline options common to all portable runners. */
+public interface PortablePipelineOptions extends PipelineOptions {
+
+  /**
+   * List of local files to make available to workers.
+   *
+   * Files are placed on the worker's classpath.
+   *
+   * The default value is the list of jars from the main program's 
classpath.
+   */
+  @Description(
+  "Files to stage to the artifact service and make available to workers. 
Files are placed on "
+  + "the worker's classpath. The default value is all files from the 
classpath.")
+  List getFilesToStage();
+  void setFilesToStage(List value);
+
+  @Description("Job service endpoint to use.")
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91990)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91986
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182267118
 
 

 ##
 File path: 
runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
 ##
 @@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import io.grpc.Server;
+import io.grpc.inprocess.InProcessServerBuilder;
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.reference.testing.InMemoryArtifactService;
+import 
org.apache.beam.runners.reference.testing.InProcessManagedChannelFactory;
+import org.apache.beam.runners.reference.testing.TestJobService;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link PortableRunner}. */
+@RunWith(JUnit4.class)
+public class PortableRunnerTest implements Serializable {
+
+  private static final String ENDPOINT_URL = "foo:3000";
+  private static final ApiServiceDescriptor ENDPOINT_DESCRIPTOR =
+  ApiServiceDescriptor.newBuilder().setUrl(ENDPOINT_URL).build();
+
+  private PipelineOptions options = createPipelineOptions();
+
+  @Rule public transient TestPipeline p = TestPipeline.fromOptions(options);
+
+  @Test
+  public void stagesAndRunsJob() throws Exception {
+try (CloseableResource server = 
createJobServer(JobState.Enum.DONE)) {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91986)
Time Spent: 7h 20m  (was: 7h 10m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91984
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182260921
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91991
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182263508
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91974
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182242374
 
 

 ##
 File path: 
runners/local-artifact-service-java/src/test/java/org/apache/beam/artifact/local/LocalFileSystemArtifactRetrievalServiceTest.java
 ##
 @@ -186,11 +187,11 @@ public void onCompleted() {
   }
 
   private void stageAndCreateRetrievalService(Map artifacts) 
throws Exception {
-List artifactFiles = new ArrayList<>();
+List artifactFiles = new ArrayList<>();
 for (Map.Entry artifact : artifacts.entrySet()) {
   File artifactFile = tmp.newFile(artifact.getKey());
   new 
FileOutputStream(artifactFile).getChannel().write(ByteBuffer.wrap(artifact.getValue()));
-  artifactFiles.add(artifactFile);
+  artifactFiles.add(FileToStage.of(artifactFile, artifactFile.getName()));
 
 Review comment:
   Done. Not sure why it was that way originally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91974)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91988
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182265271
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InMemoryArtifactService.java
 ##
 @@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference.testing;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import com.google.common.io.BaseEncoding;
+import com.google.protobuf.ByteString;
+import io.grpc.Status;
+import io.grpc.stub.StreamObserver;
+import java.security.MessageDigest;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A StagingService for tests. */
+public class InMemoryArtifactService extends ArtifactStagingServiceImplBase {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(InMemoryArtifactService.class);
+
+  private final Object artifactLock = new Object();
+
+  @GuardedBy("artifactLock")
+  private final Map artifacts = 
Maps.newHashMap();
+
+  private final boolean keepArtifacts;
+
+  @GuardedBy("artifactLock")
+  private boolean committed = false;
+
+  public InMemoryArtifactService(boolean keepArtifacts) {
+this.keepArtifacts = keepArtifacts;
+  }
+
+  @Override
+  public StreamObserver putArtifact(
+  StreamObserver responseObserver) {
+return new PutArtifactRequestObserver(responseObserver, keepArtifacts, 
this::putArtifact);
+  }
+
+  @Override
+  public void commitManifest(
+  CommitManifestRequest request, StreamObserver 
responseObserver) {
+LOG.debug("Committing manifest: {}", request);
+try {
+  Set commitMetadata =
+  Sets.newHashSet(request.getManifest().getArtifactList());
+  synchronized (artifactLock) {
+checkState(!committed, "Manifest already committed");
+// TODO: Consider comparing only by artifact name if checksums are not 
sent.
+Set missingFromServer =
+Sets.difference(commitMetadata, artifacts.keySet());
+checkArgument(
+missingFromServer.isEmpty(),
+"Artifacts from request never uploaded: %s",
+missingFromServer);
+
+Set extraOnServer = 
Sets.difference(artifacts.keySet(), commitMetadata);
+checkArgument(extraOnServer.isEmpty(), "Extraneous artifacts received: 
%s", extraOnServer);
+// NOTE: We do not attempt to commit transactionally; if a commit 
fails, the artifact server
+// is assumed to be in a bad state.
+committed = true;
+responseObserver.onNext(CommitManifestResponse.getDefaultInstance());
+responseObserver.onCompleted();
+  }
+} catch (Exception e) {
+  LOG.warn("Error committing manifest", e);
+  responseObserver.onError(
+  Status.INVALID_ARGUMENT
+  .withDescription(e.getMessage())
+  .withCause(e)
+  

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91966
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182235007
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
 
 Review comment:
   Whoops. ;)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91966)
Time Spent: 5h 20m  (was: 5h 10m)

> 

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91971
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182239755
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
 ##
 @@ -189,16 +189,18 @@ public String create(PipelineOptions options) {
   String getWorkerMachineType();
   void setWorkerMachineType(String value);
 
+  // TODO: Replace this with PortablePipelineOptions#getFilesToStage.
 
 Review comment:
   `getFilesToStage` is referenced in multiple places throughout the Dataflow 
runner code. I want to minimize the changes outside of the portable runner 
itself here and also want to prevent breaking semantics that the Dataflow 
runner expects. For example, getFilesToStage is populated with the runtime 
classpath and augmented with other stage files at some point. I'll make a bug 
and reference it from `PortablePipelineOptions`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91971)
Time Spent: 6h  (was: 5h 50m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91989
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182266695
 
 

 ##
 File path: 
runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
 ##
 @@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.junit.Assert.assertTrue;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.beam.runners.reference.CloseableResource.CloseException;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link CloseableResource}. */
+@RunWith(JUnit4.class)
+public class CloseableResourceTest {
+
+  @Test
+  public void alwaysReturnsSameResource() {
+Foo foo = new Foo();
+CloseableResource resource = CloseableResource.of(foo, (ignored) -> 
{});
+assertThat(resource.get(), is(foo));
+assertThat(resource.get(), is(foo));
+  }
+
+  @Test
+  public void callsCloser() throws Exception {
+AtomicBoolean closed = new AtomicBoolean(false);
+try (CloseableResource ignored =
+CloseableResource.of(
+new Foo(),
+(foo) -> {
+  closed.set(true);
+})) {
+  // Do nothing.
+}
+assertTrue(closed.get());
+  }
+
+  @Test
+  public void wrapsExceptionsInCloseException() {
+Exception wrapped = new Exception();
+CloseException closeException = null;
+try (CloseableResource ignored =
+CloseableResource.of(
+new Foo(),
+(foo) -> {
+  throw wrapped;
+})) {
+  // Do nothing.
+} catch (CloseException e) {
+  closeException = e;
+}
+assertThat(closeException, is(notNullValue()));
 
 Review comment:
   Good point.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91989)
Time Spent: 7h 40m  (was: 7.5h)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91978
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182234584
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
 ##
 @@ -189,16 +189,18 @@ public String create(PipelineOptions options) {
   String getWorkerMachineType();
   void setWorkerMachineType(String value);
 
+  // TODO: Replace this with PortablePipelineOptions#getFilesToStage.
   /**
* List of local files to make available to workers.
*
* Files are placed on the worker's classpath.
*
* The default value is the list of jars from the main program's 
classpath.
*/
-  @Description("Files to stage on GCS and make available to workers. "
-  + "Files are placed on the worker's classpath. "
-  + "The default value is all files from the classpath.")
+  @Description(
+  "Files to stage on GCS and make available to workers. "
 
 Review comment:
   It is (was?) for Dataflow. I didn't actually change the comment here. It was 
apparently picked up by google-java-format. I'll revert it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91978)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91970
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182244056
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
 ##
 @@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+/**
+ * An {@link AutoCloseable} that wraps a resource that needs to be cleaned up 
but does not implement
+ * {@link AutoCloseable} itself. Recipients of a {@link CloseableResource} are 
in general
+ * responsible for cleanup. Not thread-safe.
+ */
+public class CloseableResource implements AutoCloseable {
+
+  private final T resource;
+  private final Closer closer;
+
+  private boolean isClosed = false;
+
+  private CloseableResource(T resource, Closer closer) {
+this.resource = resource;
+this.closer = closer;
+  }
+
+  /** Creates a {@link CloseableResource} with the given resource and closer. 
*/
+  public static  CloseableResource of(T resource, Closer closer) {
+checkArgument(resource != null, "Resource must be non-null");
+checkArgument(closer != null, "%s must be non-null", 
Closer.class.getName());
+return new CloseableResource<>(resource, closer);
+  }
+
+  /** Gets the underlying resource. */
+  public T get() {
+checkState(!isClosed, "% is closed", CloseableResource.class.getName());
+return resource;
+  }
+
+  /**
+   * Close the underlying resource. Must only be called once.
+   *
+   * @throws CloseException wrapping any exceptions thrown while closing
+   */
+  @Override
+  public void close() throws CloseException {
+checkState(!isClosed, "% is closed", CloseableResource.class.getName());
+try {
+  closer.close(resource);
+} catch (Exception e) {
+  throw new CloseException(e);
 
 Review comment:
   That's interesting because it prevents retries on `close()` where the method 
itself attempts to be idempotent. On the other hand, without transactions on 
the underlying closer itself, you can't really safely call it multiple times 
anyway.
   
   I've changed the logic as per AutoCloseable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91970)
Time Spent: 5h 50m  (was: 5h 40m)

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=91972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91972
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 17/Apr/18 23:09
Start Date: 17/Apr/18 23:09
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5150:  
[BEAM-4071] Add Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150#discussion_r182262577
 
 

 ##
 File path: 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 ##
 @@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.reference;
+
+import static com.google.common.base.Preconditions.checkState;
+import static 
org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
+import com.google.protobuf.ByteString;
+import io.grpc.ManagedChannel;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import 
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.core.construction.ArtifactServiceStager;
+import 
org.apache.beam.runners.core.construction.ArtifactServiceStager.FileToStage;
+import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A {@link PipelineRunner} a {@link Pipeline} against a {@code JobService}. 
*/
+public class PortableRunner extends PipelineRunner {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(PortableRunner.class);
+
+  /** Provided pipeline options. */
+  private final PipelineOptions options;
+  /** Job API endpoint. */
+  private final String endpoint;
+  /** Files to stage to artifact staging service. They will ultimately be 
added to the classpath. */
+  private final Collection filesToStage;
+  /** Channel factory used to create communication channel with job and 
staging services. */
+  private final ManagedChannelFactory channelFactory;
+
+  /**
+   * Constructs a runner from the provided options.
+   *
+   * @param options Properties which configure the runner.
+   * @return The newly created runner.
+   */
+  public static PortableRunner fromOptions(PipelineOptions options) {
+if (true) {
+  throw new UnsupportedOperationException();
+}
+return createInternal(options, new DirectoryZipper(), 
getChannelFactory(options));
+  }
+
+  @VisibleForTesting
+  static PortableRunner createInternal(
+  PipelineOptions options, DirectoryZipper zipper, ManagedChannelFactory 
channelFactory) {
+PortablePipelineOptions portableOptions =
+PipelineOptionsValidator.validate(PortablePipelineOptions.class, 
options);
+
+String 

[jira] [Work logged] (BEAM-3697) Add errorprone to gradle builds

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3697?focusedWorklogId=91960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91960
 ]

ASF GitHub Bot logged work on BEAM-3697:


Author: ASF GitHub Bot
Created on: 17/Apr/18 22:58
Start Date: 17/Apr/18 22:58
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5161: [BEAM-3697] Add 
errorprone to Gradle build
URL: https://github.com/apache/beam/pull/5161#issuecomment-382183379
 
 
   @akedin I cherrypicked your change and just sickbayed one broken test. I 
think there was a broken test in HDFS which I'll let Jenkins find for me.
   
   @swegner FYI on the Gradle project management front and @lukecwik on the 
Gradle code front. Minor issue in how we clobber all configs, while errorprone 
has its own we shouldn't clobber.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91960)
Time Spent: 20m  (was: 10m)

> Add errorprone to gradle builds
> ---
>
> Key: BEAM-3697
> URL: https://issues.apache.org/jira/browse/BEAM-3697
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> [http://errorprone.info/] is a good static checker that covers a number of 
> bugs not covered by FindBugs or Checkstyle. We use it internally at Google 
> and, when run on the Beam codebase, it occasionally uncovers issues missed 
> during PR review process.
>  
> It has Maven and Gradle plugins:
> [http://errorprone.info/docs/installation]
> [https://github.com/tbroyer/gradle-errorprone-plugin]
>  
> It would be good to integrate it into our Maven and Gradle builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3697) Add errorprone to gradle builds

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3697?focusedWorklogId=91959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91959
 ]

ASF GitHub Bot logged work on BEAM-3697:


Author: ASF GitHub Bot
Created on: 17/Apr/18 22:53
Start Date: 17/Apr/18 22:53
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5161: 
[BEAM-3697] Add errorprone to Gradle build
URL: https://github.com/apache/beam/pull/5161
 
 
   This adds the nice errorprone tool to our builds. It is faster and more 
meaningful than findbugs. If this gets in, I would drop findbugs to save 
analysis time and false warnings.
   
   I have run this tool over the codebase and there are lots of actual bugs. I 
fixed the ones deemed "error" in the core SDK. Likely Jenkins will hit a few 
more and I will fix them.
   
   It also uncovered a minor issue with our Gradle config that I have worked 
around as it will take more work to actually redesign the build.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 91959)
Time Spent: 10m
Remaining Estimate: 0h

> Add errorprone to gradle builds
> ---
>
> Key: BEAM-3697
> URL: https://issues.apache.org/jira/browse/BEAM-3697
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [http://errorprone.info/] is a good static checker that covers a number of 
> bugs not covered by FindBugs or Checkstyle. We use it internally at Google 
> and, when run on the Beam codebase, it occasionally uncovers issues missed 
> during PR review process.
>  
> It has Maven and Gradle plugins:
> [http://errorprone.info/docs/installation]
> [https://github.com/tbroyer/gradle-errorprone-plugin]
>  
> It would be good to integrate it into our Maven and Gradle builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4111) Track performance of time_counter before feature enabled

[Boyuan Zhang (JIRA)]

Boyuan Zhang created BEAM-4111:
--

 Summary: Track performance of time_counter before feature enabled
 Key: BEAM-4111
 URL: https://issues.apache.org/jira/browse/BEAM-4111
 Project: Beam
  Issue Type: Task
  Components: sdk-py-core
Reporter: Boyuan Zhang
Assignee: Boyuan Zhang


Need to track performance of time_counter before turning on experimental flag 
by default



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4110) LocalResourceIdTest has had a masked failure

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-4110:
-

 Summary: LocalResourceIdTest has had a masked failure
 Key: BEAM-4110
 URL: https://issues.apache.org/jira/browse/BEAM-4110
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Discovered in https://github.com/apache/beam/pull/4929, the test should be 
sickbayed while the code is made correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4109) Support arbitrary artifact names in local file artifact staging service

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4109:


 Summary: Support arbitrary artifact names in local file artifact 
staging service
 Key: BEAM-4109
 URL: https://issues.apache.org/jira/browse/BEAM-4109
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Ben Sidhom
Assignee: Kenneth Knowles


The local-file based artifact staging service implementation stores artifacts 
in a flat directory under the exact names they are given by the artifact 
staging request. This assumes that all artifact names are safe file names and 
requires staging clients to manually escape names.

Instead, the staging service should perform its own escaping/mapping 
transparently and allow clients to specify arbitrary artifact staging names.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1387

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 1.65 KB...]
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6692840655659447202.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in 

Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0)
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4725

[Apache Jenkins Server]

See 


--
GitHub pull request #5160 of commit 3e804a0b75dfefe0e0351e399a48502ae38d1379, 
no merge conflicts.
Setting status of 3e804a0b75dfefe0e0351e399a48502ae38d1379 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/4725/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Python SDK PostCommit Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5160/*:refs/remotes/origin/pr/5160/*
 > git rev-parse refs/remotes/origin/pr/5160/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5160/merge^{commit} # timeout=10
Checking out Revision 386be26a2d58b8a502cfb9022451618155ec414b 
(refs/remotes/origin/pr/5160/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 386be26a2d58b8a502cfb9022451618155ec414b
Commit message: "Merge 3e804a0b75dfefe0e0351e399a48502ae38d1379 into 
15da6412fef89dc08de59828a5df9ba11e944a79"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins8903486943828896702.sh
+ cd src
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Remove any tox cache from previous workspace
# TODO(udim): Remove this line and add '-r' to tox invocation instead.
rm -rf sdks/python/target/.tox

# INFRA does not install these packages
pip install --user --upgrade virtualenv tox
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:339: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:137: 
InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Requirement already up-to-date: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
Requirement already up-to-date: tox in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.5.3)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (0.6.0)
Requirement not upgraded as not directly required: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.11.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: 

ERROR: invocation failed (exit code 1), logfile: 

ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', 
local('
 'sdist', '--formats=zip', '--dist-dir', 
local('

[jira] [Created] (BEAM-4108) Validate generated javadocs

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4108:
--

 Summary: Validate generated javadocs
 Key: BEAM-4108
 URL: https://issues.apache.org/jira/browse/BEAM-4108
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Scott Wegner
Assignee: Scott Wegner


Generated javadocs should match that from the Maven build.

Maven command: mvn -Prelease javadoc:jar

Gradle command: ./gradlew -p sdks/java/javadoc aggregateJavadoc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #104

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
[...truncated 18.95 MB...]
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/GBKaSVForKeys as step s23
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/ParDo(ToIsmMetadataRecordForKey) as step s24
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/Flatten.PCollections as step s25
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/CreateDataflowView as step s26
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Partition 
input as step s27
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Group by 
partition as step s28
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Batch 
mutations together as step s29
Apr 17, 2018 10:31:19 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Write 
mutations to Spanner as step s30
Apr 17, 2018 10:31:19 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0417223110-b27728f7/output/results/staging/
Apr 17, 2018 10:31:19 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <80355 bytes, hash yyzpVTUli_gWV3fN-mzROQ> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0417223110-b27728f7/output/results/staging/pipeline-yyzpVTUli_gWV3fN-mzROQ.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_ERROR
Apr 17, 2018 10:31:20 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-17_15_31_19-679023727337852941?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT
Submitted job: 2018-04-17_15_31_19-679023727337852941

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_ERROR
Apr 17, 2018 10:31:20 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-04-17_15_31_19-679023727337852941
Apr 17, 2018 10:31:20 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-04-17_15_31_19-679023727337852941 with 0 
expected assertions.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-17T22:31:19.957Z: Autoscaling is enabled for job 
2018-04-17_15_31_19-679023727337852941. The number of workers will be between 1 
and 1000.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-17T22:31:19.969Z: Autoscaling was automatically enabled for 
job 2018-04-17_15_31_19-679023727337852941.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-17T22:31:23.027Z: Checking required Cloud APIs are enabled.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-17T22:31:23.307Z: Checking permissions granted to controller 
Service Account.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-17T22:31:27.461Z: Worker configuration: n1-standard-1 in 
us-central1-f.
Apr 17, 2018 10:31:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #143

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

--
[...truncated 103.18 MB...]
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (c096fa14ae2ba2c0bb76d3c264dfd82e) switched from RUNNING 
to FINISHED.
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (c096fa14ae2ba2c0bb76d3c264dfd82e).
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (c096fa14ae2ba2c0bb76d3c264dfd82e) [FINISHED]
Apr 17, 2018 10:31:13 PM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (c096fa14ae2ba2c0bb76d3c264dfd82e)
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (c096fa14ae2ba2c0bb76d3c264dfd82e) switched from RUNNING 
to FINISHED.
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/17/2018 22:31:13   
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map)(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/17/2018 22:31:13 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map)(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
Combine.perKey(TestCombineFnWithContext)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> PAssert$292/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$292/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$292/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$292/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (806291f2370887f5aafb67b9435ca1e8) switched from 
RUNNING to FINISHED.
Apr 17, 2018 10:31:13 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
Combine.perKey(TestCombineFnWithContext)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> PAssert$292/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$292/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$292/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$292/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (806291f2370887f5aafb67b9435ca1e8).
Apr 17, 

[beam-site] branch mergebot updated (2980b13 -> 78539bb)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 2980b13  This closes #419
 discard 6094291  Update Eclipse documentation for Gradle build.
 new 71bc507  Update Eclipse documentation for Gradle build.
 new 78539bb  This closes #419

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2980b13)
\
 N -- N -- N   refs/heads/mergebot (78539bb)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #419

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 78539bbcf460427b759049c52d21b1dde7de47c8
Merge: 8c4f2a9 71bc507
Author: Mergebot 
AuthorDate: Tue Apr 17 15:23:29 2018 -0700

This closes #419

 src/contribute/eclipse.md | 66 ---
 1 file changed, 39 insertions(+), 27 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Update Eclipse documentation for Gradle build.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 71bc507544fa86df2b20e77d5aa762950ba2a682
Author: Scott Wegner 
AuthorDate: Wed Apr 11 16:23:36 2018 -0700

Update Eclipse documentation for Gradle build.
---
 src/contribute/eclipse.md | 66 ---
 1 file changed, 39 insertions(+), 27 deletions(-)

diff --git a/src/contribute/eclipse.md b/src/contribute/eclipse.md
index e0cf477..346fbaa 100644
--- a/src/contribute/eclipse.md
+++ b/src/contribute/eclipse.md
@@ -17,42 +17,54 @@ section_menu: section-menu/contribute.html
 
 ## Eclipse version
 
-Use a recent Eclipse version that includes m2e. Currently we recommend Eclipse
-Neon. Start Eclipse with a fresh workspace in a separate directory from your
-checkout.
+Use a recent Eclipse version that includes
+[Buildship](https://projects.eclipse.org/projects/tools.buildship) for Gradle
+integration. Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh
+workspace in a separate directory from your checkout.
 
 ## Initial setup
 
-1. Install m2e-apt: Beam uses apt annotation processing to provide auto
-   generated code. One example is the usage of [Google
-   AutoValue](https://github.com/google/auto/tree/master/value). By default m2e
-   does not support this and you will see compile errors.
+Before setting up Eclipse integration, verify that you can successfully build
+from the commandline by building a sample package:
 
-   Help
-   -> Eclipse Marketplace
-   -> Search for "m2 apt"
-   -> Install m2e-apt 1.2 or higher
+```
+./gradlew :beam-examples-java:build
+```
 
-2. Activate the apt processing
+If you receive any errors, first verify your environment setup based on the
+[Contribution guide]({{ site.baseurl 
}}/contribute/contribution-guide#one-time-setup).
 
-   Window
-   -> Preferences
-   -> Maven
-   -> Annotation processing
-   -> Switch to Experimental: Delegate annotation processing ...
-   -> Ok
+1. If running on Mac or Linux, launch Eclipse from a terminal. This is 
necessary
+   to ensure that Eclipse has proper environment setup from user profile
+   scripts, i.e. .bashrc.
 
-3. Import the beam projects
+1. Install [Buildship Gradle
+   
Integration](https://marketplace.eclipse.org/content/buildship-gradle-integration).
+   This will allow importing and interacting with the Gradle build.
 
-   File
-   -> Import...
-   -> Existing Maven Projects
-   -> Browse to the directory you cloned into and select "beam"
-   -> make sure all beam projects are selected
-   -> Finalize
+1. Open the project import wizard in Eclipse via "File" > "Import".
 
-You now should have all the beam projects imported into Eclipse and should see
-no compile errors.
+1. From the "Import" screen, select "Gradle" > "Existing Gradle Project", and 
click
+   Next.
+
+1. From the "Import Gradle Project" screen, fill in the Project root directory
+   with your local git path, and click Finish.
+
+Eclipse will scan the project tree and import each as a separate
+Package.
+
+Verify that your workspace is correctly configured by invoking
+'beam-runners-direct-java:build' from the "Gradle Tasks" pane. The build should
+succeed with no errors.
+
+## Building
+
+After your Eclipse workspace is properly setup, you will have a "Gradle
+Tasks" window with a set of operations. If you don't see the pane, open it
+from "Window" > "Show View" > Other.. > "Gradle" > "Gradle Tasks".
+
+From the "Gradle Tasks" window, you can build any task registered with Gradle.
+For example, if you are working on Kinesis IO, select 
'beam-sdks-java-io-kinesis:build'.
 
 ## Checkstyle
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Resolved] (BEAM-3520) Update testing documentation with gradle info

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner resolved BEAM-3520.

   Resolution: Fixed
Fix Version/s: Not applicable

> Update testing documentation with gradle info
> -
>
> Key: BEAM-3520
> URL: https://issues.apache.org/jira/browse/BEAM-3520
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing, website
>Reporter: Ismaël Mejía
>Priority: Major
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3520) Update testing documentation with gradle info

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441582#comment-16441582
 ] 

Scott Wegner commented on BEAM-3520:


I believe this is fixed with 
[PR-412|https://github.com/apache/beam-site/pull/412]. Please re-open if 
necessary.

> Update testing documentation with gradle info
> -
>
> Key: BEAM-3520
> URL: https://issues.apache.org/jira/browse/BEAM-3520
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing, website
>Reporter: Ismaël Mejía
>Priority: Major
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1386

[Apache Jenkins Server]

See 


--
[...truncated 1.85 KB...]
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins2531945073984410100.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
New python executable in 

Installing setuptools, pip, wheel...done.
. sdks/python/bin/activate
# This file must be used with "source bin/activate" *from bash*
# you cannot run it directly

deactivate () {
unset -f pydoc >/dev/null 2>&1

# reset old environment variables
# ! [ -z ${VAR+_} ] returns true if VAR is declared at all
if ! [ -z "${_OLD_VIRTUAL_PATH+_}" ] ; then
PATH="$_OLD_VIRTUAL_PATH"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if ! [ -z "${_OLD_VIRTUAL_PYTHONHOME+_}" ] ; then
PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi

if ! [ -z "${_OLD_VIRTUAL_PS1+_}" ] ; then
PS1="$_OLD_VIRTUAL_PS1"
export PS1
unset _OLD_VIRTUAL_PS1
fi

unset VIRTUAL_ENV
if [ ! "${1-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}

# unset irrelevant variables
deactivate nondestructive

VIRTUAL_ENV="
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi

if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
basename "$VIRTUAL_ENV"

# Make sure to unalias pydoc if it's already there
alias pydoc 2>/dev/null >/dev/null && unalias pydoc

pydoc () {
python -m pydoc "$@"
}

# This should detect bash and zsh, which have a hash command that must
# be called to get it to forget past commands.  Without forgetting
# past commands the $PATH changes we made may not be respected
if [ -n "${BASH-}" ] || [ -n "${ZSH_VERSION-}" ] ; then
hash -r 2>/dev/null
fi
cd sdks/python
pip install -e .[gcp,test]
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0)
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Collecting 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4724

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-3942] Add Tasks for running IOITs manually using gradle

[lukasz.gajowy] [BEAM-3942] Add task for running integration tests with

[lukasz.gajowy] [BEAM-4099] Rebuild project before running Performance tests on 
Jenkins

[Pablo] Restricting hosts where Python PreCommit runs due to environmental

[Pablo] Fix in gen_protos

[aaltay] Fix flaky Python local filesystem test (#5156)

--
Started by GitHub push by aaltay
Started by GitHub push by aaltay
Started by GitHub push by aaltay
Started by GitHub push by aaltay
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 15da6412fef89dc08de59828a5df9ba11e944a79 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 15da6412fef89dc08de59828a5df9ba11e944a79
Commit message: "Merge pull request #5003: [BEAM-3942] Update performance 
testing framework to use Gradle"
 > git rev-list --no-walk 4a32353fc379803a9b9e071b9d0380666947ca6d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins8726059949985848737.sh
+ cd src
+ bash sdks/python/run_postcommit.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Remove any tox cache from previous workspace
# TODO(udim): Remove this line and add '-r' to tox invocation instead.
rm -rf sdks/python/target/.tox

# INFRA does not install these packages
pip install --user --upgrade virtualenv tox
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:339: 
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:137: 
InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Requirement already up-to-date: virtualenv in 
/home/jenkins/.local/lib/python2.7/site-packages (15.2.0)
Requirement already up-to-date: tox in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.5.3)
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (0.6.0)
Requirement not upgraded as not directly required: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox) (1.11.0)
cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed.
apache-beam 2.5.0.dev0 requires hdfs<3.0.0,>=2.1.0, which is not installed.
apache-beam 2.5.0.dev0 requires pytz>=2018.3, which is not installed.
apache-beam 2.5.0.dev0 has requirement grpcio<2,>=1.8, but you'll have grpcio 
1.4.0 which is incompatible.

# Tox runs unit tests in a virtual environment
${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
GLOB sdist-make: 

ERROR: invocation failed (exit code 1), logfile: 

ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', 
local('
 'sdist', '--formats=zip', '--dist-dir',