[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99862
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 05:40
Start Date: 09/May/18 05:40
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387625992
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99862)
Time Spent: 18h 20m  (was: 18h 10m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 20m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99858
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 05:00
Start Date: 09/May/18 05:00
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387620449
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99858)
Time Spent: 18h 10m  (was: 18h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99856
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 09/May/18 04:48
Start Date: 09/May/18 04:48
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-387613594
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99856)
Time Spent: 4h 20m  (was: 4h 10m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99855
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 09/May/18 04:48
Start Date: 09/May/18 04:48
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-387619024
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99855)
Time Spent: 4h 10m  (was: 4h)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99853
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 04:20
Start Date: 09/May/18 04:20
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387615578
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99853)
Time Spent: 18h  (was: 17h 50m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 18h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99840=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99840
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 03:40
Start Date: 09/May/18 03:40
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387610740
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99840)
Time Spent: 17h 50m  (was: 17h 40m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 50m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4904

[Apache Jenkins Server]

See 


--
[...truncated 1.27 MB...]
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: 
dfs.namenode.safemode.min.datanodes = 0
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: 
dfs.namenode.safemode.extension = 3
namenode_1  | 18/05/09 03:37:34 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.window.num.buckets = 10
namenode_1  | 18/05/09 03:37:34 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.num.users = 10
namenode_1  | 18/05/09 03:37:34 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.windows.minutes = 1,5,25
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: Retry 
cache on namenode is enabled
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: Retry 
cache will use 0.03 of total heap and retry cache entry expiry time is 60 
millis
namenode_1  | 18/05/09 03:37:34 INFO util.GSet: Computing capacity for 
map NameNodeRetryCache
namenode_1  | 18/05/09 03:37:34 INFO util.GSet: VM type   = 64-bit
namenode_1  | 18/05/09 03:37:34 INFO util.GSet: 0.02999329447746% 
max memory 958.5 MB = 294.5 KB
namenode_1  | 18/05/09 03:37:34 INFO util.GSet: capacity  = 2^15 = 
32768 entries
namenode_1  | 18/05/09 03:37:34 INFO common.Storage: Lock on 
/hadoop/dfs/name/in_use.lock acquired by nodename 143@namenode
namenode_1  | 18/05/09 03:37:34 INFO namenode.FileJournalManager: 
Recovering unfinalized segments in /hadoop/dfs/name/current
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSImage: No edit log 
streams selected.
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSImage: Planning to 
load image: 
FSImageFile(file=/hadoop/dfs/name/current/fsimage_000, 
cpktTxId=000)
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSImageFormatPBINode: 
Loading 1 INodes.
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSImageFormatProtobuf: 
Loaded FSImage in 0 seconds.
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSImage: Loaded image 
for txid 0 from /hadoop/dfs/name/current/fsimage_000
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: Need to 
save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSEditLog: Starting log 
segment at 1
datanode_1  | 18/05/09 03:37:34 INFO ipc.Client: Retrying connect to 
server: namenode/172.18.0.2:8020. Already tried 0 time(s); retry policy is 
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
namenode_1  | 18/05/09 03:37:34 INFO namenode.NameCache: initialized 
with 0 entries 0 lookups
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: Finished 
loading FSImage in 301 msecs
namenode_1  | 18/05/09 03:37:34 INFO namenode.NameNode: RPC server is 
binding to 0.0.0.0:8020
namenode_1  | 18/05/09 03:37:34 INFO ipc.CallQueueManager: Using 
callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 
scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
namenode_1  | 18/05/09 03:37:34 INFO ipc.Server: Starting Socket 
Reader #1 for port 8020
namenode_1  | 18/05/09 03:37:34 INFO namenode.FSNamesystem: Registered 
FSNamesystemState MBean
namenode_1  | 18/05/09 03:37:34 INFO namenode.LeaseManager: Number of 
blocks under construction: 0
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
initializing replication queues
namenode_1  | 18/05/09 03:37:34 INFO hdfs.StateChange: STATE* Leaving 
safe mode after 0 secs
namenode_1  | 18/05/09 03:37:34 INFO hdfs.StateChange: STATE* Network 
topology has 0 racks and 0 datanodes
namenode_1  | 18/05/09 03:37:34 INFO hdfs.StateChange: STATE* 
UnderReplicatedBlocks has 0 blocks
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
Total number of blocks= 0
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
Number of invalid blocks  = 0
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
Number of under-replicated blocks = 0
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
Number of  over-replicated blocks = 0
namenode_1  | 18/05/09 03:37:34 INFO blockmanagement.BlockManager: 
Number of blocks being written= 0
namenode_1  | 18/05/09 03:37:34 INFO hdfs.StateChange: STATE* 
Replication Queue initialization scan for invalid, over- and under-replicated 
blocks completed in 11 msec
namenode_1  | 18/05/09 03:37:34 INFO ipc.Server: IPC Server Responder: 
starting
namenode_1  | 18/05/09 03:37:34 INFO ipc.Server: IPC Server listener 
on 8020: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #142

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[schroederc] Move textio.FileSystem and its registry to a separate package

[schroederc] Move filesystem implementations under beam/io/filesystem package

[wcn] Fix missing return in datastore connector.

[swegner] Create ValidatesRunner task for DirectRunner.

[swegner] Convert some tests from ValidatesRunner -> NeedsRunner

[apilloud] [SQL] Cleanout unneeded sqlEnv

[apilloud] [SQL] Hide details of BeamSqlEnv from rel test

[apilloud] [SQL] Simplify TableProvider interface

[apilloud] [SQL] Return map from TableProvider.getTables

[apilloud] [SQL] Add tables via TableStore in CalciteSchema

[apilloud] [SQL] Move Create and Drop Table inline with ddl

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[tgroh] Suppress Thread Leak Test

[tgroh] Rename ExecutorServiceParlallelExecutorTest#test

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 2.13 KB...]
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# ensure gcloud is version 186 or above
TMPDIR=$(mktemp -d)
mktemp -d
gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
gcloud -v | head -1 | awk '{print $4}'
if [[ "$gcloud_ver" < "186" ]]
then
  pushd $TMPDIR
  curl 
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
 --output gcloud.tar.gz
  tar xf gcloud.tar.gz
  ./google-cloud-sdk/install.sh --quiet
  . ./google-cloud-sdk/path.bash.inc
  popd
  gcloud components update --quiet || echo 'gcloud components update failed'
  gcloud -v
fi

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container

> Configure project :beam-sdks-go
Applying build_rules.gradle to beam-sdks-go
applyGoNature with default configuration for project beam-sdks-go

> Task :beam-sdks-python-container:prepare
Use project GOPATH: 


> Task :beam-sdks-go:prepare
Use project GOPATH: 


> Task :beam-sdks-python-container:dockerClean UP-TO-DATE

> Task :beam-sdks-python:setupVirtualenv
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
Collecting tox==3.0.0
  Using cached 
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
  Using cached 
https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
Collecting py>=1.4.17 (from tox==3.0.0)
  Using 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #221

[Apache Jenkins Server]

See 


Changes:

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 10.41 MB...]
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 09, 2018 3:15:30 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 09, 2018 3:15:30 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0509031524-f7f5f916/output/results/staging/
May 09, 2018 3:15:30 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71843 bytes, hash fkRBvgCDM-axZC6itONlKg> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0509031524-f7f5f916/output/results/staging/pipeline-fkRBvgCDM-axZC6itONlKg.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 09, 2018 3:15:32 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-08_20_15_31-11424088814943593274?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-08_20_15_31-11424088814943593274

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 09, 2018 3:15:32 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-08_20_15_31-11424088814943593274
May 09, 2018 3:15:32 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-08_20_15_31-11424088814943593274 with 1 
expected assertions.
May 09, 2018 3:15:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T03:15:31.267Z: Autoscaling is enabled for job 
2018-05-08_20_15_31-11424088814943593274. The number of workers will be between 
1 and 1000.
May 09, 2018 3:15:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T03:15:31.301Z: Autoscaling was automatically enabled for 
job 2018-05-08_20_15_31-11424088814943593274.
May 09, 2018 3:15:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T03:15:33.977Z: Checking required Cloud APIs are enabled.
May 09, 2018 3:15:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99827
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 03:00
Start Date: 09/May/18 03:00
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387605367
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99827)
Time Spent: 17h 40m  (was: 17.5h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 40m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1565

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 60f90c8dcb229c35a82c7be15e64a89678bae058 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 60f90c8dcb229c35a82c7be15e64a89678bae058
Commit message: "Make experiments as set attr of RuntimeValueProvider"
 > git rev-list --no-walk 60f90c8dcb229c35a82c7be15e64a89678bae058 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins498393289354643987.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
[Set GitHub commit status (universal)] ERROR on repos 
[GHRepository@bc39576[description=Apache 
Beam,homepage=,name=beam,fork=false,size=58754,milestones={},language=Java,commits={},source=,parent=,responseHeaderFields={null=[HTTP/1.1
 200 OK], Access-Control-Allow-Origin=[*], Access-Control-Expose-Headers=[ETag, 
Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, 
X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval], 
Cache-Control=[private, max-age=60, s-maxage=60], Content-Encoding=[gzip], 
Content-Security-Policy=[default-src 'none'], Content-Type=[application/json; 
charset=utf-8], Date=[Wed, 09 May 2018 03:00:11 GMT], 
ETag=[W/"f271032ca1513bfd5e28400c832a5d21"], Last-Modified=[Wed, 09 May 2018 
00:26:13 GMT], OkHttp-Received-Millis=[1525834811571], 
OkHttp-Response-Source=[NETWORK 200], OkHttp-Selected-Protocol=[http/1.1], 
OkHttp-Sent-Millis=[1525834811417], Referrer-Policy=[origin-when-cross-origin, 
strict-origin-when-cross-origin], Server=[GitHub.com], Status=[200 OK], 
Strict-Transport-Security=[max-age=31536000; includeSubdomains; preload], 
Transfer-Encoding=[chunked], Vary=[Accept, Authorization, Cookie, 
X-GitHub-OTP], X-Accepted-OAuth-Scopes=[repo], 
X-Content-Type-Options=[nosniff], X-Frame-Options=[deny], 
X-GitHub-Media-Type=[github.v3; format=json], 
X-GitHub-Request-Id=[C80A:4F3F:613F65:E692C6:5AF26434], 
X-OAuth-Scopes=[admin:repo_hook, repo, repo:status], X-RateLimit-Limit=[5000], 
X-RateLimit-Remaining=[4985], X-RateLimit-Reset=[1525836570], 
X-Runtime-rack=[0.072147], X-XSS-Protection=[1; 
mode=block]},url=https://api.github.com/repos/apache/beam,id=50904245]] 
(sha:60f90c8) with context:beam_PostCommit_Python_ValidatesRunner_Dataflow
Setting commit status on GitHub for 
https://github.com/apache/beam/commit/60f90c8dcb229c35a82c7be15e64a89678bae058
ERROR: Build step failed with exception
java.io.FileNotFoundException: 
https://api.github.com/repos/apache/beam/statuses/60f90c8dcb229c35a82c7be15e64a89678bae058
at 
com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:243)
at 
com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
at 
com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
at org.kohsuke.github.Requester.parse(Requester.java:612)
at org.kohsuke.github.Requester.parse(Requester.java:594)
at org.kohsuke.github.Requester._to(Requester.java:272)
Caused: org.kohsuke.github.GHFileNotFoundException: {"message":"Not 
Found","documentation_url":"https://developer.github.com/v3/repos/statuses/#create-a-status"}
at 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99815
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 02:20
Start Date: 09/May/18 02:20
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387599748
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99815)
Time Spent: 17.5h  (was: 17h 20m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99813
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 01:40
Start Date: 09/May/18 01:40
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387593545
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99813)
Time Spent: 17h 20m  (was: 17h 10m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4903

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 1.28 MB...]
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSNamesystem: 
dfs.namenode.safemode.extension = 3
namenode_1  | 18/05/09 01:06:29 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.window.num.buckets = 10
namenode_1  | 18/05/09 01:06:29 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.num.users = 10
namenode_1  | 18/05/09 01:06:29 INFO metrics.TopMetrics: NNTop conf: 
dfs.namenode.top.windows.minutes = 1,5,25
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSNamesystem: Retry 
cache on namenode is enabled
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSNamesystem: Retry 
cache will use 0.03 of total heap and retry cache entry expiry time is 60 
millis
namenode_1  | 18/05/09 01:06:29 INFO util.GSet: Computing capacity for 
map NameNodeRetryCache
namenode_1  | 18/05/09 01:06:29 INFO util.GSet: VM type   = 64-bit
namenode_1  | 18/05/09 01:06:29 INFO util.GSet: 0.02999329447746% 
max memory 958.5 MB = 294.5 KB
namenode_1  | 18/05/09 01:06:29 INFO util.GSet: capacity  = 2^15 = 
32768 entries
namenode_1  | 18/05/09 01:06:29 INFO common.Storage: Lock on 
/hadoop/dfs/name/in_use.lock acquired by nodename 143@namenode
test_1  | Waiting for safe mode to end.
namenode_1  | 18/05/09 01:06:29 INFO namenode.FileJournalManager: 
Recovering unfinalized segments in /hadoop/dfs/name/current
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSImage: No edit log 
streams selected.
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSImage: Planning to 
load image: 
FSImageFile(file=/hadoop/dfs/name/current/fsimage_000, 
cpktTxId=000)
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSImageFormatPBINode: 
Loading 1 INodes.
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSImageFormatProtobuf: 
Loaded FSImage in 0 seconds.
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSImage: Loaded image 
for txid 0 from /hadoop/dfs/name/current/fsimage_000
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSNamesystem: Need to 
save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
namenode_1  | 18/05/09 01:06:29 INFO namenode.FSEditLog: Starting log 
segment at 1
namenode_1  | 18/05/09 01:06:30 INFO namenode.NameCache: initialized 
with 0 entries 0 lookups
namenode_1  | 18/05/09 01:06:30 INFO namenode.FSNamesystem: Finished 
loading FSImage in 265 msecs
namenode_1  | 18/05/09 01:06:30 INFO namenode.NameNode: RPC server is 
binding to 0.0.0.0:8020
namenode_1  | 18/05/09 01:06:30 INFO ipc.CallQueueManager: Using 
callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 
scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
namenode_1  | 18/05/09 01:06:30 INFO ipc.Server: Starting Socket 
Reader #1 for port 8020
namenode_1  | 18/05/09 01:06:30 INFO namenode.FSNamesystem: Registered 
FSNamesystemState MBean
namenode_1  | 18/05/09 01:06:30 INFO namenode.LeaseManager: Number of 
blocks under construction: 0
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
initializing replication queues
namenode_1  | 18/05/09 01:06:30 INFO hdfs.StateChange: STATE* Leaving 
safe mode after 0 secs
namenode_1  | 18/05/09 01:06:30 INFO hdfs.StateChange: STATE* Network 
topology has 0 racks and 0 datanodes
namenode_1  | 18/05/09 01:06:30 INFO hdfs.StateChange: STATE* 
UnderReplicatedBlocks has 0 blocks
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
Total number of blocks= 0
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
Number of invalid blocks  = 0
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
Number of under-replicated blocks = 0
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
Number of  over-replicated blocks = 0
namenode_1  | 18/05/09 01:06:30 INFO blockmanagement.BlockManager: 
Number of blocks being written= 0
namenode_1  | 18/05/09 01:06:30 INFO hdfs.StateChange: STATE* 
Replication Queue initialization scan for invalid, over- and under-replicated 
blocks completed in 8 msec
namenode_1  | 18/05/09 01:06:30 INFO ipc.Server: IPC Server Responder: 
starting
namenode_1  | 18/05/09 01:06:30 INFO ipc.Server: IPC Server listener 
on 8020: starting
namenode_1  | 18/05/09 01:06:30 INFO namenode.NameNode: NameNode RPC 
up at: namenode/172.18.0.2:8020
namenode_1  | 18/05/09 01:06:30 INFO namenode.FSNamesystem: Starting 
services 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #220

[Apache Jenkins Server]

See 


Changes:

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[apilloud] Enable githubCommitNotifier for post commits

--
[...truncated 10.43 MB...]
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 09, 2018 12:59:03 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 09, 2018 12:59:03 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0509005858-24ad89f/output/results/staging/
May 09, 2018 12:59:04 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71843 bytes, hash 7a5i5-wbZklIqIHX39HCNw> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0509005858-24ad89f/output/results/staging/pipeline-7a5i5-wbZklIqIHX39HCNw.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 09, 2018 12:59:05 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-08_17_59_04-614006630417702889?project=apache-beam-testing

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Submitted job: 2018-05-08_17_59_04-614006630417702889

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 09, 2018 12:59:05 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-08_17_59_04-614006630417702889
May 09, 2018 12:59:05 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-08_17_59_04-614006630417702889 with 1 
expected assertions.
May 09, 2018 12:59:21 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T00:59:04.512Z: Autoscaling is enabled for job 
2018-05-08_17_59_04-614006630417702889. The number of workers will be between 1 
and 1000.
May 09, 2018 12:59:21 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T00:59:04.545Z: Autoscaling was automatically enabled for 
job 2018-05-08_17_59_04-614006630417702889.
May 09, 2018 12:59:21 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-09T00:59:07.253Z: Checking required Cloud APIs are enabled.
May 09, 2018 12:59:21 AM 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #318

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 18.85 MB...]
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 09, 2018 1:01:45 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444, For 
input string: "xxx"
May 09, 2018 1:01:45 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE ERROR,2015-11-19 09:53:53.444, 2

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_OUT
GOT user13_BisqueBilby,BisqueBilby,xxx,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_ERROR
May 09, 2018 1:01:45 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user13_BisqueBilby,BisqueBilby,xxx,144795563,2015-11-19 09:53:53.444, For 
input string: "xxx"

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_OUT
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_ERROR
May 09, 2018 1:01:45 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444, For 
input string: "xxx"

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_OUT
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_ERROR
May 09, 2018 1:01:45 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE ERROR,2015-11-19 09:53:53.444, 2

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoreSums 
STANDARD_OUT
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99812
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 09/May/18 01:00
Start Date: 09/May/18 01:00
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387587741
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99812)
Time Spent: 17h 10m  (was: 17h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1564

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 91.49 KB...]
root: DEBUG: PValue computed by ClassifyNumbers/FlatMap(some_fn) (tag odd): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/WindowInto(WindowIntoFn) (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/Create/Read (tag None): refcount: 1 
=> 0
root: DEBUG: PValue computed by assert:odd/ToVoidKey (tag None): refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/Group/pair_with_1 (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/Group/pair_with_0 (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/Group/Flatten (tag None): refcount: 
1 => 0
root: DEBUG: PValue computed by assert:odd/Group/GroupByKey (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by 
assert:odd/Group/Map(_merge_tagged_vals_under_key) (tag None): refcount: 1 => 0
root: DEBUG: PValue computed by assert:odd/Unkey (tag None): refcount: 1 => 0
root: DEBUG: PValue computed by ClassifyNumbers/FlatMap(some_fn) (tag even): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:even/WindowInto(WindowIntoFn) (tag 
None): refcount: 1 => 0
root: DEBUG: PValue computed by assert:even/Create/Read (tag None): refcount: 1 
=> 0
root: DEBUG: PValue computed by assert:even/ToVoidKey (tag None): refcount: 1 
=> 0
root: DEBUG: PValue computed by assert:even/Group/pair_with_0 (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:even/Group/pair_with_1 (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by assert:even/Group/Flatten (tag None): refcount: 
1 => 0
root: DEBUG: PValue computed by assert:even/Group/GroupByKey (tag None): 
refcount: 1 => 0
root: DEBUG: PValue computed by 
assert:even/Group/Map(_merge_tagged_vals_under_key) (tag None): refcount: 1 => 0
root: DEBUG: PValue computed by assert:even/Unkey (tag None): refcount: 1 => 0
root: DEBUG: Connecting using Google Application Default Credentials.
root: INFO: Starting GCS upload to 
gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0509003223-738898.1525825943.739107/pipeline.pb...
oauth2client.transport: INFO: Attempting refresh to obtain initial access_token
root: INFO: Completed GCS upload to 
gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0509003223-738898.1525825943.739107/pipeline.pb
root: INFO: Starting GCS upload to 
gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0509003223-738898.1525825943.739107/requirements.txt...
root: INFO: Completed GCS upload to 
gs://temp-storage-for-end-to-end-tests/staging-validatesrunner-test/beamapp-jenkins-0509003223-738898.1525825943.739107/requirements.txt
root: INFO: Executing command: 
['
 '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 
'postcommit_requirements.txt', '--no-binary', ':all:']
- >> end captured logging << -

==
ERROR: test_flatten_multiple_pcollections_having_multiple_consumers 
(apache_beam.transforms.ptransform_test.PTransformTest)
--
Traceback (most recent call last):
  File 
"
 line 558, in test_flatten_multiple_pcollections_having_multiple_consumers
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 389, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 402, in run
return self.runner.run_pipeline(self)
  File 
"
 line 48, in run_pipeline
self.result = super(TestDataflowRunner, self).run_pipeline(pipeline)
  File 
"
 line 363, in run_pipeline
self.dataflow_client.create_job(self.job), self)
  File 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #324

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 12.92 MB...]
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-19 terminating.
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-20 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-28
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-28 msg: [container-28] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-28 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-14
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-14 msg: [container-14] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-14 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-37
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-37 msg: [container-37] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-37 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-49
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-49 msg: [container-49] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-49 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-64
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-64 msg: [container-64] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-64 terminating.
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-69
May 09, 2018 12:47:56 AM com.datatorrent.stram.StreamingContainerManager 
processHeartbeat
INFO: requesting idle shutdown for container container-0
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM com.datatorrent.stram.engine.StreamingContainer 
processHeartbeatResponse
INFO: Received shutdown request type ABORT
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-69 msg: [container-69] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-0 msg: [container-0] Exiting heartbeat loop..
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-69 terminating.
May 09, 2018 12:47:56 AM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Container container-0 terminating.
May 09, 2018 12:47:56 AM 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #241

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 2.16 KB...]
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins4494020542002150853.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins18370395342691.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (39.1.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins139673392047723921.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins8499397707444235060.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (39.1.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 

Build failed in Jenkins: beam_PerformanceTests_JDBC #549

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 4.67 KB...]
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (39.1.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.2.2)
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/69/bc/230987c0dc22c763529330b2e669dbdba374d6a10c1f61232274184731be/ntlm_auth-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #145

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 567.27 KB...]
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):
 Cannot create 
file/TEXTIO_IT__1525825874817/.temp-beam-2018-05-09_00-31-15-0/6e30477e-14c9-45d4-aed8-b9858e287ca9.
 Name node is in safe mode.
The reported blocks 0 needs additional 31 blocks to reach the threshold 
0.9990 of total blocks 31.
The number of live datanodes 1 has reached the minimum number 0. Safe mode 
will be turned off automatically once the thresholds have been reached.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2447)
   

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #110

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 221.06 KB...]
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/05/09 00:36:49 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/05/09 00:36:49 Pipeline:
2018/05/09 00:36:49 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/05/09 00:36:49 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/05/09 00:36:49 Pipeline:
2018/05/09 00:36:49 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/05/09 00:36:49 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/05/09 00:36:49 wait[5] unblocked w/ 1 [false]
2018/05/09 00:36:49 wait[5] done
2018/05/09 00:36:49 Pipeline:
2018/05/09 00:36:49 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/05/09 00:36:49 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #470

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 51.03 KB...]
> Task :beam-model-pipeline:jar
> Task :beam-model-job-management:generateProto

> Task :beam-sdks-java-extensions-sql:compileJavacc
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Warning: Bad option "-grammar_encoding=UTF-8" will be ignored.
Reading from file 

 . . .
Note: UNICODE_INPUT option is specified. Please make sure you create the 
parser/lexer using a Reader with the correct character encoding.
Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD 
is more than 1.  Set option FORCE_LA_CHECK to true to force checking.
File "TokenMgrError.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.

> Task :beam-sdks-java-extensions-sql:processResources
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2
Collecting tox==3.0.0
  Using cached 
https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
  Using cached 
https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
Collecting py>=1.4.17 (from tox==3.0.0)
  Using cached 
https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in 
/usr/local/lib/python2.7/dist-packages (from 
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: pluggy, py, tox, protobuf, futures, grpcio, 
grpcio-tools
Could not install packages due to an EnvironmentError: [Errno 13] Permission 
denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.6.0.dist-info'
Consider using the `--user` option or check the permissions.


> Task :beam-model-job-management:shadowJar
> Task :beam-model-job-management:jar
> Task :beam-model-fn-execution:shadowJar
> Task 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #238

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 55.71 KB...]

> Task :beam-model-job-management:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:packageTests
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes
> Task :beam-sdks-java-extensions-sql:processResources
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2
Requirement already up-to-date: tox==3.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0)
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in 
/usr/local/lib/python2.7/dist-packages (from 
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: protobuf, futures, grpcio, grpcio-tools

> Task :beam-model-job-management:shadowJar
> Task :beam-model-job-management:jar
> Task :beam-model-job-management:extractIncludeTestProto
> Task :beam-model-job-management:generateTestProto NO-SOURCE
> Task :beam-model-job-management:compileTestJava NO-SOURCE
> Task :beam-model-job-management:testClasses UP-TO-DATE
> Task :beam-model-job-management:packageTests
> Task :beam-model-job-management:install
> Task :beam-model-fn-execution:shadowJar
> Task :beam-model-fn-execution:jar
> Task :beam-model-fn-execution:extractIncludeTestProto
> Task :beam-model-fn-execution:generateTestProto NO-SOURCE
> Task :beam-model-fn-execution:compileTestJava NO-SOURCE
> Task :beam-model-fn-execution:testClasses UP-TO-DATE
> Task :beam-model-fn-execution:packageTests
> Task :beam-model-fn-execution:install

> Task :beam-sdks-python:setupVirtualenv FAILED
Could not install packages due to an EnvironmentError: [Errno 13] Permission 
denied: 
'/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth'
Consider using the `--user` option or check the permissions.


> Task :beam-model-fn-execution:shadowTestJar
> Task :beam-sdks-java-core:shadowJar

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 43s
141 actionable tasks: 134 executed, 5 from cache, 2 up-to-date

Publishing build scan...
https://gradle.com/s/grt5qvipqhqyg


STDERR: 
FAILURE: Build failed with an exception.

* Where:
Build file 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #147

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 458.53 KB...]
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.225.24.202:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
Workflow failed. Causes: S01:Generate 
sequence/Read(BoundedCountingSource)+Produce documents/Map+Write documents to 
MongoDB/ParDo(Write) failed., A work item was attempted 4 times without 
success. Each time the worker eventually lost contact with the service. The 
work item was attempted on: 
  mongodbioit0testwriteandr-05081726-q0hn-harness-f3rw,
  mongodbioit0testwriteandr-05081726-q0hn-harness-f3rw,
  mongodbioit0testwriteandr-05081726-q0hn-harness-f3rw,
  mongodbioit0testwriteandr-05081726-q0hn-harness-f3rw
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:134)
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
at 
org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.sdk.io.mongodb.MongoDBIOIT.testWriteAndRead(MongoDBIOIT.java:118)

1 test completed, 1 failed
Finished generating test XML results (0.456 secs) into: 

Generating HTML test report...
Finished generating test html results (0.045 secs) into: 

:beam-sdks-java-io-mongodb:integrationTest (Thread[main,5,main]) completed. 
Took 9 mins 27.012 secs.

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 11m 36s
55 actionable tasks: 32 executed, 23 up-to-date

Publishing build scan...
https://gradle.com/s/qjl6lwhkeiv5e


STDERR: 
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':beam-sdks-java-io-mongodb:integrationTest'.
> There were failing tests. See the report at: 
> file://

* Try:
Run with --debug option to get more log output. Run with --scan to get 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #361

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 47.57 MB...]
May 09, 2018 12:35:09 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task Distinct/DropValues -> 
Distinct/ExtractFirstKey/ParMultiDo(Anonymous) -> 
UniqueCount/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) -> ToKeyedWorkItem 
(3629d75f9e842717f8ccf4ff580d5c10)
May 09, 2018 12:35:09 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: Distinct/DropValues -> Distinct/ExtractFirstKey/ParMultiDo(Anonymous) 
-> UniqueCount/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) -> ToKeyedWorkItem 
(1/1) (3629d75f9e842717f8ccf4ff580d5c10) switched from RUNNING to FINISHED.
May 09, 2018 12:35:09 AM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 05/09/2018 00:35:09   Distinct/DropValues -> 
Distinct/ExtractFirstKey/ParMultiDo(Anonymous) -> 
UniqueCount/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) -> ToKeyedWorkItem(1/1) 
switched to FINISHED 

org.apache.beam.sdk.io.CountingSourceTest > testBoundedSourceSplits STANDARD_OUT
05/09/2018 00:35:09 Distinct/DropValues -> 
Distinct/ExtractFirstKey/ParMultiDo(Anonymous) -> 
UniqueCount/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) -> ToKeyedWorkItem(1/1) 
switched to FINISHED 

org.apache.beam.sdk.io.CountingSourceTest > testBoundedSourceSplits 
STANDARD_ERROR
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: UniqueCount/Combine.perKey(Count) -> 
UniqueCount/Values/Values/Map/ParMultiDo(Anonymous) -> 
UniqueCount/View.AsIterable/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem (1/1) (541845db0e2b1591eb1e03a6eaa67522) switched from 
RUNNING to FINISHED.
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for UniqueCount/Combine.perKey(Count) -> 
UniqueCount/Values/Values/Map/ParMultiDo(Anonymous) -> 
UniqueCount/View.AsIterable/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem (1/1) (541845db0e2b1591eb1e03a6eaa67522).
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
UniqueCount/Combine.perKey(Count) -> 
UniqueCount/Values/Values/Map/ParMultiDo(Anonymous) -> 
UniqueCount/View.AsIterable/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem (1/1) (541845db0e2b1591eb1e03a6eaa67522) [FINISHED]
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> Map (1/1) (ddc8337f8cc4038150cbe2eced8a9a73) switched from RUNNING to 
FINISHED.
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> Map (1/1) (ddc8337f8cc4038150cbe2eced8a9a73).
May 09, 2018 12:35:09 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
UniqueCount/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> Map (1/1) (ddc8337f8cc4038150cbe2eced8a9a73) [FINISHED]
May 09, 2018 12:35:09 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task UniqueCount/Combine.perKey(Count) -> 
UniqueCount/Values/Values/Map/ParMultiDo(Anonymous) -> 
UniqueCount/View.AsIterable/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #340

[Apache Jenkins Server]

See 


Changes:

[Pablo] Make experiments as set attr of RuntimeValueProvider

--
[...truncated 588.80 KB...]
Starting process 'Gradle Test Executor 262'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 262'
Successfully started process 'Gradle Test Executor 262'

Gradle Test Executor 261 started executing tests.
Gradle Test Executor 261 finished executing tests.
Gradle Test Executor 262 started executing tests.
Gradle Test Executor 262 finished executing tests.
Gradle Test Executor 263 started executing tests.
Gradle Test Executor 263 finished executing tests.
Gradle Test Executor 264 started executing tests.
Gradle Test Executor 264 finished executing tests.
Gradle Test Executor 265 started executing tests.
Gradle Test Executor 265 finished executing tests.
Gradle Test Executor 266 started executing tests.
Gradle Test Executor 267 started executing tests.
Gradle Test Executor 267 finished executing tests.

> Task :beam-runners-spark:validatesRunnerBatch
Starting process 'Gradle Test Executor 263'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 263'
Successfully started process 'Gradle Test Executor 263'
Starting process 'Gradle Test Executor 264'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 264'
Successfully started process 'Gradle Test Executor 264'
Starting process 'Gradle Test Executor 265'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 265'
Successfully started process 'Gradle Test Executor 265'
Starting process 'Gradle Test Executor 266'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #152

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 54.24 KB...]
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE
> Task :beam-model-pipeline:jar
> Task :beam-model-job-management:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:packageTests
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2
Requirement already up-to-date: tox==3.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Collecting grpcio-tools==1.3.5
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0)
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in 
/usr/local/lib/python2.7/dist-packages (from 
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: protobuf, futures, grpcio, grpcio-tools
Could not install packages due to an EnvironmentError: [Errno 13] Permission 
denied: 
'/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth'
Consider using the `--user` option or check the permissions.


> Task :beam-model-job-management:shadowJar
> Task :beam-model-job-management:jar
> Task :beam-model-job-management:extractIncludeTestProto
> Task :beam-model-job-management:generateTestProto NO-SOURCE
> Task :beam-model-job-management:compileTestJava NO-SOURCE
> Task :beam-model-job-management:testClasses UP-TO-DATE
> Task :beam-model-job-management:packageTests
> Task :beam-model-job-management:install
> Task :beam-model-fn-execution:shadowJar
> Task :beam-model-fn-execution:jar
> Task :beam-model-fn-execution:extractIncludeTestProto
> Task :beam-model-fn-execution:generateTestProto NO-SOURCE
> Task :beam-model-fn-execution:compileTestJava NO-SOURCE
> Task :beam-model-fn-execution:testClasses UP-TO-DATE
> Task :beam-model-fn-execution:packageTests
> Task :beam-sdks-python:setupVirtualenv FAILED
> Task :beam-model-fn-execution:install
> Task :beam-sdks-java-core:shadowJar

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 41s
140 actionable tasks: 133 executed, 5 from cache, 2 up-to-date

Publishing build scan...
https://gradle.com/s/gawwn3eb6iiqi


STDERR: 
FAILURE: Build failed with an exception.

* Where:
Build file 
'
 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #488

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 50.35 KB...]
> Task :beam-sdks-java-build-tools:packageTests
> Task :beam-runners-apex:processResources NO-SOURCE
> Task :beam-runners-apex:processTestResources
> Task :beam-model-pipeline:compileJava FROM-CACHE
> Task :beam-model-pipeline:processResources
> Task :beam-model-pipeline:classes
> Task :beam-sdks-java-extensions-sql:generateFmppSources
> Task :beam-sdks-java-build-tools:install

> Task :beam-sdks-python:setupVirtualenv
done.
Running virtualenv with interpreter /usr/bin/python2
Requirement already up-to-date: tox==3.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (3.0.0)
Collecting grpcio-tools==1.3.5

> Task :beam-model-pipeline:shadowJar
> Task :beam-sdks-java-core:compileJava FROM-CACHE
> Task :beam-sdks-java-core:processResources
> Task :beam-sdks-java-core:classes

> Task :beam-sdks-java-extensions-sql:compileJavacc
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Warning: Bad option "-grammar_encoding=UTF-8" will be ignored.
Reading from file 

 . . .
Note: UNICODE_INPUT option is specified. Please make sure you create the 
parser/lexer using a Reader with the correct character encoding.
Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD 
is more than 1.  Set option FORCE_LA_CHECK to true to force checking.
File "TokenMgrError.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.

> Task :beam-model-job-management:extractIncludeProto
> Task :beam-model-fn-execution:extractIncludeProto
> Task :beam-model-pipeline:jar
> Task :beam-model-job-management:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes
> Task :beam-sdks-java-extensions-sql:processResources
> Task :beam-sdks-java-extensions-sql:processTestResources NO-SOURCE

> Task :beam-sdks-python:setupVirtualenv
  Using cached 
https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: pluggy<1.0,>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (0.6.0)
Requirement not upgraded as not directly required: six in 
/usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in 
/usr/local/lib/python2.7/dist-packages (from 
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4902

[Apache Jenkins Server]

See 


--
[...truncated 1.07 MB...]
test_type_check_valid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_duplicates_ignored 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param_cant_be_object_instance 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_nested_unions_flattened 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_nested_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_composite_type_in_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_not_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_repr (apache_beam.typehints.typehints_test.UnionHintTestCase) 
... ok
test_deprecated_with_since_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_with_since_current_message 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_since_should_fail 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_with_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_with_current_message 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_experimental_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
Tests that the filter 'once' is sufficient to print once per ... ok
test_equal_objects (apache_beam.utils.counters_test.CounterNameTest) ... ok
test_hash_two_objects (apache_beam.utils.counters_test.CounterNameTest) ... ok
test_method_forwarding_not_windows (apache_beam.utils.processes_test.Exec) ... 
ok
test_method_forwarding_windows (apache_beam.utils.processes_test.Exec) ... ok
test_call_two_objects (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_single_failure (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_two_failures (apache_beam.utils.retry_test.RetryStateTest) ... ok
test_log_calls_for_permanent_failure (apache_beam.utils.retry_test.RetryTest) 
... ok
test_log_calls_for_transient_failure (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_default_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_explicit_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_initial_delay (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_explicit_number_of_retries (apache_beam.utils.retry_test.RetryTest) 
... ok
test_with_http_error_that_should_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... ok
test_with_http_error_that_should_not_be_retried 
(apache_beam.utils.retry_test.RetryTest) ... ok
test_with_no_retry_decorator (apache_beam.utils.retry_test.RetryTest) ... ok
test_with_real_clock (apache_beam.utils.retry_test.RetryTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_of (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_precision (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_sort_order (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_str (apache_beam.utils.timestamp_test.DurationTest) ... ok
test_arithmetic (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_from_rfc3339 (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_from_rfc3339_failure (apache_beam.utils.timestamp_test.TimestampTest) ... 
ok
test_from_utc_datetime (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_of (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_precision (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_sort_order (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_str (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_utc_timestamp (apache_beam.utils.timestamp_test.TimestampTest) ... ok
test_equality (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_hash (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_pickle (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_timestamps (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_with_value (apache_beam.utils.windowed_value_test.WindowedValueTest) ... ok
test_no_partial_writeouts (apache_beam.pipeline_test.DirectRunnerRetryTests) 
... ok
test_retry_fork_graph (apache_beam.pipeline_test.DirectRunnerRetryTests) ... ok
test_element 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #146

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1245

[Apache Jenkins Server]

See 


Changes:

[github] Explictly delcare globals defined elsewhere

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

[github] Adding a microbenchmark for side input iterables. (#5294)

[apilloud] Enable githubCommitNotifier for post commits

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 5.12 KB...]
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.2.2)
Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/69/bc/230987c0dc22c763529330b2e669dbdba374d6a10c1f61232274184731be/ntlm_auth-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.22)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != 
"PyPy" in /usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.1.6)
Requirement already satisfied: asn1crypto>=0.21.0 in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (0.24.0)
Requirement already satisfied: ipaddress; python_version < "3" in 
/usr/local/lib/python2.7/dist-packages (from 
cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (1.0.22)
Requirement already satisfied: pycparser in 
/usr/local/lib/python2.7/dist-packages (from cffi>=1.7; 
platform_python_implementation != 
"PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18)
Installing collected packages: absl-py, colorama, colorlog, blinker, futures, 
pint, numpy, contextlib2, ntlm-auth, requests-ntlm, xmltodict, pywinrm
Successfully installed absl-py-0.2.0 blinker-1.4 colorama-0.3.9 colorlog-2.6.0 
contextlib2-0.5.5 futures-3.2.0 ntlm-auth-1.1.0 numpy-1.13.3 pint-0.8.1 
pywinrm-0.3.0 requests-ntlm-1.1.0 xmltodict-0.11.0
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1693927279099916115.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0)
Requirement already satisfied: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #141

[Apache Jenkins Server]

See 


--
[...truncated 38.81 MB...]
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(236): return optparse.Values(defaults)
 --- modulename: optparse, funcname: __init__
optparse.py(839): if defaults:
optparse.py(840): for (attr, val) in defaults.items():
optparse.py(841): 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #140

[Apache Jenkins Server]

See 


--
[...truncated 38.80 MB...]
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(236): return optparse.Values(defaults)
 --- modulename: optparse, funcname: __init__
optparse.py(839): if defaults:
optparse.py(840): for (attr, val) in defaults.items():
optparse.py(841): 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #139

[Apache Jenkins Server]

See 


--
[...truncated 38.81 MB...]
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(234): opt_str = option.get_opt_string()
 --- modulename: optparse, funcname: get_opt_string
optparse.py(757): if self._long_opts:
optparse.py(758): return self._long_opts[0]
baseparser.py(235): defaults[option.dest] = 
option.check_value(opt_str, default)
 --- modulename: optparse, funcname: check_value
optparse.py(766): checker = self.TYPE_CHECKER.get(self.type)
optparse.py(767): if checker is None:
optparse.py(768): return value
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(232): default = defaults.get(option.dest)
baseparser.py(233): if isinstance(default, string_types):
baseparser.py(231): for option in self._get_all_options():
baseparser.py(236): return optparse.Values(defaults)
 --- modulename: optparse, funcname: __init__
optparse.py(839): if defaults:
optparse.py(840): for (attr, val) in defaults.items():
optparse.py(841): 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #317

[Apache Jenkins Server]

See 


Changes:

[tgroh] Migrate the `portable` subpackage to Portability

--
[...truncated 18.12 MB...]
INFO: Splitting filepattern 
/tmp/junit2854183446423719364/junit2795864561835661876.tmp into bundles of size 
3 took 0 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 08, 2018 11:59:58 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo639327768974148.sh 
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo639327768974148.sh to 
/tmp/test-Echoo5959221925973039894/test-Echo639327768974148.sh 
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain2771887361065531626.sh 
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 08, 2018 11:59:59 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain2771887361065531626.sh to 
/tmp/test-Echoo5959221925973039894/test-EchoAgain2771887361065531626.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_ERROR
May 09, 2018 12:00:00 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on 
user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444, For 
input string: "xxx"
May 09, 2018 12:00:01 AM 
org.apache.beam.examples.complete.game.UserScore$ParseEventFn processElement
INFO: Parse error on THIS IS A PARSE ERROR,2015-11-19 09:53:53.444, 2

org.apache.beam.examples.complete.game.UserScoreTest > testUserScoresBadInput 
STANDARD_OUT
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT user6_AliceBlueDingo,AliceBlueDingo,4,xxx,2015-11-19 09:53:53.444
GOT 

[jira] [Work logged] (BEAM-3061) BigtableIO should support emitting a sentinel "done" value when a bundle completes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3061?focusedWorklogId=99805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99805
 ]

ASF GitHub Bot logged work on BEAM-3061:


Author: ASF GitHub Bot
Created on: 08/May/18 23:58
Start Date: 08/May/18 23:58
Worklog Time Spent: 10m 
  Work Description: steveniemitz commented on issue #3997: [BEAM-3061] Done 
notification for BigtableIO.write()
URL: https://github.com/apache/beam/pull/3997#issuecomment-387578594
 
 
   Cool, I'll probably be reopening this soon, we had stopped using it for 
awhile, but have now started again in 2.4.0.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99805)
Time Spent: 40m  (was: 0.5h)

> BigtableIO should support emitting a sentinel "done" value when a bundle 
> completes
> --
>
> Key: BEAM-3061
> URL: https://issues.apache.org/jira/browse/BEAM-3061
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Steve Niemitz
>Assignee: Steve Niemitz
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There was some discussion of this on the dev@ mailing list [1].  This 
> approach was taken based on discussion there.
> [1] 
> https://lists.apache.org/thread.html/949b33782f722a9000c9bf9e37042739c6fd0927589b99752b78d7bd@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=99802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99802
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 08/May/18 23:56
Start Date: 08/May/18 23:56
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5271: [BEAM-4073] Migrate 
DirectRunner Evaluators to use Portable Graph Components
URL: https://github.com/apache/beam/pull/5271#issuecomment-387578204
 
 
   R: @youngoli 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99802)
Time Spent: 4h 20m  (was: 4h 10m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99801
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 23:55
Start Date: 08/May/18 23:55
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387578041
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99801)
Time Spent: 17h  (was: 16h 50m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99799
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:41
Start Date: 08/May/18 23:41
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-387575768
 
 
   Updated:
   * removed postfix array support;
   * added 
[`BeamDdlNestedTypesTest`](https://github.com/apache/beam/pull/5276/files#diff-794c03679a02b7349f5070def998ce7a)
 which tests things with QuickCheck. Examples:
   
   ```
   create table tablename ( fieldName MAP> ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   create table tablename ( fieldName MAP> 
) TYPE 'text' LOCATION '/home/admin/person'
   
   create table tablename ( fieldName ARRAY ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   
   create table tablename ( fieldName MAP ) TYPE 'text' LOCATION '/home/admin/person'
   
   
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99799)
Time Spent: 3h 50m  (was: 3h 40m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99797
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:41
Start Date: 08/May/18 23:41
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-387575768
 
 
   Updated:
   * removed postfix array support;
   * added 
(`BeamDdlNestedTypesTest`)[https://github.com/apache/beam/pull/5276/files#diff-794c03679a02b7349f5070def998ce7a]
 which tests things with QuickCheck. Examples:
   
   ```
   create table tablename ( fieldName MAP> ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   create table tablename ( fieldName MAP> 
) TYPE 'text' LOCATION '/home/admin/person'
   
   create table tablename ( fieldName ARRAY ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   
   create table tablename ( fieldName MAP ) TYPE 'text' LOCATION '/home/admin/person'
   
   
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99797)
Time Spent: 3.5h  (was: 3h 20m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99798
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:41
Start Date: 08/May/18 23:41
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5276: [BEAM-4196][SQL] 
Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#issuecomment-387575768
 
 
   Updated:
   * removed postfix array support;
   * added 
(BeamDdlNestedTypesTest)[https://github.com/apache/beam/pull/5276/files#diff-794c03679a02b7349f5070def998ce7a]
 which tests things with QuickCheck. Examples:
   
   ```
   create table tablename ( fieldName MAP> ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   create table tablename ( fieldName MAP> 
) TYPE 'text' LOCATION '/home/admin/person'
   
   create table tablename ( fieldName ARRAY ) TYPE 'text' 
LOCATION '/home/admin/person'
   
   
   create table tablename ( fieldName MAP ) TYPE 'text' LOCATION '/home/admin/person'
   
   
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99798)
Time Spent: 3h 40m  (was: 3.5h)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4255) GloballyAsSingletonView is writing multiple values to the SingletonView

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4255?focusedWorklogId=99794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99794
 ]

ASF GitHub Bot logged work on BEAM-4255:


Author: ASF GitHub Bot
Created on: 08/May/18 23:38
Start Date: 08/May/18 23:38
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5311: 
[BEAM-4255] Set singleton view writers' accumulation mode to discarding
URL: https://github.com/apache/beam/pull/5311#discussion_r186897697
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
 ##
 @@ -1283,6 +1283,10 @@ private GloballyAsSingletonView(
   insertDefault,
   insertDefault ? fn.defaultValue() : null,
   combined.getCoder());
+
+  // GBK's inside view writers for singleton view should discard previous 
values.
+  
materializationInput.setWindowingStrategyInternal(input.getWindowingStrategy()
 
 Review comment:
   You should instead use `Window.configure().discardingFiredPanes()`, 
explicitly setting any other required values from the upstream 
`WindowingStrategy`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99794)
Time Spent: 0.5h  (was: 20m)

> GloballyAsSingletonView is writing multiple values to the SingletonView
> ---
>
> Key: BEAM-4255
> URL: https://issues.apache.org/jira/browse/BEAM-4255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 3.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The code snippet below is writing multiple values to the sideInput. Resulting 
> in exception: "PCollection with more than one element accessed as a singleton 
> view."
> PCollectionView globalView = input
>  .apply(Window.into(new GlobalWindows())
>     .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
>     .accumulatingFiredPanes())
>  .apply(Sum.integersGlobally().asSingletonView())
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4255) GloballyAsSingletonView is writing multiple values to the SingletonView

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4255?focusedWorklogId=99793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99793
 ]

ASF GitHub Bot logged work on BEAM-4255:


Author: ASF GitHub Bot
Created on: 08/May/18 23:29
Start Date: 08/May/18 23:29
Worklog Time Spent: 10m 
  Work Description: huygaa11 commented on issue #5311: [BEAM-4255] Set 
singleton view writers' accumulation mode to discarding
URL: https://github.com/apache/beam/pull/5311#issuecomment-387573790
 
 
   @tgroh could you review this, please?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99793)
Time Spent: 20m  (was: 10m)

> GloballyAsSingletonView is writing multiple values to the SingletonView
> ---
>
> Key: BEAM-4255
> URL: https://issues.apache.org/jira/browse/BEAM-4255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 3.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The code snippet below is writing multiple values to the sideInput. Resulting 
> in exception: "PCollection with more than one element accessed as a singleton 
> view."
> PCollectionView globalView = input
>  .apply(Window.into(new GlobalWindows())
>     .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
>     .accumulatingFiredPanes())
>  .apply(Sum.integersGlobally().asSingletonView())
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4255) GloballyAsSingletonView is writing multiple values to the SingletonView

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4255?focusedWorklogId=99792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99792
 ]

ASF GitHub Bot logged work on BEAM-4255:


Author: ASF GitHub Bot
Created on: 08/May/18 23:28
Start Date: 08/May/18 23:28
Worklog Time Spent: 10m 
  Work Description: huygaa11 opened a new pull request #5311: [BEAM-4255] 
Set singleton view writers' accumulation mode to discarding
URL: https://github.com/apache/beam/pull/5311
 
 
   The code snippet below was writing multiple values to the sideInput, 
resulting in exception: "PCollection with more than one element accessed as a 
singleton view."
   
   PCollectionView globalView = input
   .apply(Window.into(new GlobalWindows())
   .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
   .accumulatingFiredPanes())
   .apply(Sum.integersGlobally().asSingletonView())
   
   This PR sets the accumulation mode to discarding for singleton view writer, 
so that the GBK's used by view writers do not accumulate combine results.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99792)
Time Spent: 10m
Remaining Estimate: 0h

> GloballyAsSingletonView is writing multiple values to the SingletonView
> ---
>
> Key: BEAM-4255
> URL: https://issues.apache.org/jira/browse/BEAM-4255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.4.0
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 3.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The code snippet below is writing multiple values to the sideInput. Resulting 
> in exception: "PCollection with more than one element accessed as a singleton 
> view."
> PCollectionView globalView = input
>  .apply(Window.into(new GlobalWindows())
>     .triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
>     .accumulatingFiredPanes())
>  .apply(Sum.integersGlobally().asSingletonView())
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99790
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:26
Start Date: 08/May/18 23:26
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#discussion_r186895909
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCliTest.java
 ##
 @@ -41,13 +50,178 @@ public void testExecute_createTextTable() throws 
Exception {
 cli.execute(
 "create table person (\n"
 + "id int COMMENT 'id', \n"
-+ "name varchar(31) COMMENT 'name', \n"
++ "name varchar COMMENT 'name', \n"
 + "age int COMMENT 'age') \n"
 + "TYPE 'text' \n"
 + "COMMENT '' LOCATION '/home/admin/orders'"
 );
 Table table = metaStore.getTable("person");
 assertNotNull(table);
+assertEquals(
+Stream
+.of(
+Field.of("id", 
INTEGER).withDescription("id").withNullable(true),
+Field.of("name", 
VARCHAR).withDescription("name").withNullable(true),
+Field.of("age", 
INTEGER).withDescription("age").withNullable(true))
+.collect(toSchema()),
+table.getSchema());
+  }
+
+  @Test
+  public void testExecute_createTableWithArrayField() throws Exception {
+InMemoryMetaStore metaStore = new InMemoryMetaStore();
+metaStore.registerProvider(new TextTableProvider());
+
+BeamSqlCli cli = new BeamSqlCli()
+.metaStore(metaStore);
+cli.execute(
+"create table person (\n"
++ "id int COMMENT 'id', \n"
++ "name varchar COMMENT 'name', \n"
++ "age int COMMENT 'age', \n"
++ "tags VARCHAR ARRAY \n"
++ ") \n"
++ "TYPE 'text' \n"
++ "COMMENT '' LOCATION '/home/admin/orders'"
+);
+Table table = metaStore.getTable("person");
+assertNotNull(table);
+assertEquals(
+Stream
+.of(
+Field.of("id", 
INTEGER).withDescription("id").withNullable(true),
+Field.of("name", 
VARCHAR).withDescription("name").withNullable(true),
+Field.of("age", 
INTEGER).withDescription("age").withNullable(true),
+Field.of("tags",
+ 
ARRAY.type().withCollectionElementType(VARCHAR)).withNullable(true))
+.collect(toSchema()),
+table.getSchema());
+  }
+
+  @Test
+  public void testExecute_createTableWithPrefixArrayField() throws Exception {
+InMemoryMetaStore metaStore = new InMemoryMetaStore();
+metaStore.registerProvider(new TextTableProvider());
+
+BeamSqlCli cli = new BeamSqlCli()
+.metaStore(metaStore);
+cli.execute(
+"create table person (\n"
++ "id int COMMENT 'id', \n"
++ "name varchar COMMENT 'name', \n"
++ "age int COMMENT 'age', \n"
++ "tags ARRAY, \n"
++ "matrix ARRAY \n"
++ ") \n"
++ "TYPE 'text' \n"
++ "COMMENT '' LOCATION '/home/admin/orders'"
+);
+Table table = metaStore.getTable("person");
+assertNotNull(table);
+assertEquals(
+Stream
+.of(
+Field.of("id", 
INTEGER).withDescription("id").withNullable(true),
+Field.of("name", 
VARCHAR).withDescription("name").withNullable(true),
+Field.of("age", 
INTEGER).withDescription("age").withNullable(true),
+Field.of("tags",
+ 
ARRAY.type().withCollectionElementType(VARCHAR)).withNullable(true),
+Field.of("matrix",
+ ARRAY.type().withCollectionElementType(
+ 
ARRAY.type().withCollectionElementType(INTEGER))).withNullable(true))
+.collect(toSchema()),
+table.getSchema());
+  }
+
+  @Test
+  public void testExecute_createTableWithPrefixMapField() throws Exception {
+InMemoryMetaStore metaStore = new InMemoryMetaStore();
+metaStore.registerProvider(new TextTableProvider());
+
+BeamSqlCli cli = new BeamSqlCli()
+.metaStore(metaStore);
+cli.execute(
+"create table person (\n"
++ "id int COMMENT 'id', \n"
++ "name varchar COMMENT 'name', \n"
++ "age int COMMENT 'age', \n"
++ "tags MAP, \n"
++ "nestedMap MAP> \n"
 
 Review comment:
   Will do


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries 

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99788
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:26
Start Date: 08/May/18 23:26
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#discussion_r186894806
 
 

 ##
 File path: sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
 ##
 @@ -153,4 +190,115 @@ SqlDrop SqlDropTable(Span s, boolean replace) :
 }
 }
 
+Schema.FieldType FieldType() :
+{
+final SqlTypeName collectionTypeName;
+Schema.FieldType fieldType;
+final Span s = Span.of();
+}
+{
+(
+fieldType = Map()
+|
+fieldType = Array()
+|
+fieldType = Row()
+|
+fieldType = SimpleType()
+)
+[
+collectionTypeName = CollectionTypeName()
 
 Review comment:
   Yes this clause is for the postfix array support. I left it mostly because 
it was already kinda there, just limited to `MULTISETS`, not arrays. We don't 
support either at the moment, and I would rather remove it for now for 
consistency


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99788)
Time Spent: 2h 50m  (was: 2h 40m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99791
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:26
Start Date: 08/May/18 23:26
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#discussion_r186895963
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamSqlParserTest.java
 ##
 @@ -63,7 +64,7 @@ public void testParseCreateTable_withoutType() throws 
Exception {
 parseTable(
 "create table person (\n"
 + "id int COMMENT 'id', \n"
-+ "name varchar(31) COMMENT 'name') \n"
++ "name varchar COMMENT 'name') \n"
 
 Review comment:
   Adding QuickCheck tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99791)
Time Spent: 3h 20m  (was: 3h 10m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4196) [SQL] Support Complex Types in DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4196?focusedWorklogId=99789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99789
 ]

ASF GitHub Bot logged work on BEAM-4196:


Author: ASF GitHub Bot
Created on: 08/May/18 23:26
Start Date: 08/May/18 23:26
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5276: 
[BEAM-4196][SQL] Support complex types in DDL
URL: https://github.com/apache/beam/pull/5276#discussion_r186895795
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
 ##
 @@ -67,23 +72,24 @@
 this.tblProperties = tblProperties; // may be null
   }
 
+  @Override
   public List getOperandList() {
-return ImmutableNullableList.of(name, columnList, type, comment, location, 
tblProperties);
+throw new UnsupportedOperationException(
+"Getting operands CREATE TABLE is unsupported at the moment");
 
 Review comment:
   It's mostly used by implementations of the `SqlCall` to access their 
parameters. `SqlCreate` itself doesn't really need it but we might (or might 
not) need something like this for JDBC integration, depending on how `CREATE 
TABLE` parsing will be implemented there


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99789)
Time Spent: 3h  (was: 2h 50m)

> [SQL] Support Complex Types in DDL
> --
>
> Key: BEAM-4196
> URL: https://issues.apache.org/jira/browse/BEAM-4196
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Neither our DDL parser we copied from calcite-server or the calcite-server 
> don't support complex types in DDL. If we want to model something like JSON 
> objects we need to support at least Arrays and nested Rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99785
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 23:24
Start Date: 08/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/worker/opcounters.pxd 
b/sdks/python/apache_beam/runners/worker/opcounters.pxd
index 0bcd42848d2..1d7f296c5ce 100644
--- a/sdks/python/apache_beam/runners/worker/opcounters.pxd
+++ b/sdks/python/apache_beam/runners/worker/opcounters.pxd
@@ -19,13 +19,14 @@ cimport cython
 cimport libc.stdint
 
 from apache_beam.utils.counters cimport Counter
+from apache_beam.runners.worker cimport statesampler_fast
 
 
 cdef class TransformIOCounter(object):
   cdef readonly object _counter_factory
   cdef readonly object _state_sampler
   cdef Counter bytes_read_counter
-  cdef object scoped_state
+  cdef statesampler_fast.ScopedState scoped_state
   cdef object _latest_step
 
   cpdef update_current_step(self)
diff --git a/sdks/python/apache_beam/runners/worker/sideinputs.py 
b/sdks/python/apache_beam/runners/worker/sideinputs.py
index d2599fd59a5..a63109b4d08 100644
--- a/sdks/python/apache_beam/runners/worker/sideinputs.py
+++ b/sdks/python/apache_beam/runners/worker/sideinputs.py
@@ -160,7 +160,8 @@ def __iter__(self):
 try:
   while True:
 try:
-  element = self.element_queue.get()
+  with self.read_counter:
+element = self.element_queue.get()
   if element is READER_THREAD_IS_DONE_SENTINEL:
 num_readers_finished += 1
 if num_readers_finished == self.num_reader_threads:
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd 
b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
new file mode 100644
index 000..ae1ce90e145
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
@@ -0,0 +1,42 @@
+cimport cython
+
+from apache_beam.metrics.execution cimport MetricsContainer
+
+from cpython cimport pythread
+from libc.stdint cimport int32_t, int64_t
+
+cdef class StateSampler(object):
+  """Tracks time spent in states during pipeline execution."""
+  cdef int _sampling_period_ms
+
+  cdef list scoped_states_by_index
+
+  cdef public bint started
+  cdef public bint finished
+  cdef object sampling_thread
+
+  # This lock guards members that are shared between threads, specificaly
+  # finished, scoped_states_by_index, and the nsecs field of each state 
therein.
+  cdef pythread.PyThread_type_lock lock
+
+  cdef public int64_t state_transition_count
+  cdef public int64_t time_since_transition
+
+  cdef int32_t current_state_index
+
+  cpdef _scoped_state(self, counter_name, output_counter, metrics_container)
+
+cdef class ScopedState(object):
+  """Context manager class managing transitions for a given sampler state."""
+
+  cdef readonly StateSampler sampler
+  cdef readonly int32_t state_index
+  cdef readonly object counter
+  cdef readonly object name
+  cdef readonly int64_t _nsecs
+  cdef int32_t old_state_index
+  cdef readonly MetricsContainer _metrics_container
+
+  cpdef __enter__(self)
+
+  cpdef __exit__(self, unused_exc_type, unused_exc_value, unused_traceback)
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx 
b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
index 0fc58445f3b..21b09e626ec 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
@@ -70,23 +70,6 @@ cdef inline int64_t get_nsec_time() nogil:
 
 cdef class StateSampler(object):
   """Tracks time spent in states during pipeline execution."""
-  cdef int _sampling_period_ms
-
-  cdef list scoped_states_by_index
-
-  cdef public bint started
-  cdef public bint finished
-  cdef object sampling_thread
-
-  # This lock guards members that are shared between threads, specificaly
-  # finished, scoped_states_by_index, and the nsecs field of each state 
therein.
-  cdef pythread.PyThread_type_lock lock
-
-  cdef public int64_t state_transition_count
-  cdef public int64_t time_since_transition
-
-  cdef int32_t current_state_index
-
   def __init__(self, sampling_period_ms, *args):
 self._sampling_period_ms = sampling_period_ms
 self.started = False
@@ -157,7 +140,7 @@ cdef class StateSampler(object):
 return 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1563

[Apache Jenkins Server]

See 


Changes:

[tgroh] Migrate the `portable` subpackage to Portability

--
Started by GitHub push by tgroh
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7848764a070b3bcd02528014838792b88c5ea2e4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7848764a070b3bcd02528014838792b88c5ea2e4
Commit message: "Merge pull request #5270: Migrate the `portable` subpackage to 
use Portable representations"
 > git rev-list --no-walk e7a3578abb919021be95c9c7148fd2d4fc6331fd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6501802628843931600.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user schroed...@google.com

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99786
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 23:24
Start Date: 08/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: pabloem opened a new pull request #5309: [BEAM-3042] 
Adding time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309
 
 
   This PR improves Cython tags for some classes, and uses them for tracking of 
time spent reading side inputs.
   
   NOTE: This PR should add flag versioning before merging in any case.
   
   This has been benchmarked with the new 
[`sideinput_microbenchmark.py`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/sideinput_microbenchmark.py),
 and here are the results:
   
   Current performance with 500 runs:
   - Average runtime: 0.422656793594
   - Time per element: 2.64160495996e-05
   - Regression: 0% (it's the baseline)
   
   With change and flag deactivated:
   - Average runtime: 0.424214161396
   - Time per element: 2.65133850873e-05
   - Regression: 0.36%
   
   With change and flag activated:
   - Average runtime: 0.425546179771
   - Time per element: 2.65966362357e-05
   - Regression: 0.68%
   
   This represents a really small regression in a microbenchmark that 
specifically exercises this feature.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99786)
Time Spent: 5h  (was: 4h 50m)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99781
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 23:16
Start Date: 08/May/18 23:16
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186891548
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
+A BeamRunner that executes Python pipelines via the Beam Job API.
 
-  By default, this runner executes in process but still uses GRPC to 
communicate
-  pipeline and worker state.  It can also be configured to use inline calls
-  rather than GRPC (for speed) or launch completely separate subprocesses for
-  the runner and worker(s).
+This runner is a stub and does not run the actual job.
+This runner schedules the job on a job service. The responsibility of
+running and managing the job lies with the job service used.
   """
 
-  def __init__(
-  self,
-  use_grpc=True,
-  use_subprocesses=False,
-  runner_api_address=None,
-  docker_image=None):
-if use_subprocesses and not use_grpc:
-  raise ValueError("GRPC must be used with subprocesses")
+  # TODO(angoenka): Read all init parameters from pipeline_options.
+  def __init__(self,
+   runner_api_address=None,
+   job_service_address=None,
 
 Review comment:
   Not sure, I need to check with @robertwb about the intent of 
runner_api_address.
   docker_image and other parameters in constructor should be moved to 
pipeline_options as job_service will interpret them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99781)
Time Spent: 1h 50m  (was: 1h 40m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99779
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 23:15
Start Date: 08/May/18 23:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387571526
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99779)
Time Spent: 16h 50m  (was: 16h 40m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99777
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 23:00
Start Date: 08/May/18 23:00
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186891838
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/job_service.py
 ##
 @@ -0,0 +1,294 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import logging
+import os
+import Queue as queue
+import subprocess
+import threading
+import time
+import traceback
+import uuid
+from concurrent import futures
+
+import grpc
+from google.protobuf import text_format
+
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.portability.api import beam_job_api_pb2
+from apache_beam.portability.api import beam_job_api_pb2_grpc
+from apache_beam.portability.api import endpoints_pb2
+from apache_beam.runners.portability import fn_api_runner
+
+TERMINAL_STATES = [
+beam_job_api_pb2.JobState.DONE,
+beam_job_api_pb2.JobState.STOPPED,
+beam_job_api_pb2.JobState.FAILED,
+beam_job_api_pb2.JobState.CANCELLED,
+]
+
+
+class JobServicer(beam_job_api_pb2_grpc.JobServiceServicer):
+  """
+Experimental: No backward compatibility guaranteed.
+Servicer for the Beam Job API.
+
+Manages one or more pipelines, possibly concurrently.
+
+This JobService uses a basic local implementation of runner to run the job.
 
 Review comment:
   Agree, Renaming it to LocalJobService.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99777)
Time Spent: 1h 40m  (was: 1.5h)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99776
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:59
Start Date: 08/May/18 22:59
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186891744
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
+A BeamRunner that executes Python pipelines via the Beam Job API.
 
-  By default, this runner executes in process but still uses GRPC to 
communicate
-  pipeline and worker state.  It can also be configured to use inline calls
-  rather than GRPC (for speed) or launch completely separate subprocesses for
-  the runner and worker(s).
+This runner is a stub and does not run the actual job.
+This runner schedules the job on a job service. The responsibility of
 
 Review comment:
   I didn't see a usecase for it other than the test.
   We can recreate the functionality when needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99776)
Time Spent: 1.5h  (was: 1h 20m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99775
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:58
Start Date: 08/May/18 22:58
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186891548
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
+A BeamRunner that executes Python pipelines via the Beam Job API.
 
-  By default, this runner executes in process but still uses GRPC to 
communicate
-  pipeline and worker state.  It can also be configured to use inline calls
-  rather than GRPC (for speed) or launch completely separate subprocesses for
-  the runner and worker(s).
+This runner is a stub and does not run the actual job.
+This runner schedules the job on a job service. The responsibility of
+running and managing the job lies with the job service used.
   """
 
-  def __init__(
-  self,
-  use_grpc=True,
-  use_subprocesses=False,
-  runner_api_address=None,
-  docker_image=None):
-if use_subprocesses and not use_grpc:
-  raise ValueError("GRPC must be used with subprocesses")
+  # TODO(angoenka): Read all init parameters from pipeline_options.
+  def __init__(self,
+   runner_api_address=None,
+   job_service_address=None,
 
 Review comment:
   Not sure, I need to check with @robertwb about the intent of 
runner_api_address.
   docker_image and other parameters in constructor should be moved to 
pipeline_options as job_service will interpret it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99775)
Time Spent: 1h 20m  (was: 1h 10m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99774
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:56
Start Date: 08/May/18 22:56
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186891219
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
 
 Review comment:
   I was thinking about it and wanted to do in a subsequent PR.
   I will go ahead with the rename in this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99774)
Time Spent: 1h 10m  (was: 1h)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99771
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:51
Start Date: 08/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309#issuecomment-387565860
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99771)
Time Spent: 4h 20m  (was: 4h 10m)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99773
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:51
Start Date: 08/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: pabloem opened a new pull request #5309: [BEAM-3042] 
Adding time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309
 
 
   This PR improves Cython tags for some classes, and uses them for tracking of 
time spent reading side inputs.
   
   NOTE: This PR should add flag versioning before merging in any case.
   
   This has been benchmarked with the new 
[`sideinput_microbenchmark.py`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/sideinput_microbenchmark.py),
 and here are the results:
   
   Current performance with 500 runs:
   - Average runtime: 0.422656793594
   - Time per element: 2.64160495996e-05
   - Regression: 0% (it's the baseline)
   
   With change and flag deactivated:
   - Average runtime: 0.424214161396
   - Time per element: 2.65133850873e-05
   - Regression: 0.36%
   
   With change and flag activated:
   - Average runtime: 0.425546179771
   - Time per element: 2.65966362357e-05
   - Regression: 0.68%
   
   This represents a really small regression in a microbenchmark that 
specifically exercises this feature.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99773)
Time Spent: 4h 40m  (was: 4.5h)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99772
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:51
Start Date: 08/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/worker/opcounters.pxd 
b/sdks/python/apache_beam/runners/worker/opcounters.pxd
index 0bcd42848d2..1d7f296c5ce 100644
--- a/sdks/python/apache_beam/runners/worker/opcounters.pxd
+++ b/sdks/python/apache_beam/runners/worker/opcounters.pxd
@@ -19,13 +19,14 @@ cimport cython
 cimport libc.stdint
 
 from apache_beam.utils.counters cimport Counter
+from apache_beam.runners.worker cimport statesampler_fast
 
 
 cdef class TransformIOCounter(object):
   cdef readonly object _counter_factory
   cdef readonly object _state_sampler
   cdef Counter bytes_read_counter
-  cdef object scoped_state
+  cdef statesampler_fast.ScopedState scoped_state
   cdef object _latest_step
 
   cpdef update_current_step(self)
diff --git a/sdks/python/apache_beam/runners/worker/sideinputs.py 
b/sdks/python/apache_beam/runners/worker/sideinputs.py
index d2599fd59a5..a63109b4d08 100644
--- a/sdks/python/apache_beam/runners/worker/sideinputs.py
+++ b/sdks/python/apache_beam/runners/worker/sideinputs.py
@@ -160,7 +160,8 @@ def __iter__(self):
 try:
   while True:
 try:
-  element = self.element_queue.get()
+  with self.read_counter:
+element = self.element_queue.get()
   if element is READER_THREAD_IS_DONE_SENTINEL:
 num_readers_finished += 1
 if num_readers_finished == self.num_reader_threads:
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd 
b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
new file mode 100644
index 000..ae1ce90e145
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pxd
@@ -0,0 +1,42 @@
+cimport cython
+
+from apache_beam.metrics.execution cimport MetricsContainer
+
+from cpython cimport pythread
+from libc.stdint cimport int32_t, int64_t
+
+cdef class StateSampler(object):
+  """Tracks time spent in states during pipeline execution."""
+  cdef int _sampling_period_ms
+
+  cdef list scoped_states_by_index
+
+  cdef public bint started
+  cdef public bint finished
+  cdef object sampling_thread
+
+  # This lock guards members that are shared between threads, specificaly
+  # finished, scoped_states_by_index, and the nsecs field of each state 
therein.
+  cdef pythread.PyThread_type_lock lock
+
+  cdef public int64_t state_transition_count
+  cdef public int64_t time_since_transition
+
+  cdef int32_t current_state_index
+
+  cpdef _scoped_state(self, counter_name, output_counter, metrics_container)
+
+cdef class ScopedState(object):
+  """Context manager class managing transitions for a given sampler state."""
+
+  cdef readonly StateSampler sampler
+  cdef readonly int32_t state_index
+  cdef readonly object counter
+  cdef readonly object name
+  cdef readonly int64_t _nsecs
+  cdef int32_t old_state_index
+  cdef readonly MetricsContainer _metrics_container
+
+  cpdef __enter__(self)
+
+  cpdef __exit__(self, unused_exc_type, unused_exc_value, unused_traceback)
diff --git a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx 
b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
index 0fc58445f3b..21b09e626ec 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
+++ b/sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
@@ -70,23 +70,6 @@ cdef inline int64_t get_nsec_time() nogil:
 
 cdef class StateSampler(object):
   """Tracks time spent in states during pipeline execution."""
-  cdef int _sampling_period_ms
-
-  cdef list scoped_states_by_index
-
-  cdef public bint started
-  cdef public bint finished
-  cdef object sampling_thread
-
-  # This lock guards members that are shared between threads, specificaly
-  # finished, scoped_states_by_index, and the nsecs field of each state 
therein.
-  cdef pythread.PyThread_type_lock lock
-
-  cdef public int64_t state_transition_count
-  cdef public int64_t time_since_transition
-
-  cdef int32_t current_state_index
-
   def __init__(self, sampling_period_ms, *args):
 self._sampling_period_ms = sampling_period_ms
 self.started = False
@@ -157,7 +140,7 @@ cdef class StateSampler(object):
 return 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #219

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99768
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:45
Start Date: 08/May/18 22:45
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309#issuecomment-387565860
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99768)
Time Spent: 4h 10m  (was: 4h)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99767
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:44
Start Date: 08/May/18 22:44
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309#issuecomment-387564575
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99767)
Time Spent: 4h  (was: 3h 50m)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=99766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99766
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 08/May/18 22:43
Start Date: 08/May/18 22:43
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5180: [BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r186888994
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,259 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+#NOW = calendar.timegm(time.strptime('14-03-2018_13-30-27', TIME_PATTERN)) 
#left for testing
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBOOK_URL = os.getenv('SLACK_WEBOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
+results = query_job.result()
+if return_type == "list":
+# Queries that have multiple elements in output displayed as 
query_result
+result_list = []
+for row in results:
+result_list.append(row.query_result)
+return result_list
+elif return_type == "value":
+# All queries must have single element in output displayed as 
query_result
+for row in results:
+return (row.query_result)
+else:
+print("This type is not supported")
+return None
+
+def count_queries(table_name, time_start, time_stop, metric):
+# This function checks how many data was inserted in time interval.
+sql_command = 'select count(*) as query_result from {} where TIMESTAMP > 
{} and TIMESTAMP < {} and METRIC=\'{}\''.format(
+table_name,
+time_start,
+time_stop,
+metric
+)
+count = submit_big_query_job(sql_command, "value")
+print("Number of records inserted into {} between {} - {}: {}".format(
+table_name,
+time.strftime(TIME_PATTERN, time.gmtime(time_start)),
+time.strftime(TIME_PATTERN, time.gmtime(time_stop)),
+count))
+return count
+
+def get_average_from(table_name, time_start, time_stop, metric):
+# This function return average value of the provided metric in time 
interval.
+sql_command = 'select avg(value) as query_result from {} where TIMESTAMP > 
{} and TIMESTAMP < {} and 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4899

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99764
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:38
Start Date: 08/May/18 22:38
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5309: [BEAM-3042] Adding 
time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309#issuecomment-387564575
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99764)
Time Spent: 3h 50m  (was: 3h 40m)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #316

[Apache Jenkins Server]

See 


Changes:

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

--
[...truncated 18.60 MB...]
org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testFormatCounts 
STANDARD_ERROR
May 08, 2018 10:37:34 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testExtractTornadoes 
STANDARD_ERROR
May 08, 2018 10:37:34 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testNoTornadoes 
STANDARD_ERROR
May 08, 2018 10:37:34 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.TriggerExampleTest > testExtractTotalFlow 
STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > 
testFilterSingleMonthDataFn STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 08, 2018 10:37:35 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern /tmp/junit599937022893102938/junit645337529883236470.tmp 
matched 1 files with total size 54
May 08, 2018 10:37:35 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit599937022893102938/junit645337529883236470.tmp into bundles of size 3 
took 0 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 08, 2018 10:37:36 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo7946709399233507469.sh 
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo7946709399233507469.sh to 
/tmp/test-Echoo386350855995896034/test-Echo7946709399233507469.sh 
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain3964695487178783493.sh 
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 08, 2018 10:37:37 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain3964695487178783493.sh to 
/tmp/test-Echoo386350855995896034/test-EchoAgain3964695487178783493.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT 

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=99763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99763
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 08/May/18 22:37
Start Date: 08/May/18 22:37
Worklog Time Spent: 10m 
  Work Description: pabloem opened a new pull request #5309: [BEAM-3042] 
Adding time tracking of batch side inputs
URL: https://github.com/apache/beam/pull/5309
 
 
   This PR improves Cython tags for some classes, and uses them for tracking of 
time spent reading side inputs.
   
   NOTE: This PR should add flag versioning before merging in any case.
   
   This has been benchmarked with the new 
[`sideinput_microbenchmark.py`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/sideinput_microbenchmark.py),
 and here are the results:
   
   Current performance with 500 runs:
   - Average runtime: 0.422656793594
   - Time per element: 2.64160495996e-05
   - Regression: 0% (it's the baseline)
   
   With change and flag deactivated:
   - Average runtime: 0.424214161396
   - Time per element: 2.65133850873e-05
   - Regression: 0.36%
   
   With change and flag activated:
   - Average runtime: 0.425546179771
   - Time per element: 2.65966362357e-05
   - Regression: 0.68%
   
   This represents a really small regression in a microbenchmark that 
specifically exercises this feature.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99763)
Time Spent: 3h 40m  (was: 3.5h)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99761
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 22:35
Start Date: 08/May/18 22:35
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387563867
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99761)
Time Spent: 16h 40m  (was: 16.5h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2597) FlinkRunner ExecutableStage batch operator

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2597?focusedWorklogId=99759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99759
 ]

ASF GitHub Bot logged work on BEAM-2597:


Author: ASF GitHub Bot
Created on: 08/May/18 22:25
Start Date: 08/May/18 22:25
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5285: 
[BEAM-2597] Flink batch ExecutableStage operator
URL: https://github.com/apache/beam/pull/5285#discussion_r186884859
 
 

 ##
 File path: 
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
 ##
 @@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.protobuf.Struct;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.runners.flink.ArtifactSourcePool;
+import org.apache.beam.runners.flink.FlinkBundleFactory;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.Collector;
+import org.hamcrest.Matchers;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.InOrder;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.mockito.stubbing.Answer;
+
+/** Tests for {@link FlinkExecutableStageFunction}. */
+@RunWith(JUnit4.class)
+public class FlinkExecutableStageFunctionTest {
+  @Rule public ExpectedException thrown = ExpectedException.none();
+
+  @Mock private RuntimeContext runtimeContext;
+  @Mock private DistributedCache distributedCache;
+  @Mock private Collector collector;
+  @Mock private StageBundleFactory stageBundleFactory;
+  @Mock private ArtifactSourcePool cachePool;
+  @Mock private StateRequestHandler stateRequestHandler;
+
+  // NOTE: ExecutableStage.fromPayload expects exactly one input, so we 
provide one here. These unit
+  // tests in general ignore the executable stage itself and mock around it.
+  private final ExecutableStagePayload stagePayload =
+  ExecutableStagePayload.newBuilder()
+  .setInput("input")
+  .setComponents(
+  Components.newBuilder()
+  .putPcollections("input", PCollection.getDefaultInstance())
+  .build())
+  .build();
+  private final JobInfo jobInfo = JobInfo.create("job-id", "job-name", 
Struct.getDefaultInstance());
+
+  @Before
+  public void setUpMocks() {
+MockitoAnnotations.initMocks(this);
+

[jira] [Work logged] (BEAM-2597) FlinkRunner ExecutableStage batch operator

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2597?focusedWorklogId=99758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99758
 ]

ASF GitHub Bot logged work on BEAM-2597:


Author: ASF GitHub Bot
Created on: 08/May/18 22:25
Start Date: 08/May/18 22:25
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5285: 
[BEAM-2597] Flink batch ExecutableStage operator
URL: https://github.com/apache/beam/pull/5285#discussion_r186884593
 
 

 ##
 File path: 
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
 ##
 @@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.protobuf.Struct;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.runners.flink.ArtifactSourcePool;
+import org.apache.beam.runners.flink.FlinkBundleFactory;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.flink.api.common.cache.DistributedCache;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.Collector;
+import org.hamcrest.Matchers;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.InOrder;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.mockito.stubbing.Answer;
+
+/** Tests for {@link FlinkExecutableStageFunction}. */
+@RunWith(JUnit4.class)
+public class FlinkExecutableStageFunctionTest {
+  @Rule public ExpectedException thrown = ExpectedException.none();
+
+  @Mock private RuntimeContext runtimeContext;
+  @Mock private DistributedCache distributedCache;
+  @Mock private Collector collector;
+  @Mock private StageBundleFactory stageBundleFactory;
+  @Mock private ArtifactSourcePool cachePool;
+  @Mock private StateRequestHandler stateRequestHandler;
+
+  // NOTE: ExecutableStage.fromPayload expects exactly one input, so we 
provide one here. These unit
+  // tests in general ignore the executable stage itself and mock around it.
+  private final ExecutableStagePayload stagePayload =
+  ExecutableStagePayload.newBuilder()
+  .setInput("input")
+  .setComponents(
+  Components.newBuilder()
+  .putPcollections("input", PCollection.getDefaultInstance())
+  .build())
+  .build();
+  private final JobInfo jobInfo = JobInfo.create("job-id", "job-name", 
Struct.getDefaultInstance());
+
+  @Before
+  public void setUpMocks() {
+MockitoAnnotations.initMocks(this);
+

[jira] [Work logged] (BEAM-2597) FlinkRunner ExecutableStage batch operator

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2597?focusedWorklogId=99757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99757
 ]

ASF GitHub Bot logged work on BEAM-2597:


Author: ASF GitHub Bot
Created on: 08/May/18 22:25
Start Date: 08/May/18 22:25
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5285: 
[BEAM-2597] Flink batch ExecutableStage operator
URL: https://github.com/apache/beam/pull/5285#discussion_r186884010
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
 ##
 @@ -38,36 +53,117 @@
 public class FlinkExecutableStageFunction
 extends RichMapPartitionFunction {
 
+  // Main constructor fields. All must be Serializable because Flink 
distributes Functions to
+  // task managers via java serialization.
+
   // The executable stage this function will run.
   private final RunnerApi.ExecutableStagePayload stagePayload;
   // Pipeline options. Used for provisioning api.
-  private final Struct pipelineOptions;
+  private final JobInfo jobInfo;
   // Map from PCollection id to the union tag used to represent this 
PCollection in the output.
   private final Map outputMap;
+  private final SerializableSupplier bundleFactorySupplier;
+  private final DistributedCachePool.Factory cachePoolFactory;
+  private final FlinkStateRequestHandlerFactory stateHandlerFactory;
+
+  // Worker-local fields. These should only be constructed and consumed on 
Flink TaskManagers.
+  private transient RuntimeContext runtimeContext;
+  private transient StateRequestHandler stateRequestHandler;
+  private transient StageBundleFactory stageBundleFactory;
+  private transient AutoCloseable distributedCacheCloser;
 
   public FlinkExecutableStageFunction(
   RunnerApi.ExecutableStagePayload stagePayload,
-  Struct pipelineOptions,
-  Map outputMap) {
+  JobInfo jobInfo,
+  Map outputMap,
+  SerializableSupplier bundleFactorySupplier,
 
 Review comment:
   Hm I wonder if it makes sense then to encapsulate these three things into a 
"FlinkExecutableStageContext" and have a ContextFactory, so that at least we 
avoid the proliferation of factories.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99757)
Time Spent: 2h 40m  (was: 2.5h)

> FlinkRunner ExecutableStage batch operator
> --
>
> Key: BEAM-2597
> URL: https://issues.apache.org/jira/browse/BEAM-2597
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This operator will execute user code in the context of an SDK harness by 
> constructing a ProcessBundleDescriptor from an ExecutableStage (physical 
> stage plan) and sending instructions/elements over the control and data 
> planes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #138

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit 902b357bd9bc48de738c81b67463a0e275c4a7b2, 
no merge conflicts.
Setting status of 902b357bd9bc48de738c81b67463a0e275c4a7b2 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/138/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision b47d54dc80952c109c51e73d20190a39377ddc2b 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b47d54dc80952c109c51e73d20190a39377ddc2b
Commit message: "Merge 902b357bd9bc48de738c81b67463a0e275c4a7b2 into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins330929832631580181.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #137

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit 404add30918a5a15efcfd33eea810c3ea797963e, 
no merge conflicts.
Setting status of 404add30918a5a15efcfd33eea810c3ea797963e to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/137/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision bb45a63d4d1a0d52c153948be971d6383a7c0212 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb45a63d4d1a0d52c153948be971d6383a7c0212
Commit message: "Merge 404add30918a5a15efcfd33eea810c3ea797963e into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5936959178773576857.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #136

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit d3434ce1f1982affa0c542e1507794dcad209921, 
no merge conflicts.
Setting status of d3434ce1f1982affa0c542e1507794dcad209921 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/136/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision addd4404361a49ce7a3790e077eafb6708928ac7 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f addd4404361a49ce7a3790e077eafb6708928ac7
Commit message: "Merge d3434ce1f1982affa0c542e1507794dcad209921 into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4376380555207253935.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99749
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:12
Start Date: 08/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186881926
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
 
 Review comment:
   Should this runner be renamed perhaps, to PortableRunner now? It's no longer 
"local"


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99749)
Time Spent: 40m  (was: 0.5h)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99748
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:12
Start Date: 08/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186882495
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
+A BeamRunner that executes Python pipelines via the Beam Job API.
 
-  By default, this runner executes in process but still uses GRPC to 
communicate
-  pipeline and worker state.  It can also be configured to use inline calls
-  rather than GRPC (for speed) or launch completely separate subprocesses for
-  the runner and worker(s).
+This runner is a stub and does not run the actual job.
+This runner schedules the job on a job service. The responsibility of
 
 Review comment:
   Do we need to keep also the old functionality of this class, perhaps as a 
wrapper? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99748)
Time Spent: 0.5h  (was: 20m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99750
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:12
Start Date: 08/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186882048
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/universal_local_runner.py
 ##
 @@ -50,42 +37,30 @@
 
 
 class UniversalLocalRunner(runner.PipelineRunner):
-  """A BeamRunner that executes Python pipelines via the Beam Job API.
+  """
+Experimental: No backward compatibility guaranteed.
+A BeamRunner that executes Python pipelines via the Beam Job API.
 
-  By default, this runner executes in process but still uses GRPC to 
communicate
-  pipeline and worker state.  It can also be configured to use inline calls
-  rather than GRPC (for speed) or launch completely separate subprocesses for
-  the runner and worker(s).
+This runner is a stub and does not run the actual job.
+This runner schedules the job on a job service. The responsibility of
+running and managing the job lies with the job service used.
   """
 
-  def __init__(
-  self,
-  use_grpc=True,
-  use_subprocesses=False,
-  runner_api_address=None,
-  docker_image=None):
-if use_subprocesses and not use_grpc:
-  raise ValueError("GRPC must be used with subprocesses")
+  # TODO(angoenka): Read all init parameters from pipeline_options.
+  def __init__(self,
+   runner_api_address=None,
+   job_service_address=None,
 
 Review comment:
   Do we need runner_api_address and docker_image?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99750)
Time Spent: 50m  (was: 40m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=99751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99751
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 08/May/18 22:12
Start Date: 08/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5301: 
[BEAM-4204] Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#discussion_r186883116
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/job_service.py
 ##
 @@ -0,0 +1,294 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import logging
+import os
+import Queue as queue
+import subprocess
+import threading
+import time
+import traceback
+import uuid
+from concurrent import futures
+
+import grpc
+from google.protobuf import text_format
+
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.portability.api import beam_job_api_pb2
+from apache_beam.portability.api import beam_job_api_pb2_grpc
+from apache_beam.portability.api import endpoints_pb2
+from apache_beam.runners.portability import fn_api_runner
+
+TERMINAL_STATES = [
+beam_job_api_pb2.JobState.DONE,
+beam_job_api_pb2.JobState.STOPPED,
+beam_job_api_pb2.JobState.FAILED,
+beam_job_api_pb2.JobState.CANCELLED,
+]
+
+
+class JobServicer(beam_job_api_pb2_grpc.JobServiceServicer):
+  """
+Experimental: No backward compatibility guaranteed.
+Servicer for the Beam Job API.
+
+Manages one or more pipelines, possibly concurrently.
+
+This JobService uses a basic local implementation of runner to run the job.
 
 Review comment:
   Perhaps this one should be called UniversalLocalJobService then? (or 
something like that)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99751)
Time Spent: 1h  (was: 50m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #135

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit 3ecf5c05329dd2b38aa333464b332a0f1051ef70, 
no merge conflicts.
Setting status of 3ecf5c05329dd2b38aa333464b332a0f1051ef70 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/135/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision 7d4035727c1e36cdc03fa458ec1e5edf4146c0dc 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7d4035727c1e36cdc03fa458ec1e5edf4146c0dc
Commit message: "Merge 3ecf5c05329dd2b38aa333464b332a0f1051ef70 into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins634408782215180840.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

[jira] [Work logged] (BEAM-3061) BigtableIO should support emitting a sentinel "done" value when a bundle completes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3061?focusedWorklogId=99743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99743
 ]

ASF GitHub Bot logged work on BEAM-3061:


Author: ASF GitHub Bot
Created on: 08/May/18 22:02
Start Date: 08/May/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #3997: [BEAM-3061] Done 
notification for BigtableIO.write()
URL: https://github.com/apache/beam/pull/3997#issuecomment-387556596
 
 
   @steveniemitz Please note, though, that now we have a proper way to do this 
sort of done notifications, using the Wait transform. E.g. see how the Spanner 
connector does this: https://github.com/apache/beam/pull/4264


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99743)
Time Spent: 0.5h  (was: 20m)

> BigtableIO should support emitting a sentinel "done" value when a bundle 
> completes
> --
>
> Key: BEAM-3061
> URL: https://issues.apache.org/jira/browse/BEAM-3061
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Steve Niemitz
>Assignee: Steve Niemitz
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> There was some discussion of this on the dev@ mailing list [1].  This 
> approach was taken based on discussion there.
> [1] 
> https://lists.apache.org/thread.html/949b33782f722a9000c9bf9e37042739c6fd0927589b99752b78d7bd@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #134

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit a14dfd35f65d89a232fdb9548bcdf8eede056f1f, 
no merge conflicts.
Setting status of a14dfd35f65d89a232fdb9548bcdf8eede056f1f to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/134/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision 6b120eddd6c0d00f8793df2b382e0125a49c9b16 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6b120eddd6c0d00f8793df2b382e0125a49c9b16
Commit message: "Merge a14dfd35f65d89a232fdb9548bcdf8eede056f1f into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins5516541468781274172.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

[beam] 01/01: Merge pull request #4264: [BEAM-4061] Introduced SpannerWriteResult

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e7a3578abb919021be95c9c7148fd2d4fc6331fd
Merge: 4a9b826 ee288f2
Author: Eugene Kirpichov 
AuthorDate: Tue May 8 14:59:55 2018 -0700

Merge pull request #4264: [BEAM-4061] Introduced SpannerWriteResult

[BEAM-4061] Introduced SpannerWriteResult

 .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java  | 134 +
 .../sdk/io/gcp/spanner/SpannerWriteResult.java |  72 +++
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java |  44 +++
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java| 125 ---
 4 files changed, 305 insertions(+), 70 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1562

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e7a3578abb919021be95c9c7148fd2d4fc6331fd (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e7a3578abb919021be95c9c7148fd2d4fc6331fd
Commit message: "Merge pull request #4264: [BEAM-4061] Introduced 
SpannerWriteResult"
 > git rev-list --no-walk e7a3578abb919021be95c9c7148fd2d4fc6331fd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins927005579096213696.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user schroed...@google.com

[jira] [Work logged] (BEAM-3433) Allow BigQueryIO to use a different project for the load job in batch mode.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3433?focusedWorklogId=99742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99742
 ]

ASF GitHub Bot logged work on BEAM-3433:


Author: ASF GitHub Bot
Created on: 08/May/18 22:00
Start Date: 08/May/18 22:00
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5178: [BEAM-3433] Allow a GCP 
project to be explicitly set for a load job
URL: https://github.com/apache/beam/pull/5178#issuecomment-387556158
 
 
   Hey @kvncp , any updates on this one?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99742)
Time Spent: 50m  (was: 40m)

> Allow BigQueryIO to use a different project for the load job in batch mode.
> ---
>
> Key: BEAM-3433
> URL: https://issues.apache.org/jira/browse/BEAM-3433
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Peterson
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> BigQueryIO is currently configured to always run a batch load job using the 
> same projects as the destination table: 
> https://github.com/apache/beam/blob/192b4c70927901860312f8c8acd27bd47e4a4259/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L256
> This may not always be desirable, since a pipeline may have write access to a 
> dataset in a different project, but not jobs.create access in that project. 
> This parameter should be settable in the interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #133

[Apache Jenkins Server]

See 


--
GitHub pull request #5283 of commit a14dfd35f65d89a232fdb9548bcdf8eede056f1f, 
no merge conflicts.
Setting status of a14dfd35f65d89a232fdb9548bcdf8eede056f1f to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/133/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesContainer 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5283/*:refs/remotes/origin/pr/5283/*
 > git rev-parse refs/remotes/origin/pr/5283/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5283/merge^{commit} # timeout=10
Checking out Revision 6b120eddd6c0d00f8793df2b382e0125a49c9b16 
(refs/remotes/origin/pr/5283/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6b120eddd6c0d00f8793df2b382e0125a49c9b16
Commit message: "Merge a14dfd35f65d89a232fdb9548bcdf8eede056f1f into 
e7a3578abb919021be95c9c7148fd2d4fc6331fd"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4668147772686800911.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Create a tarball
./gradlew :beam-sdks-python:clean --info
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project 

[jira] [Work logged] (BEAM-4061) Chaining SpannerIO#write() transforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4061?focusedWorklogId=99741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99741
 ]

ASF GitHub Bot logged work on BEAM-4061:


Author: ASF GitHub Bot
Created on: 08/May/18 21:59
Start Date: 08/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #4264: [BEAM-4061] Introduced 
SpannerWriteResult
URL: https://github.com/apache/beam/pull/4264
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index b950964411c..c3924d861f3 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -23,12 +23,12 @@
 import com.google.auto.value.AutoValue;
 import com.google.cloud.ServiceFactory;
 import com.google.cloud.Timestamp;
-import com.google.cloud.spanner.AbortedException;
 import com.google.cloud.spanner.Key;
 import com.google.cloud.spanner.KeySet;
 import com.google.cloud.spanner.Mutation;
 import com.google.cloud.spanner.PartitionOptions;
 import com.google.cloud.spanner.Spanner;
+import com.google.cloud.spanner.SpannerException;
 import com.google.cloud.spanner.SpannerOptions;
 import com.google.cloud.spanner.Statement;
 import com.google.cloud.spanner.Struct;
@@ -47,6 +47,7 @@
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.ApproximateQuantiles;
@@ -60,16 +61,15 @@
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.Wait;
 import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.sdk.util.BackOff;
-import org.apache.beam.sdk.util.BackOffUtils;
-import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.sdk.util.Sleeper;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PDone;
-import org.joda.time.Duration;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Experimental {@link PTransform Transforms} for reading from and writing to 
, 
PDone> {
+  public abstract static class Write extends PTransform {
 
 abstract SpannerConfig getSpannerConfig();
 
@@ -659,6 +670,8 @@ public CreateTransaction withTimestampBound(TimestampBound 
timestampBound) {
 
 abstract int getNumSamples();
 
+abstract FailureMode getFailureMode();
+
 @Nullable
  abstract PTransform>, 
PCollection>, PCollection

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1561

[Apache Jenkins Server]

See 


Changes:

[mairbek] Introduced SpannerWriteResult that

[mairbek] Addressed comments

[mairbek] Happy checkstyle

--
Started by GitHub push by jkff
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e7a3578abb919021be95c9c7148fd2d4fc6331fd (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e7a3578abb919021be95c9c7148fd2d4fc6331fd
Commit message: "Merge pull request #4264: [BEAM-4061] Introduced 
SpannerWriteResult"
 > git rev-list --no-walk 4a9b82625b3242f7561ac4a0b92bb075ecc0cd84 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins3959255492282716597.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user schroed...@google.com

[beam] branch master updated (4a9b826 -> e7a3578)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4a9b826  Merge pull request #5305: Enable githubCommitNotifier for 
post commits
 add f914fa7  Introduced SpannerWriteResult that
 add 553af41  Addressed comments
 add ee288f2  Happy checkstyle
 new e7a3578  Merge pull request #4264: [BEAM-4061] Introduced 
SpannerWriteResult

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java  | 134 +
 .../SpannerWriteResult.java}   |  51 
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java |  44 +++
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java| 125 ---
 4 files changed, 260 insertions(+), 94 deletions(-)
 copy 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/{bigquery/WriteResult.java
 => spanner/SpannerWriteResult.java} (56%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-4061) Chaining SpannerIO#write() transforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4061?focusedWorklogId=99740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99740
 ]

ASF GitHub Bot logged work on BEAM-4061:


Author: ASF GitHub Bot
Created on: 08/May/18 21:59
Start Date: 08/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #4264: [BEAM-4061] Introduced 
SpannerWriteResult
URL: https://github.com/apache/beam/pull/4264#issuecomment-387555870
 
 
   OK, the test failure is flaky, and I confirmed locally that the tests pass. 
Merging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99740)
Time Spent: 3h  (was: 2h 50m)

> Chaining SpannerIO#write() transforms
> -
>
> Key: BEAM-4061
> URL: https://issues.apache.org/jira/browse/BEAM-4061
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Mairbek Khadikov
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> It should be possible to chain several Cloud Spanner writes. In practice, we 
> can leverage Wait.on transform by returning a result object from 
> SpannerIO#write.
> One particular example, when this feature is useful is full database import. 
> When data in parent tables should be injected before interleaved tables. See 
> more about table hierarchies in Spanner here 
> https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99738
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 21:55
Start Date: 08/May/18 21:55
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387554832
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99738)
Time Spent: 16.5h  (was: 16h 20m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4898

[Apache Jenkins Server]

See 


Changes:

[apilloud] Enable githubCommitNotifier for post commits

--
[...truncated 1.18 MB...]
  "is_pair_like": true
}, 
{
  "@type": "kind:interval_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "count.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s6"
}, 
"serialized_fn": "ref_AppliedPTransform_count_10", 
"user_name": "count"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s8", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "format_result"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:interval_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "ref_AppliedPTransform_format_11", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s9", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": "kind:interval_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "WriteStringsToPubSub/EncodeString.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s8"
}, 
"serialized_fn": 
"ref_AppliedPTransform_WriteStringsToPubSub/EncodeString_13", 
"user_name": "WriteStringsToPubSub/EncodeString"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s10", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"format": "pubsub", 
"parallel_input": {
  "@type": 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #315

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2594) Python shim for submitting to the ULR

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467995#comment-16467995
 ] 

Ben Sidhom commented on BEAM-2594:
--

I'm not sure exactly what this entails. It depends exactly what we want the 
"ULR" to be responsible for. To start with, I think it's reasonable to just 
submit to a running job service with the appropriate pipeline options. 
Eventually, it should actually spin up the job service it will use. However, 
before it can actually start up any job services, we need to nail down our job 
service "entry point" story.

> Python shim for submitting to the ULR
> -
>
> Key: BEAM-2594
> URL: https://issues.apache.org/jira/browse/BEAM-2594
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: portability
>
> Python SDK should support submission of portable pipelines to the ULR, as per 
> https://s.apache.org/beam-job-api.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99732
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 21:25
Start Date: 08/May/18 21:25
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5290: [BEAM-3983] 
Restore BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387547506
 
 
   OK well I think what we really want is a separate job so we don't pollute 
the build history anyhow (aka "run java ITs" where both 
job_beam_precommit_java_ITs and job_beam_postcommit_java_ITs would share the 
definition of which ITs)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99732)
Time Spent: 16h 20m  (was: 16h 10m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 20m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=99728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99728
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 08/May/18 21:20
Start Date: 08/May/18 21:20
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5290: [BEAM-3983] Restore 
BigQuery SQL Support with copied enums
URL: https://github.com/apache/beam/pull/5290#issuecomment-387546062
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 99728)
Time Spent: 16h 10m  (was: 16h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 16h 10m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #314

[Apache Jenkins Server]

See 


--
[...truncated 18.33 MB...]
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.
May 08, 2018 9:18:55 PM org.apache.beam.sdk.metrics.MetricsEnvironment 
getCurrentContainer
WARNING: Reporting metrics are not supported in the current execution 
environment.

org.apache.beam.examples.cookbook.MaxPerKeyExamplesTest > testFormatMaxesFn 
STANDARD_ERROR
May 08, 2018 9:18:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.MaxPerKeyExamplesTest > testExtractTempFn 
STANDARD_ERROR
May 08, 2018 9:18:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testFormatCounts 
STANDARD_ERROR
May 08, 2018 9:18:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testExtractTornadoes 
STANDARD_ERROR
May 08, 2018 9:18:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.BigQueryTornadoesTest > testNoTornadoes 
STANDARD_ERROR
May 08, 2018 9:18:55 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.TriggerExampleTest > testExtractTotalFlow 
STANDARD_ERROR
May 08, 2018 9:18:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > 
testFilterSingleMonthDataFn STANDARD_ERROR
May 08, 2018 9:18:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
May 08, 2018 9:18:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
May 08, 2018 9:18:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
May 08, 2018 9:18:57 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
May 08, 2018 9:18:58 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit1205534688646397911/junit1248993894376922863.tmp matched 1 files with 
total size 54
May 08, 2018 9:18:58 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit1205534688646397911/junit1248993894376922863.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
May 08, 2018 9:18:58 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
May 08, 2018 9:18:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo7124570876517926590.sh 
May 08, 2018 9:18:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
May 08, 2018 9:18:59 PM org.apache.beam.examples.subprocess.utils.FileUtils 
copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo7124570876517926590.sh to 
/tmp/test-Echoo3899543248381835617/test-Echo7124570876517926590.sh 
May 08, 2018 9:18:59 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain8492113030457273721.sh 
May 08, 2018 9:18:59 PM 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1560

[Apache Jenkins Server]

See 


Changes:

[apilloud] Enable githubCommitNotifier for post commits

--
Started by GitHub push by jasonkuster
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4a9b82625b3242f7561ac4a0b92bb075ecc0cd84 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4a9b82625b3242f7561ac4a0b92bb075ecc0cd84
Commit message: "Merge pull request #5305: Enable githubCommitNotifier for post 
commits"
 > git rev-list --no-walk 08c91f6e80cae7f322a243e547f971743563fac2 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins9031397058745956926.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ro...@frantil.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user 
re...@relax-macbookpro2.roam.corp.google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user pger...@us.ibm.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user jasonkus...@google.com
Not sending mail to unregistered user axelm...@gmail.com
Not sending mail to unregistered user sweg...@google.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user aal...@gmail.com
Not sending mail to unregistered user szewi...@gmail.com
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user ankurgoe...@gmail.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user katarzyna.kucharc...@polidea.com
Not sending mail to unregistered user schroed...@google.com