Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4845

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3574) [SQL] Support schema qualifiers for field names

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3574:
--
Description: 
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate field names, and without 
any means of distinguishing them or getting a value from specific field by 
name. We don't keep any metadata in BeamRecordType to help identify a field in 
this case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 

  was:
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate field names, and without 
any means of distinguishing them and getting a value from specific field by 
name. We don't keep any metadata in BeamRecordType to help identify a field in 
this case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 


> [SQL] Support schema qualifiers for field names
> ---
>
> Key: BEAM-3574
> URL: https://issues.apache.org/jira/browse/BEAM-3574
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Currently there are utility methods in BeamRecord to get field values by 
> name, e.g. BeamRecord.getFieldValue(String name). Internally they call 
> fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.
> This works as long as there is only one field with such name in the record. 
> But when joining 2 records you can end up with duplicate field names, and 
> without any means of distinguishing them or getting a value from specific 
> field by name. We don't keep any metadata in BeamRecordType to help identify 
> a field in this case. 
> It feels that this can lead to obscure bugs.
> We probably should keep more detailed schema information attached to the 
> fields, so that we could reference them using qualifiers like 
> "[schemaA].[pcollectionB].[fieldC]".
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3574) [SQL] Support schema qualifiers for field names

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3574:
--
Description: 
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate field names, and without 
any means of distinguishing them and getting a value from specific field by 
name. We don't keep any metadata in BeamRecordType to help identify a field in 
this case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 

  was:
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate field nameswithout any 
means of distinguishing them and getting a value from specific field by name. 
We don't keep any metadata in BeamRecordType to help identify a field in this 
case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 


> [SQL] Support schema qualifiers for field names
> ---
>
> Key: BEAM-3574
> URL: https://issues.apache.org/jira/browse/BEAM-3574
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Currently there are utility methods in BeamRecord to get field values by 
> name, e.g. BeamRecord.getFieldValue(String name). Internally they call 
> fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.
> This works as long as there is only one field with such name in the record. 
> But when joining 2 records you can end up with duplicate field names, and 
> without any means of distinguishing them and getting a value from specific 
> field by name. We don't keep any metadata in BeamRecordType to help identify 
> a field in this case. 
> It feels that this can lead to obscure bugs.
> We probably should keep more detailed schema information attached to the 
> fields, so that we could reference them using qualifiers like 
> "[schemaA].[pcollectionB].[fieldC]".
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3574) [SQL] Support schema qualifiers for field names

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3574:
-

 Summary: [SQL] Support schema qualifiers for field names
 Key: BEAM-3574
 URL: https://issues.apache.org/jira/browse/BEAM-3574
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Anton Kedin


Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate fields without any means 
of distinguishing them and getting a value from specific field by name. We 
don't keep any metadata in BeamRecordType to help identify a field in this 
case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3574) [SQL] Support schema qualifiers for field names

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3574:
--
Description: 
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate field nameswithout any 
means of distinguishing them and getting a value from specific field by name. 
We don't keep any metadata in BeamRecordType to help identify a field in this 
case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 

  was:
Currently there are utility methods in BeamRecord to get field values by name, 
e.g. BeamRecord.getFieldValue(String name). Internally they call 
fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But 
when joining 2 records you can end up with duplicate fields without any means 
of distinguishing them and getting a value from specific field by name. We 
don't keep any metadata in BeamRecordType to help identify a field in this 
case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the 
fields, so that we could reference them using qualifiers like 
"[schemaA].[pcollectionB].[fieldC]".

 


> [SQL] Support schema qualifiers for field names
> ---
>
> Key: BEAM-3574
> URL: https://issues.apache.org/jira/browse/BEAM-3574
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Currently there are utility methods in BeamRecord to get field values by 
> name, e.g. BeamRecord.getFieldValue(String name). Internally they call 
> fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.
> This works as long as there is only one field with such name in the record. 
> But when joining 2 records you can end up with duplicate field nameswithout 
> any means of distinguishing them and getting a value from specific field by 
> name. We don't keep any metadata in BeamRecordType to help identify a field 
> in this case. 
> It feels that this can lead to obscure bugs.
> We probably should keep more detailed schema information attached to the 
> fields, so that we could reference them using qualifiers like 
> "[schemaA].[pcollectionB].[fieldC]".
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3345

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #856

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5499283578059472838.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins527850684334946878.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1097187561501222493.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7306039795591851249.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2310814785634687574.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3740887293761302288.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #86

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3074574041432318959.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8263755375435022907.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8850893343154091176.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4172879130382086019.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6434471366690219720.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Downloading/unpacking absl-py (from -r PerfKitBenchmarker/requirements.txt 
(line 14))
  Running setup.py 
(path:
 egg_info for package absl-py

Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Downloading/unpacking colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Downloading colorlog-2.6.0-py2.py3-none-any.whl
Downloading/unpacking blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt 
(line 18))
  Running setup.py 
(path:
 egg_info for package blinker

Downloading/unpacking futures>=3.0.3 (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
  Downloading futures-3.2.0-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Downloading/unpacking pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt 
(line 21))
  Running setup.py 
(path:
 egg_info for package pint

no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
warning: no previously-included files matching '*~' found anywhere in 
distribution
warning: no previously-included files matching '.DS_Store' found anywhere 
in distribution
warning: no previously-included files matching '*__pycache__*' found 
anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in 
distribution
warning: no previously-included files matching '.travis-exclude.yml' found 
anywhere in distribution
Downloading/unpacking numpy (from -r PerfKitBenchmarker/requirements.txt (line 
22))
  Running setup.py 
(path:
 egg_info for package numpy
Running 

Build failed in Jenkins: beam_PerformanceTests_Spark #1300

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1785580539226881264.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7529992403315819531.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7269213716009218919.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7193372487339830744.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5170549270468161894.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #84

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2060408636259449725.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins129536160906735337.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5201035847705142276.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1301156985999643135.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8930084552274670034.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7101817040538182008.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #100

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8545714045679161033.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins399266992076426348.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8331163036319532401.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins597711880646903.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5535651422161936249.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #83

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

[kedin] [Nexmark][SQL] Implement sql query 3

[tgroh] Add CoderTranslatorRegistrar

[chamikara] Updates PTransform overriding to create a new AppliedPTransform 
object

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 645e1b51e1c83ec4b0b54a774d6615624f3c42ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Commit message: "Merge pull request #4448: [SQL] Refactor Variance"
 > git rev-list 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2927773544539186127.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5480544320425912715.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5577185747979399113.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6186452495235065801.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3895282036698786580.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Downloading/unpacking absl-py (from -r PerfKitBenchmarker/requirements.txt 
(line 14))
  Running setup.py 
(path:
 egg_info for package absl-py

Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Downloading/unpacking colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Downloading colorlog-2.6.0-py2.py3-none-any.whl
Downloading/unpacking blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt 
(line 18))
  Running setup.py 
(path:
 egg_info for package blinker

Downloading/unpacking futures>=3.0.3 (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
  Downloading futures-3.2.0-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Downloading/unpacking pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt 
(line 21))
  Running setup.py 
(path:
 egg_info for package pint

no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
warning: no previously-included files matching '*~' found anywhere in 
distribution
warning: no previously-included files matching '.DS_Store' found anywhere 
in distribution
warning: no previously-included files matching '*__pycache__*' found 
anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in 
distribution
warning: no previously-included files matching '.travis-exclude.yml' found 
anywhere in distribution
Downloading/unpacking numpy (from -r PerfKitBenchmarker/requirements.txt (line 
22))
  Running setup.py 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4844

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3321) Update gax-grpc dependency to latest

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346274#comment-16346274
 ] 

Jacob Marble commented on BEAM-3321:


I ended up moving my beam-sdks-java-io-google-cloud-platform code in a project 
separate from my google-cloud-storage code.

> Update gax-grpc dependency to latest
> 
>
> Key: BEAM-3321
> URL: https://issues.apache.org/jira/browse/BEAM-3321
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Mairbek Khadikov
>Priority: Major
>
> Beam uses 0.20 latest release is 1.15: 
> https://mvnrepository.com/artifact/com.google.api/gax-grpc
> This results in compatibility issues when Beam and Datastore client are used 
> together: 
> https://lists.apache.org/thread.html/8a99f4c0dbb1138f1067ef73b2ba7f44f035b309af41b1876ffc1ff2@%3Cuser.beam.apache.org%3E
> Assigning to Mairbek who added this dependency recently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5819

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Refactor Variance

--
[...truncated 1.27 MB...]
2018-01-31T05:00:07.910 [INFO] Excluding com.google.inject:guice:jar:3.0 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding javax.inject:javax.inject:jar:1 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding aopalliance:aopalliance:jar:1.0 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 
from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.sun.jersey.contribs:jersey-guice:jar:1.9 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding jline:jline:jar:2.11 from the shaded 
jar.
2018-01-31T05:00:07.910 [INFO] Excluding org.apache.ant:ant:jar:1.9.2 from the 
shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding org.apache.ant:ant-launcher:jar:1.9.2 
from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding net.engio:mbassador:jar:1.1.9 from the 
shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding net.lingala.zip4j:zip4j:jar:1.3.2 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding commons-codec:commons-codec:jar:1.10 
from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.3 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding org.jctools:jctools-core:jar:1.1 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
org.apache.beam:beam-model-pipeline:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Including com.google.guava:guava:jar:20.0 in the 
shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1 from the shaded 
jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.6.8 
from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-01-31T05:00:07.910 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-01-31T05:00:07.910 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.14 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.4.0-SNAPSHOT from the 
shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
org.apache.beam:beam-model-job-management:jar:2.4.0-SNAPSHOT from the shaded 
jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
org.apache.beam:beam-runners-core-java:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.6 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2018-01-31T05:00:07.911 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2018-01-31T05:00:09.874 [INFO] Replacing original artifact with shaded artifact.
2018-01-31T05:00:09.874 [INFO] Replacing 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3344

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3573) Test jars should export only tests

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3573:
-

 Summary: Test jars should export only tests
 Key: BEAM-3573
 URL: https://issues.apache.org/jira/browse/BEAM-3573
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Today, we have test-jars that are used as libraries for testing. That is not 
what "test jar" means, and dependency management actually does not work 
correctly for this. It is OK to depend on a test jar in order to run the tests 
therein, and not really OK to depend on one for another reason.

This ticket is a bucket ticket for fixes to this situation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (53466d1 -> 645e1b5)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 53466d1  Merge pull request #4535:[BEAM-3512] Updates PTransform 
overriding to create new AppliedPTransform objects
 add e8c663d  [SQL] Refactor Variance
 new 645e1b5  Merge pull request #4448: [SQL] Refactor Variance

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../impl/transform/BeamAggregationTransforms.java  |  22 +-
 .../impl/transform/BeamBuiltinAggregations.java| 270 +
 .../impl/transform/agg/BigDecimalConverter.java|  55 +
 .../impl/transform/agg/VarianceAccumulator.java|  97 
 .../sql/impl/transform/agg/VarianceFn.java | 138 +++
 .../sql/impl/transform/agg}/package-info.java  |   4 +-
 .../sql/BeamSqlDslAggregationVarianceTest.java | 114 +
 .../transform/agg/BigDecimalConverterTest.java |  64 +
 .../transform/agg/VarianceAccumulatorTest.java | 132 ++
 .../sql/impl/transform/agg/VarianceFnTest.java | 104 
 .../extensions/sql/utils/BeamRecordAsserts.java|  58 +
 11 files changed, 786 insertions(+), 272 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/BigDecimalConverter.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceAccumulator.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
 copy 
{runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/metrics
 => 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg}/package-info.java
 (86%)
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/BigDecimalConverterTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceAccumulatorTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFnTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/BeamRecordAsserts.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4448: [SQL] Refactor Variance

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 645e1b51e1c83ec4b0b54a774d6615624f3c42ed
Merge: 53466d1 e8c663d
Author: Kenn Knowles 
AuthorDate: Tue Jan 30 20:03:33 2018 -0800

Merge pull request #4448: [SQL] Refactor Variance

 .../impl/transform/BeamAggregationTransforms.java  |  22 +-
 .../impl/transform/BeamBuiltinAggregations.java| 270 +
 .../impl/transform/agg/BigDecimalConverter.java|  55 +
 .../impl/transform/agg/VarianceAccumulator.java|  97 
 .../sql/impl/transform/agg/VarianceFn.java | 138 +++
 .../sql/impl/transform/agg/package-info.java   |  26 ++
 .../sql/BeamSqlDslAggregationVarianceTest.java | 114 +
 .../transform/agg/BigDecimalConverterTest.java |  64 +
 .../transform/agg/VarianceAccumulatorTest.java | 132 ++
 .../sql/impl/transform/agg/VarianceFnTest.java | 104 
 .../extensions/sql/utils/BeamRecordAsserts.java|  58 +
 11 files changed, 810 insertions(+), 270 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam-site] branch snapshot-release created (now 9464aed)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch snapshot-release
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


  at 9464aed  Add nightly snapshot build to the release guide

This branch includes the following new commits:

 new 9464aed  Add nightly snapshot build to the release guide

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam-site] 01/01: Add nightly snapshot build to the release guide

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch snapshot-release
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9464aed94aab143564606c21af464a6275fc56ed
Author: Kenn Knowles 
AuthorDate: Tue Jan 30 19:57:45 2018 -0800

Add nightly snapshot build to the release guide
---
 src/contribute/release-guide.md | 28 ++--
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md
index ea6..7feeba7 100644
--- a/src/contribute/release-guide.md
+++ b/src/contribute/release-guide.md
@@ -226,21 +226,29 @@ In the release branch, update the Python SDK version to 
the release version (e.g
 
 1. Update archetypes:

[example](https://github.com/apache/beam/commit/d375cfa126fd7be9c34f39c2b9b856f324bf)
-1. Update runner specific configurations:
+2. Update runner specific configurations:

[example](https://github.com/apache/beam/commit/f572328ce23e70adee8001e3d10f1479bd9a380d)
 
+### Start a snapshot build
+
+Start a build of [the nightly 
snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/).
+Some processes, including our archetype tests, rely on having a live SNAPSHOT 
of the current version
+from the `master` branch. Once the release branch is cut, these SNAPSHOT 
versions are no longer found,
+so builds will be broken until a new snapshot is available.
+
 ### Checklist to proceed to the next step
 
 1. Release Manager’s GPG key is published to `dist.apache.org`
-1. Release Manager’s GPG key is configured in `git` configuration
-1. Release Manager has `org.apache.beam` listed under `Staging Profiles` in 
Nexus
-1. Release Manager’s Nexus User Token is configured in `settings.xml`
-1. JIRA release item for the subsequent release has been created
-1. There are no release blocking JIRA issues
-1. Release Notes in JIRA have been audited and adjusted
-1. Combined javadoc has the appropriate contents.
-1. Release branch has been created
-1. Originating branch has the version information updated to the new version
+2. Release Manager’s GPG key is configured in `git` configuration
+3. Release Manager has `org.apache.beam` listed under `Staging Profiles` in 
Nexus
+4. Release Manager’s Nexus User Token is configured in `settings.xml`
+5. JIRA release item for the subsequent release has been created
+6. There are no release blocking JIRA issues
+7. Release Notes in JIRA have been audited and adjusted
+8. Combined javadoc has the appropriate contents.
+9. Release branch has been created
+10. Originating branch has the version information updated to the new version
+11. Nightly snapshot is in progress (do revisit it continually)
 
 **
 

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4535:[BEAM-3512] Updates PTransform overriding to create new AppliedPTransform objects

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 53466d161b432c2a95535bca85294f44ad477967
Merge: a14b34e dac0e3f
Author: Chamikara Jayalath 
AuthorDate: Tue Jan 30 19:51:50 2018 -0800

Merge pull request #4535:[BEAM-3512] Updates PTransform overriding to 
create new AppliedPTransform objects

 sdks/python/apache_beam/pipeline.py | 64 +
 1 file changed, 44 insertions(+), 20 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (a14b34e -> 53466d1)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a14b34e  Merge pull request #4352: [BEAM-3182][Nexmark][SQL] Implement 
sql query 3
 add dac0e3f  Updates PTransform overriding to create a new 
AppliedPTransform object instead of updating the existing node. I think this 
approach is cleaner.
 new 53466d1  Merge pull request #4535:[BEAM-3512] Updates PTransform 
overriding to create new AppliedPTransform objects

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline.py | 64 +
 1 file changed, 44 insertions(+), 20 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (750981a -> a14b34e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 750981a  Merge pull request #4496: Add CoderTranslatorRegistrar
 add 0b033da  [Nexmark][SQL] Implement sql query 3
 new a14b34e  Merge pull request #4352: [BEAM-3182][Nexmark][SQL] Implement 
sql query 3

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/nexmark/NexmarkLauncher.java   |   4 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java| 148 +
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java | 147 
 3 files changed, 298 insertions(+), 1 deletion(-)
 create mode 100644 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3.java
 create mode 100644 
sdks/java/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3Test.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4352: [BEAM-3182][Nexmark][SQL] Implement sql query 3

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a14b34ebd52d3d169e6353cd1c216ca4b1411c10
Merge: 750981a 0b033da
Author: Kenn Knowles 
AuthorDate: Tue Jan 30 19:40:45 2018 -0800

Merge pull request #4352: [BEAM-3182][Nexmark][SQL] Implement sql query 3

 .../apache/beam/sdk/nexmark/NexmarkLauncher.java   |   4 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java| 148 +
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java | 147 
 3 files changed, 298 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5815

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4842

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3321) Update gax-grpc dependency to latest

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346127#comment-16346127
 ] 

Jacob Marble commented on BEAM-3321:


Are there any common workarounds for this bug?

> Update gax-grpc dependency to latest
> 
>
> Key: BEAM-3321
> URL: https://issues.apache.org/jira/browse/BEAM-3321
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Mairbek Khadikov
>Priority: Major
>
> Beam uses 0.20 latest release is 1.15: 
> https://mvnrepository.com/artifact/com.google.api/gax-grpc
> This results in compatibility issues when Beam and Datastore client are used 
> together: 
> https://lists.apache.org/thread.html/8a99f4c0dbb1138f1067ef73b2ba7f44f035b309af41b1876ffc1ff2@%3Cuser.beam.apache.org%3E
> Assigning to Mairbek who added this dependency recently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #5814

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #855

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add slf4j_simple to the top level Gradle build

[tgroh] Implement FnService in FnApiControlClientPoolService

[tgroh] Add a Timeout to GrpcDataService#send

[tgroh] Use a Data Service in SdkHarnessClient

[XuMingmin] [BEAM-3525] Fix KafkaIO metric (#4524)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4
Commit message: "[BEAM-3525] Fix KafkaIO metric (#4524)"
 > git rev-list a7226cba18fb0366bc8b1d516ccab1fa65d753ed # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3405383127224712091.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1526764464314917463.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6993315847457157443.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins152595654224580139.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8022485230757275373.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2377601178439960802.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 

Jenkins build is back to normal : beam_PerformanceTests_Spark #1299

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4841

[Apache Jenkins Server]

See 

[beam] branch master updated: [BEAM-3525] Fix KafkaIO metric (#4524)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 34d8bc9  [BEAM-3525] Fix KafkaIO metric (#4524)
34d8bc9 is described below

commit 34d8bc9079a5a4b3d3a5476b2d4b48072be0dff4
Author: Raghu Angadi 
AuthorDate: Tue Jan 30 15:39:29 2018 -0800

[BEAM-3525] Fix KafkaIO metric (#4524)

* Fix a KafkaIO metric

Couple of fixes :

'checkpointMarkCommits' was incremented outside a Beam API context (inside 
consumerPollLoop),
it is not supported. Instead, increment new metric 'enqueued'. 'enqueued' - 
'skipped' gives
total number of actual commits.

Reverted an erlier PR that removed a test dependendency. It is required for 
tests in order
to see test output.

* Verify commits metric in unit test.
---
 sdks/java/io/kafka/pom.xml |  6 ++
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 18 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 14 ++
 3 files changed, 29 insertions(+), 9 deletions(-)

diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index 38afa00..b04f5bf 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -127,5 +127,11 @@
   junit
   test
 
+
+
+  org.slf4j
+  slf4j-jdk14
+  test
+
   
 
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 83d702c..996a460 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -911,9 +911,9 @@ public class KafkaIO {
 @VisibleForTesting
 static final String METRIC_NAMESPACE = "KafkaIOReader";
 @VisibleForTesting
-static final String CHECKPOINT_MARK_COMMITS_METRIC = 
"checkpointMarkCommits";
-private static final String CHECKPOINT_MARK_COMMIT_SKIPS_METRIC = 
"checkpointMarkCommitSkips";
-
+static final String CHECKPOINT_MARK_COMMITS_ENQUEUED_METRIC = 
"checkpointMarkCommitsEnqueued";
+private static final String CHECKPOINT_MARK_COMMITS_SKIPPED_METRIC =
+  "checkpointMarkCommitsSkipped";
 
 private final UnboundedKafkaSource source;
 private final String name;
@@ -932,11 +932,11 @@ public class KafkaIO {
 private final Counter bytesReadBySplit;
 private final Gauge backlogBytesOfSplit;
 private final Gauge backlogElementsOfSplit;
-private final Counter checkpointMarkCommits = Metrics.counter(
-  METRIC_NAMESPACE, CHECKPOINT_MARK_COMMITS_METRIC);
+private final Counter checkpointMarkCommitsEnqueued = Metrics.counter(
+  METRIC_NAMESPACE, CHECKPOINT_MARK_COMMITS_ENQUEUED_METRIC);
 // Checkpoint marks skipped in favor of newer mark (only the latest needs 
to be committed).
-private final Counter checkpointMarkCommitSkips = Metrics.counter(
-  METRIC_NAMESPACE, CHECKPOINT_MARK_COMMIT_SKIPS_METRIC);
+private final Counter checkpointMarkCommitsSkipped = Metrics.counter(
+  METRIC_NAMESPACE, CHECKPOINT_MARK_COMMITS_SKIPPED_METRIC);
 
 /**
  * The poll timeout while reading records from Kafka.
@@ -1130,7 +1130,6 @@ public class KafkaIO {
 p -> new OffsetAndMetadata(p.getNextOffset())
   ))
   );
-  checkpointMarkCommits.inc();
 }
 
 /**
@@ -1142,8 +1141,9 @@ public class KafkaIO {
  */
 void finalizeCheckpointMarkAsync(KafkaCheckpointMark checkpointMark) {
   if (finalizedCheckpointMark.getAndSet(checkpointMark) != null) {
-checkpointMarkCommitSkips.inc();
+checkpointMarkCommitsSkipped.inc();
   }
+  checkpointMarkCommitsEnqueued.inc();
 }
 
 private void nextBatch() {
diff --git 
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
 
b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
index 08b9d7c..08338d8 100644
--- 
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
+++ 
b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.io.kafka;
 
 import static 
org.apache.beam.sdk.metrics.MetricResultsMatchers.attemptedMetricsResult;
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.hasItem;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertNull;
@@ -714,6 +715,19 @@ public class KafkaIOTest {
 
 // since gauge values may be inconsistent in some environments assert only 
on their existence.
 assertThat(backlogBytesMetrics.gauges(), 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5813

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add slf4j_simple to the top level Gradle build

[tgroh] Implement FnService in FnApiControlClientPoolService

[tgroh] Add a Timeout to GrpcDataService#send

[tgroh] Use a Data Service in SdkHarnessClient

--
[...truncated 1.51 MB...]
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.spark:spark-launcher_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.spark:spark-network-common_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.9.3 
from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding javax.activation:activation:jar:1.1.1 
from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding mx4j:mx4j:jar:3.0.2 from the shaded 
jar.
2018-01-30T23:20:00.986 [INFO] Excluding javax.mail:mail:jar:1.4.7 from the 
shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.bouncycastle:bcprov-jdk15on:jar:1.51 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
com.jamesmurty.utils:java-xmlbuilder:jar:1.0 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding net.iharder:base64:jar:2.3.8 from the 
shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.6.0 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.6.0 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
javax.servlet:javax.servlet-api:jar:3.1.0 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.16 from 
the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.16 
from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2018-01-30T23:20:00.986 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.16 
from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.11.8 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding 
org.json4s:json4s-jackson_2.11:jar:3.2.11 from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding org.json4s:json4s-core_2.11:jar:3.2.11 
from the shaded jar.
2018-01-30T23:20:00.986 [INFO] Excluding org.json4s:json4s-ast_2.11:jar:3.2.11 
from the shaded jar.

[jira] [Commented] (BEAM-3572) Reduce inefficient allocations in coders

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345912#comment-16345912
 ] 

Kenneth Knowles commented on BEAM-3572:
---

I think I see what you mean in terms of excess allocation. The buffering was 
added as an optimization :-)

While the {{Coder}} itself should be observably immutable, there is no problem 
with mutation under the hood to manage a pool of buffers. The real issue, which 
you alluded to, is that coders are required to be thread safe. The reason that 
{{BufferedElementCountingOutputStream}} can be used despite lack of thread 
safety is that it is only local.

Having either {{IterableLikeCoder}} or {{BufferedElementCountingOutputStream}} 
do their own suballocation makes sense, with the usual caveats of bugs and 
leaks from that sort of code. Definitely better encapsulation for 
{{BufferedElementCountingOutputStream}} to own it unless it doesn't have enough 
info to do it well. I'm willing to trust that you came to this because you 
actually hit this in practice, or are at least driven by a benchmark.

> Reduce inefficient allocations in coders
> 
>
> Key: BEAM-3572
> URL: https://issues.apache.org/jira/browse/BEAM-3572
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> BufferedElementCountingOutputStream's constructor allocates a new buffer to 
> wrap the input OutputStream. This gets called on each invocation of encode() 
> from IterableLikeCoder. Since Coder is designed to be stateless, but this 
> buffer holds state and isn't threadsafe, we can't just have the caller manage 
> the buffer. Modifying the constructor to use a pool of buffers to reduce the 
> number of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Qi Cui (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345910#comment-16345910
 ] 

Qi Cui edited comment on BEAM-3531 at 1/30/18 10:19 PM:


Created PR:

[https://github.com/apache/beam/pull/4541]


was (Author: jackyq2015):
Create PR:

https://github.com/apache/beam/pull/4541

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run 

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Qi Cui (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345910#comment-16345910
 ] 

Qi Cui commented on BEAM-3531:
--

Create PR:

https://github.com/apache/beam/pull/4541

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run ***
> ==
> [WARNING] 
> 

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345899#comment-16345899
 ] 

Ismaël Mejía commented on BEAM-3531:


I think the best thing for the moment is to create a quick fix (3), maybe 
[~jackyq2015] can contribute this just by doing a work around by just adding 
the extra null validation.

[~kedin] since you are working on a proper fix we can maybe create a Jira to 
fix this in a proper way. (1) sounds reasonable. Extra comment, pay attention 
to update the documentation in the website in case some of the 
refactorings/improvements you are working on change the current behavior, it 
would be also nice if you add some doc on the SQL benchmarks and how to run 
them.

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> 

[beam] 01/01: Merge pull request #4472

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit eddfa0d92779f636458954a9292fc0bfa965a123
Merge: a7226cb 16aefc1
Author: Thomas Groh 
AuthorDate: Tue Jan 30 14:02:21 2018 -0800

Merge pull request #4472

[BEAM-3326] Use a Data Service in SdkHarnessClient

 build.gradle   |   1 +
 runners/java-fn-execution/build.gradle |   2 +
 runners/java-fn-execution/pom.xml  |  16 ++
 .../control/FnApiControlClientPoolService.java |   9 +-
 .../fnexecution/control/SdkHarnessClient.java  | 126 +++
 .../runners/fnexecution/data/GrpcDataService.java  |   8 +-
 .../fnexecution/control/SdkHarnessClientTest.java  | 231 -
 .../apache/beam/fn/harness/FnApiDoFnRunner.java|   8 +-
 8 files changed, 345 insertions(+), 56 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (a7226cb -> eddfa0d)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a7226cb  Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo
 add b50abc8  Add slf4j_simple to the top level Gradle build
 add 8d5fdae  Implement FnService in FnApiControlClientPoolService
 add 8e8386e  Add a Timeout to GrpcDataService#send
 add 16aefc1  Use a Data Service in SdkHarnessClient
 new eddfa0d  Merge pull request #4472

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build.gradle   |   1 +
 runners/java-fn-execution/build.gradle |   2 +
 runners/java-fn-execution/pom.xml  |  16 ++
 .../control/FnApiControlClientPoolService.java |   9 +-
 .../fnexecution/control/SdkHarnessClient.java  | 126 +++
 .../runners/fnexecution/data/GrpcDataService.java  |   8 +-
 .../fnexecution/control/SdkHarnessClientTest.java  | 231 -
 .../apache/beam/fn/harness/FnApiDoFnRunner.java|   8 +-
 8 files changed, 345 insertions(+), 56 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Updated] (BEAM-3572) Reduce inefficient allocations in coders

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer updated BEAM-3572:

Description: BufferedElementCountingOutputStream's constructor allocates a 
new buffer to wrap the input OutputStream. This gets called on each invocation 
of encode() from IterableLikeCoder. Since Coder is designed to be stateless, 
but this buffer holds state and isn't threadsafe, we can't just have the caller 
manage the buffer. Modifying the constructor to use a pool of buffers to reduce 
the number of allocations will help performance.  (was: 
BufferedElementCountingOutputStream's constructor allocates a new buffer to 
wrap the input OutputStream. This gets called on each invocation of encode() 
from IterableLikeCoder. Since Coder is designed to be stateless, but this coder 
holds state and isn't threadsafe, we can't just have the caller manage the 
buffer. Modifying the constructor to use a pool of buffers to reduce the number 
of allocations will help performance.)

> Reduce inefficient allocations in coders
> 
>
> Key: BEAM-3572
> URL: https://issues.apache.org/jira/browse/BEAM-3572
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> BufferedElementCountingOutputStream's constructor allocates a new buffer to 
> wrap the input OutputStream. This gets called on each invocation of encode() 
> from IterableLikeCoder. Since Coder is designed to be stateless, but this 
> buffer holds state and isn't threadsafe, we can't just have the caller manage 
> the buffer. Modifying the constructor to use a pool of buffers to reduce the 
> number of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3572) Reduce inefficient allocations in coders

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345875#comment-16345875
 ] 

Kenneth Knowles commented on BEAM-3572:
---

My reading of {{IterableLikeCoder}} is that it is stateless. Did I miss 
something?

> Reduce inefficient allocations in coders
> 
>
> Key: BEAM-3572
> URL: https://issues.apache.org/jira/browse/BEAM-3572
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> BufferedElementCountingOutputStream's constructor allocates a new buffer to 
> wrap the input OutputStream. This gets called on each invocation of encode() 
> from IterableLikeCoder. Since Coder is designed to be stateless, but this 
> buffer holds state and isn't threadsafe, we can't just have the caller manage 
> the buffer. Modifying the constructor to use a pool of buffers to reduce the 
> number of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3572) Reduce inefficient allocations in coders

[Bill Neubauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Neubauer reassigned BEAM-3572:
---

Assignee: Bill Neubauer  (was: Kenneth Knowles)

> Reduce inefficient allocations in coders
> 
>
> Key: BEAM-3572
> URL: https://issues.apache.org/jira/browse/BEAM-3572
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> BufferedElementCountingOutputStream's constructor allocates a new buffer to 
> wrap the input OutputStream. This gets called on each invocation of encode() 
> from IterableLikeCoder. Since Coder is designed to be stateless, but this 
> coder holds state and isn't threadsafe, we can't just have the caller manage 
> the buffer. Modifying the constructor to use a pool of buffers to reduce the 
> number of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3572) Reduce inefficient allocations in coders

[Bill Neubauer (JIRA)]

Bill Neubauer created BEAM-3572:
---

 Summary: Reduce inefficient allocations in coders
 Key: BEAM-3572
 URL: https://issues.apache.org/jira/browse/BEAM-3572
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Bill Neubauer
Assignee: Kenneth Knowles


BufferedElementCountingOutputStream's constructor allocates a new buffer to 
wrap the input OutputStream. This gets called on each invocation of encode() 
from IterableLikeCoder. Since Coder is designed to be stateless, but this coder 
holds state and isn't threadsafe, we can't just have the caller manage the 
buffer. Modifying the constructor to use a pool of buffers to reduce the number 
of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345862#comment-16345862
 ] 

Anton Kedin commented on BEAM-3531:
---

 

[~iemejia], you're right. I added the check that `options.getQuery()` is within 
supported range. This value is null if it's not specified in the command line. 
I can think of few of ways to fix it:

 1. move queries creation out of NexmarkLauncher into Main.java and perform 
these kind of checks there, right after suite configurations are loaded:
 * NexmarkLauncher is supposed to run a single query. The fact that it creates 
all queries and performs such checks is incorrect;
 * in this case `Main.runAll()` would analyze whether any of the loaded 
NexmarkConfigurations includes unsupported queries, and throw the exception 
there. E.g. right after `options.getSuite().getConfigurations(options)`;

2. instead of `options.getQuery()` use `configuration.query` to check whether 
current query is supported:
 * and instead of throwing and stopping the run we could just return empty 
results and log a warning;
 * problem is that some queries may have already been executed before this 
check is hit when trying to run unsupported query;

3. remove the check completely for now:
 * it will work for java as expected, and I don't believe anyone uses Nexmark 
SQL at the moment. I am planning to work on it soon, so I will likely touch 
this code and fix it the right way;

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> 

[jira] [Updated] (BEAM-3302) Go CoGBK support

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-3302:

Description: 
Consider using the java approach and implement as union coder over GBK, given 
that runners may not have CoGBK support.

Short design doc: 
https://docs.google.com/document/d/18SjId_0OQLUX2v5EbNjWfJHLnx96R0V8poLSUKmymig/edit#

  was:Consider using the java approach and implement as union coder over GBK, 
given that runners may not have CoGBK support.


> Go CoGBK support
> 
>
> Key: BEAM-3302
> URL: https://issues.apache.org/jira/browse/BEAM-3302
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>
> Consider using the java approach and implement as union coder over GBK, given 
> that runners may not have CoGBK support.
> Short design doc: 
> https://docs.google.com/document/d/18SjId_0OQLUX2v5EbNjWfJHLnx96R0V8poLSUKmymig/edit#



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345854#comment-16345854
 ] 

Kenneth Knowles commented on BEAM-3531:
---

[~kedin] do you know about this perhaps?

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run ***
> ==
> [WARNING] 
> 

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345769#comment-16345769
 ] 

Ismaël Mejía commented on BEAM-3531:


The issue is in org.apache.beam.sdk.nexmark.NexmarkLauncher#getNexmarkQuery 
after some changes in a recent refactor, it ignores the fact that query can be 
null when we are running the full suite e.g. --suite=DEFAULT or --suite=SMOKE

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #79

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #80

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #79

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #93

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3547) [SQL] Nested Query Generates Incompatible Trigger

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345728#comment-16345728
 ] 

Anton Kedin commented on BEAM-3547:
---

There is also this issue: https://issues.apache.org/jira/browse/BEAM-3481

> [SQL] Nested Query Generates Incompatible Trigger
> -
>
> Key: BEAM-3547
> URL: https://issues.apache.org/jira/browse/BEAM-3547
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>
> From 
> [https://stackoverflow.com/questions/48335383/nested-queries-in-beam-sql] :
>  
> SQL:
> {code:java}
> PCollection Query_Output = Query.apply(
> BeamSql.queryMulti("Select Orders.OrderID From Orders Where 
> Orders.CustomerID IN (Select Customers.CustomerID From Customers WHERE 
> Customers.CustomerID = 2)"));{code}
>  
> Error:
> {code:java}
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
> validateAndConvert
> INFO: SQL:
> SELECT `Orders`.`OrderID`
> FROM `Orders` AS `Orders`
> WHERE `Orders`.`CustomerID` IN (SELECT `Customers`.`CustomerID`
> FROM `Customers` AS `Customers`
> WHERE `Customers`.`CustomerID` = 2)
> Jan 19, 2018 11:56:36 AM 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner 
> convertToBeamRel
> INFO: SQLPlan>
> LogicalProject(OrderID=[$0])
>   LogicalJoin(condition=[=($1, $3)], joinType=[inner])
> LogicalTableScan(table=[[Orders]])
> LogicalAggregate(group=[{0}])
>   LogicalProject(CustomerID=[$0])
> LogicalFilter(condition=[=($0, 2)])
>   LogicalTableScan(table=[[Customers]])
> Exception in thread "main" java.lang.IllegalStateException: 
> java.lang.IllegalStateException: Inputs to Flatten had incompatible triggers: 
> DefaultTrigger, Repeatedly.forever(AfterWatermark.pastEndOfWindow())
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:165)
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:116)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:160)
> at com.bitwise.cloud.ExampleOfJoins.main(ExampleOfJoins.java:91)
> Caused by: java.lang.IllegalStateException: Inputs to Flatten had 
> incompatible triggers: DefaultTrigger, 
> Repeatedly.forever(AfterWatermark.pastEndOfWindow())
> at 
> org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:123)
> at 
> org.apache.beam.sdk.transforms.Flatten$PCollections.expand(Flatten.java:101)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:182)
> at 
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:124)
> at 
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:74)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:533)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:465)
> at 
> org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:107)
> at org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(Join.java:59)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.standardJoin(BeamJoinRel.java:217)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.buildBeamPipeline(BeamJoinRel.java:161)
> at 
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:68)
> at 
> org.apache.beam.sdk.extensions.sql.BeamSql$QueryTransform.expand(BeamSql.java:163)
> ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3171) convert a join into lookup

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3171.

   Resolution: Fixed
Fix Version/s: 2.3.0

> convert a join into lookup
> --
>
> Key: BEAM-3171
> URL: https://issues.apache.org/jira/browse/BEAM-3171
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Labels: experimental
> Fix For: 2.3.0
>
>
> We use BeamSQL to run streaming jobs mostly, and  add a join_as_lookup 
> improvement(internal branch) to cover the streaming-to-batch case(similar as 
> [1]). I could submit a PR as experimental if people are interested. 
> The rough solution is, if one source of join node implements 
> {{BeamSeekableTable}} and the other is not, then the join node is converted 
> to a fact-lookup operation.
> Ref:
> [1] 
> https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing
>  
> [~xumingming] [~takidau] for any comments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3558) aggregation expression can't apply to math or arithmetic expressions

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3558:


Assignee: (was: Xu Mingmin)

> aggregation expression can't apply to math or arithmetic expressions
> 
>
> Key: BEAM-3558
> URL: https://issues.apache.org/jira/browse/BEAM-3558
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Priority: Major
>
> fails when executing sql
> 'select sum(c1)+2 from PCOLLECTION group by c2'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3481) Query with subquery and aggregates cannot be implemented.

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3481:


Assignee: (was: Xu Mingmin)

> Query with subquery and aggregates cannot be implemented.
> -
>
> Key: BEAM-3481
> URL: https://issues.apache.org/jira/browse/BEAM-3481
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> This query results in the error below:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> This works correctly:
> {noformat}
> "SELECT (COUNT(`p`))
>  FROM (SELECT `p`, CURRENT_TIME
>FROM `contains`
>GROUP BY `p`) AS `t1`"{noformat}
> Error:
>  
> {noformat}
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQL:
> SELECT COUNT(`t1`.`p`)
> FROM (SELECT `contains`.`p`
> FROM `contains` AS `contains`
> GROUP BY `contains`.`p`) AS `t1`
> [nREPL-worker-5] INFO 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner - SQLPlan>
> LogicalAggregate(group=[{}], EXPR$0=[COUNT()])
>  LogicalAggregate(group=[{0}])
>  LogicalProject(p=[$0])
>  LogicalTableScan(table=[[contains]])
> CannotPlanException Node [rel#157:Subset#3.BEAM_LOGICAL.[]] could not be 
> implemented; planner state:
> Root: rel#157:Subset#3.BEAM_LOGICAL.[]
> Original rel:
> LogicalAggregate(subset=[rel#157:Subset#3.BEAM_LOGICAL.[]], group=[{}], 
> EXPR$0=[COUNT()]): rowcount = 1.0, cumulative cost = {1.125 rows, 0.0 cpu, 
> 0.0 io}, id = 155
>  LogicalAggregate(subset=[rel#154:Subset#2.NONE.[]], group=[{0}]): rowcount = 
> 10.0, cumulative cost = {10.0 rows, 0.0 cpu, 0.0 io}, id = 153
>  LogicalProject(subset=[rel#152:Subset#1.NONE.[]], p=[$0]): rowcount = 100.0, 
> cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 151
>  LogicalTableScan(subset=[rel#150:Subset#0.NONE.[]], table=[[contains]]): 
> rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 146
> Sets:
> Set#0, type: RecordType(VARCHAR p, VARCHAR s, BIGINT c)
>  rel#150:Subset#0.NONE.[], best=null, importance=0.6561
>  rel#146:LogicalTableScan.NONE.[](table=[contains]), rowcount=100.0, 
> cumulative cost={inf}
>  rel#162:Subset#0.BEAM_LOGICAL.[], best=rel#164, importance=0.32805
>  rel#164:BeamIOSourceRel.BEAM_LOGICAL.[](table=[contains]), rowcount=100.0, 
> cumulative cost={100.0 rows, 101.0 cpu, 0.0 io}
> Set#1, type: RecordType(VARCHAR p)
>  rel#152:Subset#1.NONE.[], best=null, importance=0.7291
>  rel#151:LogicalProject.NONE.[](input=rel#150:Subset#0.NONE.[],p=$0), 
> rowcount=100.0, cumulative cost={inf}
>  rel#159:Subset#1.BEAM_LOGICAL.[], best=rel#163, 
> importance=0.36455
>  
> rel#163:BeamProjectRel.BEAM_LOGICAL.[](input=rel#162:Subset#0.BEAM_LOGICAL.[],p=$0),
>  rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io}
> Set#2, type: RecordType(VARCHAR p)
>  rel#154:Subset#2.NONE.[], best=null, importance=0.81
>  rel#153:LogicalAggregate.NONE.[](input=rel#152:Subset#1.NONE.[],group={0}), 
> rowcount=10.0, cumulative cost={inf}
>  rel#161:Subset#2.BEAM_LOGICAL.[], best=rel#160, importance=0.405
>  
> rel#160:BeamAggregationRel.BEAM_LOGICAL.[](group={0},window=org.apache.beam.sdk.transforms.windowing.GlobalWindows,trigger=Repeatedly.forever(AfterWatermark.pastEndOfWindow())),
>  rowcount=10.0, cumulative cost={210.0 rows, 201.0 cpu, 0.0 io}
> Set#3, type: RecordType(BIGINT EXPR$0)
>  rel#156:Subset#3.NONE.[], best=null, importance=0.9
>  
> rel#155:LogicalAggregate.NONE.[](input=rel#154:Subset#2.NONE.[],group={},EXPR$0=COUNT()),
>  rowcount=1.0, cumulative cost={inf}
>  rel#157:Subset#3.BEAM_LOGICAL.[], best=null,
>  importance=1.0
>  
> rel#158:AbstractConverter.BEAM_LOGICAL.[](input=rel#156:Subset#3.NONE.[],convention=BEAM_LOGICAL,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit
>  (RelSubset.java:441)
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3386:


Assignee: (was: Xu Mingmin)

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Austin Haas
>Priority: Major
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3509) PARTITION BY in Beam SQL In Select Command

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-3509:


Assignee: (was: Xu Mingmin)

> PARTITION BY in Beam SQL In Select Command
> --
>
> Key: BEAM-3509
> URL: https://issues.apache.org/jira/browse/BEAM-3509
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Priority: Major
>  Labels: performance
>
> Partition By Option Will Be Very Help Full for DataFlow Developer To Migrate 
> Query and Do Transformation on That because of Many *Netezza Query and Oracle 
> Query* Consists Of Partition By Which Makes SQL Query More Efficient. *The 
> alternative is Making Joins And Filtering It Can Be Done But It Makes Code 
> Unreadable And Performance Become bad for DataFlow Job.*
> Examples: SELECT MIN(COLUMN) OVER (PARTITION BY COLUMN NAME) FROM TABLENAME



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3348) Beam SQL DSL not support non-ascii characters in sql

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3348.

   Resolution: Fixed
Fix Version/s: 2.3.0

> Beam SQL DSL not support non-ascii characters in sql
> 
>
> Key: BEAM-3348
> URL: https://issues.apache.org/jira/browse/BEAM-3348
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Yiyan Lu
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.3.0
>
>
> When I have non-ascii in Beam SQL:
> ```
> SELECT * FROM TABLE_A WHERE f_string = '第四行'
> ```
>  calcite planner will throw exception:
> ```
> Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode 
> '第四行' in character set 'ISO-8859-1'
> ```
> cause by calcite using ISO-8859-1 for default charset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-3110.

   Resolution: Fixed
Fix Version/s: 2.3.0

> The transform Read(UnboundedKafkaSource) is currently not supported
> ---
>
> Key: BEAM-3110
> URL: https://issues.apache.org/jira/browse/BEAM-3110
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.3.0
>
>
> I see this issue when submitting a job to Flink cluster. It appears after 
> build {{2.2.0-20170912.083349-51}}.
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>   at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
> Caused by: java.lang.UnsupportedOperationException: The transform 
> Read(UnboundedKafkaSource) is currently not supported.
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
>   at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
>   at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
>   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
> {code} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oscar Korz updated BEAM-3569:
-
Component/s: (was: runner-core)
 sdk-java-gcp

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner

[Oscar Korz (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345591#comment-16345591
 ] 

Oscar Korz commented on BEAM-1542:
--

PR #4014 is required to do deletes with SpannerIO. When will this be released?

> Need Source/Sink for Spanner
> 
>
> Key: BEAM-1542
> URL: https://issues.apache.org/jira/browse/BEAM-1542
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Guy Molinari
>Assignee: Mairbek Khadikov
>Priority: Major
>
> Is there a source/sink for Spanner in the works?   If not I would gladly give 
> this a shot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345586#comment-16345586
 ] 

Kenneth Knowles commented on BEAM-3531:
---

Great. Thanks!

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run ***
> ==
> [WARNING] 
> 

[jira] [Assigned] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3531:
-

Assignee: Qi Cui  (was: Kenneth Knowles)

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Qi Cui
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run ***
> ==
> [WARNING] 
> java.lang.reflect.InvocationTargetException
>  

[jira] [Commented] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345572#comment-16345572
 ] 

Oscar Korz commented on BEAM-3569:
--

It looks like a fix for this was merged to master already before 2.2.0 was 
released (2017-12-02). When can we expect this patch to be in a released 
version?

{noformat}
commit 3ba96003d31ce98a54c0c51c1c0a9cf7c06e2fa2
Author: Mairbek Khadikov 
Date:   2017-10-18 15:26:11 -0700

[BEAM-1542] SpannerIO: mutation encoding and size estimation improvements

 static long sizeOf(Mutation m) {
+if (m.getOperation() == Mutation.Op.DELETE) {
+  return sizeOf(m.getKeySet());
+}
{noformat}

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3571) Validate Go SDK encodes values the same as other SDKs

[Robert Burke (JIRA)]

Robert Burke created BEAM-3571:
--

 Summary: Validate Go SDK encodes values the same as other SDKs
 Key: BEAM-3571
 URL: https://issues.apache.org/jira/browse/BEAM-3571
 Project: Beam
  Issue Type: Task
  Components: sdk-go
Reporter: Robert Burke
Assignee: Henning Rohde


EvenTime, windows, length prefixing, etc encodings and decodings should be 
validated to match the other SDKs. This issue is to sanity check for and 
resolve these differences when the are found.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3531) nexmark failed with NPE with DEFAULT suite

[Qi Cui (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345538#comment-16345538
 ] 

Qi Cui commented on BEAM-3531:
--

Sure. I can help with this with your guidance. could you please reassign to me?

> nexmark failed with NPE with DEFAULT suite
> --
>
> Key: BEAM-3531
> URL: https://issues.apache.org/jira/browse/BEAM-3531
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Qi Cui
>Assignee: Kenneth Knowles
>Priority: Major
>
> @:~/src/beam/sdks/java/nexmark$ mvn exec:java 
> -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
> -Dexec.args="--runner=DirectRunner \
> > --suite=DEFAULT --streaming=false --manageResources=false 
> > --monitorJobs=true --enforceEncodability=true --enforceImmutability=true"
> [INFO] Scanning for projects...
> [WARNING] 
> [WARNING] Some problems were encountered while building the effective model 
> for org.apache.beam:beam-sdks-java-nexmark:jar:2.3.0-SNAPSHOT
> [WARNING] The expression ${parent.version} is deprecated. Please use 
> ${project.parent.version} instead.
> [WARNING] 
> [WARNING] It is highly recommended to fix these problems because they 
> threaten the stability of your build.
> [WARNING] 
> [WARNING] For this reason, future Maven versions might no longer support 
> building such malformed projects.
> [WARNING] 
> [INFO] 
> 
> [INFO] Detecting the operating system and CPU architecture
> [INFO] 
> 
> [INFO] os.detected.name: linux
> [INFO] os.detected.arch: x86_64
> [INFO] os.detected.version: 4.13
> [INFO] os.detected.version.major: 4
> [INFO] os.detected.version.minor: 13
> [INFO] os.detected.release: ubuntu
> [INFO] os.detected.release.version: 17.10
> [INFO] os.detected.release.like.ubuntu: true
> [INFO] os.detected.release.like.debian: true
> [INFO] os.detected.classifier: linux-x86_64
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] 
> 
> [INFO] Building Apache Beam :: SDKs :: Java :: Nexmark 2.3.0-SNAPSHOT
> [INFO] 
> 
> [WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
> no dependency information available
> [WARNING] Failed to retrieve plugin descriptor for 
> org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
> org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not 
> be resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced
> [INFO] 
> [INFO] --- exec-maven-plugin:1.5.0:java (default-cli) @ 
> beam-sdks-java-nexmark ---
> 2018-01-25T02:08:13.056Z Running query:0
> ==
> Run started 2018-01-25T02:08:12.987Z and ran for PT0.249S
> Default configuration:
> {"debug":true,"query":0,"sourceType":"DIRECT","sinkType":"DEVNULL","pubSubMode":"COMBINED","numEvents":10,"numEventGenerators":100,"rateShape":"SINE","firstEventRate":1,"nextEventRate":1,"rateUnit":"PER_SECOND","ratePeriodSec":600,"preloadSeconds":0,"streamTimeout":240,"isRateLimited":false,"useWallclockEventTime":false,"avgPersonByteSize":200,"avgAuctionByteSize":500,"avgBidByteSize":100,"hotAuctionRatio":2,"hotSellersRatio":4,"hotBiddersRatio":4,"windowSizeSec":10,"windowPeriodSec":5,"watermarkHoldbackSec":0,"numInFlightAuctions":100,"numActivePeople":1000,"coderStrategy":"HAND","cpuDelayMs":0,"diskBusyBytes":0,"auctionSkip":123,"fanout":5,"maxAuctionsWaitingTime":600,"occasionalDelaySec":3,"probDelayedEvent":0.1,"maxLogEvents":10,"usePubsubPublishTime":false,"outOfOrderGroupSize":1}
> Configurations:
>  Conf Description
>   query:0
> Performance:
>  Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>   *** not run ***
> ==
> 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5812

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list 5391053039f2e9951f05d50898530f3b81d54252 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Discovered a new module org.apache.beam:beam-parent Apache Beam :: Parent
ERROR: Processing failed due to a bug in the code. Please report this to the 
issue tracker (https://jenkins.io/redirect/report-an-issue).
java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:1035)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:691)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1724)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
project=hudson.maven.MavenModuleSet@141d582[beam_PostCommit_Java_MavenInstall]
project.getModules()=[]
project.getRootModule()=null
FATAL: null
java.lang.UnsupportedOperationException
at java.util.AbstractMap.put(AbstractMap.java:209)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:1035)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:691)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1724)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)
Not sending mail to unregistered user joey.bar...@gmail.com
Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com

Build failed in Jenkins: beam_PerformanceTests_Spark #1298

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5583166646440965852.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8939725341747624148.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins754180760910310008.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7291080484683591782.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2616506633722833239.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #79

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam5 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3628290200175741188.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6934488639919749585.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5188318602834395633.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1934368782951125910.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins875720719933340664.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Downloading/unpacking absl-py (from -r PerfKitBenchmarker/requirements.txt 
(line 14))
  Running setup.py 
(path:
 egg_info for package absl-py

Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Downloading/unpacking colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Downloading colorlog-2.6.0-py2.py3-none-any.whl
Downloading/unpacking blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt 
(line 18))
  Running setup.py 
(path:
 egg_info for package blinker

Downloading/unpacking futures>=3.0.3 (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
  Downloading futures-3.2.0-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Downloading/unpacking pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt 
(line 21))
  Running setup.py 
(path:
 egg_info for package pint

no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'
warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
warning: no previously-included files matching '*~' found anywhere in 
distribution
warning: no previously-included files matching '.DS_Store' found anywhere 
in distribution
warning: no previously-included files matching '*__pycache__*' found 
anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in 
distribution
warning: no previously-included files matching '.travis-exclude.yml' found 
anywhere in distribution
Downloading/unpacking numpy (from -r PerfKitBenchmarker/requirements.txt (line 
22))
  Running setup.py 
(path:
 egg_info for package numpy
Running from numpy source directory.

Build failed in Jenkins: beam_PerformanceTests_Python #854

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8862466731767327857.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3706425129200079782.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2344431064361173270.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8852298751330289034.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1557445552538784123.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6838029990535680905.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

[jira] [Assigned] (BEAM-3570) More elegant interface for RuntimeValueProvider

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada reassigned BEAM-3570:
---

Assignee: Pablo Estrada  (was: Ahmet Altay)

> More elegant interface for RuntimeValueProvider
> ---
>
> Key: BEAM-3570
> URL: https://issues.apache.org/jira/browse/BEAM-3570
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>
> The RuntimeValueProvider/ValueProvider classes require creating a new 
> instance whenever we want to retrieve a value from them. It may be desirable 
> to have a class method that pulls the value without creating a new instance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #92

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7726035918316764169.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7515729833675000987.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5505273016772069676.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins948955522260615292.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7268601321007938882.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1270370480462910008.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #78

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5682804500441400410.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1000114434601909177.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins922293324751137034.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3663836969683025453.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6243262151540122660.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3501085780783893307.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #78

[Apache Jenkins Server]

See 


Changes:

[kenn] Add a test for an event time timer loop in ParDo

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a7226cba18fb0366bc8b1d516ccab1fa65d753ed (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Commit message: "Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo"
 > git rev-list dc24beb9f3651dc05749b8fac9bc6f6acacf7022 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8042324721229415374.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8207577064356153634.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4224986380187498008.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8401851253241525814.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3115905917642258342.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4529463927636036342.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

[beam-site] branch asf-site updated (f0e45db -> 21477ff)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from f0e45db  Prepare repository for deployment.
 add 985fefd  Fix broken links due to code directory moves
 add d910f9b  This closes #380
 new 21477ff  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../2016/10/11/strata-hadoop-world-and-beam.html   |   2 +-
 content/blog/2016/10/20/test-stream.html   |   4 +-
 content/documentation/io/testing/index.html|   2 +-
 content/documentation/programming-guide/index.html |   7 +-
 content/feed.xml   |   4 +-
 .../get-started/mobile-gaming-example/index.html   | 193 +++--
 .../2016-10-12-strata-hadoop-world-and-beam.md |   2 +-
 src/_posts/2016-10-20-test-stream.md   |   4 +-
 src/documentation/io/testing.md|   2 +-
 src/documentation/programming-guide.md |   8 +-
 src/get-started/mobile-gaming-example.md   |  30 ++--
 11 files changed, 136 insertions(+), 122 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 21477ff9ba506f84da4dc3e6155046098503
Author: Mergebot 
AuthorDate: Tue Jan 30 09:59:27 2018 -0800

Prepare repository for deployment.
---
 .../2016/10/11/strata-hadoop-world-and-beam.html   |   2 +-
 content/blog/2016/10/20/test-stream.html   |   4 +-
 content/documentation/io/testing/index.html|   2 +-
 content/documentation/programming-guide/index.html |   7 +-
 content/feed.xml   |   4 +-
 .../get-started/mobile-gaming-example/index.html   | 193 +++--
 6 files changed, 113 insertions(+), 99 deletions(-)

diff --git a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html 
b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
index 9c939a8..f42292e 100644
--- a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
+++ b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
@@ -95,7 +95,7 @@
 
 
 
-If you want to take a look at the tutorial materials, we’ve put them up https://github.com/eljefe6a/beamexample;>on GitHub. This includes the 
https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf;>actual
 slides as well as the https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game;>exercises
 that we covered. If you’re looking to learn a little about Beam, this is  [...]
+If you want to take a look at the tutorial materials, we’ve put them up https://github.com/eljefe6a/beamexample;>on GitHub. This includes the 
https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf;>actual
 slides as well as the https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game;>exercises
 that we covered. If you’re looking to learn a little about Beam, this is  [...]
 
 I want to share some of takeaways I had about Beam during the 
conference.
 
diff --git a/content/blog/2016/10/20/test-stream.html 
b/content/blog/2016/10/20/test-stream.html
index f98d239..f2cc8bb 100644
--- a/content/blog/2016/10/20/test-stream.html
+++ b/content/blog/2016/10/20/test-stream.html
@@ -128,8 +128,8 @@ from the Mobile Gaming example series.
 
 LeaderBoard and the Mobile 
Gaming Example
 
-https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177;>LeaderBoard
-is part of the https://github.com/apache/beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game;>Beam
 mobile gaming examples
+https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177;>LeaderBoard
+is part of the https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete/game;>Beam
 mobile gaming examples
 (and walkthroughs)
 which produces a continuous accounting of user and team scores. User scores are
 calculated over the lifetime of the program, while team scores are calculated
diff --git a/content/documentation/io/testing/index.html 
b/content/documentation/io/testing/index.html
index bb0b1b5..4a3eb11 100644
--- a/content/documentation/io/testing/index.html
+++ b/content/documentation/io/testing/index.html
@@ -832,7 +832,7 @@ dynamic_pipeline_options:
   Having your test take a pipeline option that decides whether to generate 
a small or large amount of test data (where small and large are sizes 
appropriate to your data store)
 
 
-An example of this is https://github.com/apache/beam/tree/master/sdks/java/io/hadoop/input-format;>HadoopInputFormatIO’s
 tests.
+An example of this is https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-input-format;>HadoopInputFormatIO’s
 tests.
 
 

[beam-site] 02/02: This closes #380

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d910f9b1da4cca61820aab2a10ee213703c81cda
Merge: f0e45db 985fefd
Author: Mergebot 
AuthorDate: Tue Jan 30 09:54:40 2018 -0800

This closes #380

 .../2016-10-12-strata-hadoop-world-and-beam.md |  2 +-
 src/_posts/2016-10-20-test-stream.md   |  4 +--
 src/documentation/io/testing.md|  2 +-
 src/documentation/programming-guide.md |  8 +++---
 src/get-started/mobile-gaming-example.md   | 30 +++---
 5 files changed, 23 insertions(+), 23 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (1aa4590 -> d910f9b)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 1aa4590  This closes #380
 discard 51836e2  Fix broken links due to code directory moves
 new 985fefd  Fix broken links due to code directory moves
 new d910f9b  This closes #380

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1aa4590)
\
 N -- N -- N   refs/heads/mergebot (d910f9b)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/documentation/programming-guide.md | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Fix broken links due to code directory moves

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 985fefd87d21bfed0d66c832acff46ff66c2067c
Author: melissa 
AuthorDate: Mon Jan 29 10:28:50 2018 -0800

Fix broken links due to code directory moves
---
 .../2016-10-12-strata-hadoop-world-and-beam.md |  2 +-
 src/_posts/2016-10-20-test-stream.md   |  4 +--
 src/documentation/io/testing.md|  2 +-
 src/documentation/programming-guide.md |  8 +++---
 src/get-started/mobile-gaming-example.md   | 30 +++---
 5 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md 
b/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
index 1642edd..3b8eb37 100644
--- a/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
+++ b/src/_posts/2016-10-12-strata-hadoop-world-and-beam.md
@@ -12,7 +12,7 @@ Tyler Akidau and I gave a [three-hour 
tutorial](http://conferences.oreilly.com/s
 
 
 
-If you want to take a look at the tutorial materials, we’ve put them up [on 
GitHub](https://github.com/eljefe6a/beamexample). This includes the [actual 
slides](https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf)
 as well as the 
[exercises](https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game)
 that we covered. If you’re looking to learn a little about Beam, this is a 
good way to start. The exercises a [...]
+If you want to take a look at the tutorial materials, we’ve put them up [on 
GitHub](https://github.com/eljefe6a/beamexample). This includes the [actual 
slides](https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf)
 as well as the 
[exercises](https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game)
 that we covered. If you’re looking to learn a little about Beam, this is a 
good way to start. The exercises a [...]
 
 I want to share some of takeaways I had about Beam during the conference.
 
diff --git a/src/_posts/2016-10-20-test-stream.md 
b/src/_posts/2016-10-20-test-stream.md
index 500680a..a65a8b7 100644
--- a/src/_posts/2016-10-20-test-stream.md
+++ b/src/_posts/2016-10-20-test-stream.md
@@ -45,8 +45,8 @@ from the Mobile Gaming example series.
 
 ## LeaderBoard and the Mobile Gaming Example
 
-[LeaderBoard](https://github.com/apache/beam/blob/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177)
-is part of the [Beam mobile gaming 
examples](https://github.com/apache/beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game)
+[LeaderBoard](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java#L177)
+is part of the [Beam mobile gaming 
examples](https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete/game)
 (and [walkthroughs]({{ site.baseurl }}/get-started/mobile-gaming-example/))
 which produces a continuous accounting of user and team scores. User scores are
 calculated over the lifetime of the program, while team scores are calculated
diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
index 0c3f439..cac7b8a 100644
--- a/src/documentation/io/testing.md
+++ b/src/documentation/io/testing.md
@@ -561,7 +561,7 @@ You can do this by:
 1.  Creating two Kubernetes scripts: one for a small instance of the data 
store, and one for a large instance.
 1.  Having your test take a pipeline option that decides whether to generate a 
small or large amount of test data (where small and large are sizes appropriate 
to your data store)
 
-An example of this is 
[HadoopInputFormatIO](https://github.com/apache/beam/tree/master/sdks/java/io/hadoop/input-format)'s
 tests.
+An example of this is 
[HadoopInputFormatIO](https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-input-format)'s
 tests.
 
 

[jira] [Created] (BEAM-3570) More elegant interface for RuntimeValueProvider

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-3570:
---

 Summary: More elegant interface for RuntimeValueProvider
 Key: BEAM-3570
 URL: https://issues.apache.org/jira/browse/BEAM-3570
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Ahmet Altay


The RuntimeValueProvider/ValueProvider classes require creating a new instance 
whenever we want to retrieve a value from them. It may be desirable to have a 
class method that pulls the value without creating a new instance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (33930f4 -> 1aa4590)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 33930f4  This closes #380
 new 1aa4590  This closes #380

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (33930f4)
\
 N -- N -- N   refs/heads/mergebot (1aa4590)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: This closes #380

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 1aa459057df3a808b70fe2fffcd8b2b23f2a0b63
Merge: f0e45db 51836e2
Author: Mergebot 
AuthorDate: Tue Jan 30 09:39:55 2018 -0800

This closes #380

 .../2016-10-12-strata-hadoop-world-and-beam.md |  2 +-
 src/_posts/2016-10-20-test-stream.md   |  4 +--
 src/documentation/io/testing.md|  2 +-
 src/documentation/programming-guide.md |  8 +++---
 src/get-started/mobile-gaming-example.md   | 30 +++---
 5 files changed, 23 insertions(+), 23 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Comment Edited] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345402#comment-16345402
 ] 

Oscar Korz edited comment on BEAM-3569 at 1/30/18 5:22 PM:
---

I've attached a minimal example to demonstrate the issue. It seems reproducing 
the error does not even require a valid Spanner deployment to write to. The 
example can be built and run with "./gradlew run".

{code}
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)
{code}


was (Author: okorz001):
I've attached a minimal example to demonstrate the issue. It seems reproducing 
the error does not even require a valid Spanner deployment to write to:
{code}
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)
{code}

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345402#comment-16345402
 ] 

Oscar Korz commented on BEAM-3569:
--

I've attached a minimal example to demonstrate the issue. It seems reproducing 
the error does not even require a valid Spanner deployment to write to:

Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345402#comment-16345402
 ] 

Oscar Korz edited comment on BEAM-3569 at 1/30/18 5:20 PM:
---

I've attached a minimal example to demonstrate the issue. It seems reproducing 
the error does not even require a valid Spanner deployment to write to:
{code}
Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)
{code}


was (Author: okorz001):
I've attached a minimal example to demonstrate the issue. It seems reproducing 
the error does not even require a valid Spanner deployment to write to:

Exception in thread "main" 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.IllegalStateException: values() cannot be called for a DELETE mutation
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:344)
at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:314)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at Main.main(Main.java:42)
Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:77)

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oscar Korz updated BEAM-3569:
-
Attachment: beam-spanner-io-delete.tar.gz

> SpannerIO.write throws on delete mutations
> --
>
> Key: BEAM-3569
> URL: https://issues.apache.org/jira/browse/BEAM-3569
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.2.0
>Reporter: Oscar Korz
>Assignee: Kenneth Knowles
>Priority: Major
> Attachments: beam-spanner-io-delete.tar.gz
>
>
> It is currently impossible to delete a Spanner row in Beam with SpannerIO. 
> The exception is generated by trying to guess the size of a delete mutation 
> which cannot contain any values (deletes are simply by key).
> The root exception stack trace:
> {code:java}
>  Caused by: java.lang.IllegalStateException: values() cannot be called for a 
> DELETE mutation
>   at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
>   at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
> {code}
> I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
> either 0 or 1 for Mutations with getOperation() = Op.DELETE.
> The workaround is to avoid using SpannerIO and use Spanner client API 
> directly in a custom DoFn, but this forces users to either reimplement all 
> the intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3569) SpannerIO.write throws on delete mutations

[Oscar Korz (JIRA)]

Oscar Korz created BEAM-3569:


 Summary: SpannerIO.write throws on delete mutations
 Key: BEAM-3569
 URL: https://issues.apache.org/jira/browse/BEAM-3569
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Affects Versions: 2.2.0
Reporter: Oscar Korz
Assignee: Kenneth Knowles


It is currently impossible to delete a Spanner row in Beam with SpannerIO. The 
exception is generated by trying to guess the size of a delete mutation which 
cannot contain any values (deletes are simply by key).

The root exception stack trace:
{code:java}
 Caused by: java.lang.IllegalStateException: values() cannot be called for a 
DELETE mutation
  at com.google.common.base.Preconditions.checkState(Preconditions.java:456)
  at com.google.cloud.spanner.Mutation.getValues(Mutation.java:233)
  at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:33)
  at 
org.apache.beam.sdk.io.gcp.spanner.MutationSizeEstimator.sizeOf(MutationSizeEstimator.java:51)
{code}

I believe this can be fixed by special casing MutationSizeEstimator.sizeOf to 
either 0 or 1 for Mutations with getOperation() = Op.DELETE.

The workaround is to avoid using SpannerIO and use Spanner client API directly 
in a custom DoFn, but this forces users to either reimplement all the 
intelligent batching that SpannerIO does or suffer poor performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (5391053 -> a7226cb)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5391053  Merge pull request #4243: [BEAM-2469] Handling Kinesis shards 
splits and merges
 add 4c3bac8  Add a test for an event time timer loop in ParDo
 new a7226cb  Merge pull request #3745: [BEAM-2791] Add a test for an event 
time timer loop in ParDo

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/sdk/transforms/ParDoTest.java  | 45 ++
 1 file changed, 45 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #3745: [BEAM-2791] Add a test for an event time timer loop in ParDo

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a7226cba18fb0366bc8b1d516ccab1fa65d753ed
Merge: 5391053 4c3bac8
Author: Kenn Knowles 
AuthorDate: Tue Jan 30 07:30:39 2018 -0800

Merge pull request #3745: [BEAM-2791] Add a test for an event time timer 
loop in ParDo

 .../org/apache/beam/sdk/transforms/ParDoTest.java  | 45 ++
 1 file changed, 45 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5811

[Apache Jenkins Server]

See 


Changes:

[pawel.pk.kaczmarczyk] [BEAM-2469] Handling Kinesis shards splits and merges

--
[...truncated 1.41 MB...]
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.spark:spark-launcher_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.spark:spark-network-common_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.9.3 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding javax.activation:activation:jar:1.1.1 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding mx4j:mx4j:jar:3.0.2 from the shaded 
jar.
2018-01-30T14:48:56.447 [INFO] Excluding javax.mail:mail:jar:1.4.7 from the 
shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.bouncycastle:bcprov-jdk15on:jar:1.51 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
com.jamesmurty.utils:java-xmlbuilder:jar:1.0 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding net.iharder:base64:jar:2.3.8 from the 
shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.6.0 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.6.0 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
javax.servlet:javax.servlet-api:jar:3.1.0 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.16 from 
the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.16 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.16 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.11.8 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 
org.json4s:json4s-jackson_2.11:jar:3.2.11 from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.json4s:json4s-core_2.11:jar:3.2.11 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.json4s:json4s-ast_2.11:jar:3.2.11 
from the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding org.scala-lang:scalap:jar:2.11.0 from 
the shaded jar.
2018-01-30T14:48:56.447 [INFO] Excluding 

[jira] [Created] (BEAM-3568) Overlapping sessions with zero allowed lateness due to window expiry rules

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3568:
-

 Summary: Overlapping sessions with zero allowed lateness due to 
window expiry rules
 Key: BEAM-3568
 URL: https://issues.apache.org/jira/browse/BEAM-3568
 Project: Beam
  Issue Type: Bug
  Components: beam-model, runner-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Consider this sequence, with session gap durations of 5:

 - element arrives with timestamp 0, assigned to proto-window [0, 5)
 - watermark advances to 6, emitting the session and discarding it
 - element arrives with timestamp 3, assigned to proto-window [3, 8) so it is 
not dropped as the window is not expired
 - watermark advances to 8+, emitting that session

While "technically correct" according to spec, this seems undesirable. It was 
introduced when late data dropping was tied to window expiry. I think either 
dropping the second element or including it and emitting a merged window would 
be OK.

In the case of sessions, we could just retain the window until it cannot 
possibly merge with other non-expired data. Even with allowed lateness zero 
this is double the gap duration. The window would be in an interesting state 
where it would be expired and ineligible for further output but could still 
merge and the greater window could be output.

The challenge is that sessions are just one kind of merging window - the 
merging logic has to be assumed opaque. So we cannot simply reason about how 
sessions work. The other, more drastic option, is to rethink how late data 
dropping is defined for merging windows, particularly in the "proto-window" 
phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-2943) Beam Flink deployment results in ClassNotFoundException

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345099#comment-16345099
 ] 

Grzegorz Kołakowski edited comment on BEAM-2943 at 1/30/18 2:26 PM:


[~guenhter]

I tried to reproduce your problem and eventually I managed to observe the same 
exception.

The key question is how you built the project? You didn't mention it in the 
description but it turned out to be a crucial step.

The class 
{{org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat}} is a 
part of {{org.apache.beam:beam-runners-flink_2.10}} dependency, which is 
included into uber-jar only if the {{flink-runner}} profile is used. I guess 
you just simply run {{mvn package}} instead of {{mvn package -Pflink-runner}}.


was (Author: grzegorz_kolakowski):
[~guenhter]

I tried to reproduce your problem and eventually I managed to observe the same 
exception.

The key question is how you built the project? You didn't mentioned it in the 
desciption but it turned out to be a crucial step.

The class 
{{org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat}} is a 
part of {{org.apache.beam:beam-runners-flink_2.10}} dependency, which is 
included into uber-jar only if the {{flink-runner}} profile is used. I guess 
you just simply run {{mvn package}} instead of {{mvn package -Pflink-runner}}.

> Beam Flink deployment results in ClassNotFoundException
> ---
>
> Key: BEAM-2943
> URL: https://issues.apache.org/jira/browse/BEAM-2943
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
> Environment: Debian 9.1 / 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 
> (2017-08-06) x86_64 GNU/Linux
>Reporter: Guenther Grill
>Assignee: Aljoscha Krettek
>Priority: Major
>  Labels: flink
>
> Hi,
> I followed the guide https://beam.apache.org/get-started/quickstart-java/ to 
> run beam program within a flink cluster. 
> The output of the dependency-command is:
> {code}
> mvn dependency:tree -Pflink-runner |grep flink
>   
> [INFO] \- org.apache.beam:beam-runners-flink_2.10:jar:2.1.0:runtime
> [INFO]+- org.apache.flink:flink-clients_2.10:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-optimizer_2.10:jar:1.3.0:runtime
> [INFO]|  \- org.apache.flink:force-shading:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-core:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-annotations:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-metrics-core:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-java:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-shaded-hadoop2:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-runtime_2.10:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-streaming-java_2.10:jar:1.3.0:runtime
> {code}
> Then I started the flink cluster with the correct version with docker-compose
> {code}
> export JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> export FLINK_DOCKER_IMAGE_NAME=flink:1.3.0-hadoop27-scala_2.10
> docker-compose up -d
> {code}
> The compose file looks like this:
> {code}
> version: '3.3'
> services:
>   jobmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6123"
> ports:
>   - "6123:6123"
>   - "8081:8081"
> volumes:
>   - /tmp:/tmp
> command: jobmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
>   taskmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6121"
>   - "6122"
> depends_on:
>   - jobmanager
> command: taskmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> {code}
> The flink cluster works, but when I execute 
> {code}
> mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
> -Pflink-runner \
> -Dexec.args="--runner=FlinkRunner \
>   --inputFile=pom.xml \
>   --output=/path/to/counts \
>   --flinkMaster=[HOST_IP]:6123 \
>   --filesToStage=target/word-count-beam-bundled-0.1.jar"
> {code}
> I get:
> {code}
> 2017-09-12 06:39:57,226 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Submitting job a913f922506053e65e732eeb8336b3bd 
> (wordcount-grg-0912063956-c7ea6199).
> 2017-09-12 06:39:57,227 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Using restart strategy NoRestartStrategy for 
> a913f922506053e65e732eeb8336b3bd.
> 2017-09-12 06:39:57,227 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> 2017-09-12 06:39:57,229 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Running initialization on master for job 

[jira] [Commented] (BEAM-2943) Beam Flink deployment results in ClassNotFoundException

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345099#comment-16345099
 ] 

Grzegorz Kołakowski commented on BEAM-2943:
---

[~guenhter]

I tried to reproduce your problem and eventually I managed to observe the same 
exception.

The key question is how you built the project? You didn't mentioned it in the 
desciption but it turned out to be a crucial step.

The class 
{{org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat}} is a 
part of {{org.apache.beam:beam-runners-flink_2.10}} dependency, which is 
included into uber-jar only if the {{flink-runner}} profile is used. I guess 
you just simply run {{mvn package}} instead of {{mvn package -Pflink-runner}}.

> Beam Flink deployment results in ClassNotFoundException
> ---
>
> Key: BEAM-2943
> URL: https://issues.apache.org/jira/browse/BEAM-2943
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.1.0
> Environment: Debian 9.1 / 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 
> (2017-08-06) x86_64 GNU/Linux
>Reporter: Guenther Grill
>Assignee: Aljoscha Krettek
>Priority: Major
>  Labels: flink
>
> Hi,
> I followed the guide https://beam.apache.org/get-started/quickstart-java/ to 
> run beam program within a flink cluster. 
> The output of the dependency-command is:
> {code}
> mvn dependency:tree -Pflink-runner |grep flink
>   
> [INFO] \- org.apache.beam:beam-runners-flink_2.10:jar:2.1.0:runtime
> [INFO]+- org.apache.flink:flink-clients_2.10:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-optimizer_2.10:jar:1.3.0:runtime
> [INFO]|  \- org.apache.flink:force-shading:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-core:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-annotations:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-metrics-core:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-java:jar:1.3.0:runtime
> [INFO]|  +- org.apache.flink:flink-shaded-hadoop2:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-runtime_2.10:jar:1.3.0:runtime
> [INFO]+- org.apache.flink:flink-streaming-java_2.10:jar:1.3.0:runtime
> {code}
> Then I started the flink cluster with the correct version with docker-compose
> {code}
> export JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> export FLINK_DOCKER_IMAGE_NAME=flink:1.3.0-hadoop27-scala_2.10
> docker-compose up -d
> {code}
> The compose file looks like this:
> {code}
> version: '3.3'
> services:
>   jobmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6123"
> ports:
>   - "6123:6123"
>   - "8081:8081"
> volumes:
>   - /tmp:/tmp
> command: jobmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
>   taskmanager:
> image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
> expose:
>   - "6121"
>   - "6122"
> depends_on:
>   - jobmanager
> command: taskmanager
> environment:
>   - JOB_MANAGER_RPC_ADDRESS=[HOST_IP]
> {code}
> The flink cluster works, but when I execute 
> {code}
> mvn exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
> -Pflink-runner \
> -Dexec.args="--runner=FlinkRunner \
>   --inputFile=pom.xml \
>   --output=/path/to/counts \
>   --flinkMaster=[HOST_IP]:6123 \
>   --filesToStage=target/word-count-beam-bundled-0.1.jar"
> {code}
> I get:
> {code}
> 2017-09-12 06:39:57,226 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Submitting job a913f922506053e65e732eeb8336b3bd 
> (wordcount-grg-0912063956-c7ea6199).
> 2017-09-12 06:39:57,227 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Using restart strategy NoRestartStrategy for 
> a913f922506053e65e732eeb8336b3bd.
> 2017-09-12 06:39:57,227 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> 2017-09-12 06:39:57,229 INFO  org.apache.flink.runtime.jobmanager.JobManager  
>   - Running initialization on master for job 
> wordcount-grg-0912063956-c7ea6199 (a913f922506053e65e732eeb8336b3bd).
> 2017-09-12 06:39:57,230 ERROR org.apache.flink.runtime.jobmanager.JobManager  
>   - Failed to submit job a913f922506053e65e732eeb8336b3bd 
> (wordcount-grg-0912063956-c7ea6199)
> org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 
> 'DataSource (at Read(CompressedSource) 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat))': 
> Deserializing the InputFormat 
> (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat@58e7a91a)
>  failed: Could not read the user code wrapper: 
> 

[jira] [Resolved] (BEAM-2469) Handling Kinesis shards splits and merges

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-2469.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Handling Kinesis shards splits and merges
> -
>
> Key: BEAM-2469
> URL: https://issues.apache.org/jira/browse/BEAM-2469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Paweł Kaczmarczyk
>Assignee: Paweł Kaczmarczyk
>Priority: Major
> Fix For: 2.4.0
>
>
> Kinesis stream consists of 
> [shards|http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard]
>  that allow for capacity scaling. In order to increase/decrease the capacity 
> shards have to be split/merged together. Such operations are currently not 
> handled properly and will end with errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (dc24beb -> 5391053)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from dc24beb  Merge pull request #3839: Add javadoc to ConsoleIO
 add 994c7f3  [BEAM-2469] Handling Kinesis shards splits and merges
 new 5391053  Merge pull request #4243: [BEAM-2469] Handling Kinesis shards 
splits and merges

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/io/kinesis/DynamicCheckpointGenerator.java |  27 ++-
 .../apache/beam/sdk/io/kinesis/KinesisReader.java  |  12 +-
 ...-info.java => KinesisShardClosedException.java} |  10 +-
 .../beam/sdk/io/kinesis/ShardReadersPool.java  | 167 +++--
 .../beam/sdk/io/kinesis/ShardRecordsIterator.java  |  27 ++-
 .../sdk/io/kinesis/StartingPointShardsFinder.java  | 193 +++
 .../io/kinesis/DynamicCheckpointGeneratorTest.java |  35 ++-
 .../beam/sdk/io/kinesis/KinesisReaderTest.java |  20 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  | 126 --
 .../sdk/io/kinesis/ShardRecordsIteratorTest.java   |   6 +-
 .../io/kinesis/StartingPointShardsFinderTest.java  | 264 +
 11 files changed, 836 insertions(+), 51 deletions(-)
 copy 
sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/{package-info.java
 => KinesisShardClosedException.java} (80%)
 create mode 100644 
sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinder.java
 create mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinderTest.java

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 01/01: Merge pull request #4243: [BEAM-2469] Handling Kinesis shards splits and merges

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5391053039f2e9951f05d50898530f3b81d54252
Merge: dc24beb 994c7f3
Author: Ismaël Mejía 
AuthorDate: Tue Jan 30 14:50:02 2018 +0100

Merge pull request #4243: [BEAM-2469] Handling Kinesis shards splits and 
merges

 .../sdk/io/kinesis/DynamicCheckpointGenerator.java |  27 ++-
 .../apache/beam/sdk/io/kinesis/KinesisReader.java  |  12 +-
 .../io/kinesis/KinesisShardClosedException.java|  28 +++
 .../beam/sdk/io/kinesis/ShardReadersPool.java  | 167 +++--
 .../beam/sdk/io/kinesis/ShardRecordsIterator.java  |  27 ++-
 .../sdk/io/kinesis/StartingPointShardsFinder.java  | 193 +++
 .../io/kinesis/DynamicCheckpointGeneratorTest.java |  35 ++-
 .../beam/sdk/io/kinesis/KinesisReaderTest.java |  20 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  | 126 --
 .../sdk/io/kinesis/ShardRecordsIteratorTest.java   |   6 +-
 .../io/kinesis/StartingPointShardsFinderTest.java  | 264 +
 11 files changed, 856 insertions(+), 49 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5810

[Apache Jenkins Server]

See 


--
[...truncated 1.41 MB...]
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.spark:spark-launcher_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.spark:spark-network-common_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.11:jar:2.2.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.9.3 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding javax.activation:activation:jar:1.1.1 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding mx4j:mx4j:jar:3.0.2 from the shaded 
jar.
2018-01-30T13:06:08.968 [INFO] Excluding javax.mail:mail:jar:1.4.7 from the 
shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.bouncycastle:bcprov-jdk15on:jar:1.51 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
com.jamesmurty.utils:java-xmlbuilder:jar:1.0 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding net.iharder:base64:jar:2.3.8 from the 
shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.6.0 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.6.0 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
javax.servlet:javax.servlet-api:jar:3.1.0 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.16 from 
the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.16 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.16 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.11.8 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.json4s:json4s-jackson_2.11:jar:3.2.11 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.json4s:json4s-core_2.11:jar:3.2.11 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.json4s:json4s-ast_2.11:jar:3.2.11 
from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding org.scala-lang:scalap:jar:2.11.0 from 
the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 
org.scala-lang:scala-compiler:jar:2.11.0 from the shaded jar.
2018-01-30T13:06:08.968 [INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_Python #853

[Apache Jenkins Server]

See 


Changes:

[joey.baruch] Add javadoc to ConsoleIO

[lcwik] [BEAM-2500, BEAM-3249] Add amazon-web-services gradle build rules.

[github] [BEAM-3557] Sets parent pointer of AppliedPTransform objects correctly

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision dc24beb9f3651dc05749b8fac9bc6f6acacf7022 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f dc24beb9f3651dc05749b8fac9bc6f6acacf7022
Commit message: "Merge pull request #3839: Add javadoc to ConsoleIO"
 > git rev-list 21623d892302717a3ad5b3dbc447ebc38268bb29 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8710428979684075735.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6186107474566433790.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4473822791037069439.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6458451769936506786.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins702031711613407868.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #78

[Apache Jenkins Server]

See