Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3505

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4290

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2544

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #59

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243
 > git rev-list 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson5319729041671723195.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson910036321182999356.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson6678792084990695645.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson8206565567529375711.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
  Installing extra requirements: 'gcp,test'
Requirement 

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072945#comment-16072945
 ] 

Reuven Lax commented on BEAM-2353:
--

n/m - just saw this is bumped to 2.2. In that case all of these changes
will go in for 2.2.0, which is fine.




> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3504

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4289

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2543

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #58

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2530] Fix compilation of modules with Java 9 that depend on

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243
 > git rev-list 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson398738363712742199.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson6562891134953476816.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson705767665452622758.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson6397254117961652724.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072737#comment-16072737
 ] 

Reuven Lax commented on BEAM-2353:
--

+JB




> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3503

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072582#comment-16072582
 ] 

Reuven Lax commented on BEAM-2353:
--

In that case we should take https://github.com/apache/beam/pull/3356 as
well, as that also is a breaking change to the FilenamePolicy interface
(removing baseDirectory and extension). Last word from Eugene was that he
was happy with it but was going on vacation. He told me to relay the
message that he was happy for any other committer to merge the PR.

An Apex runner test currently breaks with this PR. As far as I can tell
debugging, this is simply a bug in the Apex runner. The root cause appears
to be the Apex runner throwing some sort of socket exception when one of
the ParDos calls output. I've started a thread on the dev list about this,
but IMO should not be a blocker for this PR.

On Tue, May 23, 2017 at 2:07 PM, Kenneth Knowles (JIRA) 



> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072570#comment-16072570
 ] 

Jean-Baptiste Onofré commented on BEAM-2523:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick on the 2.1.0 release branch if you want.

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2534) KafkaIO should allow gaps in message offsets

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072571#comment-16072571
 ] 

Jean-Baptiste Onofré commented on BEAM-2534:


Any update on this Jira for 2.1.0 release ?

> KafkaIO should allow gaps in message offsets
> 
>
> Key: BEAM-2534
> URL: https://issues.apache.org/jira/browse/BEAM-2534
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: 2.1.0
>
>
> KafkaIO reader logs a warning when it notices gaps in offsets for messages. 
> While such gaps are not expected for normal Kafka topics, there could be gaps 
> when log compaction is enabled (which deletes older messages for a key). 
> This warning log is not very useful. Also we should take such gaps while 
> estimating backlog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2549) gcsio should set timeouts for http requests

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072574#comment-16072574
 ] 

Jean-Baptiste Onofré commented on BEAM-2549:


Any update about this Jira for the 2.1.0 release ?

> gcsio should set timeouts for http requests
> ---
>
> Key: BEAM-2549
> URL: https://issues.apache.org/jira/browse/BEAM-2549
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: 2.1.0
>
>
> Use an http client with timeout value for apitool requests. Once this is, it 
> is also possible to simplify gcs read operations (i.e. remove the custom 
> solution for handling timeouts.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2508) Fix javaDoc of Stateful DoFn

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072567#comment-16072567
 ] 

Jean-Baptiste Onofré commented on BEAM-2508:


Any update on this Jira for 2.1.0 release ?

> Fix javaDoc of Stateful DoFn
> 
>
> Key: BEAM-2508
> URL: https://issues.apache.org/jira/browse/BEAM-2508
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Jingsong Lee
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>
> StateSpec > StateSpec



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2523) GCP IO exposes protobuf on its API surface, causing user pain

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2523:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> GCP IO exposes protobuf on its API surface, causing user pain
> -
>
> Key: BEAM-2523
> URL: https://issues.apache.org/jira/browse/BEAM-2523
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Putting the SDK, DataflowRunner, and GCP IO on the same classpath, results in 
> (at least) three versions of protobuf getting pulled in. These should be made 
> to converge. We should consider using maven enforcer, which I think can check 
> this.
> {code}
> [INFO] com.example:foo:jar:0.1
> [INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.0.0:compile
> [INFO] +- 
> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.0.0:compile
> [INFO] |  +- 
> org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.0.0:compile
> [INFO] |  |  \- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> [INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  |  \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
> [INFO] |  | \- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- 
> com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
> [INFO] |  |  +- 
> (com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.http-client:google-http-client:jar:1.20.0:compile 
> - omitted for conflict with 1.22.0)
> [INFO] |  |  +- 
> com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
> [INFO] |  |  |  +- 
> (com.google.http-client:google-http-client:jar:1.20.0:compile - omitted for 
> conflict with 1.22.0)
> [INFO] |  |  |  \- (com.google.protobuf:protobuf-java:jar:2.4.1:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.0.0:compile - 
> omitted for conflict with 3.2.0)
> [INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.9.6.2:compile
> [INFO] |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted 
> for duplicate)
> [INFO] |  |  +- (com.google.protobuf:protobuf-java:jar:3.2.0:compile - 
> omitted for duplicate)
> {code}
> Incidentally, the dependency plugin stopped supporting the verbose tree, so 
> we can't even visually inspect this except by downgrading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[Tarush Grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072568#comment-16072568
 ] 

Tarush Grover commented on BEAM-2424:
-

PR link - https://github.com/apache/beam/pull/3386

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2502) Processing time timers for expired windows are not ignored

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072565#comment-16072565
 ] 

Jean-Baptiste Onofré commented on BEAM-2502:


Any update on this Jira for the 2.1.0 release ?

> Processing time timers for expired windows are not ignored
> --
>
> Key: BEAM-2502
> URL: https://issues.apache.org/jira/browse/BEAM-2502
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.1.0
>
>
> If the ReduceFnRunner receives a processing time timer for an expired window, 
> it may produce output even though the window is expired (and may have already 
> sent a final output!)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2498:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Dataflow runner should shade Runner/Fn API protos
> -
>
> Key: BEAM-2498
> URL: https://issues.apache.org/jira/browse/BEAM-2498
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Just checked, and runners-core-construction is shaded but not the Runner API 
> protos. There may be a technical reason this cannot be done trivially, but we 
> need to work at it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2498) Dataflow runner should shade Runner/Fn API protos

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072564#comment-16072564
 ] 

Jean-Baptiste Onofré commented on BEAM-2498:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick on 2.1.0 release branch if you want.

> Dataflow runner should shade Runner/Fn API protos
> -
>
> Key: BEAM-2498
> URL: https://issues.apache.org/jira/browse/BEAM-2498
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Just checked, and runners-core-construction is shaded but not the Runner API 
> protos. There may be a technical reason this cannot be done trivially, but we 
> need to work at it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2486) Should throws some useful messages when statefulParDo use non-KV input

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2486:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Should throws some useful messages when statefulParDo use non-KV input
> --
>
> Key: BEAM-2486
> URL: https://issues.apache.org/jira/browse/BEAM-2486
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Jingsong Lee
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Now Flink runner will throws a ClassCastException without detail messages 
> when a statefulParDo use non-KV input. It is not easy for users to find 
> errors and causes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2486) Should throws some useful messages when statefulParDo use non-KV input

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072563#comment-16072563
 ] 

Jean-Baptiste Onofré commented on BEAM-2486:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick on 2.1.0 release branch if you want.

> Should throws some useful messages when statefulParDo use non-KV input
> --
>
> Key: BEAM-2486
> URL: https://issues.apache.org/jira/browse/BEAM-2486
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Jingsong Lee
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Now Flink runner will throws a ClassCastException without detail messages 
> when a statefulParDo use non-KV input. It is not easy for users to find 
> errors and causes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2424:
---
Fix Version/s: (was: 2.1.0)

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2390:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> allow user to use .setTimePartitioning in BigQueryIO.write
> --
>
> Key: BEAM-2390
> URL: https://issues.apache.org/jira/browse/BEAM-2390
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Eric Johston
>Assignee: Eric Johston
>  Labels: easyfix, features, newbie
> Fix For: 2.2.0
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently when writing to a table with BigQueryIO sink, there is no way to 
> create a new table that is date partitioned. This would be very useful, since 
> currently the only way to do this is  by manually creating a table ahead of 
> time. We should be able to leverage the automatic table creation 
> functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the 
> BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072559#comment-16072559
 ] 

Jean-Baptiste Onofré commented on BEAM-2390:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick on 2.1.0 release branch if you want.

> allow user to use .setTimePartitioning in BigQueryIO.write
> --
>
> Key: BEAM-2390
> URL: https://issues.apache.org/jira/browse/BEAM-2390
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Eric Johston
>Assignee: Eric Johston
>  Labels: easyfix, features, newbie
> Fix For: 2.2.0
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently when writing to a table with BigQueryIO sink, there is no way to 
> create a new table that is date partitioned. This would be very useful, since 
> currently the only way to do this is  by manually creating a table ahead of 
> time. We should be able to leverage the automatic table creation 
> functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the 
> BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2424) CAST operator supporting numeric, date and timestamp types

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072560#comment-16072560
 ] 

Jean-Baptiste Onofré commented on BEAM-2424:


Can you attach the PR link to the Jira ?

> CAST operator supporting numeric, date and timestamp types
> --
>
> Key: BEAM-2424
> URL: https://issues.apache.org/jira/browse/BEAM-2424
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2359) SparkTimerInternals inputWatermarkTime does not get updated in cluster mode

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072557#comment-16072557
 ] 

Jean-Baptiste Onofré commented on BEAM-2359:


Any update on this Jira for 2.1.0 release ?

> SparkTimerInternals inputWatermarkTime does not get updated in cluster mode
> ---
>
> Key: BEAM-2359
> URL: https://issues.apache.org/jira/browse/BEAM-2359
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
> Fix For: 2.1.0
>
>
> {{SparkTimerInternals#inputWatermarkTime}} does not get updated in cluster 
> mode.
> This causes windows to not get closed and state to increase forever in memory 
> and processing time to increase leading to eventual application crash (also, 
> triggers based on the watermark do not fire).
> The root cause is 
> a call from within the {{updateStateByKey}} operation in 
> [SparkGroupAlsoByWindowViaWindowSet|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java#L241-L242]
>  which tries to access a static reference to a {{GlobalWatermarkHolder}} 
> broadcast variable, however, in cluster mode this static reference would be a 
> different one in the executor's JVM and is null (this works in local mode 
> since the executor and driver are on the same JVM).
> Alternative Solutions (And viability of solution):
> * -Broadcast variable passed to the {{updateStateByKey}} operator- - Not 
> viable since even if we use the broadcast correctly, broadcast variables 
> can't be used in this case (from within {{updateStateByKey}}) since  
> {{updateStateByKey}} is a {{DStream}} operator and not an {{RDD}} operator so 
> it will not be updated every micro-batch but rather will retain the same 
> initial value.
> * -Broadcast variable to update the data in an additional transform- - Create 
> an additional transform on the {{DStream}}'s RDDs prior to the {{DStream}} 
> operator {{updateStateByKey}} and use a broadcast which will be updated 
> (since this is an {{RDD}} operator), and add this value to the keyed datum 
> itself so it will be available in the {{DStream}} operator 
> {{updateStateByKey}}. Not viable since this will only update keys which have 
> had new data appear in the microbatch, however we also want to update the 
> watermark value for keys which did not have new data appear in the microbatch.
> * -Broadcast variable to update a static reference- - Create an additional 
> transform on the {{DStream}}'s RDDs prior to the {{DStream}} operator 
> {{updateStateByKey}} and use a broadcast which will be updated (since this is 
> an {{RDD}} operator), and set this value in a static reference within the 
> executor. Not viable since we cannot ensure that all executors will receive 
> partitions to process in each microbatch.
> * Server to be polled lazily every microbatch from within the 
> {{updateStateByKey}} operator - Spin a server on some configured port on the 
> driver which will serve the current watermarks upon request. Lazily poll this 
> value every microbatch from within the {{updateStateByKey}} operator and 
> update a static reference within the executor. Viable, however does not use 
> Spark native operations and incurs code maintenance for this and operational 
> cost for the user (open ports in firewalls, etc.).
> * Drop/register watermarks as a block in BlockManager and request remote 
> version from within the {{updateStateByKey}} operator - Update watermarks as 
> a block in the BlockManager on the driver by dropping and reregistering the 
> block every microbatch. Lazily poll this value every microbatch from within 
> the {{updateStateByKey}} operator and update a static reference within the 
> executor. Viable, less "ugly" than the server version and requires less 
> operational cost.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072556#comment-16072556
 ] 

Jean-Baptiste Onofré commented on BEAM-2353:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick to 2.1.0 release branch if wanted.

> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072554#comment-16072554
 ] 

Jean-Baptiste Onofré commented on BEAM-2345:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick on the release branch if applicable.

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.2.0
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2353:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2345:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>Priority: Minor
> Fix For: 2.2.0
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2353) FileNamePolicy context parameters allow backwards compatibility where we really don't want any

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072556#comment-16072556
 ] 

Jean-Baptiste Onofré edited comment on BEAM-2353 at 7/3/17 2:51 PM:


I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick to 2.1.0 release branch you want.


was (Author: jbonofre):
I'm bumping fix version to 2.2.0. Please, feel free to update this Jira and 
cherry-pick to 2.1.0 release branch if wanted.

> FileNamePolicy context parameters allow backwards compatibility where we 
> really don't want any
> --
>
> Key: BEAM-2353
> URL: https://issues.apache.org/jira/browse/BEAM-2353
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>
> Currently, in {{FileBasedSink}} the {{FileNamePolicy}} object accepts 
> parameters of type {{Context}} and {{WindowedContext}} respectively.
> These contexts are a coding technique to allow easy backwards compatibility 
> when adding new parameters. However, if a new parameter is added to the file 
> name policy it is likely data loss for the user to not incorporate it, so in 
> fact that is never a safe backwards compatible change.
> These are brand-new APIs and marked experimental. This is important enough I 
> think we should make the breaking change.
> We should inline all the parameters of the context, so that we _cannot_ add 
> parameters and maintain compatibility. Instead, if we have new ones we want 
> to add, it will have to be a new method or some such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2328) Introduce Apache Tika Input component

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072549#comment-16072549
 ] 

Jean-Baptiste Onofré commented on BEAM-2328:


I'm still reviewing the PR. It's short to include in 2.1.0 release. Let's 
target on 2.2.0 release.

> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2328) Introduce Apache Tika Input component

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2328:
---
Fix Version/s: (was: 2.1.0)

> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2299) Beam repo build fails in Windows OS

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2299:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Beam repo build fails in Windows OS
> ---
>
> Key: BEAM-2299
> URL: https://issues.apache.org/jira/browse/BEAM-2299
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Jason Kuster
> Fix For: 2.2.0
>
>
> Need to run unit tests in Windows OS.
> Currently, many unit tests fail when doing "mvn clean install" in Windows OS.
> [ERROR] Errors:
> [ERROR]   AvroSourceTest.testCreationWithSchema:403 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   AvroSourceTest.testGetCurrentFromUnstartedReader:305 ? IllegalState 
> Un
> able to ...
> [ERROR]   AvroSourceTest.testGetProgressFromUnstartedReader:204 ? 
> IllegalState U
> nable to...
> [ERROR]   AvroSourceTest.testMultipleFiles:390 ? IllegalState Unable to find 
> reg
> istrar f...
> [ERROR]   AvroSourceTest.testProgress:225 ? IllegalState Unable to find 
> registra
> r for c
> [ERROR]   AvroSourceTest.testProgressEmptySource:278 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   AvroSourceTest.testReadMetadataWithCodecs:676 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testReadSchemaString:688 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testReadWithDifferentCodecs:158 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   AvroSourceTest.testSchemaIsInterned:460 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testSchemaStringIsInterned:441 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testSchemaUpdate:425 ? IllegalState Unable to find 
> regi
> strar fo...
> [ERROR]   AvroSourceTest.testSplitAtFraction:176 ? IllegalState Unable to 
> find r
> egistrar...
> [ERROR]   AvroSourceTest.testSplitAtFractionExhaustive:322 ? IllegalState 
> Unable
>  to find...
> [ERROR]   AvroSourceTest.testSplitsWithSmallBlocks:341 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   CompressedSourceTest.testEmptyGzipProgress:646 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   CompressedSourceTest.testGzipProgress:673 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   CompressedSourceTest.testSplittableProgress:739 ? IllegalState 
> Unable
> to find ...
> [ERROR]   CompressedSourceTest.testUncompressedFileIsSplittable:333 ? 
> IllegalSta
> te Unabl...
> [ERROR]   CompressedSourceTest.testUnsplittable:715 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   FileBasedSinkTest.testCopyToOutputFiles:301 ? IllegalState Unable 
> to f
> ind regi...
> [ERROR]   
> FileBasedSinkTest.testFinalize:154->generateTemporaryFilesForFinalize:
> 189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeMultipleCalls:161->generateTemporaryFile
> sForFinalize:189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeWithIntermediateState:171->generateTempo
> raryFilesForFinalize:189 ? IO
> [ERROR]   FileBasedSourceTest.testCloseUnstartedFilePatternReader:390 ? 
> IllegalS
> tate Una...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFile:746 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFilePattern:772 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testFractionConsumedWhenReadingFilepattern:422 
> ? I
> llegalState
> [ERROR]   FileBasedSourceTest.testFullyReadFilePattern:370 ? IllegalState 
> Unable
>  to find...
> [ERROR]   FileBasedSourceTest.testFullyReadFilePatternFirstRecordEmpty:461 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testFullyReadSingleFile:346 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfFilePattern:792 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfSingleFile:681 ? 
> IllegalState U
> nable to...
> [ERROR]   FileBasedSourceTest.testReadEverythingFromFileWithSplits:502 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadFileWithSplitsWithEmptyRange:578 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadRangeAtEnd:659 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   FileBasedSourceTest.testReadRangeAtMiddle:637 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   FileBasedSourceTest.testReadRangeAtStart:472 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddle:546 ? 
> Il
> legalState
> [ERROR]   
> FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddleOfHeader:
> 615 ? IllegalState
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromStart:517 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testSplitAtFraction:815 ? IllegalState Unable 
> to f
> ind regi...
> 

[jira] [Commented] (BEAM-2299) Beam repo build fails in Windows OS

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072547#comment-16072547
 ] 

Jean-Baptiste Onofré commented on BEAM-2299:


I'm bumping fix version to 2.2.0. If you can fix that for 2.1.0, please update 
this Jira and cherry pick on the 2.1.0 release branch.

> Beam repo build fails in Windows OS
> ---
>
> Key: BEAM-2299
> URL: https://issues.apache.org/jira/browse/BEAM-2299
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Jason Kuster
> Fix For: 2.2.0
>
>
> Need to run unit tests in Windows OS.
> Currently, many unit tests fail when doing "mvn clean install" in Windows OS.
> [ERROR] Errors:
> [ERROR]   AvroSourceTest.testCreationWithSchema:403 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   AvroSourceTest.testGetCurrentFromUnstartedReader:305 ? IllegalState 
> Un
> able to ...
> [ERROR]   AvroSourceTest.testGetProgressFromUnstartedReader:204 ? 
> IllegalState U
> nable to...
> [ERROR]   AvroSourceTest.testMultipleFiles:390 ? IllegalState Unable to find 
> reg
> istrar f...
> [ERROR]   AvroSourceTest.testProgress:225 ? IllegalState Unable to find 
> registra
> r for c
> [ERROR]   AvroSourceTest.testProgressEmptySource:278 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   AvroSourceTest.testReadMetadataWithCodecs:676 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testReadSchemaString:688 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testReadWithDifferentCodecs:158 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   AvroSourceTest.testSchemaIsInterned:460 ? IllegalState Unable to 
> find
> registra...
> [ERROR]   AvroSourceTest.testSchemaStringIsInterned:441 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   AvroSourceTest.testSchemaUpdate:425 ? IllegalState Unable to find 
> regi
> strar fo...
> [ERROR]   AvroSourceTest.testSplitAtFraction:176 ? IllegalState Unable to 
> find r
> egistrar...
> [ERROR]   AvroSourceTest.testSplitAtFractionExhaustive:322 ? IllegalState 
> Unable
>  to find...
> [ERROR]   AvroSourceTest.testSplitsWithSmallBlocks:341 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   CompressedSourceTest.testEmptyGzipProgress:646 ? IllegalState 
> Unable t
> o find r...
> [ERROR]   CompressedSourceTest.testGzipProgress:673 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   CompressedSourceTest.testSplittableProgress:739 ? IllegalState 
> Unable
> to find ...
> [ERROR]   CompressedSourceTest.testUncompressedFileIsSplittable:333 ? 
> IllegalSta
> te Unabl...
> [ERROR]   CompressedSourceTest.testUnsplittable:715 ? IllegalState Unable to 
> fin
> d regist...
> [ERROR]   FileBasedSinkTest.testCopyToOutputFiles:301 ? IllegalState Unable 
> to f
> ind regi...
> [ERROR]   
> FileBasedSinkTest.testFinalize:154->generateTemporaryFilesForFinalize:
> 189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeMultipleCalls:161->generateTemporaryFile
> sForFinalize:189 ? IO
> [ERROR]   
> FileBasedSinkTest.testFinalizeWithIntermediateState:171->generateTempo
> raryFilesForFinalize:189 ? IO
> [ERROR]   FileBasedSourceTest.testCloseUnstartedFilePatternReader:390 ? 
> IllegalS
> tate Una...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFile:746 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testEstimatedSizeOfFilePattern:772 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testFractionConsumedWhenReadingFilepattern:422 
> ? I
> llegalState
> [ERROR]   FileBasedSourceTest.testFullyReadFilePattern:370 ? IllegalState 
> Unable
>  to find...
> [ERROR]   FileBasedSourceTest.testFullyReadFilePatternFirstRecordEmpty:461 ? 
> Ill
> egalState
> [ERROR]   FileBasedSourceTest.testFullyReadSingleFile:346 ? IllegalState 
> Unable
> to find ...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfFilePattern:792 ? 
> IllegalState
> Unable t...
> [ERROR]   FileBasedSourceTest.testReadAllSplitsOfSingleFile:681 ? 
> IllegalState U
> nable to...
> [ERROR]   FileBasedSourceTest.testReadEverythingFromFileWithSplits:502 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadFileWithSplitsWithEmptyRange:578 ? 
> Illegal
> State Un...
> [ERROR]   FileBasedSourceTest.testReadRangeAtEnd:659 ? IllegalState Unable to 
> fi
> nd regis...
> [ERROR]   FileBasedSourceTest.testReadRangeAtMiddle:637 ? IllegalState Unable 
> to
>  find re...
> [ERROR]   FileBasedSourceTest.testReadRangeAtStart:472 ? IllegalState Unable 
> to
> find reg...
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddle:546 ? 
> Il
> legalState
> [ERROR]   
> FileBasedSourceTest.testReadRangeFromFileWithSplitsFromMiddleOfHeader:
> 615 ? IllegalState
> [ERROR]   FileBasedSourceTest.testReadRangeFromFileWithSplitsFromStart:517 ? 
> Ill
> 

[jira] [Commented] (BEAM-2298) Java WordCount doesn't work in Window OS for glob expressions or file: prefixed paths

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072546#comment-16072546
 ] 

Jean-Baptiste Onofré commented on BEAM-2298:


I'm bumping fix version to 2.2.0. If you can fix that for 2.1.0, please update 
this Jira and cherry pick on the 2.1.0 release branch.

> Java WordCount doesn't work in Window OS for glob expressions or file: 
> prefixed paths
> -
>
> Key: BEAM-2298
> URL: https://issues.apache.org/jira/browse/BEAM-2298
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Flavio Fiszman
> Fix For: 2.2.0
>
>
> I am not able to build beam repo in Windows OS, so I copied the jar file from 
> my Mac.
> WordCount failed with the following cmd:
> java -cp beam-examples-java-2.0.0-jar-with-dependencies.jar
>  org.apache.beam.examples.WordCount --inputFile=input.txt --output=counts
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource 
> getEstimatedSizeB
> ytes
> INFO: Filepattern input.txt matched 1 files with total size 0
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource 
> expandFilePattern
> INFO: Matched 1 files for pattern input.txt
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource split
> INFO: Splitting filepattern input.txt into bundles of size 0 took 0 ms and 
> produ
> ced 1 files and 0 bundles
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.WriteFiles$2 processElement
> INFO: Finalizing write operation 
> TextWriteOperation{tempDirectory=C:\Users\Pei\D
> esktop\.temp-beam-2017-05-135_13-09-48-1\, windowedWrites=false}.
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.WriteFiles$2 processElement
> INFO: Creating 1 empty output shards in addition to 0 written for a total of 
> 1.
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionExcepti
> on: java.lang.IllegalStateException: Unable to find registrar for c
> at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.wait
> UntilFinish(DirectRunner.java:322)
> at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.wait
> UntilFinish(DirectRunner.java:292)
> at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:200
> )
> at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:63)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281)
> at org.apache.beam.examples.WordCount.main(WordCount.java:184)
> Caused by: java.lang.IllegalStateException: Unable to find registrar for c
> at 
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.
> java:447)
> at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:111)
> at 
> org.apache.beam.sdk.io.FileSystems.matchResources(FileSystems.java:17
> 4)
> at 
> org.apache.beam.sdk.io.FileSystems.filterMissingFiles(FileSystems.jav
> a:367)
> at org.apache.beam.sdk.io.FileSystems.copy(FileSystems.java:251)
> at 
> org.apache.beam.sdk.io.FileBasedSink$WriteOperation.copyToOutputFiles
> (FileBasedSink.java:641)
> at 
> org.apache.beam.sdk.io.FileBasedSink$WriteOperation.finalize(FileBase
> dSink.java:529)
> at 
> org.apache.beam.sdk.io.WriteFiles$2.processElement(WriteFiles.java:59
> 2)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2298) Java WordCount doesn't work in Window OS for glob expressions or file: prefixed paths

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2298:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Java WordCount doesn't work in Window OS for glob expressions or file: 
> prefixed paths
> -
>
> Key: BEAM-2298
> URL: https://issues.apache.org/jira/browse/BEAM-2298
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Flavio Fiszman
> Fix For: 2.2.0
>
>
> I am not able to build beam repo in Windows OS, so I copied the jar file from 
> my Mac.
> WordCount failed with the following cmd:
> java -cp beam-examples-java-2.0.0-jar-with-dependencies.jar
>  org.apache.beam.examples.WordCount --inputFile=input.txt --output=counts
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource 
> getEstimatedSizeB
> ytes
> INFO: Filepattern input.txt matched 1 files with total size 0
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource 
> expandFilePattern
> INFO: Matched 1 files for pattern input.txt
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.FileBasedSource split
> INFO: Splitting filepattern input.txt into bundles of size 0 took 0 ms and 
> produ
> ced 1 files and 0 bundles
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.WriteFiles$2 processElement
> INFO: Finalizing write operation 
> TextWriteOperation{tempDirectory=C:\Users\Pei\D
> esktop\.temp-beam-2017-05-135_13-09-48-1\, windowedWrites=false}.
> May 15, 2017 6:09:48 AM org.apache.beam.sdk.io.WriteFiles$2 processElement
> INFO: Creating 1 empty output shards in addition to 0 written for a total of 
> 1.
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionExcepti
> on: java.lang.IllegalStateException: Unable to find registrar for c
> at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.wait
> UntilFinish(DirectRunner.java:322)
> at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.wait
> UntilFinish(DirectRunner.java:292)
> at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:200
> )
> at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:63)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281)
> at org.apache.beam.examples.WordCount.main(WordCount.java:184)
> Caused by: java.lang.IllegalStateException: Unable to find registrar for c
> at 
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.
> java:447)
> at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:111)
> at 
> org.apache.beam.sdk.io.FileSystems.matchResources(FileSystems.java:17
> 4)
> at 
> org.apache.beam.sdk.io.FileSystems.filterMissingFiles(FileSystems.jav
> a:367)
> at org.apache.beam.sdk.io.FileSystems.copy(FileSystems.java:251)
> at 
> org.apache.beam.sdk.io.FileBasedSink$WriteOperation.copyToOutputFiles
> (FileBasedSink.java:641)
> at 
> org.apache.beam.sdk.io.FileBasedSink$WriteOperation.finalize(FileBase
> dSink.java:529)
> at 
> org.apache.beam.sdk.io.WriteFiles$2.processElement(WriteFiles.java:59
> 2)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2272) Test scope & runtime dependencies need reevaluation

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072542#comment-16072542
 ] 

Jean-Baptiste Onofré commented on BEAM-2272:


I'm bumping fix version to 2.2.0. If you can fix that for 2.1.0, please update 
this Jira and cherry pick on the 2.1.0 release branch.

> Test scope & runtime dependencies need reevaluation
> ---
>
> Key: BEAM-2272
> URL: https://issues.apache.org/jira/browse/BEAM-2272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Now that SDK core has been thinned out, many of the modules that have it in 
> their {{dependenciesToScan}} will need to update their test-scoped 
> dependencies for things that moved out of SDK.
> Known examples include protobuf and XML and perhaps others.
> We should re-enable dependency analysis on test scope and manually validate 
> and ignore specific known runtime/test-runtime dependencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072543#comment-16072543
 ] 

Jean-Baptiste Onofré commented on BEAM-2273:


Any update on this Jira for 2.1.0 release ?

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jason Kuster
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2272) Test scope & runtime dependencies need reevaluation

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2272:
---
Fix Version/s: (was: 2.1.0)
   2.2.0

> Test scope & runtime dependencies need reevaluation
> ---
>
> Key: BEAM-2272
> URL: https://issues.apache.org/jira/browse/BEAM-2272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
> Fix For: 2.2.0
>
>
> Now that SDK core has been thinned out, many of the modules that have it in 
> their {{dependenciesToScan}} will need to update their test-scoped 
> dependencies for things that moved out of SDK.
> Known examples include protobuf and XML and perhaps others.
> We should re-enable dependency analysis on test scope and manually validate 
> and ignore specific known runtime/test-runtime dependencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072540#comment-16072540
 ] 

Jean-Baptiste Onofré commented on BEAM-2271:


Any update on this Jira for 2.1.0 release ?

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Ahmet Altay
> Fix For: 2.1.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2177) Support file scheme

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2177:
---
Fix Version/s: (was: 2.1.0)

> Support file scheme
> ---
>
> Key: BEAM-2177
> URL: https://issues.apache.org/jira/browse/BEAM-2177
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> Now, we support "new" filesystems using schema. For instance, it's possible 
> to do:
> {code}
> .apply(TextIO.write().to("hdfs://foo"))
> {code}
> Of course, if:
> {code}
> .apply(TextIO.write().to("/path/to/foo"))
> {code}
> works, users may be tempted to use:
> {code}
> .apply(TextIO.write().to("file://path/to/foo"))
> {code}
> which actually doesn't work today.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1612) Support real Bundle in Flink runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072536#comment-16072536
 ] 

Jean-Baptiste Onofré commented on BEAM-1612:


Any update on this one for 2.1.0 ?

> Support real Bundle in Flink runner
> ---
>
> Key: BEAM-1612
> URL: https://issues.apache.org/jira/browse/BEAM-1612
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> The Bundle is very important in the beam model. Users can use the bundle to 
> flush buffer, can reuse many heavyweight resources in a bundle. Most IO 
> plugins use the bundle to flush. 
> Moreover, FlinkRunner can also use Bundle to reduce access to the FlinkState, 
> such as first placed in JavaHeap, flush into RocksDbState when invoke 
> finishBundle , this can reduce the number of serialization.
> But now FlinkRunner calls the finishBundle every processElement. We need 
> support real Bundle.
> I think we can have the following implementations:
> 1.Invoke finishBundle and next startBundle in {{snapshot}} of Flink. But 
> sometimes this "Bundle" maybe too big. This depends on the user's checkpoint 
> configuration.
> 2.Manually control the size of the bundle. The half-bundle will be flushed to 
> a full-bundle by count or eventTime or processTime or {{snapshot}}. We do not 
> need to wait, just call the startBundle and finishBundle at the right time.
> [Proposal 
> document|https://docs.google.com/document/d/1UzELM4nFu8SIeu-QJkbs0sv7Uzd1Ux4aXXM3cw4s7po/edit?usp=sharing]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-893) Update Mobile Gaming Example

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072535#comment-16072535
 ] 

Jean-Baptiste Onofré commented on BEAM-893:
---

I'm bumping fix version to 2.2.0. If you can fix that for 2.1.0, please update 
this Jira and cherry pick on the 2.1.0 release branch.

> Update Mobile Gaming Example
> 
>
> Key: BEAM-893
> URL: https://issues.apache.org/jira/browse/BEAM-893
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: 2.2.0
>
>
> - In the intro: As well as offline, this doc should also mention delays that 
> can happen in the network at various points, so that data may arrive delayed 
> and out of order.
> - For Figure 1: Explain more clearly what the two axes mean and how a 
> specific point might have a given position ('ideal' line (x=y) is very 
> helpful here).
> - In section "What Does Leaderboard Do?": These aren't exact 10 minute 
> intervals - possibly because of the arrival pattern of the data. Nothing will 
> be triggered if there's no data that arrives. @kennknowles can probably 
> provide more precise phrasing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-893) Update Mobile Gaming Example

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-893:
--
Fix Version/s: (was: 2.1.0)
   2.2.0

> Update Mobile Gaming Example
> 
>
> Key: BEAM-893
> URL: https://issues.apache.org/jira/browse/BEAM-893
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: 2.2.0
>
>
> - In the intro: As well as offline, this doc should also mention delays that 
> can happen in the network at various points, so that data may arrive delayed 
> and out of order.
> - For Figure 1: Explain more clearly what the two axes mean and how a 
> specific point might have a given position ('ideal' line (x=y) is very 
> helpful here).
> - In section "What Does Leaderboard Do?": These aren't exact 10 minute 
> intervals - possibly because of the arrival pattern of the data. Nothing will 
> be triggered if there's no data that arrives. @kennknowles can probably 
> provide more precise phrasing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072533#comment-16072533
 ] 

Jean-Baptiste Onofré commented on BEAM-659:
---

[~kenn] I guess this Jira can be fixed, right ?

> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Huafeng Wang
>Priority: Minor
>  Labels: easy, easyfix, starter
> Fix For: 2.1.0
>
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-667) Include code snippets from real examples

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072531#comment-16072531
 ] 

Jean-Baptiste Onofré commented on BEAM-667:
---

I'm bumping fix version to 2.2.0. If you can fix that for 2.1.0, please update 
this Jira and cherry pick on the 2.1.0 release branch.

> Include code snippets from real examples
> 
>
> Key: BEAM-667
> URL: https://issues.apache.org/jira/browse/BEAM-667
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Melissa Pashniak
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-667) Include code snippets from real examples

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-667:
--
Fix Version/s: (was: 2.1.0)
   2.2.0

> Include code snippets from real examples
> 
>
> Key: BEAM-667
> URL: https://issues.apache.org/jira/browse/BEAM-667
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Melissa Pashniak
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3502

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2542

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1900

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2541

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2530) Make Beam compatible with Java 9

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-2530.

   Resolution: Fixed
 Assignee: Ismaël Mejía
Fix Version/s: 2.1.0

> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.1.0
>
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2530) Make Beam compatible with Java 9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072340#comment-16072340
 ] 

ASF GitHub Bot commented on BEAM-2530:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3460


> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3460: [BEAM-2530] Fix compilation of modules with Java 9 ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3460


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: [BEAM-2530] This closes #3460

[jbonofre]

[BEAM-2530] This closes #3460


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0bd47c07
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0bd47c07
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0bd47c07

Branch: refs/heads/master
Commit: 0bd47c07659909c6b3c2d04fbdb96cc30a6e0243
Parents: 11010cf 75475ef
Author: Jean-Baptiste Onofré 
Authored: Mon Jul 3 14:23:18 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 14:23:18 2017 +0200

--
 runners/apex/pom.xml  |  7 +++
 runners/spark/pom.xml |  7 +++
 sdks/java/io/hbase/pom.xml|  7 +++
 sdks/java/io/hcatalog/pom.xml | 12 
 4 files changed, 33 insertions(+)
--

[1/2] beam git commit: [BEAM-2530] Fix compilation of modules with Java 9 that depend on jdk.tools

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master 11010cf08 -> 0bd47c076


[BEAM-2530] Fix compilation of modules with Java 9 that depend on jdk.tools


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/75475ef3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/75475ef3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/75475ef3

Branch: refs/heads/master
Commit: 75475ef3dc23a09fa9bbba478d6fdbc468f7dd2e
Parents: 11010cf
Author: Ismaël Mejía 
Authored: Wed Jun 28 16:58:55 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 10:10:04 2017 +0200

--
 runners/apex/pom.xml  |  7 +++
 runners/spark/pom.xml |  7 +++
 sdks/java/io/hbase/pom.xml|  7 +++
 sdks/java/io/hcatalog/pom.xml | 12 
 4 files changed, 33 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 88ff0f2..20f2d28 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -75,6 +75,13 @@
   apex-engine
   ${apex.core.version}
   runtime
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 1d93427..8a69496 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -149,6 +149,13 @@
   org.apache.hadoop
   hadoop-common
   provided
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
   org.apache.hadoop

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/sdks/java/io/hbase/pom.xml
--
diff --git a/sdks/java/io/hbase/pom.xml b/sdks/java/io/hbase/pom.xml
index 9d5e2aa..40ac8df 100644
--- a/sdks/java/io/hbase/pom.xml
+++ b/sdks/java/io/hbase/pom.xml
@@ -121,6 +121,13 @@
   org.apache.hadoop
   hadoop-common
   test
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/75475ef3/sdks/java/io/hcatalog/pom.xml
--
diff --git a/sdks/java/io/hcatalog/pom.xml b/sdks/java/io/hcatalog/pom.xml
index 8af740d..a31ff86 100644
--- a/sdks/java/io/hcatalog/pom.xml
+++ b/sdks/java/io/hcatalog/pom.xml
@@ -61,6 +61,13 @@
 
   org.apache.hadoop
   hadoop-common
+  
+
+
+  jdk.tools
+  jdk.tools
+
+  
 
 
 
@@ -109,6 +116,11 @@
   com.google.protobuf
   protobuf-java
 
+
+
+  jdk.tools
+  jdk.tools
+
   
 
 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1899

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
 > git rev-list 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 53950
maven33-agent.jar already up to date
maven33-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_ValidatesRunner_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS 
-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -cp 
/home/jenkins/jenkins-slave/maven33-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven33Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven33-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 53950
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--activate-profiles validates-runner-tests --activate-profiles 
local-validates-runner-tests
2017-07-03T12:03:10.605 [INFO] Error stacktraces are turned on.
2017-07-03T12:03:10.668 [INFO] Scanning for projects...
2017-07-03T12:03:11.453 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
2017-07-03T12:03:11.841 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
 (7 KB at 15.9 KB/sec)
2017-07-03T12:03:11.848 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-07-03T12:03:11.878 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 213.8 KB/sec)
2017-07-03T12:03:11.886 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-07-03T12:03:11.915 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 113.9 KB/sec)
2017-07-03T12:03:11.917 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-07-03T12:03:11.955 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 565.6 KB/sec)
2017-07-03T12:03:11.958 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-07-03T12:03:11.998 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 816.0 KB/sec)
2017-07-03T12:03:12.001 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-07-03T12:03:12.032 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 440.2 KB/sec)
2017-07-03T12:03:12.038 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-07-03T12:03:12.070 [INFO] Downloaded: 

Build failed in Jenkins: beam_PerformanceTests_Python #57

[Apache Jenkins Server]

See 


Changes:

[jbonofre] Make modules that depend on Hadoop and Spark use the same version

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
 > git rev-list 0e429b33ff85eba08da5018c9febd0b99b44f720 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson848114572567495280.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson5110391616487205278.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson2987471893144896789.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson3200198662023629685.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4286

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2540

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2539

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3501

[Apache Jenkins Server]

See 


Changes:

[jbonofre] Make modules that depend on Hadoop and Spark use the same version

--
[...truncated 920.06 KB...]
2017-07-03T07:38:11.701 [INFO] 
2017-07-03T07:38:11.701 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:11.704 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-07-03T07:38:11.762 [INFO] 
2017-07-03T07:38:11.762 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:11.813 [INFO] 
2017-07-03T07:38:11.813 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:11.867 [INFO] 
2017-07-03T07:38:11.867 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:11.924 [INFO] 
2017-07-03T07:38:11.924 [INFO] --- maven-checkstyle-plugin:2.17:check (default) 
@ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.414 [INFO] Starting audit...
Audit done.
2017-07-03T07:38:12.465 [INFO] 
2017-07-03T07:38:12.465 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.515 [INFO] 
2017-07-03T07:38:12.515 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.615 [INFO] 
2017-07-03T07:38:12.615 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.616 [INFO] Skipping packaging of the jar
2017-07-03T07:38:12.715 [INFO] 
2017-07-03T07:38:12.715 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.716 [INFO] Skipping packaging of the test-jar
2017-07-03T07:38:12.766 [INFO] 
2017-07-03T07:38:12.766 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.770 [INFO] Replacing original artifact with shaded artifact.
2017-07-03T07:38:12.854 [INFO] 
2017-07-03T07:38:12.854 [INFO] --- maven-dependency-plugin:3.0.1:analyze-only 
(default) @ beam-sdks-java-extensions-parent ---
2017-07-03T07:38:12.855 [INFO] Skipping pom project
[JENKINS] Archiving disabled
2017-07-03T07:38:16.849 [INFO]  
   
2017-07-03T07:38:16.849 [INFO] 

2017-07-03T07:38:16.849 [INFO] Building Apache Beam :: SDKs :: Java :: 
Extensions :: Google Cloud Platform Core 2.1.0-SNAPSHOT
2017-07-03T07:38:16.849 [INFO] 

2017-07-03T07:38:16.856 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/http-client/google-http-client-jackson2/1.22.0/google-http-client-jackson2-1.22.0.pom
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
2017-07-03T07:38:16.915 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-oauth2-http/0.6.1/google-auth-library-oauth2-http-0.6.1.pom
2017-07-03T07:38:16.950 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-oauth2-http/0.6.1/google-auth-library-oauth2-http-0.6.1.pom
 (3 KB at 61.0 KB/sec)
2017-07-03T07:38:16.952 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-parent/0.6.1/google-auth-library-parent-0.6.1.pom
2017-07-03T07:38:16.978 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-parent/0.6.1/google-auth-library-parent-0.6.1.pom
 (8 KB at 306.0 KB/sec)
2017-07-03T07:38:16.980 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auth/google-auth-library-credentials/0.6.1/google-auth-library-credentials-0.6.1.pom
2017-07-03T07:38:17.006 [INFO] Downloaded: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3500

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3291: Make all Hadoop related IOs use the same version pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3291


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3291

[jbonofre]

This closes #3291


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/11010cf0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/11010cf0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/11010cf0

Branch: refs/heads/master
Commit: 11010cf08ec9ca90a843cc15ea8f0fd4910f7fab
Parents: 0e429b3 ce4e517
Author: Jean-Baptiste Onofré 
Authored: Mon Jul 3 09:26:40 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 09:26:40 2017 +0200

--
 examples/java/pom.xml   | 18 --
 examples/java8/pom.xml  | 18 --
 pom.xml | 65 +++-
 runners/apex/pom.xml|  2 +-
 runners/spark/pom.xml   |  7 ---
 sdks/java/extensions/sorter/pom.xml |  6 --
 sdks/java/io/hadoop-file-system/pom.xml | 31 --
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml|  2 -
 sdks/java/io/hbase/pom.xml  |  9 ++-
 sdks/java/io/hcatalog/pom.xml   |  6 +-
 sdks/java/io/jdbc/pom.xml   |  2 -
 sdks/java/io/pom.xml| 31 --
 sdks/java/javadoc/pom.xml   |  2 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 15 files changed, 98 insertions(+), 103 deletions(-)
--

[1/2] beam git commit: Make modules that depend on Hadoop and Spark use the same version property

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master 0e429b33f -> 11010cf08


Make modules that depend on Hadoop and Spark use the same version property


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ce4e5174
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ce4e5174
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ce4e5174

Branch: refs/heads/master
Commit: ce4e5174750ae2c4b1691c6994bd0f92e161
Parents: 0e429b3
Author: Ismaël Mejía 
Authored: Sun Jun 4 22:55:05 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Mon Jul 3 08:19:02 2017 +0200

--
 examples/java/pom.xml   | 18 --
 examples/java8/pom.xml  | 18 --
 pom.xml | 65 +++-
 runners/apex/pom.xml|  2 +-
 runners/spark/pom.xml   |  7 ---
 sdks/java/extensions/sorter/pom.xml |  6 --
 sdks/java/io/hadoop-file-system/pom.xml | 31 --
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml|  2 -
 sdks/java/io/hbase/pom.xml  |  9 ++-
 sdks/java/io/hcatalog/pom.xml   |  6 +-
 sdks/java/io/jdbc/pom.xml   |  2 -
 sdks/java/io/pom.xml| 31 --
 sdks/java/javadoc/pom.xml   |  2 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 .../main/resources/archetype-resources/pom.xml  |  1 -
 15 files changed, 98 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 701e4fe..7ae4e6a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -34,10 +34,6 @@
 
   jar
 
-  
-1.6.2
-  
-
   
 
 
+2.7.3
 1.3
 2.8.9
 3.0.1
@@ -139,7 +145,7 @@
 v1-rev10-1.22.0
 1.7.14
 0.20.0-beta
-1.6.2
+1.6.3
 4.3.5.RELEASE
 3.1.4
 v1-rev71-1.22.0
@@ -1075,6 +1081,42 @@
 ${snappy-java.version}
   
 
+  
+org.apache.hadoop
+hadoop-client
+${hadoop.version}
+  
+
+  
+org.apache.hadoop
+hadoop-common
+${hadoop.version}
+  
+
+  
+org.apache.hadoop
+hadoop-mapreduce-client-core
+${hadoop.version}
+  
+
+  
+org.apache.spark
+spark-core_2.10
+${spark.version}
+  
+
+  
+org.apache.spark
+spark-streaming_2.10
+${spark.version}
+  
+
+  
+org.apache.spark
+spark-network-common_2.10
+${spark.version}
+  
+
   
 
   
@@ -1144,6 +1186,27 @@
 test
   
 
+  
+org.apache.hadoop
+hadoop-minicluster
+${hadoop.version}
+test
+  
+
+  
+org.apache.hadoop
+hadoop-hdfs
+${hadoop.version}
+test
+  
+
+  
+org.apache.hadoop
+hadoop-hdfs
+${hadoop.version}
+tests
+test
+  
 
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 2c54654..88ff0f2 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -261,7 +261,7 @@
 
com.esotericsoftware.kryo:kryo::${apex.kryo.version}
 
com.datatorrent:netlet::1.3.0
 
org.slf4j:slf4j-api:jar:1.7.14
-
org.apache.hadoop:hadoop-common:jar:2.6.0
+
org.apache.hadoop:hadoop-common:jar:${hadoop.version}
 
joda-time:joda-time:jar:2.4
 
com.google.guava:guava:jar:20.0
   

http://git-wip-us.apache.org/repos/asf/beam/blob/ce4e5174/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index ee72dd9..1d93427 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -34,8 +34,6 @@
   
 UTF-8
 UTF-8
-1.6.3
-2.2.0
 0.9.0.1
 2.4.4
 3.1.2
@@ -135,31 +133,26 @@
 
   org.apache.spark
   spark-core_2.10
-  ${spark.version}
   provided
 
 
   org.apache.spark
   spark-streaming_2.10
-  ${spark.version}
   provided
 
 
   org.apache.spark
   spark-network-common_2.10
-  ${spark.version}
   provided
 
 
   org.apache.hadoop
   hadoop-common
-  ${hadoop.version}
   provided
 
 
   org.apache.hadoop
   

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4285

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2384) CEIL, FLOOR, TRUNCATE, RAND_INTEGER, ATAN2 math function

[Tarush Grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarush Grover updated BEAM-2384:

Summary: CEIL, FLOOR, TRUNCATE, RAND_INTEGER, ATAN2 math function  (was: 
TRUNCATE math function)

> CEIL, FLOOR, TRUNCATE, RAND_INTEGER, ATAN2 math function
> 
>
> Key: BEAM-2384
> URL: https://issues.apache.org/jira/browse/BEAM-2384
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2538

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #56

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e429b33ff85eba08da5018c9febd0b99b44f720 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e429b33ff85eba08da5018c9febd0b99b44f720
 > git rev-list 0e429b33ff85eba08da5018c9febd0b99b44f720 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson4576748700706139736.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson6903800885240862810.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson5586061990740441910.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson6962585333128719668.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
  Installing extra requirements: 'gcp,test'
Requirement