date:20171023

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3348

2017-10-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Python #482

2017-10-23 Thread Apache Jenkins Server

See 


Changes:

[github] Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4

[kenn] Make the main seed job standalone

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fefe153cdb6ed19545352a622f88c657a67901f1 (origin/master)
Commit message: "This closes #4019: Make the main seed job standalone"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fefe153cdb6ed19545352a622f88c657a67901f1
 > git rev-list 77c77add12fb9b118f4ff1418d7fe5dec5154895 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5899155917459927528.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2250498170732466012.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3420801690099024119.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins791821222141556734.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #3347

2017-10-23 Thread Apache Jenkins Server

See 


Changes:

[kenn] Make the main seed job standalone

--
[...truncated 637.79 KB...]
2017-10-24T04:42:26.938 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-10-24T04:42:26.938 [INFO] 
:
 Recompile with -Xlint:unchecked for details.
2017-10-24T04:42:27.248 [INFO] 
2017-10-24T04:42:27.248 [INFO] --- maven-resources-plugin:3.0.2:testResources 
(default-testResources) @ beam-runners-spark ---
2017-10-24T04:42:27.249 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-10-24T04:42:27.250 [INFO] Copying 3 resources
2017-10-24T04:42:27.257 [INFO] Copying 3 resources
2017-10-24T04:42:27.365 [INFO] 
2017-10-24T04:42:27.365 [INFO] --- maven-compiler-plugin:3.6.2:testCompile 
(default-testCompile) @ beam-runners-spark ---
2017-10-24T04:42:27.404 [INFO] Changes detected - recompiling the module!
2017-10-24T04:42:27.407 [INFO] Compiling 25 source files to 

2017-10-24T04:42:30.400 [INFO] 
:
 

 uses or overrides a deprecated API.
2017-10-24T04:42:30.400 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-10-24T04:42:30.403 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-10-24T04:42:30.403 [INFO] 
:
 Recompile with -Xlint:unchecked for details.
2017-10-24T04:42:30.681 [INFO] 
2017-10-24T04:42:30.682 [INFO] --- maven-checkstyle-plugin:2.17:check (default) 
@ beam-runners-spark ---
2017-10-24T04:42:33.984 [INFO] Starting audit...
Audit done.
2017-10-24T04:42:34.096 [INFO] 
2017-10-24T04:42:34.096 [INFO] --- maven-surefire-plugin:2.20:test 
(default-test) @ beam-runners-spark ---
2017-10-24T04:42:34.103 [INFO] Surefire report directory: 

2017-10-24T04:42:34.107 [INFO] Using configured provider 
org.apache.maven.surefire.junitcore.JUnitCoreProvider
2017-10-24T04:42:34.108 [INFO] parallel='none', perCoreThreadCount=true, 
threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-10-24T04:42:34.109 [INFO] 
2017-10-24T04:42:34.115 [INFO] 
---
2017-10-24T04:42:34.115 [INFO]  T E S T S
2017-10-24T04:42:34.115 [INFO] 
---
2017-10-24T04:42:35.677 [INFO] Running 
org.apache.beam.runners.spark.metrics.SparkBeamMetricTest
2017-10-24T04:42:35.755 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.072 s - in 
org.apache.beam.runners.spark.metrics.SparkBeamMetricTest
2017-10-24T04:42:39.603 [INFO] Running 
org.apache.beam.runners.spark.GlobalWatermarkHolderTest
2017-10-24T04:42:48.704 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 9.1 s - in 
org.apache.beam.runners.spark.GlobalWatermarkHolderTest
2017-10-24T04:42:50.877 [INFO] Running 
org.apache.beam.runners.spark.SparkRunnerDebuggerTest
2017-10-24T04:43:02.769 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 11.89 s - in 
org.apache.beam.runners.spark.SparkRunnerDebuggerTest
2017-10-24T04:43:04.256 [INFO] Running 
org.apache.beam.runners.spark.io.ReaderToIteratorAdapterTest
2017-10-24T04:43:04.686 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.428 s - in 
org.apache.beam.runners.spark.io.ReaderToIteratorAdapterTest
2017-10-24T04:43:09.098 [INFO] Running 
org.apache.beam.runners.spark.io.NumShardsTest
2017-10-24T04:43:22.439 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 13.34 s - in

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5095

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 300.22 KB...]
2017-10-24T04:34:52.623 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.623 [INFO] 

2017-10-24T04:34:52.623 [INFO]  
   
2017-10-24T04:34:52.623 [INFO] 

2017-10-24T04:34:52.623 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-24T04:34:52.623 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.623 [INFO] 

2017-10-24T04:34:52.623 [INFO]  
   
2017-10-24T04:34:52.623 [INFO] 

2017-10-24T04:34:52.623 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-24T04:34:52.623 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.623 [INFO] 

2017-10-24T04:34:52.623 [INFO]  
   
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-24T04:34:52.624 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO]  
   
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-24T04:34:52.624 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO]  
   
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-24T04:34:52.624 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.624 [INFO]  
   
2017-10-24T04:34:52.624 [INFO] 

2017-10-24T04:34:52.625 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-24T04:34:52.625 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.625 [INFO] 

2017-10-24T04:34:52.625 [INFO]  
   
2017-10-24T04:34:52.625 [INFO] 

2017-10-24T04:34:52.625 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-24T04:34:52.625 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.625 [INFO] 

2017-10-24T04:34:52.625 [INFO]  
   
2017-10-24T04:34:52.625 [INFO] 

2017-10-24T04:34:52.625 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-24T04:34:52.625 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.625 [INFO] 

2017-10-24T04:34:52.625 [INFO]  
   
2017-10-24T04:34:52.626 [INFO] 

2017-10-24T04:34:52.626 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-24T04:34:52.626 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T04:34:52.626 [INFO] 

2017-10-24T04:34:52.626 [INFO]

[jira] [Updated] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

2017-10-23 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-793:
--
Fix Version/s: (was: 2.2.0)
   2.3.0

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.3.0
>
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3412

2017-10-23 Thread Apache Jenkins Server

See 


Changes:

[kenn] Make the main seed job standalone

--
[...truncated 932.13 KB...]
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py ->

[jira] [Commented] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

2017-10-23 Thread Xu Mingmin (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216282#comment-16216282
 ] 

Xu Mingmin commented on BEAM-3093:
--

Mostly it's about case (1) and (2), but I think it's more clear if we list the 
4 scenarios together in one method.

Reset offset with admin commands may not be easy for developers, especially in 
production environment. (I don't have the permission.)

> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2947) SDK harnesses should be able to indicate panic in FnAPI

2017-10-23 Thread Henning Rohde (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-2947:
---

Assignee: Henning Rohde

> SDK harnesses should be able to indicate panic in FnAPI
> ---
>
> Key: BEAM-2947
> URL: https://issues.apache.org/jira/browse/BEAM-2947
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
>
> If the SDK harness encounters a permanent problem (local file corruption or 
> bad staged files, say) it should be able to rely that information to the 
> runner as a "panic" message, say. Such conditions may also happen during the 
> boot stage.
> Using process/container exit codes would likely not work well, because that 
> information may not flow easily to the runner logic depending on how 
> containers are managed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3005) Provision API should optionally include SDK harness memory/cpu/disk limits

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216264#comment-16216264
 ] 

ASF GitHub Bot commented on BEAM-3005:
--

GitHub user herohde opened a pull request:

https://github.com/apache/beam/pull/4029

[BEAM-3005] Add resource limits to provisioning

* Use memory limit for JVM max heap size.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/herohde/beam containers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4029.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4029


commit 1514f5a36376c12bef8597b9b37505a01cfec9f7
Author: Henning Rohde 
Date:   2017-10-20T21:38:54Z

[BEAM-3005] Add resource limits to provision proto

commit b37e4d1c23b9d7291d8063e525a554ae343fe0ab
Author: Henning Rohde 
Date:   2017-10-23T16:56:55Z

[BEAM-3005] Set JVM max heap size in java container

commit 9e8cc1b77e6afbccb181f81db6bc201f40dde68d
Author: Henning Rohde 
Date:   2017-10-23T16:59:08Z

Declare .go and Dockerfile as text in gitattributes




> Provision API should optionally include SDK harness memory/cpu/disk limits
> --
>
> Key: BEAM-3005
> URL: https://issues.apache.org/jira/browse/BEAM-3005
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
>
> SDK harness containers should be told what resources they can use via the 
> provision API, notably to configure the JVM heap limit. Disk usage (in the 
> semi-persistent location) and cpu share can also be added. These limits -- if 
> provided -- may be enforced at the discretion of the runner (or the reality 
> of the execution environment).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4029: [BEAM-3005] Add resource limits to provisioning

2017-10-23 Thread herohde

GitHub user herohde opened a pull request:

https://github.com/apache/beam/pull/4029

[BEAM-3005] Add resource limits to provisioning

* Use memory limit for JVM max heap size.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/herohde/beam containers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4029.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4029


commit 1514f5a36376c12bef8597b9b37505a01cfec9f7
Author: Henning Rohde 
Date:   2017-10-20T21:38:54Z

[BEAM-3005] Add resource limits to provision proto

commit b37e4d1c23b9d7291d8063e525a554ae343fe0ab
Author: Henning Rohde 
Date:   2017-10-23T16:56:55Z

[BEAM-3005] Set JVM max heap size in java container

commit 9e8cc1b77e6afbccb181f81db6bc201f40dde68d
Author: Henning Rohde 
Date:   2017-10-23T16:59:08Z

Declare .go and Dockerfile as text in gitattributes




---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5094

2017-10-23 Thread Apache Jenkins Server

See 


Changes:

[kenn] Make the main seed job standalone

--
[...truncated 300.15 KB...]
2017-10-24T03:46:37.175 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO]  
   
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-24T03:46:37.175 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO]  
   
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-24T03:46:37.175 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO]  
   
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.175 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-24T03:46:37.175 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.175 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO]  
   
2017-10-24T03:46:37.176 [INFO] 

2017-10-24T03:46:37.176 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-24T03:46:37.176 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-24T03:46:37.176 [INFO]

[jira] [Comment Edited] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

2017-10-23 Thread Raghu Angadi (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216107#comment-16216107
 ] 

Raghu Angadi edited comment on BEAM-3093 at 10/24/17 12:13 AM:
---

(3) is same as setting "auto.offset.reset" Kafka consumer config to "earliest".
(4) is the default behavior. Same as setting ""auto.offset.reset" config to 
"latest".

It is better not to mix 'checkpointed' offsets and 'auto_committed' offsets. If 
you restart a job from scratch, 'checkpointed' offsets are discarded out as 
well.

(1) & (2) might be useful in the case of 'auto_committed' offsets. User can 
always remove auto_committed offsets in Kafka through admin commands. In that 
sense, (1) is same as '(3) with auto committed offsets reset.'. In fact, 
resetting these offsets using Kafka admin commands gives you much better 
control on where you want to start processing. E.g. you could resume from 24 
hours ago rather then from max retention period for Kafka.





was (Author: rangadi):
(3) is same as setting "auto.offset.reset" Kafka consumer config to "earliest".
(4) is the default behavior. Same as setting ""auto.offset.reset" config to 
"latest".

It is better not to mix 'checkpointed' offsets and 'auto_committed' offsets. If 
you restart a job scratch, 'checkpointed' offsets are thrown out as well.

(1) & (2) might be useful in the case of 'auto_committed' offsets. User can 
always remove auto_committed offsets in Kafka through admin commands. In that 
sense, (1) is same as '(3) with auto committed offsets reset.'. In fact, 
resetting these offsets using Kafka admin commands gives you much better 
control on where you want to start processing. E.g. you could resume from 24 
hours ago rather then from max retention period for Kafka.




> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

2017-10-23 Thread Raghu Angadi (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216107#comment-16216107
 ] 

Raghu Angadi commented on BEAM-3093:


(3) is same as setting "auto.offset.reset" Kafka consumer config to "earliest".
(4) is the default behavior. Same as setting ""auto.offset.reset" config to 
"latest".

It is better not to mix 'checkpointed' offsets and 'auto_committed' offsets. If 
you restart a job scratch, 'checkpointed' offsets are thrown out as well.

(1) & (2) might be useful in the case of 'auto_committed' offsets. User can 
always remove auto_committed offsets in Kafka through admin commands. In that 
sense, (1) is same as '(3) with auto committed offsets reset.'. In fact, 
resetting these offsets using Kafka admin commands gives you much better 
control on where you want to start processing. E.g. you could resume from 24 
hours ago rather then from max retention period for Kafka.




> add an option 'FirstPollOffsetStrategy' to KafkaIO
> --
>
> Key: BEAM-3093
> URL: https://issues.apache.org/jira/browse/BEAM-3093
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>
> This is a feature borrowed from Storm KafkaSpout.
> *What's the issue?*
> In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
> cannot be changed in application, to force to read from earliest/latest. 
> --This feature is important to reset the start offset when relaunching a job.
> *Proposed solution:*
> By borrowing the FirstPollOffsetStrategy concept, users can have more options:
> 1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
> checkpoint/auto_commit;
> 2). *{{LATEST}}*: always start_from_latest no matter of what's in 
> checkpoint/auto_commit;
> 3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_beginning if, otherwise start_from_previous_offset;
> 4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
> start_from_latest, otherwise start_from_previous_offset;
> [~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3093) add an option 'FirstPollOffsetStrategy' to KafkaIO

2017-10-23 Thread Xu Mingmin (JIRA)

Xu Mingmin created BEAM-3093:


 Summary: add an option 'FirstPollOffsetStrategy' to KafkaIO
 Key: BEAM-3093
 URL: https://issues.apache.org/jira/browse/BEAM-3093
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Xu Mingmin
Assignee: Kenneth Knowles


This is a feature borrowed from Storm KafkaSpout.

*What's the issue?*
In KafkaIO, when offset is stored either in checkpoint or auto_committed, it 
cannot be changed in application, to force to read from earliest/latest. --This 
feature is important to reset the start offset when relaunching a job.

*Proposed solution:*
By borrowing the FirstPollOffsetStrategy concept, users can have more options:
1). *{{EARLIEST}}*: always start_from_beginning no matter of what's in 
checkpoint/auto_commit;
2). *{{LATEST}}*: always start_from_latest no matter of what's in 
checkpoint/auto_commit;
3). *{{UNCOMMITTED_EARLIEST}}*: if no offset in checkpoint/auto_commit then 
start_from_beginning if, otherwise start_from_previous_offset;
4). *{{UNCOMMITTED_LATEST}}*: if no offset in checkpoint/auto_commit then 
start_from_latest, otherwise start_from_previous_offset;

[~rangadi], any comments?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2227) add a post to explain how Beam SQL works

2017-10-23 Thread Xu Mingmin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2227.

   Resolution: Fixed
Fix Version/s: Not applicable

> add a post to explain how Beam SQL works
> 
>
> Key: BEAM-2227
> URL: https://issues.apache.org/jira/browse/BEAM-2227
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql, website
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
> Fix For: Not applicable
>
>
> As mentioned in maillist, it's important to clarify what Beam SQL does, and 
> how it works.
> {quote}
> There's some confusion where people think we're just doing a pass through
> to the framework's SQL engine. We'll have to make sure we're clear on how
> Beam's SQL works in the docs.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-1705) visualize the DAG of Pipeline

2017-10-23 Thread Xu Mingmin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-1705.

   Resolution: Won't Fix
Fix Version/s: Not applicable

> visualize the DAG of Pipeline
> -
>
> Key: BEAM-1705
> URL: https://issues.apache.org/jira/browse/BEAM-1705
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> After pipeline is compiled and run in different runners, such as Spark, 
> Flink, it shows the data flow as a graph. There's limitation here:
> 1. the job must be run first;
> 2. The pipeline in a runner is not exact same as original designed, some 
> steps are merged for example.
> What to do:
> a method or tool to show the DAG of pipeline in a graph, to display the 
> native logic components, flows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2125) update JavaDoc of BoundedWindow

2017-10-23 Thread Xu Mingmin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2125.

   Resolution: Fixed
Fix Version/s: Not applicable

> update JavaDoc of BoundedWindow
> ---
>
> Key: BEAM-2125
> URL: https://issues.apache.org/jira/browse/BEAM-2125
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> {{BoundedWindow}} represents the window information of element, not 
> represents a finite grouping of elements. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2672) test merge in BeamSqlUdaf

2017-10-23 Thread Xu Mingmin (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2672.

   Resolution: Fixed
Fix Version/s: Not applicable

> test merge in BeamSqlUdaf
> -
>
> Key: BEAM-2672
> URL: https://issues.apache.org/jira/browse/BEAM-2672
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Minor
> Fix For: Not applicable
>
>
> The integration test with {{DirectRunner}} cannot cover method {{merge}} in 
> {{BeamSqlUdaf}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3566: (WIP) Display data translation

2017-10-23 Thread kennknowles

GitHub user kennknowles reopened a pull request:

https://github.com/apache/beam/pull/3566

(WIP) Display data translation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

@swegner and @bjchambers I hacked this pretty quick. I need to fulfill more 
translation and add tests, but putting this up so I have code to point at when 
I ask questions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DisplayDataTranslation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3566.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3566


commit 7a6b70f8344ccd296e65d2e6c0b474c521db2cd7
Author: Kenneth Knowles 
Date:   2017-07-14T22:38:22Z

Fix Runner API DisplayData proto

commit e9b5c8a561bc4c5db808814a2ade59c177e7ddb9
Author: Kenneth Knowles 
Date:   2017-07-14T22:38:35Z

Add DisplayDataTranslation




---

[GitHub] beam pull request #3566: (WIP) Display data translation

2017-10-23 Thread kennknowles

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/3566


---

[jira] [Resolved] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

2017-10-23 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3066.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Jenkins seed job should be resilient and independent of other jobs' 
> configurations
> --
>
> Key: BEAM-3066
> URL: https://issues.apache.org/jira/browse/BEAM-3066
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3411

2017-10-23 Thread Apache Jenkins Server

See 


Changes:

[github] Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4

--
[...truncated 931.26 KB...]
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying

[jira] [Commented] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215947#comment-16215947
 ] 

ASF GitHub Bot commented on BEAM-3066:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4019


> Jenkins seed job should be resilient and independent of other jobs' 
> configurations
> --
>
> Key: BEAM-3066
> URL: https://issues.apache.org/jira/browse/BEAM-3066
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4019: [BEAM-3066] Make the main seed job standalone

2017-10-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4019


---

[1/2] beam git commit: Make the main seed job standalone

2017-10-23 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/master 9e9300214 -> fefe153cd


Make the main seed job standalone


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/073601b0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/073601b0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/073601b0

Branch: refs/heads/master
Commit: 073601b019a99e16a0eec5a4a07209c3a876e77d
Parents: 9e93002
Author: Kenneth Knowles 
Authored: Thu Oct 19 15:55:03 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 23 15:11:47 2017 -0700

--
 .test-infra/jenkins/job_seed.groovy | 93 ++--
 1 file changed, 77 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/073601b0/.test-infra/jenkins/job_seed.groovy
--
diff --git a/.test-infra/jenkins/job_seed.groovy 
b/.test-infra/jenkins/job_seed.groovy
index 2d1b07c..1ef4950 100644
--- a/.test-infra/jenkins/job_seed.groovy
+++ b/.test-infra/jenkins/job_seed.groovy
@@ -16,30 +16,91 @@
  * limitations under the License.
  */
 
-import common_job_properties
-
 // Defines the seed job, which creates or updates all other Jenkins projects.
 job('beam_SeedJob') {
   description('Automatically configures all Apache Beam Jenkins projects 
based' +
   ' on Jenkins DSL groovy files checked into the code repository.')
 
-  previousNames('beam_SeedJob_Main')
+  properties {
+githubProjectUrl('https://github.com/apache/beam/')
+  }
+
+  // Restrict to only run on Jenkins executors labeled 'beam'
+  label('beam')
+
+  logRotator {
+daysToKeep(14)
+  }
+
+  scm {
+git {
+  remote {
+github('apache/beam')
+
+// ${ghprBuildId} is not interpolated by groovy, but passed through to 
Jenkins where it
+// refers to the environment variable
+refspec(['+refs/head/*:refs/remotes/origin/*',
+ 
'+refs/pull/${ghprPullId}/*:refs/remotes/origin/pr/${ghprPullId}/*']
+.join(' '))
+
+// The variable ${sha1} is not interpolated by groovy, but a parameter 
of the Jenkins job
+branch('${sha1}')
+
+extensions {
+  cleanAfterCheckout()
+}
+  }
+}
+  }
+
+  parameters {
+// Setup for running this job from a pull request
+stringParam(
+'sha1',
+'master',
+'Commit id or refname (eg: origin/pr/4001/head) you want to build 
against.')
+  }
+
+  wrappers {
+timeout {
+  absolute(5)
+  abortBuild()
+}
+  }
+
+  triggers {
+// Run once per day
+cron('0 */6 * * *')
 
-  // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(delegate)
+githubPullRequest {
+  admins(['asfbot'])
+  useGitHubHooks()
+  orgWhitelist(['apache'])
+  allowMembersOfWhitelistedOrgsAsAdmin()
+  permitAll()
 
-  // This is a post-commit job that runs once per day, not for every push.
-  common_job_properties.setPostCommit(
-  delegate,
-  '0 6 * * *',
-  false,
-  'd...@beam.apache.org')
+  // Also run when manually kicked on a pull request
+  triggerPhrase('Run Seed Job')
+  onlyTriggerPhrase()
 
-  // Allows triggering this build against pull requests.
-  common_job_properties.enablePhraseTriggeringFromPullRequest(
-delegate,
-'Seed Job',
-'Run Seed Job')
+  extensions {
+commitStatus {
+  context("Jenkins: Seed Job")
+}
+
+buildStatus {
+  completedStatus('SUCCESS', '--none--')
+  completedStatus('FAILURE', '--none--')
+  completedStatus('ERROR', '--none--')
+}
+  }
+}
+  }
+
+  // If anything goes wrong, mail the main dev list, because it is a big deal
+  publishers {
+mailer('d...@beam.apache.org', false, true)
+  }
 
   steps {
 dsl {

[jira] [Commented] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215916#comment-16215916
 ] 

Kenneth Knowles commented on BEAM-3092:
---

It turns out having Jenkins clone into the same directory as all our other 
stuff made this problem worse. We have exclusions for things like 
{{.repository}} which are not even part of our repository or built artifacts. 
When we discovered that we had to add exclusions for things that were actually 
not part of our source or build tree, that is our hint that we really must 
clone to a directory that Jenkins isn't going to drop random things in.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4028: Increase job_beam_PreCommit_Java_MavenInstall timeo...

2017-10-23 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4028


---

[1/2] beam git commit: Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours.

2017-10-23 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/master 77c77add1 -> 9e9300214


Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours.

Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ced7b9c5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ced7b9c5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ced7b9c5

Branch: refs/heads/master
Commit: ced7b9c50df8ba4a8ae35557575374118d057b0d
Parents: 77c77ad
Author: tvalentyn 
Authored: Mon Oct 23 14:09:12 2017 -0700
Committer: GitHub 
Committed: Mon Oct 23 14:09:12 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ced7b9c5/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
--
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
index 8721468..d2d2485 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy
@@ -32,7 +32,7 @@ mavenJob('beam_PreCommit_Java_MavenInstall') {
   common_job_properties.setTopLevelMainJobProperties(
 delegate,
 'master',
-150)
+240)
 
   // Set Maven parameters.
   common_job_properties.setMavenConfig(delegate)

[2/2] beam git commit: This closes #4028: Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours

2017-10-23 Thread kenn

This closes #4028: Increase job_beam_PreCommit_Java_MavenInstall timeout from 
2.5 to 4 hours


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9e930021
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9e930021
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9e930021

Branch: refs/heads/master
Commit: 9e9300214c6f868f93d62493e94e399cd01b636f
Parents: 77c77ad ced7b9c
Author: Kenneth Knowles 
Authored: Mon Oct 23 14:12:16 2017 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 23 14:12:16 2017 -0700

--
 .test-infra/jenkins/job_beam_PreCommit_Java_MavenInstall.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #4028: Increase job_beam_PreCommit_Java_MavenInstall timeo...

2017-10-23 Thread tvalentyn

GitHub user tvalentyn opened a pull request:

https://github.com/apache/beam/pull/4028

Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hâ¦

â¦ours.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4028.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4028


commit ced7b9c50df8ba4a8ae35557575374118d057b0d
Author: tvalentyn 
Date:   2017-10-23T21:09:12Z

Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours.




---

[jira] [Commented] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215810#comment-16215810
 ] 

ASF GitHub Bot commented on BEAM-3092:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4027

[BEAM-3092] Fix RAT plugin configuration

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam RAT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4027


commit 57c1e57d5e7d78e5b0ca3b1d434ea4b528da036a
Author: Kenneth Knowles 
Date:   2017-10-23T19:36:38Z

Rearrange .gitignore slightly

commit dfb6a6c75ff2fca342bf0695cdd547c03f04148c
Author: Kenneth Knowles 
Date:   2017-10-23T20:26:08Z

Make pom.xml match .gitignore

commit 3e045eeef67a2218559743904099c2e329203481
Author: Kenneth Knowles 
Date:   2017-10-23T20:26:27Z

Manage RAT plugin more centrally; only toggle skipping




> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4027: [BEAM-3092] Fix RAT plugin configuration

2017-10-23 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4027

[BEAM-3092] Fix RAT plugin configuration

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam RAT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4027.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4027


commit 57c1e57d5e7d78e5b0ca3b1d434ea4b528da036a
Author: Kenneth Knowles 
Date:   2017-10-23T19:36:38Z

Rearrange .gitignore slightly

commit dfb6a6c75ff2fca342bf0695cdd547c03f04148c
Author: Kenneth Knowles 
Date:   2017-10-23T20:26:08Z

Make pom.xml match .gitignore

commit 3e045eeef67a2218559743904099c2e329203481
Author: Kenneth Knowles 
Date:   2017-10-23T20:26:27Z

Manage RAT plugin more centrally; only toggle skipping




---

[jira] [Comment Edited] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215706#comment-16215706
 ] 

Kenneth Knowles edited comment on BEAM-3092 at 10/23/17 7:28 PM:
-

Looks like the universe of files to check respects {{ ${basedir} }} but the 
exclusions do not.


was (Author: kenn):
Looks like the universe of files to check respects {{${basedir}}} but the 
exclusions do not.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215706#comment-16215706
 ] 

Kenneth Knowles edited comment on BEAM-3092 at 10/23/17 7:28 PM:
-

Looks like the universe of files to check respects {{$\{basedir\}}} but the 
exclusions do not.


was (Author: kenn):
Looks like the universe of files to check respects {{ ${basedir} }} but the 
exclusions do not.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215704#comment-16215704
 ] 

Kenneth Knowles edited comment on BEAM-3092 at 10/23/17 7:28 PM:
-

The docs on the RAT plugin look like it should default to correctly using 
{{$\{basedir\}}} so there might be some further thing going on.


was (Author: kenn):
The docs on the RAT plugin look like it should default to correctly using 
`${basedir}` so there might be some further thing going on.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215706#comment-16215706
 ] 

Kenneth Knowles commented on BEAM-3092:
---

Looks like the universe of files to check respects {{${basedir}}} but the 
exclusions do not.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215704#comment-16215704
 ] 

Kenneth Knowles commented on BEAM-3092:
---

The docs on the RAT plugin look like it should default to correctly using 
`${basedir}` so there might be some further thing going on.

> RAT configuration is sensitive to $PWD
> --
>
> Key: BEAM-3092
> URL: https://issues.apache.org/jira/browse/BEAM-3092
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> This is currently breaking the build, as our Jenkins config does the 
> equivalent of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3092) RAT configuration is sensitive to $PWD

2017-10-23 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-3092:
-

 Summary: RAT configuration is sensitive to $PWD
 Key: BEAM-3092
 URL: https://issues.apache.org/jira/browse/BEAM-3092
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Blocker


This is currently breaking the build, as our Jenkins config does the equivalent 
of {{mvn -f src/pom.xml}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3091) RAT exclusions are inconsistent with .gitignore

2017-10-23 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-3091:
-

 Summary: RAT exclusions are inconsistent with .gitignore
 Key: BEAM-3091
 URL: https://issues.apache.org/jira/browse/BEAM-3091
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


We have generated files that can cause a RAT failure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #481

2017-10-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 77c77add12fb9b118f4ff1418d7fe5dec5154895 (origin/master)
Commit message: "Closes #4020"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77c77add12fb9b118f4ff1418d7fe5dec5154895
 > git rev-list 77c77add12fb9b118f4ff1418d7fe5dec5154895 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7729697332960612213.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6912033402847650315.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5986127598943570076.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins164264320587388556.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied:

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5092

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 299.98 KB...]
2017-10-23T18:07:49.637 [INFO] 

2017-10-23T18:07:49.637 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-23T18:07:49.638 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.638 [INFO] 

2017-10-23T18:07:49.638 [INFO]  
   
2017-10-23T18:07:49.638 [INFO] 

2017-10-23T18:07:49.638 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-23T18:07:49.638 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.638 [INFO] 

2017-10-23T18:07:49.638 [INFO]  
   
2017-10-23T18:07:49.639 [INFO] 

2017-10-23T18:07:49.639 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-23T18:07:49.639 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.639 [INFO] 

2017-10-23T18:07:49.639 [INFO]  
   
2017-10-23T18:07:49.639 [INFO] 

2017-10-23T18:07:49.639 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-23T18:07:49.639 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.639 [INFO] 

2017-10-23T18:07:49.639 [INFO]  
   
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-23T18:07:49.640 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO]  
   
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-23T18:07:49.640 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO]  
   
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-23T18:07:49.640 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.640 [INFO] 

2017-10-23T18:07:49.640 [INFO]  
   
2017-10-23T18:07:49.641 [INFO] 

2017-10-23T18:07:49.641 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-23T18:07:49.641 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.641 [INFO] 

2017-10-23T18:07:49.641 [INFO]  
   
2017-10-23T18:07:49.641 [INFO] 

2017-10-23T18:07:49.641 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-23T18:07:49.641 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T18:07:49.641 [INFO] 

2017-10-23T18:07:49.642 [INFO]  
   
2017-10-23T18:07:49.642 [INFO] 

2017-10-23T18:07:49.642 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-23T18:07:49.642 [INFO] This project has been banned from the build due 
to previous failures.

[jira] [Created] (BEAM-3090) Update Documentation of BigQuery source/sink to include the exact format of data read/written

2017-10-23 Thread Chamikara Jayalath (JIRA)

Chamikara Jayalath created BEAM-3090:


 Summary: Update Documentation of BigQuery source/sink to include 
the exact format of data read/written
 Key: BEAM-3090
 URL: https://issues.apache.org/jira/browse/BEAM-3090
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Currently documentation at following location simply mentions that BigQuery 
source/sink reads/writes dictionaries. This documentation should be updated to 
include examples.

https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L22




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4026: Add python_requires to setup.py

2017-10-23 Thread aaltay

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/4026

Add python_requires to setup.py

@elmer-garduno

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam pyr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4026.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4026


commit 947a613d2ae2fff3f7dc3ed0212d4e9cd2080f34
Author: Ahmet Altay 
Date:   2017-10-23T16:49:54Z

Add python_requires to setup.py




---

[jira] [Assigned] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not

2017-10-23 Thread Justin Tumale (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Tumale reassigned BEAM-2817:
---

Assignee: Justin Tumale

> Bigquery queries should allow options to run in batch mode or not
> -
>
> Key: BEAM-2817
> URL: https://issues.apache.org/jira/browse/BEAM-2817
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Lara Schmidt
>Assignee: Justin Tumale
>  Labels: newbie, starter
>
> When bigquery read does a query it sets the mode to batch. A batch query can 
> be very slow to schedule as it batches it with other queries. However it 
> doesn't use batch quota which is better for some cases. However, in some 
> cases a fast query is better (especially in timed tests). It would be a good 
> idea to have a configuration to the bigquery source to set this per-read.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3410

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 929.01 KB...]
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2660

2017-10-23 Thread Apache Jenkins Server

See

[jira] [Assigned] (BEAM-2854) Add SQL versions of the Nexmark queries (when possible)

2017-10-23 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-2854:
--

Assignee: Ismaël Mejía

> Add SQL versions of the Nexmark queries (when possible)
> ---
>
> Key: BEAM-2854
> URL: https://issues.apache.org/jira/browse/BEAM-2854
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: newbie, starter
>
> Wit the new support for SQL we can translate some of the Nexmark queries and 
> eventually also add new ones to expand the Nexmark suite.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #480

2017-10-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 77c77add12fb9b118f4ff1418d7fe5dec5154895 (origin/master)
Commit message: "Closes #4020"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77c77add12fb9b118f4ff1418d7fe5dec5154895
 > git rev-list 77c77add12fb9b118f4ff1418d7fe5dec5154895 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins68143806552377005.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9083709111995373564.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8921773840445706742.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8858938991712217850.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied:

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5091

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 299.98 KB...]
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-23T12:11:27.485 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO]  
   
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-23T12:11:27.485 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO]  
   
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-23T12:11:27.485 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO]  
   
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-23T12:11:27.485 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO]  
   
2017-10-23T12:11:27.485 [INFO] 

2017-10-23T12:11:27.485 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-23T12:11:27.486 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.486 [INFO] 

2017-10-23T12:11:27.486 [INFO]  
   
2017-10-23T12:11:27.486 [INFO] 

2017-10-23T12:11:27.486 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-23T12:11:27.486 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.486 [INFO] 

2017-10-23T12:11:27.486 [INFO]  
   
2017-10-23T12:11:27.486 [INFO] 

2017-10-23T12:11:27.486 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-23T12:11:27.486 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.486 [INFO] 

2017-10-23T12:11:27.487 [INFO]  
   
2017-10-23T12:11:27.487 [INFO] 

2017-10-23T12:11:27.487 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-23T12:11:27.487 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.488 [INFO] 

2017-10-23T12:11:27.488 [INFO]  
   
2017-10-23T12:11:27.488 [INFO] 

2017-10-23T12:11:27.488 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-23T12:11:27.488 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T12:11:27.488 [INFO] 

2017-10-23T12:11:27.488 [INFO]  
   
2017-10-23T12:11:27.488 [INFO] 

2017-10-23T12:11:27.488 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-23T12:11:27.489 [INFO] This project has been banned from the build due 
to previous failures.

Build failed in Jenkins: beam_PostCommit_Python_Verify #3409

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 933.24 KB...]
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py ->

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4213

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 21.15 MB...]
[INFO] 2017-10-23T07:39:19.628Z: (b6ee9ab08b71231): Checking required Cloud 
APIs are enabled.
[INFO] 2017-10-23T07:39:23.390Z: (b6ee9ab08b71581): Expanding CoGroupByKey 
operations into optimizable parts.
[INFO] 2017-10-23T07:39:23.407Z: (b6ee9ab08b71aea): Expanding GroupByKey 
operations into optimizable parts.
[INFO] 2017-10-23T07:39:23.411Z: (b6ee9ab08b71a84): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
[INFO] 2017-10-23T07:39:23.442Z: (b6ee9ab08b71754): Fusing adjacent ParDo, 
Read, Write, and Flatten operations
[INFO] 2017-10-23T07:39:23.444Z: (b6ee9ab08b716ee): Elided trivial flatten 
[INFO] 2017-10-23T07:39:23.446Z: (b6ee9ab08b71688): Elided trivial flatten 
[INFO] 2017-10-23T07:39:23.448Z: (b6ee9ab08b71622): Elided trivial flatten 
[INFO] 2017-10-23T07:39:23.453Z: (b6ee9ab08b715bc): Unzipping flatten s42 for 
input s37.37
[INFO] 2017-10-23T07:39:23.455Z: (b6ee9ab08b71556): Fusing unzipped copy of 
PAssert$335/GroupGlobally/GroupDummyAndContents/Reify, through flatten , into 
producer PAssert$335/GroupGlobally/KeyForDummy/AddKeys/Map
[INFO] 2017-10-23T07:39:23.457Z: (b6ee9ab08b714f0): Fusing consumer 
PAssert$335/GroupGlobally/GroupDummyAndContents/GroupByWindow into 
PAssert$335/GroupGlobally/GroupDummyAndContents/Read
[INFO] 2017-10-23T07:39:23.459Z: (b6ee9ab08b7148a): Unzipping flatten s42-u97 
for input s44-reify-value79-c95
[INFO] 2017-10-23T07:39:23.462Z: (b6ee9ab08b71424): Fusing unzipped copy of 
PAssert$335/GroupGlobally/GroupDummyAndContents/Write, through flatten , into 
producer PAssert$335/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2017-10-23T07:39:23.464Z: (b6ee9ab08b713be): Fusing consumer 
PAssert$335/GroupGlobally/ParDo(Concat) into 
PAssert$335/GroupGlobally/Values/Values/Map
[INFO] 2017-10-23T07:39:23.466Z: (b6ee9ab08b71358): Fusing consumer 
PAssert$335/RunChecks into PAssert$335/GetPane/Map
[INFO] 2017-10-23T07:39:23.468Z: (b6ee9ab08b712f2): Fusing consumer 
PAssert$335/VerifyAssertions/ParDo(DefaultConclude) into PAssert$335/RunChecks
[INFO] 2017-10-23T07:39:23.470Z: (b6ee9ab08b7128c): Fusing consumer 
PAssert$335/GetPane/Map into PAssert$335/GroupGlobally/ParDo(Concat)
[INFO] 2017-10-23T07:39:23.472Z: (b6ee9ab08b71226): Fusing consumer 
PAssert$335/GroupGlobally/Values/Values/Map into 
PAssert$335/GroupGlobally/GroupDummyAndContents/GroupByWindow
[INFO] 2017-10-23T07:39:23.474Z: (b6ee9ab08b711c0): Fusing consumer ParDo(Test) 
into Create.Values/Read(CreateSource)
[INFO] 2017-10-23T07:39:23.477Z: (b6ee9ab08b7115a): Fusing consumer 
PAssert$335/GroupGlobally/Window.Into()/Window.Assign into ParDo(Test)
[INFO] 2017-10-23T07:39:23.479Z: (b6ee9ab08b710f4): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/Write into 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
[INFO] 2017-10-23T07:39:23.481Z: (b6ee9ab08b7108e): Fusing consumer 
PAssert$335/GroupGlobally/KeyForDummy/AddKeys/Map into 
PAssert$335/GroupGlobally/RewindowActuals/Window.Assign
[INFO] 2017-10-23T07:39:23.484Z: (b6ee9ab08b71028): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/Read
[INFO] 2017-10-23T07:39:23.485Z: (b6ee9ab08b71fc2): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/Values/Values/Map into 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
[INFO] 2017-10-23T07:39:23.487Z: (b6ee9ab08b71f5c): Fusing consumer 
PAssert$335/GroupGlobally/RewindowActuals/Window.Assign into 
PAssert$335/GroupGlobally/GatherAllOutputs/Values/Values/Map
[INFO] 2017-10-23T07:39:23.489Z: (b6ee9ab08b71ef6): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows) 
into PAssert$335/GroupGlobally/Window.Into()/Window.Assign
[INFO] 2017-10-23T07:39:23.491Z: (b6ee9ab08b71e90): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into 
PAssert$335/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows)
[INFO] 2017-10-23T07:39:23.493Z: (b6ee9ab08b71e2a): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign into 
PAssert$335/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
[INFO] 2017-10-23T07:39:23.495Z: (b6ee9ab08b71dc4): Fusing consumer 
PAssert$335/GroupGlobally/GatherAllOutputs/GroupByKey/Reify into 
PAssert$335/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign
[INFO] 2017-10-23T07:39:23.498Z: (b6ee9ab08b71d5e): Fusing consumer 
PAssert$335/GroupGlobally/GroupDummyAndContents/Reify into 
PAssert$335/GroupGlobally/WindowIntoDummy/Window.Assign
[INFO] 2017-10-23T07:39:23.500Z: (b6ee9ab08b71cf8): Fusing consumer 
PAssert$335/GroupGlobally/GroupDummyAndContents/Write into 
PAssert$335/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2017-10-23T07:39:23.502Z:

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #2659

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 566.34 KB...]
2017-10-23T06:11:22.490 [INFO] Excluding 
org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
2017-10-23T06:11:22.490 [INFO] Excluding io.netty:netty:jar:3.7.0.Final from 
the shaded jar.
2017-10-23T06:11:22.490 [INFO] Excluding com.datatorrent:netlet:jar:1.3.0 from 
the shaded jar.
2017-10-23T06:11:22.490 [INFO] Excluding 
com.esotericsoftware.kryo:kryo:jar:2.24.0 from the shaded jar.
2017-10-23T06:11:22.490 [INFO] Excluding 
com.esotericsoftware.minlog:minlog:jar:1.2 from the shaded jar.
2017-10-23T06:11:22.490 [INFO] Excluding org.objenesis:objenesis:jar:2.1 from 
the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
javax.validation:validation-api:jar:1.1.0.Final from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 
from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.apache.apex:malhar-library:jar:3.4.0 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.eclipse.jetty:jetty-servlet:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.eclipse.jetty:jetty-security:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.eclipse.jetty:jetty-server:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding 
org.eclipse.jetty:jetty-continuation:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.491 [INFO] Excluding com.sun.mail:javax.mail:jar:1.5.0 from 
the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding javax.activation:activation:jar:1.1 
from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 
from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding javax.jms:jms-api:jar:1.1-rev-1 from 
the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
org.apache.activemq:activemq-client:jar:5.8.0 from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1 from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding org.fusesource.hawtbuf:hawtbuf:jar:1.9 
from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
org.apache.geronimo.specs:geronimo-j2ee-management_1.1_spec:jar:1.0.1 from the 
shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from 
the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
com.github.tony19:named-regexp:jar:0.2.3 from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
org.codehaus.janino:commons-compiler:jar:2.7.8 from the shaded jar.
2017-10-23T06:11:22.492 [INFO] Excluding 
org.eclipse.jetty:jetty-websocket:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
org.eclipse.jetty:jetty-util:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
org.eclipse.jetty:jetty-io:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
org.eclipse.jetty:jetty-http:jar:8.1.10.v20130312 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
commons-beanutils:commons-beanutils:jar:1.8.3 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
commons-logging:commons-logging:jar:1.1.1 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding it.unimi.dsi:fastutil:jar:7.0.6 from 
the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding 
org.apache.apex:apex-shaded-ning19:jar:1.0.0 from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding org.apache.apex:apex-engine:jar:3.6.0 
from the shaded jar.
2017-10-23T06:11:22.493 [INFO] Excluding org.apache.bval:bval-jsr303:jar:0.5 
from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding org.apache.bval:bval-core:jar:0.5 from 
the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding 
org.apache.apex:apex-bufferserver:jar:3.6.0 from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.3.6 from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.3.3 from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding 
com.sun.jersey.contribs:jersey-apache-client4:jar:1.9 from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-client:jar:2.6.0 from the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding commons-lang:commons-lang:jar:2.6 from 
the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding commons-cli:commons-cli:jar:1.2 from 
the shaded jar.
2017-10-23T06:11:22.494 [INFO] Excluding

Build failed in Jenkins: beam_PerformanceTests_Python #479

2017-10-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 77c77add12fb9b118f4ff1418d7fe5dec5154895 (origin/master)
Commit message: "Closes #4020"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77c77add12fb9b118f4ff1418d7fe5dec5154895
 > git rev-list 77c77add12fb9b118f4ff1418d7fe5dec5154895 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2610446804139255464.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5336702620289384012.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1501891837591306216.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4717427013893627958.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied:

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5090

2017-10-23 Thread Apache Jenkins Server

See 


--
[...truncated 299.98 KB...]
2017-10-23T06:02:33.082 [INFO] 

2017-10-23T06:02:33.082 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
MQTT
2017-10-23T06:02:33.082 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.082 [INFO] 

2017-10-23T06:02:33.082 [INFO]  
   
2017-10-23T06:02:33.082 [INFO] 

2017-10-23T06:02:33.082 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Redis
2017-10-23T06:02:33.082 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.082 [INFO] 

2017-10-23T06:02:33.082 [INFO]  
   
2017-10-23T06:02:33.082 [INFO] 

2017-10-23T06:02:33.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2017-10-23T06:02:33.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.083 [INFO] 

2017-10-23T06:02:33.083 [INFO]  
   
2017-10-23T06:02:33.083 [INFO] 

2017-10-23T06:02:33.083 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2017-10-23T06:02:33.083 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.083 [INFO] 

2017-10-23T06:02:33.084 [INFO]  
   
2017-10-23T06:02:33.084 [INFO] 

2017-10-23T06:02:33.084 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: XML
2017-10-23T06:02:33.084 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.084 [INFO] 

2017-10-23T06:02:33.084 [INFO]  
   
2017-10-23T06:02:33.084 [INFO] 

2017-10-23T06:02:33.084 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2017-10-23T06:02:33.085 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.085 [INFO] 

2017-10-23T06:02:33.085 [INFO]  
   
2017-10-23T06:02:33.085 [INFO] 

2017-10-23T06:02:33.085 [INFO] Skipping Apache Beam :: Examples :: Java
2017-10-23T06:02:33.085 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.086 [INFO] 

2017-10-23T06:02:33.086 [INFO]  
   
2017-10-23T06:02:33.086 [INFO] 

2017-10-23T06:02:33.086 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2017-10-23T06:02:33.086 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.086 [INFO] 

2017-10-23T06:02:33.086 [INFO]  
   
2017-10-23T06:02:33.086 [INFO] 

2017-10-23T06:02:33.087 [INFO] Skipping Apache Beam :: Examples :: Java 8
2017-10-23T06:02:33.087 [INFO] This project has been banned from the build due 
to previous failures.
2017-10-23T06:02:33.087 [INFO] 

2017-10-23T06:02:33.087 [INFO]  
   
2017-10-23T06:02:33.087 [INFO] 

2017-10-23T06:02:33.087 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples - Java 8
2017-10-23T06:02:33.087 [INFO] This project has been banned from the build due 
to previous failures.

54 matches

Mail list logo