date:20180104

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4655

2018-01-04 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

2018-01-04 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312595#comment-16312595
 ] 

Jean-Baptiste Onofré commented on BEAM-981:
---

Let me take a look on the PR. And yes, agree to assign the Jira to you.

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
> at 
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
> at 
> io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:845)
> at 
>

[jira] [Assigned] (BEAM-3371) Add ability to stage directories with compiled classes to Spark

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3371:
--

Assignee: Jean-Baptiste Onofré  (was: Amit Sela)

> Add ability to stage directories with compiled classes to Spark
> ---
>
> Key: BEAM-3371
> URL: https://issues.apache.org/jira/browse/BEAM-3371
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Łukasz Gajowy
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> This one is basically the same issue as
>  [this Flink's one|https://issues.apache.org/jira/browse/BEAM-3370], except 
> of two things:
> - a detection of files to stage has to be provided in Spark, which is already 
> being developed [here|https://issues.apache.org/jira/browse/BEAM-981]
> - the test execution is not interrupted by FileNotFoundException but by *the 
> effect* of the directory not being staged (absence of test classes on the 
> Spark's classpath, hence ClassNotFoundException).
> Again, this probably could be resolved analogously as in flink, while 
> BEAM-981 issue is resolved. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

2018-01-04 Thread huangjianhuang (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312593#comment-16312593
 ] 

huangjianhuang commented on BEAM-3414:
--

the trigger would never fire after 06:34:55.505 if i didnot send new data. how 
could i fix it without new input?

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Kenneth Knowles
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

2018-01-04 Thread huangjianhuang (JIRA)

huangjianhuang created BEAM-3414:


 Summary: AfterProcessingTime trigger issue with Flink Runner
 Key: BEAM-3414
 URL: https://issues.apache.org/jira/browse/BEAM-3414
 Project: Beam
  Issue Type: Bug
  Components: runner-core, runner-flink
Affects Versions: 2.2.0
 Environment: idea, ubuntu 16.04, FlinkRunner
Reporter: huangjianhuang
Assignee: Kenneth Knowles


in my demo, I read data from kafka and count globally, finally output the total 
count of recieved data, as follow:

{code:java}
FlinkPipelineOptions options = 
PipelineOptionsFactory.fromArgs(args).withValidation()
.as(FlinkPipelineOptions.class);

options.setStreaming(true);
options.setRunner(FlinkRunner.class);
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply("Read from kafka",
KafkaIO.read()
//.withTimestampFn(kafkaData -> 
TimeUtil.timeMillisToInstant(kafkaData.getKey()))
.withBootstrapServers("localhost:9092")
.withTopic("recharge")
.withKeyDeserializer(StringDeserializer.class)
.withValueDeserializer(StringDeserializer.class)
.withoutMetadata()
)
.apply(Values.create())
.apply(Window.into(new GlobalWindows())
.triggering(Repeatedly.forever(

AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
.accumulatingFiredPanes()
)
.apply(Count.globally())
.apply("output",
ParDo.of(new DoFn() {
@ProcessElement
public void process(ProcessContext context) {
System.out.println("---get at: " + 
Instant.now() + "--");
System.out.println(context.element());
}
}));
{code}

the result should be displayed after (5s) I sent first data, but sometimes 
there were nothing display after I sent data. the pic shows the outputs i got 
in a test:

(cant upload a pic, desc as text)

{code:java}
Send 681Msg at: 2018-01-05T06:34:31.436

---get at: 2018-01-05T06:34:36.668Z--
681

Send 681Msg at: 2018-01-05T06:34:47.166

---get at: 2018-01-05T06:34:52.284Z--
1362

Send 681Msg at: 2018-01-05T06:34:55.505

Send 681Msg at: 2018-01-05T06:35:22.068

---get at: 2018-01-05T06:35:22.112Z--
2044
{code}

btw, the code works fine with direct runner.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5586

2018-01-04 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3827

2018-01-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #61

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 54.73 KB...]
2018-01-05 06:22:46,999 e7a85060 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 06:22:47,000 e7a85060 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 06:23:16,379 e7a85060 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json
2018-01-05 06:23:16,985 e7a85060 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 06:23:16,986 e7a85060 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 06:23:39,267 e7a85060 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json
2018-01-05 06:23:39,802 e7a85060 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 06:23:39,802 e7a85060 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 06:24:01,454 e7a85060 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json
2018-01-05 06:24:02,130 e7a85060 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-05 06:24:02,130 e7a85060 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-05 06:24:25,202 e7a85060 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json
2018-01-05 06:24:25,853 e7a85060 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/e7a85060/perfkit-bq-pubRFMGq4.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for

Build failed in Jenkins: beam_PerformanceTests_Python #754

2018-01-04 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 63e269604a437f7110e15a8fcd27257864c9100d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 63e269604a437f7110e15a8fcd27257864c9100d
Commit message: "Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix"
 > git rev-list 63e269604a437f7110e15a8fcd27257864c9100d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8772306903516702061.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2982479366169020516.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1333598146164767458.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4623

2018-01-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Spark #1198

2018-01-04 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 63e269604a437f7110e15a8fcd27257864c9100d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 63e269604a437f7110e15a8fcd27257864c9100d
Commit message: "Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix"
 > git rev-list 63e269604a437f7110e15a8fcd27257864c9100d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2129734858316552135.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6309806416790934507.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins301983200947095599.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already

Build failed in Jenkins: beam_PostCommit_Python_Verify #3881

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 170.31 KB...]
[32mtest_1  |[0m credentials._refresh(orig_request_method)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 128, in _refresh
[32mtest_1  |[0m self._retrieve_info(http)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 111, in _retrieve_info
[32mtest_1  |[0m service_account=self.service_account_email or 
'default')
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 96, in get_service_account_info
[32mtest_1  |[0m recursive=True)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 60, in get
[32mtest_1  |[0m http, url, headers=METADATA_HEADERS)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
 line 282, in request
[32mtest_1  |[0m connection_type=connection_type)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1609, in request
[32mtest_1  |[0m (response, content) = self._request(conn, authority, 
uri, request_uri, method, body, headers, redirections, cachekey)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1351, in _request
[32mtest_1  |[0m (response, content) = self._conn_request(conn, 
request_uri, method, body, headers)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1278, in _conn_request
[32mtest_1  |[0m raise ServerNotFoundError("Unable to find the server 
at %s" % conn.host)
[32mtest_1  |[0m 
[32mtest_1  |[0m INFO:root:Starting the size estimation of the input
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m WARNING:root:Retry with exponential backoff: waiting for 
14.9098196375 seconds before retrying size_of_files_in_glob because we caught 
exception: ServerNotFoundError: Unable to find the server at 
metadata.google.internal
[32mtest_1  |[0m  Traceback for above exception (most recent call last):
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
 line 175, in wrapper
[32mtest_1  |[0m return fun(*args, **kwargs)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
 line 438, in size_of_files_in_glob
[32mtest_1  |[0m response = self.client.objects.List(request)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
 line 984, in List
[32mtest_1  |[0m config, request, global_params=global_params)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod
[32mtest_1  |[0m http, http_request, **opts)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest
[32mtest_1  |[0m check_response_func=check_response_func)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 396, in _MakeRequestNoRetry
[32mtest_1  |[0m redirections=redirections, 
connection_type=connection_type)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
 line 159, in new_request
[32mtest_1  |[0m credentials._refresh(orig_request_method)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 128, in _refresh
[32mtest_1  |[0m self._retrieve_info(http)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 111, in _retrieve_info
[32mtest_1  |[0m service_account=self.service_account_email or 
'default')
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 96, in

Build failed in Jenkins: beam_PostCommit_Python_Verify #3880

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 1.05 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py ->

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #583

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 74.53 KB...]
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[str, NoneType], 
Tuple[str, int]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[Any, List[Any]], 
Tuple[Any, NoneType]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.3.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[NoneType, 
Tuple[Any, List[Any]]], Tuple[NoneType, Tuple[Any, NoneType]]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:133:
 UserWarning: Using fallback coder for typehint: Union[Tuple[Any, List[Any]], 
Tuple[Any, NoneType]].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-38.3.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz

Build failed in Jenkins: beam_PostCommit_Python_Verify #3879

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 1.64 MB...]
[32mtest_1  |[0m credentials._refresh(orig_request_method)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 128, in _refresh
[32mtest_1  |[0m self._retrieve_info(http)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 111, in _retrieve_info
[32mtest_1  |[0m service_account=self.service_account_email or 
'default')
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 96, in get_service_account_info
[32mtest_1  |[0m recursive=True)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 60, in get
[32mtest_1  |[0m http, url, headers=METADATA_HEADERS)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
 line 282, in request
[32mtest_1  |[0m connection_type=connection_type)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1609, in request
[32mtest_1  |[0m (response, content) = self._request(conn, authority, 
uri, request_uri, method, body, headers, redirections, cachekey)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1351, in _request
[32mtest_1  |[0m (response, content) = self._conn_request(conn, 
request_uri, method, body, headers)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/httplib2/__init__.py", 
line 1278, in _conn_request
[32mtest_1  |[0m raise ServerNotFoundError("Unable to find the server 
at %s" % conn.host)
[32mtest_1  |[0m 
[32mtest_1  |[0m INFO:root:Starting the size estimation of the input
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m INFO:oauth2client.transport:Attempting refresh to obtain 
initial access_token
[32mtest_1  |[0m WARNING:root:Retry with exponential backoff: waiting for 
15.9650243917 seconds before retrying size_of_files_in_glob because we caught 
exception: ServerNotFoundError: Unable to find the server at 
metadata.google.internal
[32mtest_1  |[0m  Traceback for above exception (most recent call last):
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
 line 175, in wrapper
[32mtest_1  |[0m return fun(*args, **kwargs)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
 line 438, in size_of_files_in_glob
[32mtest_1  |[0m response = self.client.objects.List(request)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
 line 984, in List
[32mtest_1  |[0m config, request, global_params=global_params)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod
[32mtest_1  |[0m http, http_request, **opts)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest
[32mtest_1  |[0m check_response_func=check_response_func)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 396, in _MakeRequestNoRetry
[32mtest_1  |[0m redirections=redirections, 
connection_type=connection_type)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/transport.py",
 line 159, in new_request
[32mtest_1  |[0m credentials._refresh(orig_request_method)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 128, in _refresh
[32mtest_1  |[0m self._retrieve_info(http)
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/gce.py",
 line 111, in _retrieve_info
[32mtest_1  |[0m service_account=self.service_account_email or 
'default')
[32mtest_1  |[0m   File 
"/usr/local/envs/beamenv/lib/python2.7/site-packages/oauth2client/contrib/_metadata.py",
 line 96, in

Build failed in Jenkins: beam_PostCommit_Python_Verify #3878

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 1.04 MB...]
copying apache_beam/io/gcp/gcsio_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp
copying apache_beam/io/gcp/pubsub.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp
copying apache_beam/io/gcp/pubsub_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp
copying apache_beam/io/gcp/datastore/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore
copying apache_beam/io/gcp/datastore/v1/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/adaptive_throttler.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/adaptive_throttler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/datastoreio.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/datastoreio_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/fake_datastore.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/helper.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/helper_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/query_splitter.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/query_splitter_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/datastore/v1/util_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/datastore/v1
copying apache_beam/io/gcp/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal
copying apache_beam/io/gcp/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients
copying apache_beam/io/gcp/internal/clients/bigquery/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/bigquery
copying apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/bigquery
copying apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_messages.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/bigquery
copying apache_beam/io/gcp/internal/clients/storage/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/storage
copying apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/storage
copying apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/internal/clients/storage
copying apache_beam/io/gcp/tests/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/tests
copying apache_beam/io/gcp/tests/bigquery_matcher.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/tests
copying apache_beam/io/gcp/tests/bigquery_matcher_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/tests
copying apache_beam/io/gcp/tests/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/tests
copying apache_beam/io/gcp/tests/utils_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/io/gcp/tests
copying apache_beam/metrics/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/cells.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/cells_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/execution.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/execution.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/execution_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/metric.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/metric_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/metrics/metricbase.py -> 
apache-beam-2.3.0.dev0/apache_beam/metrics
copying apache_beam/options/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/pipeline_options.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/pipeline_options_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/pipeline_options_validator.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/pipeline_options_validator_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/value_provider.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying apache_beam/options/value_provider_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/options
copying

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5585

2018-01-04 Thread Apache Jenkins Server

See

[jira] [Created] (BEAM-3413) Support copying GCs objects using rewrite operation

2018-01-04 Thread Chamikara Jayalath (JIRA)

Chamikara Jayalath created BEAM-3413:


 Summary: Support copying GCs objects using rewrite operation
 Key: BEAM-3413
 URL: https://issues.apache.org/jira/browse/BEAM-3413
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Chamikara Jayalath


Currently Java GCS FileSystem users GCS copy() operation to copy objects.
https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/GcsUtil.java#L744

According to GCS website this can result in HTTP 413 errors if copying takes 
more than 30 seconds: 
https://cloud.google.com/storage/docs/json_api/v1/status-codes

The solution suggested there is to use the GCS rewrite() method.

We should add support for this.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5584

2018-01-04 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312263#comment-16312263
 ] 

ASF GitHub Bot commented on BEAM-3412:
--

chamikaramj opened a new pull request #4347: [BEAM-3412] Updates BigTable 
client version to 1.0
URL: https://github.com/apache/beam/pull/4347
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3412) Update BigTable client version to 1.0

2018-01-04 Thread Chamikara Jayalath (JIRA)

Chamikara Jayalath created BEAM-3412:


 Summary: Update BigTable client version to 1.0
 Key: BEAM-3412
 URL: https://issues.apache.org/jira/browse/BEAM-3412
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-gcp
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4653

2018-01-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Python #753

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[github] Use beam-master-$DATE pattern for default Dataflow containers in

[github] Update dependency.py

[github] fixup: lint

[chamikara] Updates common.py due to a Cython bug.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 63e269604a437f7110e15a8fcd27257864c9100d (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 63e269604a437f7110e15a8fcd27257864c9100d
Commit message: "Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix"
 > git rev-list 4fef942195d0fc1052e6888240d44668580a5013 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2804095853822157673.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5279134051679760874.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8091346111996690315.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt

Jenkins build is back to normal : beam_PerformanceTests_Spark #1197

2018-01-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #60

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[github] Use beam-master-$DATE pattern for default Dataflow containers in

[github] Update dependency.py

[github] fixup: lint

[chamikara] Updates common.py due to a Cython bug.

--
[...truncated 685.05 KB...]
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/973352c8/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:test-jar (default-test-jar) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Building jar: 
/tmp/perfkitbenchmarker/runs/973352c8/beam/sdks/java/io/file-based-io-tests/target/beam-sdks-java-io-file-based-io-tests-2.3.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (bundle-and-repackage) @ 
beam-sdks-java-io-file-based-io-tests ---
[INFO] Excluding com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from 
the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from the shaded jar.
[INFO] Excluding joda-time:joda-time:jar:2.4 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.3.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.3.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.3.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.1 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding

[jira] [Commented] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

2018-01-04 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312235#comment-16312235
 ] 

Ahmet Altay commented on BEAM-3411:
---

Here is one of the failing dataflow jobs:

https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-01-04_13_16_26-2332266460968770664?project=apache-beam-testing=433637338589

I see the following error in the worker logs:

I  2018/01/04 21:20:37 Traceback (most recent call last): 
I  2018/01/04 21:20:37   File "/usr/lib/python2.7/runpy.py", line 162, in 
_run_module_as_main 
I  2018/01/04 21:20:37 "__main__", fname, loader, pkg_name) 
I  2018/01/04 21:20:37   File "/usr/lib/python2.7/runpy.py", line 72, in 
_run_code 
I  2018/01/04 21:20:37 exec code in run_globals 
I  2018/01/04 21:20:37   File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 195, in  
I  2018/01/04 21:20:37 main(sys.argv) 
I  2018/01/04 21:20:37   File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 134, in main 
I  2018/01/04 21:20:37 
worker_count=_get_worker_count(sdk_pipeline_options)).run() 
I  2018/01/04 21:20:37   File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 97, in run 
I  2018/01/04 21:20:37 work_request) 
I  2018/01/04 21:20:37   File 
"/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
 line 162, in _request_process_bundle_progress 
I  2018/01/04 21:20:37 worker = 
self._instruction_id_vs_worker[request.instruction_id] 
I  2018/01/04 21:20:37 KeyError: u'-39'

The error started happening likely after 
https://github.com/apache/beam/commit/8188db40ee369dd54d69c7ef6020cf47463c8e85 
which started using a newer fnapi worker container. Potentially a PR between 
12/19 - 12/22 introduce this issue, but was not tested. (Assigning to 
[~angoenka], because it might be related his changes.)

(cc: [~alanmyrvold], with automated testing using containers built at head we 
should be able to notice these issues earlier.)
 


> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
>

[jira] [Assigned] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

2018-01-04 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3411:
-

Assignee: Ankur Goenka  (was: Ahmet Altay)

> Test apache_beam.examples.wordcount_it_test.WordCountIT times out
> -
>
> Key: BEAM-3411
> URL: https://issues.apache.org/jira/browse/BEAM-3411
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>
> Failed run: 
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console
> Log snippet:
> test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
> ... ERROR
> ==
> ERROR: test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 812, in run
> test(orig)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 45, in __call__
> return self.run(*arg, **kwarg)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 133, in run
> self.runTest(result)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
>  line 151, in runTest
> test(result)
>   File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
> return self.run(*args, **kwds)
>   File "/usr/lib/python2.7/unittest/case.py", line 331, in run
> testMethod()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
>  line 77, in test_wordcount_fnapi_it
> on_success_matcher=PipelineStateMatcher()))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
>  line 130, in run
> result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 956, in wait_until_finish
> time.sleep(5.0)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
>  line 276, in signalhandler
> raise TimedOutException()
> TimedOutException: 'test_wordcount_fnapi_it 
> (apache_beam.examples.wordcount_it_test.WordCountIT)'
> --
> Ran 3 tests in 901.290s
> FAILED (errors=1)
> Build step 'Execute shell' marked build as failure



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3877

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[chamikara] Updates common.py due to a Cython bug.

--
[...truncated 1.05 MB...]
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct

[jira] [Commented] (BEAM-3397) beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because of too long commandline

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312170#comment-16312170
 ] 

ASF GitHub Bot commented on BEAM-3397:
--

alanmyrvold opened a new pull request #4345: [BEAM-3397] Fix the failure in the 
dataflow integration test by removing the spark and flink pipeline options
URL: https://github.com/apache/beam/pull/4345
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   
   Fix the failure in the dataflow integration test by removing the spark and 
flink pipeline options
   Otherwise there are 3 copies of the filesToStage option in the 
pipelineoptions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because 
> of too long commandline
> -
>
> Key: BEAM-3397
> URL: https://issues.apache.org/jira/browse/BEAM-3397
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, testing
>Reporter: Luke Cwik
> Fix For: 2.3.0
>
>
> The Dataflow worker fails to launch the process because the command line is 
> too long. The worker logs are riddled with a very long command line followed 
> by this error message:
> `java failed with fork/exec /usr/local/bin/java: argument list too long`
> Example failures from run from master @ 
> 49981f15009de434af9c09c3c9f50003c7417b88
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/16428/
> Job ids (project: apache-beam-testing):
> 2017-12-28_01_32_55-16084854344397840285
> 2017-12-28_01_32_55-213040034028880428
> 2017-12-28_01_32_55-16742595557951289122



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3411) Test apache_beam.examples.wordcount_it_test.WordCountIT times out

2018-01-04 Thread Henning Rohde (JIRA)

Henning Rohde created BEAM-3411:
---

 Summary: Test apache_beam.examples.wordcount_it_test.WordCountIT 
times out
 Key: BEAM-3411
 URL: https://issues.apache.org/jira/browse/BEAM-3411
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Henning Rohde
Assignee: Ahmet Altay


Failed run: 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/3876/console

Log snippet:

test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ERROR

==
ERROR: test_wordcount_fnapi_it 
(apache_beam.examples.wordcount_it_test.WordCountIT)
--
Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
 line 812, in run
test(orig)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
 line 133, in run
self.runTest(result)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/case.py",
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
 line 77, in test_wordcount_fnapi_it
on_success_matcher=PipelineStateMatcher()))
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/wordcount_fnapi.py",
 line 130, in run
result.wait_until_finish()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 956, in wait_until_finish
time.sleep(5.0)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/.eggs/nose-1.3.7-py2.7.egg/nose/plugins/multiprocess.py",
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_wordcount_fnapi_it 
(apache_beam.examples.wordcount_it_test.WordCountIT)'

--
Ran 3 tests in 901.290s

FAILED (errors=1)
Build step 'Execute shell' marked build as failure



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3825

2018-01-04 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-3410) Jenkins post-commit fails with no error and truncated logs

2018-01-04 Thread Henning Rohde (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312154#comment-16312154
 ] 

Henning Rohde commented on BEAM-3410:
-

A burst happened around the same time, so it is perhaps just some transient 
issue.

> Jenkins post-commit fails with no error and truncated logs
> --
>
> Key: BEAM-3410
> URL: https://issues.apache.org/jira/browse/BEAM-3410
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Henning Rohde
>Assignee: Jason Kuster
>
> Some runs failed without an obvious cause (taking 0 ms) and seemingly 
> truncated logs: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/3128/
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5582/
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/3824/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3410) Jenkins post-commit fails with no error and truncated logs

2018-01-04 Thread Henning Rohde (JIRA)

Henning Rohde created BEAM-3410:
---

 Summary: Jenkins post-commit fails with no error and truncated logs
 Key: BEAM-3410
 URL: https://issues.apache.org/jira/browse/BEAM-3410
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Henning Rohde
Assignee: Jason Kuster


Some runs failed without an obvious cause (taking 0 ms) and seemingly truncated 
logs: 

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/3128/
https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/5582/
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/3824/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1630) Add Splittable DoFn to Python SDK

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312125#comment-16312125
 ] 

ASF GitHub Bot commented on BEAM-1630:
--

chamikaramj closed pull request #4342: [BEAM-1630] Updates common.py due to a 
Cython bug.
URL: https://github.com/apache/beam/pull/4342
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/common.pxd 
b/sdks/python/apache_beam/runners/common.pxd
index dd7f3e45953..fed0c2c95c2 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -47,10 +47,11 @@ cdef class DoFnSignature(object):
 
 cdef class DoFnInvoker(object):
   cdef public DoFnSignature signature
-  cdef _OutputProcessor output_processor
+  cdef OutputProcessor output_processor
 
   cpdef invoke_process(self, WindowedValue windowed_value,
-   restriction_tracker=*, output_processor=*)
+   restriction_tracker=*,
+   OutputProcessor output_processor=*)
   cpdef invoke_start_bundle(self)
   cpdef invoke_finish_bundle(self)
   cpdef invoke_split(self, element, restriction)
@@ -85,16 +86,15 @@ cdef class DoFnRunner(Receiver):
 
 
 cdef class OutputProcessor(object):
-  pass
+  @cython.locals(windowed_value=WindowedValue)
+  cpdef process_outputs(self, WindowedValue element, results)
+
 
 cdef class _OutputProcessor(OutputProcessor):
   cdef object window_fn
   cdef Receiver main_receivers
   cdef object tagged_receivers
 
-  @cython.locals(windowed_value=WindowedValue)
-  cpdef process_outputs(self, WindowedValue element, results)
-
 
 cdef class DoFnContext(object):
   cdef object label
diff --git a/sdks/python/apache_beam/runners/common.py 
b/sdks/python/apache_beam/runners/common.py
index 57c9f389cd9..d5ca68307aa 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -264,7 +264,8 @@ def __init__(self, output_processor, signature):
 
   def invoke_process(self, windowed_value, restriction_tracker=None,
  output_processor=None):
-output_processor = output_processor or self.output_processor
+if not output_processor:
+  output_processor = self.output_processor
 output_processor.process_outputs(
 windowed_value, self.process_method(windowed_value.value))
 
@@ -349,7 +350,8 @@ def __init__(self, placeholder):
 
   def invoke_process(self, windowed_value, restriction_tracker=None,
  output_processor=None):
-output_processor = output_processor or self.output_processor
+if not output_processor:
+  output_processor = self.output_processor
 self.context.set_element(windowed_value)
 # Call for the process function for each window if has windowed side inputs
 # or if the process accesses the window parameter. We can just call it once


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Splittable DoFn to Python SDK
> -
>
> Key: BEAM-1630
> URL: https://issues.apache.org/jira/browse/BEAM-1630
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Splittable DoFn [1] is currently being implemented for Java SDK [2]. We 
> should add this to Python SDK as well.
> Following document proposes an API for this.
> https://docs.google.com/document/d/1h_zprJrOilivK2xfvl4L42vaX4DMYGfH1YDmi-s_ozM/edit?usp=sharing
> [1] https://s.apache.org/splittable-do-fn
> [2] 
> https://lists.apache.org/thread.html/0ce61ac162460a149d5c93cdface37cc383f8030fe86ca09e5699b18@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam] 01/01: Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix

2018-01-04 Thread chamikara

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 63e269604a437f7110e15a8fcd27257864c9100d
Merge: d28cc55 8445048
Author: Chamikara Jayalath 
AuthorDate: Thu Jan 4 14:26:08 2018 -0800

Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix

[BEAM-1630] Updates common.py due to a Cython bug.

 sdks/python/apache_beam/runners/common.pxd | 12 ++--
 sdks/python/apache_beam/runners/common.py  |  6 --
 2 files changed, 10 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch master updated (d28cc55 -> 63e2696)

2018-01-04 Thread chamikara

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d28cc55  Merge pull request #4322 from tvalentyn/patch-4
 add 8445048  Updates common.py due to a Cython bug.
 new 63e2696  Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/common.pxd | 12 ++--
 sdks/python/apache_beam/runners/common.py  |  6 --
 2 files changed, 10 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

2018-01-04 Thread Chamikara Jayalath (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312119#comment-16312119
 ] 

Chamikara Jayalath commented on BEAM-3406:
--

I don't have access to that project. But [~jasonkuster] or [~alanmyrvold] might 
have access.

Alternatively, you can specify the schema here so that somebody who have access 
can create the table.


> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2774) Add I/O source for VCF files (python)

2018-01-04 Thread Asha Rostamianfar (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Asha Rostamianfar closed BEAM-2774.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

The main code is done. Additional features (e.g. ReadAllFromVcf to handle 
massive number of files) will be done in separate issues.

> Add I/O source for VCF files (python)
> -
>
> Key: BEAM-2774
> URL: https://issues.apache.org/jira/browse/BEAM-2774
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Asha Rostamianfar
>Assignee: Miles Saul
> Fix For: 2.3.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> A new I/O source for reading (and eventually writing) VCF files [1] for 
> Python. The design doc is available at 
> https://docs.google.com/document/d/1jsdxOPALYYlhnww2NLURS8NKXaFyRSJrcGbEDpY9Lkw/edit
> [1] http://samtools.github.io/hts-specs/VCFv4.3.pdf



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

2018-01-04 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312082#comment-16312082
 ] 

Ismaël Mejía commented on BEAM-981:
---

[~ŁukaszG] excellent work ! Are you planning to work on BEAM-3371 too ?
I have noticed that after this change I get exceptions like this one:
{quote}
ERROR org.apache.spark.SparkContext  - Failed to add 
/home/ismael/workspace/beam2/runners/spark/target/test-classes to Spark 
environment
java.lang.IllegalArgumentException: Directory 
/home/ismael/workspace/beam2/runners/spark/target/test-classes is not allowed 
for addJar
{quote}
Do you think that it would make sense to filter the list returned on 
`PipelineResources#detectClassPathResourcesToStage` in the SparkRunner to 
include only the jar files and avoid this error message at least until 
BEAM-3371 is done ?

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
>

Build failed in Jenkins: beam_PostCommit_Python_Verify #3876

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[github] Use beam-master-$DATE pattern for default Dataflow containers in

[github] Update dependency.py

[github] fixup: lint

--
[...truncated 1.04 MB...]
copying apache_beam/portability/api/endpoints_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/endpoints_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/sdf_common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py ->

[jira] [Assigned] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-981:
-

Assignee: Łukasz Gajowy  (was: Kobi Salant)

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
> at 
> org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
> at 
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
> at 
> io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:845)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:750)
> at

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #581

2018-01-04 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312041#comment-16312041
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

robertwb opened a new pull request #4344: [BEAM-1866] Plumb user metrics 
through Fn API.
URL: https://github.com/apache/beam/pull/4344
 
 
   The SDK worker is now periodically querried for progress
   and user metrics.
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #59

2018-01-04 Thread Apache Jenkins Server

See 


--
[...truncated 50.59 KB...]
2018-01-04 18:22:52,050 83dc1a11 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 18:22:52,051 83dc1a11 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 18:23:17,698 83dc1a11 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json
2018-01-04 18:23:18,310 83dc1a11 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 18:23:18,310 83dc1a11 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 18:23:47,138 83dc1a11 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json
2018-01-04 18:23:47,827 83dc1a11 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 18:23:47,828 83dc1a11 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 18:24:12,808 83dc1a11 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json
2018-01-04 18:24:13,387 83dc1a11 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 18:24:13,387 83dc1a11 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 18:24:35,500 83dc1a11 MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json
2018-01-04 18:24:36,082 83dc1a11 MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/83dc1a11/perfkit-bq-pubUU6wuW.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for

[beam] 01/01: Merge pull request #4322 from tvalentyn/patch-4

2018-01-04 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d28cc55fd6b1287adca142b8b3c517cb516aec06
Merge: 4fef942 eaf4918
Author: Ahmet Altay 
AuthorDate: Thu Jan 4 10:28:50 2018 -0800

Merge pull request #4322 from tvalentyn/patch-4

Use beam-master-$DATE pattern for default Dataflow containers in unre…

 .../apache_beam/runners/dataflow/internal/dependency.py | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch master updated (4fef942 -> d28cc55)

2018-01-04 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4fef942  Merge pull request #4294 from jkff/other-sinks
 add 8188db4  Use beam-master-$DATE pattern for default Dataflow containers 
in unreleased BEAM SDK.
 add fe46663  Update dependency.py
 add eaf4918  fixup: lint
 new d28cc55  Merge pull request #4322 from tvalentyn/patch-4

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/runners/dataflow/internal/dependency.py | 13 ++---
 1 file changed, 6 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

Build failed in Jenkins: beam_PerformanceTests_Python #752

2018-01-04 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4fef942195d0fc1052e6888240d44668580a5013 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4fef942195d0fc1052e6888240d44668580a5013
Commit message: "Merge pull request #4294 from jkff/other-sinks"
 > git rev-list 4fef942195d0fc1052e6888240d44668580a5013 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9006877130268138335.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3711894467861903065.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1631747829998619761.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already

[jira] [Commented] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311767#comment-16311767
 ] 

Ismaël Mejía commented on BEAM-3409:


For some extra context Alexey found this one with a test that was failing 
because the teardown part of the DoFn took a long time but the thread execution 
on TestPipeline was finishing before completion. I tested this with other 
TestPipelines e.g.TestSparkPipeline and TestFlinkPipeline and it worked ok, so 
I suppose this is probably closer to the direct runner and junit thread 
creation/finalization.
We are a bit surprised of this behavior but it could be easily a source for 
other flaky tests.

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3409:
---
Labels: test  (was: )

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3409:
--

Assignee: Thomas Groh  (was: Jason Kuster)

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3409:
---
Component/s: runner-direct

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Jason Kuster
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3409:
---
Affects Version/s: 2.3.0

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, testing
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Jason Kuster
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Spark #1196

2018-01-04 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4fef942195d0fc1052e6888240d44668580a5013 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4fef942195d0fc1052e6888240d44668580a5013
Commit message: "Merge pull request #4294 from jkff/other-sinks"
 > git rev-list 4fef942195d0fc1052e6888240d44668580a5013 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5882404396841376778.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins342448344786314088.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5165496937501994818.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied:

[jira] [Updated] (BEAM-3399) Documentation wrong for Session Windows (Python SDK)

2018-01-04 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3399:
--
Labels: starter  (was: )

> Documentation wrong for Session Windows (Python SDK)
> 
>
> Key: BEAM-3399
> URL: https://issues.apache.org/jira/browse/BEAM-3399
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Affects Versions: 2.2.0
>Reporter: Ralf Hein
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>
> The documentation for setting Session Windows via python SDK is wrong 
> (Section 7.3.3).
> https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function
> It says "time gap of at least 10 minutes" when introducing the example code, 
> but the code actually only sets a duration gap of 10 seconds.
> Reason: The duration needs to be passed in seconds, not minutes. So the 
> example code should read something like this:
> beam.WindowInto(window.Sessions(10*60)))
> Took me quite some time to figure this out ...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3403) Ingesting json file ValidationError: Expected type

2018-01-04 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3403:
-

Assignee: Chamikara Jayalath  (was: Ahmet Altay)

> Ingesting json file ValidationError: Expected type 
> ---
>
> Key: BEAM-3403
> URL: https://issues.apache.org/jira/browse/BEAM-3403
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Akash Patel
>Assignee: Chamikara Jayalath
>
> Reading a json file from GCS file pattern using Beam Python SDK 2.2.0 in 
> Dataflow yields the following warning:
> {code:bash}
> Retry with exponential backoff: waiting for 4.21317187833 seconds before 
> retrying report_completion_status because we caught exception: 
> ValidationError: Expected type  for field name, found 
> s05-s34-reify20-process-msecs (type  'apache_beam.utils.counters.CounterName'>) Traceback for above exception 
> (most recent call last): File 
> "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 
> 175, in wrapper return fun(*args, **kwargs) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 
> 491, in report_completion_status exception_details=exception_details) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 
> 299, in report_status work_executor=self._work_executor) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py", 
> line 316, in report_status append_counter(work_item_status, counter, 
> tentative=not completed) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py", 
> line 43, in append_counter status_object, counter.name, kind, 
> counter.accumulator, setter) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py", 
> line 95, in append_counter_update 
> add_unstructured_name_and_kind(metric_update, metric_name, kind) File 
> "/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py", 
> line 63, in add_unstructured_name_and_kind metric_update.nameAndKind.name = 
> metric_name File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 973, in __setattr__ object.__setattr__(self, name, value) File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 1299, in __set__ value = self.validate(value) File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 1406, in validate return self.__validate(value, self.validate_element) 
> File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 1364, in __validate return validate_element(value) File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 1549, in validate_element return super(StringField, 
> self).validate_element(value) File 
> "/usr/local/lib/python2.7/dist-packages/apitools/base/protorpclite/messages.py",
>  line 1346, in validate_element (self.type, name, value, type(value)))
> {code}
> The job does not fail but rather gets stuck on trying to read the file. The 
> above warning is thrown for every retry read.
> However running the job with Beam Python SDK 2.1.1 works perfectly fine.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

2018-01-04 Thread Alexey Romanenko (JIRA)

Alexey Romanenko created BEAM-3409:
--

 Summary: Unexpected behavior of DoFn teardown method running in 
unit tests 
 Key: BEAM-3409
 URL: https://issues.apache.org/jira/browse/BEAM-3409
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Alexey Romanenko
Assignee: Jason Kuster
Priority: Minor


Writing a unit test, I found out a strange behaviour of Teardown method of DoFn 
implementation when I run this method in unit tests using TestPipeline.

To be more precise, it doesn’t wait until teardown() method will be finished, 
it just exits from this method after about 1 sec (on my machine) even if it 
should take longer (very simple example - running infinite loop inside this 
method or put thread in sleep). In the same time, when I run the same code from 
main() with ordinary Pipeline and direct runner, then it’s ok and it works as 
expected - teardown() method will be performed completely despite how much time 
it will take.

I created two test cases to reproduce this issue - the first one to run with 
main() and the second one to run with junit. They use the same implementation 
of DoFn (class LongTearDownFn) and expects that teardown method will be running 
at least for SLEEP_TIME ms. In case of running as junit test it's not a case 
(see output log).

- run with main()
https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java

- run with junit
https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3406) FIeBasedIO IT fails due to "BigQuery error"

2018-01-04 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311570#comment-16311570
 ] 

Łukasz Gajowy commented on BEAM-3406:
-

In short: to fix this issue right away I think we need access to 
apache-beam-testing project to set schema in bigquery table.
 
While creating the dashboard for FileBased ioit we created the table in the 
bigquery underneath but the schema was not set and it is needed there. 
According to this:  
[documentation|https://cloud.google.com/bigquery/docs/schemas#specify-schema-manual-cli]
 we can set the schema while loading data or right after creating the table. 
Perfkit does not set the schema in any way for now.

We could set the schema manually and fix the issue, but AFAIK we do not have 
access to "apache-beam-testing" where bigquery seems to be located. Could you 
grant it to us [~chamikara]?

I also created [an 
issue|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/1552] on 
PKBs github to add an --autodetect flag while loading json files to bigquery - 
as i checked locally, the schema we use can be autodetected. If this issue gets 
resolved and merged in perfkit we won't have to worry about the schema anymore 
as it will be set automatically.



> FIeBasedIO IT fails due to "BigQuery error"
> ---
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>
> For example, 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results 
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return 
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR: 
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
>  UserWarning: You have requested explicit scopes to be used with a GCE 
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
>   warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_FileBasedIO_IT #58

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] Supports a sink for XmlIO

[kirpichov] Adds TFRecordIO.sink()

--
[...truncated 54.72 KB...]
2018-01-04 12:27:08,946 b93b56ad MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 12:27:08,947 b93b56ad MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 12:27:38,868 b93b56ad MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json
2018-01-04 12:27:40,396 b93b56ad MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 12:27:40,396 b93b56ad MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 12:27:58,111 b93b56ad MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json
2018-01-04 12:27:58,637 b93b56ad MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 12:27:58,637 b93b56ad MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 12:28:28,595 b93b56ad MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json
2018-01-04 12:28:29,109 b93b56ad MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-01-04 12:28:29,110 b93b56ad MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-01-04 12:28:50,011 b93b56ad MainThread INFO Running: bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json
2018-01-04 12:28:50,688 b93b56ad MainThread INFO Ran bq load 
--source_format=NEWLINE_DELIMITED_JSON 
beam_performance.filebasedioit_pkb_results 
/tmp/perfkitbenchmarker/runs/b93b56ad/perfkit-bq-pubfjgj7Z.json. Got return 
code (1).
STDOUT: BigQuery error in load operation: No schema specified on job or table.

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be

Build failed in Jenkins: beam_PerformanceTests_Python #751

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] Supports a sink for XmlIO

[kirpichov] Adds TFRecordIO.sink()

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4fef942195d0fc1052e6888240d44668580a5013 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4fef942195d0fc1052e6888240d44668580a5013
Commit message: "Merge pull request #4294 from jkff/other-sinks"
 > git rev-list 0892e23940a584a170e2668d86b7bdc50c03e024 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1320070348921939371.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6975765461444100490.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6911215458677708533.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from

Build failed in Jenkins: beam_PerformanceTests_Spark #1195

2018-01-04 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] Supports a sink for XmlIO

[kirpichov] Adds TFRecordIO.sink()

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4fef942195d0fc1052e6888240d44668580a5013 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4fef942195d0fc1052e6888240d44668580a5013
Commit message: "Merge pull request #4294 from jkff/other-sinks"
 > git rev-list 0892e23940a584a170e2668d86b7bdc50c03e024 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins9156352944654133560.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins1648841912361959179.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3398830651624078306.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from

[jira] [Commented] (BEAM-3394) Trying to run Apache Beam 2.3.0-SNAPSHOT pipeline on Spark 2.2.1 standalone cluster and I am facing some ClassCastException.

2018-01-04 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311167#comment-16311167
 ] 

Jean-Baptiste Onofré commented on BEAM-3394:


The versions looks good. The ClassCastException is between spark core (RDD) and 
scala Collections. So, the Scala Collections versions expected by Spark core 
doesn't seem to the right one.
I would double check the Scala version on the cluster. In the mean time, I'm 
boostrapping my Spark 2.2.1 cluster to try to reproduce (tweaking the scala 
versions).

> Trying to run Apache Beam 2.3.0-SNAPSHOT pipeline on Spark 2.2.1 standalone 
> cluster and I am facing some ClassCastException.
> 
>
> Key: BEAM-3394
> URL: https://issues.apache.org/jira/browse/BEAM-3394
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.3.0
>Reporter: ethanhunt999
>Assignee: Jean-Baptiste Onofré
>
> As Jira Issue for [Spark 2.x support in Spark 
> runner](https://issues.apache.org/jira/browse/BEAM-1920) is resolved, Now I 
> am able to execute pipeline in Local mode successfully. But if am trying to 
> run on Spark standalone cluster its showing some _ClassCastException_
> Here is some tech versioning details which am currently using may be will 
> helpful to you:
>  -Scala 2.11.8
>  -Spark 2.2.1
>  -Apache Beam 2.3.0-SNAPSHOT
> Here is the full trace for error
> {noformat}
> java.lang.ClassCastException: cannot assign instance of 
> scala.collection.immutable.List$SerializationProxy to field 
> org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type 
> scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
>   at 
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>   at 
> java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2288)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
>   at 
> scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2173)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
>   at 
> scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2173)
>   at 
>

[jira] [Commented] (BEAM-3394) Trying to run Apache Beam 2.3.0-SNAPSHOT pipeline on Spark 2.2.1 standalone cluster and I am facing some ClassCastException.

2018-01-04 Thread ethanhunt999 (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311163#comment-16311163
 ] 

ethanhunt999 commented on BEAM-3394:


If you see here in dependency detail which I am using, Seems to be everything 
is fine. Spark specific dependencies have runtime scope but still I am facing 
the same error. Even I tried with new Spark binary to run standalone spark but 
no luck. I just don't understand Where I am doing wrong. Locally Project is 
working fine but not working on Standalone cluster.
{noformat}
 val javaCore="org.apache.beam" % "beam-sdks-java-core" % beamVersion

//Spark
val sparkRunner="org.apache.beam" % "beam-runners-spark" % beamVersion % 
"runtime"
val sparkStream="org.apache.spark" % "spark-streaming_2.11" % sparkVersion 
% "runtime"


val protobuf="com.google.protobuf" % "protobuf-java" % "3.2.0"

val jacksonModuleScala="com.fasterxml.jackson.module" % 
"jackson-module-scala_2.11" %  "2.8.9" % "runtime"

{noformat}


> Trying to run Apache Beam 2.3.0-SNAPSHOT pipeline on Spark 2.2.1 standalone 
> cluster and I am facing some ClassCastException.
> 
>
> Key: BEAM-3394
> URL: https://issues.apache.org/jira/browse/BEAM-3394
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.3.0
>Reporter: ethanhunt999
>Assignee: Jean-Baptiste Onofré
>
> As Jira Issue for [Spark 2.x support in Spark 
> runner](https://issues.apache.org/jira/browse/BEAM-1920) is resolved, Now I 
> am able to execute pipeline in Local mode successfully. But if am trying to 
> run on Spark standalone cluster its showing some _ClassCastException_
> Here is some tech versioning details which am currently using may be will 
> helpful to you:
>  -Scala 2.11.8
>  -Spark 2.2.1
>  -Apache Beam 2.3.0-SNAPSHOT
> Here is the full trace for error
> {noformat}
> java.lang.ClassCastException: cannot assign instance of 
> scala.collection.immutable.List$SerializationProxy to field 
> org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type 
> scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
>   at 
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>   at 
> java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2288)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
>   at 
> scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2173)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1568)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:428)
>   at 
> scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at

[jira] [Closed] (BEAM-2377) Cross compile flink runner to scala 2.11

2018-01-04 Thread Aljoscha Krettek (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2377.
--
   Resolution: Invalid
Fix Version/s: (was: 2.3.0)
   Not applicable

Now that we always build against Flink_2.11 this becomes obsolete.

> Cross compile flink runner to scala 2.11
> 
>
> Key: BEAM-2377
> URL: https://issues.apache.org/jira/browse/BEAM-2377
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Ole Langbehn
>Assignee: Aljoscha Krettek
> Fix For: Not applicable
>
>
> The flink runner is compiled for flink built against scala 2.10. flink cross 
> compiles its scala artifacts against 2.10 and 2.11.
> In order to make it possible to use beam with the flink runner in scala 2.11 
> projects, it would be nice if you could publish the flink runner for 2.11 
> next to 2.10.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

63 matches

Mail list logo