[jira] [Created] (BEAM-3740) Prototype of retractions support by DirectRunner

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3740:
-

 Summary: Prototype of retractions support by DirectRunner
 Key: BEAM-3740
 URL: https://issues.apache.org/jira/browse/BEAM-3740
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Reporter: Anton Kedin
Assignee: Anton Kedin


Build prototype retractions support with DirectRunner to identify correct path 
forward



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3743) Support for SDF splitting protocol in ULR

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-3743:
--

 Summary: Support for SDF splitting protocol in ULR
 Key: BEAM-3743
 URL: https://issues.apache.org/jira/browse/BEAM-3743
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


If I understand correctly what ULR does and where it currently stands - this is 
the task for a reference implementation of the runner side of things from 
https://s.apache.org/beam-breaking-fusion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #981

[Apache Jenkins Server]

See 


--
[...truncated 118.22 KB...]
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 839, in to_runner_api_parameter
source=self.source.to_runner_api(context),
  File 
"
 line 94, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 82, in 
pickle_urn, wrappers_pb2.BytesValue(value=pickler.dumps(self
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 94, in wrapper
obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 400, in save_reduce
save(func)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1314, in save_function
log.info("F2: %s" % obj)
  File "/usr/lib/python2.7/logging/__init__.py", line 1151, in info
if self.isEnabledFor(INFO):
  File "/usr/lib/python2.7/logging/__init__.py", line 1351, in isEnabledFor
return level >= self.getEffectiveLevel()
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 

[jira] [Commented] (BEAM-3681) S3Filesystem fails when copying empty files

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375031#comment-16375031
 ] 

Ismaël Mejía commented on BEAM-3681:


It was quite tricky to find, I really found it by pure luck while trying to 
test the write for the first time with exactly the same example I mentioned.

> S3Filesystem fails when copying empty files
> ---
>
> Key: BEAM-3681
> URL: https://issues.apache.org/jira/browse/BEAM-3681
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When executing a simple write on S3 with the direct runner. It breaks 
> sometimes when it ends up trying to write 'empty' shards to S3.
> {code:java}
> Pipeline pipeline = Pipeline.create(options);
> pipeline
>  .apply("CreateSomeData", Create.of("1", "2", "3"))
>  .apply("WriteToFS", TextIO.write().to(options.getOutput()));
> pipeline.run();{code}
> The related exception is:
> {code:java}
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:312)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:206)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>     at 
> org.apache.beam.samples.ingest.amazon.IngestToS3.main(IngestToS3.java:82)
> Caused by: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at org.apache.beam.sdk.io.aws.s3.S3FileSystem.copy(S3FileSystem.java:563)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$copy$4(S3FileSystem.java:495)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$callTasks$8(S3FileSystem.java:642)
>     at 
> org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:100)
>     at 
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The XML you 
> provided was not well-formed or did not validate against our published schema 
> (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
>     at 
> 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #982

[Apache Jenkins Server]

See 


Changes:

[mairbek] Changes multipe structuredValue implementations. Most notably, updates

--
[...truncated 117.95 KB...]
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1057, in save_cell
pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 686, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 142, in new_save_module_dict
@dill.dill.register(dict)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 

[jira] [Closed] (BEAM-3698) Support SDF over Fn API

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-3698.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> Support SDF over Fn API
> ---
>
> Key: BEAM-3698
> URL: https://issues.apache.org/jira/browse/BEAM-3698
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>
> SDF is the only API for emitting unbounded data in Fn API. This issue is 
> about supporting SDF in an SDK harness, with its checkpointing controlled by 
> the runner harness.
> ++Umbrella issue: includes design and implementation in various SDK languages.
> First practically useful goal in sight: Kafka source implemented in the 
> Python SDK, running over Fn API using the Dataflow runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2939) Fn API streaming SDF support

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-2939:
--

Assignee: Eugene Kirpichov

> Fn API streaming SDF support
> 
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Eugene Kirpichov
>Priority: Major
>  Labels: portability
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-591] KafkaIO : Improve watermarks and support server side timestamps (#4680)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 9f84beb  [BEAM-591] KafkaIO : Improve watermarks and support server 
side timestamps (#4680)
9f84beb is described below

commit 9f84bebc4b602551b2c719702cbb4dbab7c5b258
Author: Raghu Angadi 
AuthorDate: Fri Feb 23 16:42:18 2018 -0800

[BEAM-591] KafkaIO : Improve watermarks and support server side timestamps 
(#4680)

* Redesign how timestamps and watermarks are handled in KafkaIO.
 - Added TimestampPolicy that provides both record timestamps and
 watermarks.
 - built in policies for 'LogAppendTime' (server-time) ProcessingTime
 (default)
 - Ensure idle partitions don't hold watermark back
 - deprecated previous API to set functions for custom timestamps and
 watermarks.

* minor
---
 .../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java |  20 +-
 .../beam/sdk/io/kafka/KafkaCheckpointMark.java |  32 ++--
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |  83 -
 .../org/apache/beam/sdk/io/kafka/KafkaRecord.java  |  14 +-
 .../apache/beam/sdk/io/kafka/KafkaRecordCoder.java |   3 +
 .../beam/sdk/io/kafka/KafkaTimestampType.java  |  44 +
 .../beam/sdk/io/kafka/KafkaUnboundedReader.java| 200 +---
 .../apache/beam/sdk/io/kafka/TimestampPolicy.java  |  68 +++
 .../beam/sdk/io/kafka/TimestampPolicyFactory.java  | 206 +
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 145 +--
 10 files changed, 706 insertions(+), 109 deletions(-)

diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
index a3bd439..f615ad6 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
@@ -80,10 +80,10 @@ class ConsumerSpEL {
 }
   }
 
-  public void evaluateSeek2End(Consumer consumer, TopicPartition 
topicPartitions) {
+  public void evaluateSeek2End(Consumer consumer, TopicPartition 
topicPartition) {
 StandardEvaluationContext mapContext = new StandardEvaluationContext();
 mapContext.setVariable("consumer", consumer);
-mapContext.setVariable("tp", topicPartitions);
+mapContext.setVariable("tp", topicPartition);
 seek2endExpression.getValue(mapContext);
   }
 
@@ -95,11 +95,19 @@ class ConsumerSpEL {
   }
 
   public long getRecordTimestamp(ConsumerRecord rawRecord) {
-long timestamp;
-if (!hasRecordTimestamp || (timestamp = rawRecord.timestamp()) <= 0L) {
-  timestamp = System.currentTimeMillis();
+if (hasRecordTimestamp) {
+  return rawRecord.timestamp();
+}
+return -1L; // This is the timestamp used in Kafka for older messages 
without timestamps.
+  }
+
+  public KafkaTimestampType getRecordTimestamptType(
+ConsumerRecord rawRecord) {
+if (hasRecordTimestamp) {
+  return 
KafkaTimestampType.forOrdinal(rawRecord.timestampType().ordinal());
+} else {
+  return KafkaTimestampType.NO_TIMESTAMP_TYPE;
 }
-return timestamp;
   }
 
   public boolean hasOffsetsForTimes() {
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
index 791e594..95ec7ca 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
@@ -18,13 +18,14 @@
 package org.apache.beam.sdk.io.kafka;
 
 import com.google.common.base.Joiner;
-import java.io.IOException;
 import java.io.Serializable;
 import java.util.List;
+import java.util.Optional;
 import org.apache.avro.reflect.AvroIgnore;
 import org.apache.beam.sdk.coders.AvroCoder;
 import org.apache.beam.sdk.coders.DefaultCoder;
 import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 
 /**
  * Checkpoint for a {@link KafkaUnboundedReader}. Consists of Kafka topic 
name, partition id,
@@ -36,12 +37,12 @@ public class KafkaCheckpointMark implements 
UnboundedSource.CheckpointMark {
   private List partitions;
 
   @AvroIgnore
-  private KafkaUnboundedReader reader; // Non-null when offsets need to 
be committed.
+  private Optional reader; // Present when offsets 
need to be committed.
 
   private KafkaCheckpointMark() {} // for Avro
 
   public KafkaCheckpointMark(List partitions,
- KafkaUnboundedReader reader) {
+ Optional reader) {
 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6047

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4299

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job

[jira] [Created] (BEAM-3741) Proto changes for splitting over Fn API

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-3741:
--

 Summary: Proto changes for splitting over Fn API
 Key: BEAM-3741
 URL: https://issues.apache.org/jira/browse/BEAM-3741
 Project: Beam
  Issue Type: Sub-task
  Components: beam-model
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2939) Fn API streaming SDF support

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375125#comment-16375125
 ] 

Eugene Kirpichov commented on BEAM-2939:


Proposed design: https://s.apache.org/beam-breaking-fusion ; start with the 
simple case.

> Fn API streaming SDF support
> 
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Eugene Kirpichov
>Priority: Major
>  Labels: portability
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #177

[Apache Jenkins Server]

See 


Changes:

[mairbek] Changes multipe structuredValue implementations. Most notably, updates

--
[...truncated 32.00 KB...]
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0-beta from the 
shaded jar.
[INFO] Excluding 

[jira] [Created] (BEAM-3744) Support full PubsubMessages

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-3744:
---

 Summary: Support full PubsubMessages
 Key: BEAM-3744
 URL: https://issues.apache.org/jira/browse/BEAM-3744
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Udi Meiri


Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3742) Support for running a streaming SDF in Python SDK

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-3742:
--

 Summary: Support for running a streaming SDF in Python SDK
 Key: BEAM-3742
 URL: https://issues.apache.org/jira/browse/BEAM-3742
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-harness
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 14b17ad  [BEAM-3634] Refactor BeamRelNode to return a PTransform 
(#4705)
14b17ad is described below

commit 14b17ad574342a875c8f99278e18c605aa5b4bc3
Author: Kenn Knowles 
AuthorDate: Fri Feb 23 16:39:21 2018 -0800

[BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

* Refactor BeamRelNode to return a PTransform

* Refactor SQL toPTransform to have no parameters
---
 .../beam/sdk/extensions/sql/QueryTransform.java|   3 +-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  13 +-
 .../sql/impl/planner/BeamQueryPlanner.java |  21 +-
 .../extensions/sql/impl/planner/BeamRuleSets.java  |  54 ++--
 .../sql/impl/rel/BeamAggregationRel.java   |  19 +-
 .../sdk/extensions/sql/impl/rel/BeamFilterRel.java |  34 ++-
 .../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java |  85 --
 .../extensions/sql/impl/rel/BeamIOSourceRel.java   |  45 +--
 .../extensions/sql/impl/rel/BeamIntersectRel.java  |  33 ++-
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   | 328 -
 .../sdk/extensions/sql/impl/rel/BeamMinusRel.java  |  16 +-
 .../extensions/sql/impl/rel/BeamProjectRel.java|  37 ++-
 .../sdk/extensions/sql/impl/rel/BeamRelNode.java   |  15 +-
 .../sql/impl/rel/BeamSetOperatorRelBase.java   |  14 +-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  71 +++--
 .../sdk/extensions/sql/impl/rel/BeamUnionRel.java  |  15 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  40 +--
 .../extensions/sql/impl/rule/BeamIOSinkRule.java   |  58 ++--
 .../extensions/sql/impl/rule/BeamIOSourceRule.java |  30 +-
 .../sdk/extensions/sql/impl/rule/BeamJoinRule.java |  33 ++-
 .../extensions/sql/impl/rule/BeamProjectRule.java  |  37 +--
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java |   6 +-
 .../interpreter/BeamSqlFnExecutorTestBase.java |  42 +--
 .../transform/agg/VarianceAccumulatorTest.java |   6 +-
 .../sql/impl/transform/agg/VarianceFnTest.java |   3 +-
 25 files changed, 599 insertions(+), 459 deletions(-)

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
index 8a7335f..67ccbb1 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
@@ -67,10 +67,11 @@ public abstract class QueryTransform extends 
PTransform(16);
 schema = Frameworks.createRootSchema(true);
-planner = new BeamQueryPlanner(schema);
+planner = new BeamQueryPlanner(this, schema);
   }
 
   /**
@@ -151,7 +150,7 @@ public class BeamSqlEnv implements Serializable {
 schema.add(tableName, new BeamCalciteTable(table.getRowType()));
   }
 }
-planner = new BeamQueryPlanner(schema);
+planner = new BeamQueryPlanner(this, schema);
   }
 
   /**
@@ -216,6 +215,6 @@ public class BeamSqlEnv implements Serializable {
 
 tables = 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4300

[Apache Jenkins Server]

See 


Changes:

[mairbek] Changes multipe structuredValue implementations. Most notably, updates

--
[...truncated 1.03 MB...]
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying 

Build failed in Jenkins: beam_PerformanceTests_Spark #1393

[Apache Jenkins Server]

See 


Changes:

[mairbek] Changes multipe structuredValue implementations. Most notably, updates

--
[...truncated 91.66 KB...]
'apache-beam-testing:bqjob_r2418b1e0a6f9ee32_0161c5301541_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 00:22:00,134 1e29391e MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 00:22:17,834 1e29391e MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 00:22:20,166 1e29391e MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.32s,  CPU:0.26s,  MaxMemory:25464kb 
STDOUT: Upload complete.
Waiting on bqjob_r4e22da6e982e34b6_0161c5306300_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4e22da6e982e34b6_0161c5306300_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4e22da6e982e34b6_0161c5306300_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 00:22:20,166 1e29391e MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 00:22:44,994 1e29391e MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 00:22:47,184 1e29391e MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.18s,  CPU:0.26s,  MaxMemory:25424kb 
STDOUT: Upload complete.
Waiting on bqjob_r3de643ab1bb349f7_0161c530cd0b_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r3de643ab1bb349f7_0161c530cd0b_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3de643ab1bb349f7_0161c530cd0b_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 00:22:47,185 1e29391e MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 00:23:14,493 1e29391e MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 00:23:18,647 1e29391e MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:04.11s,  CPU:0.41s,  MaxMemory:25488kb 
STDOUT: Upload complete.
Waiting on bqjob_r238beb3757fc0ada_0161c531472a_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r238beb3757fc0ada_0161c531472a_1 ... (0s) 
Current 

Build failed in Jenkins: beam_PerformanceTests_Python #949

[Apache Jenkins Server]

See 


Changes:

[mairbek] Changes multipe structuredValue implementations. Most notably, updates

--
[...truncated 15 B...]
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1e84e49e253f8833f28f1268bec3813029f582d0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1e84e49e253f8833f28f1268bec3813029f582d0
Commit message: "Merge pull request #4249: Update StructuredValue 
Implementations"
 > git rev-list --no-walk 0259636661998b8aabd3bf20eff8d519bcb9fb47 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7392072764576549038.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8907063838636694413.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8373409395755446827.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins523815507295817920.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5773654253742787451.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3728158331247681406.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #984

[Apache Jenkins Server]

See 


--
[...truncated 121.93 KB...]
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 839, in to_runner_api_parameter
source=self.source.to_runner_api(context),
  File 
"
 line 94, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 82, in 
pickle_urn, wrappers_pb2.BytesValue(value=pickler.dumps(self
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 637, in _batch_appends
write(APPEND)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4302

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #178

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #194

[Apache Jenkins Server]

See 


Changes:

[XuMingmin] [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[XuMingmin] [BEAM-591] KafkaIO : Improve watermarks and support server side

--
[...truncated 14.71 KB...]
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2439467635122702844.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.textioit_pkb_results 
--temp_dir= 
--official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/194/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-24 06:00:49,933 6475c4b6 MainThread INFO Verbose logging to: 

2018-02-24 06:00:49,934 6475c4b6 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-369-g5f134e6
2018-02-24 06:00:49,934 6475c4b6 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/194/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.textioit_pkb_results
--official
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-24 06:00:50,180 6475c4b6 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-24 06:00:50,181 6475c4b6 MainThread INFO Initializing the edw 
service decoder
2018-02-24 06:00:50,324 6475c4b6 MainThread beam_integration_benchmark(1/1) 
INFO Provisioning resources for benchmark beam_integration_benchmark
2018-02-24 06:00:50,326 6475c4b6 MainThread beam_integration_benchmark(1/1) 
INFO Preparing benchmark beam_integration_benchmark
2018-02-24 06:00:50,326 6475c4b6 MainThread beam_integration_benchmark(1/1) 
INFO Running: git clone https://github.com/apache/beam.git
2018-02-24 06:00:59,962 6475c4b6 MainThread beam_integration_benchmark(1/1) 
INFO Running benchmark beam_integration_benchmark
2018-02-24 06:00:59,974 6475c4b6 MainThread beam_integration_benchmark(1/1) 
INFO Running: /home/jenkins/tools/maven/latest/bin/mvn -e verify 
-Dit.test=org.apache.beam.sdk.io.text.TextIOIT -DskipITs=false -pl 
sdks/java/io/file-based-io-tests -Pio-it -Pdataflow-runner -Dfilesystem=gcs 
-DintegrationTestPipelineOptions=["--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-perf-tests","--numberOfRecords=100","--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/194/","--runner=TestDataflowRunner"]
2018-02-24 06:20:59,995 6475c4b6 Thread-2 ERRORIssueCommand timed out after 
1200 seconds. Killing command "/home/jenkins/tools/maven/latest/bin/mvn -e 
verify -Dit.test=org.apache.beam.sdk.io.text.TextIOIT -DskipITs=false -pl 
sdks/java/io/file-based-io-tests -Pio-it -Pdataflow-runner -Dfilesystem=gcs 

Build failed in Jenkins: beam_PerformanceTests_Spark #1394

[Apache Jenkins Server]

See 


Changes:

[XuMingmin] [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[XuMingmin] [BEAM-591] KafkaIO : Improve watermarks and support server side

--
[...truncated 96.21 KB...]
'apache-beam-testing:bqjob_rf0b50098350cd70_0161c677db82_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 06:20:01,055 24ec1ea2 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 06:20:26,354 24ec1ea2 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 06:20:28,516 24ec1ea2 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.14s,  CPU:0.34s,  MaxMemory:25252kb 
STDOUT: Upload complete.
Waiting on bqjob_r58713a00a356ebed_0161c67846e1_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r58713a00a356ebed_0161c67846e1_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r58713a00a356ebed_0161c67846e1_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 06:20:28,516 24ec1ea2 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 06:20:52,166 24ec1ea2 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 06:20:54,526 24ec1ea2 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.35s,  CPU:0.31s,  MaxMemory:25396kb 
STDOUT: Upload complete.
Waiting on bqjob_r7d6685f87befeab0_0161c678ab89_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7d6685f87befeab0_0161c678ab89_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r7d6685f87befeab0_0161c678ab89_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-24 06:20:54,527 24ec1ea2 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-24 06:21:17,254 24ec1ea2 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-24 06:21:21,766 24ec1ea2 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:04.49s,  CPU:0.39s,  MaxMemory:25448kb 
STDOUT: Upload complete.
Waiting on bqjob_r522756f429df071b_0161c679120e_1 ... (0s) Current status: 
RUNNING 
 Waiting 

Build failed in Jenkins: beam_PerformanceTests_Python #950

[Apache Jenkins Server]

See 


Changes:

[XuMingmin] [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[XuMingmin] [BEAM-591] KafkaIO : Improve watermarks and support server side

--
[...truncated 550 B...]
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 9f84bebc4b602551b2c719702cbb4dbab7c5b258 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9f84bebc4b602551b2c719702cbb4dbab7c5b258
Commit message: "[BEAM-591] KafkaIO : Improve watermarks and support server 
side timestamps (#4680)"
 > git rev-list --no-walk 1e84e49e253f8833f28f1268bec3813029f582d0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4561445707007385111.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5660431915900580840.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9011575078856724443.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4032368630127496130.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2220808428130949536.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8648819835608540323.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #983

[Apache Jenkins Server]

See 


Changes:

[XuMingmin] [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[XuMingmin] [BEAM-591] KafkaIO : Improve watermarks and support server side

--
[...truncated 120.22 KB...]
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 859, in to_runner_api_parameter
picked_pardo_fn_data = pickler.dumps(self._pardo_fn_data())
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 258, in dumps
file = StringIO()
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 580, in to_runner_api
root_transform_id = 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4301

[Apache Jenkins Server]

See 


Changes:

[XuMingmin] [BEAM-3634] Refactor BeamRelNode to return a PTransform (#4705)

[XuMingmin] [BEAM-591] KafkaIO : Improve watermarks and support server side

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 

[jira] [Assigned] (BEAM-2219) All TestRunners or no TestRunners should be registered

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2219:
-

Assignee: Kenneth Knowles

> All TestRunners or no TestRunners should be registered
> --
>
> Key: BEAM-2219
> URL: https://issues.apache.org/jira/browse/BEAM-2219
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
>
> Right now, {{TestXRunner}} is key to get unit tests working but is probably 
> not a user-facing runner. However, they seem to be in the {{main}} (not test) 
> artifacts [at least for Dataflow] and to be registered pipeline runners.
> We should make all the runners behave the same way w.r.t. test runner 
> registration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4298

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job

[jira] [Created] (BEAM-3739) @Parameter annotation does not work for UDFs in Beam SQL

[Samuel Waggoner (JIRA)]

Samuel Waggoner created BEAM-3739:
-

 Summary: @Parameter annotation does not work for UDFs in Beam SQL
 Key: BEAM-3739
 URL: https://issues.apache.org/jira/browse/BEAM-3739
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Affects Versions: 2.3.0
Reporter: Samuel Waggoner
Assignee: Xu Mingmin


BeamSqlUdf javadoc indicates you can have optional parameters, but this 
functionality is not working. I implemented the following copy/paste from the 
doc 
https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/extensions/sql/BeamSqlUdf.html:
{code:java}
public static class MyLeftFunction implements BeamSqlUdf {
 public String eval(
 @Parameter(name = "s") String s,
 @Parameter(name = "n", optional = true) Integer n) {
 return s.substring(0, n == null ? 1 : n);
 }
}{code}
I modify a query in BeamSqlExample.java to use it. With all parameters 
supplied, it completes successfully:
{code:java}
//Case 1. run a simple SQL query over input PCollection with 
BeamSql.simpleQuery;
PCollection outputStream = inputTable.apply(
BeamSql.query("select c1, leftfn('string1', 1) as c2, c3 from PCOLLECTION 
where c1 > 1")
.registerUdf("leftfn", MyLeftFunction.class));{code}
With the optional parameter left off, I get an exception:
{code:java}
//Case 1. run a simple SQL query over input PCollection with 
BeamSql.simpleQuery;
PCollection outputStream = inputTable.apply(
 BeamSql.query("select c1, leftfn('string1') as c2, c3 from PCOLLECTION where 
c1 > 1")
 .registerUdf("leftfn", MyLeftFunction.class));{code}
{code:java}
Exception in thread "main" java.lang.IllegalStateException: 
java.lang.UnsupportedOperationException: Operator: DEFAULT is not supported yet!
 at 
org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:75)
 at 
org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
 at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
 at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
 at 
org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:76)
Caused by: java.lang.UnsupportedOperationException: Operator: DEFAULT is not 
supported yet!
 at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:424)
 at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.buildExpression(BeamSqlFnExecutor.java:201)
 at 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor.(BeamSqlFnExecutor.java:125)
 at 
org.apache.beam.sdk.extensions.sql.impl.rel.BeamProjectRel.buildBeamPipeline(BeamProjectRel.java:70)
 at 
org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
 ... 5 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3573) Test jars should export only tests

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3573:
-

Assignee: Kenneth Knowles

> Test jars should export only tests
> --
>
> Key: BEAM-3573
> URL: https://issues.apache.org/jira/browse/BEAM-3573
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> Today, we have test-jars that are used as libraries for testing. That is not 
> what "test jar" means, and dependency management actually does not work 
> correctly for this. It is OK to depend on a test jar in order to run the 
> tests therein, and not really OK to depend on one for another reason.
> This ticket is a bucket ticket for fixes to this situation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3138) Stop depending on Test JARs

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3138:
-

Assignee: Kenneth Knowles

> Stop depending on Test JARs
> ---
>
> Key: BEAM-3138
> URL: https://issues.apache.org/jira/browse/BEAM-3138
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Minor
>
> Testing components can be in a testing or otherwise signaled package, but 
> shouldn't really be depended on by depending on a test jar in the test scope.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2219) All TestRunners or no TestRunners should be registered

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2219.
---
   Resolution: Fixed
Fix Version/s: Not applicable

Looks like they are all registered. No need to backport release notes for this 
so leaving fix version unspecified.

> All TestRunners or no TestRunners should be registered
> --
>
> Key: BEAM-2219
> URL: https://issues.apache.org/jira/browse/BEAM-2219
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, runner-direct, runner-flink, 
> runner-spark
>Reporter: Daniel Halperin
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> Right now, {{TestXRunner}} is key to get unit tests working but is probably 
> not a user-facing runner. However, they seem to be in the {{main}} (not test) 
> artifacts [at least for Dataflow] and to be registered pipeline runners.
> We should make all the runners behave the same way w.r.t. test runner 
> registration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (817cf97 -> 0be4c54)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 817cf97  Merge pull request #4735 from angoenka/change_default_thread
 add bd9c6aa  `extractOutput()` ended up resetting underlying aggregation. 
This is due to use of `extractOrderedList()` which removes all the elements 
from the heap. `extractOrderedList()` is costly and is not required either. 
`extractOutput()` does not mutate now and is cheaper too.
 add 2cde00b  review comments.
 add 8bc6f23  Use TreeSet in place of PriorityQueue.
 add c24aebd  update checksWithMultipleMerges() to check for multiple 
merges by keeping track of number of merges in accumulator.
 new 0be4c54  Merge pull request #4688: [BEAM-3705] ApproximateUnique 
resets its accumulator with each firing.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/testing/CombineFnTester.java   |  1 +
 .../beam/sdk/transforms/ApproximateUnique.java | 85 +-
 .../beam/sdk/testing/CombineFnTesterTest.java  | 33 +
 .../beam/sdk/transforms/ApproximateUniqueTest.java | 60 +++
 4 files changed, 112 insertions(+), 67 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_Verify #4294

[Apache Jenkins Server]

See 


Changes:

[rangadi] `extractOutput()` ended up resetting underlying aggregation. This is 
due

[rangadi] review comments.

[rangadi] Use TreeSet in place of PriorityQueue.

[rangadi] update checksWithMultipleMerges() to check for multiple merges by

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #980

[Apache Jenkins Server]

See 


--
[...truncated 124.32 KB...]
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 324, in run_pipeline
self.dataflow_client.create_job(self.job), self)
  File 
"
 line 179, in wrapper
return fun(*args, **kwargs)
  File 
"
 line 461, in create_job
self.create_job_description(job)
  File 
"
 line 487, in create_job_description
StringIO(job.proto_pipeline.SerializeToString()))
  File 
"
 line 443, in stage_file
response = self._storage_client.objects.Insert(request, upload=upload)
  File 
"
 line 971, in Insert
download=download)
  File 
"
 line 720, in _RunMethod
http, http_request, **opts)
  File 
"
 line 346, in MakeRequest
check_response_func=check_response_func)
  File 
"
 line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
  File 
"
 line 175, in new_request
redirections, connection_type)
  File 
"
 line 282, in request
connection_type=connection_type)
  File 
"
 line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, 
method, body, headers, redirections, cachekey)
  File 
"
 line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, 
headers)
  File 
"
 line 1272, in _conn_request
conn.connect()
  File 
"
 line 1036, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
  File 
"
 line 80, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 243, in __init__
self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake
self._sslobj.do_handshake()
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: 

[jira] [Commented] (BEAM-3681) S3Filesystem fails when copying empty files

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374827#comment-16374827
 ] 

Jacob Marble commented on BEAM-3681:


Strange that I haven't bumped into this, but likely because I usually write 
gzip'd files to S3, so "empty" is 20 bytes.

> S3Filesystem fails when copying empty files
> ---
>
> Key: BEAM-3681
> URL: https://issues.apache.org/jira/browse/BEAM-3681
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When executing a simple write on S3 with the direct runner. It breaks 
> sometimes when it ends up trying to write 'empty' shards to S3.
> {code:java}
> Pipeline pipeline = Pipeline.create(options);
> pipeline
>  .apply("CreateSomeData", Create.of("1", "2", "3"))
>  .apply("WriteToFS", TextIO.write().to(options.getOutput()));
> pipeline.run();{code}
> The related exception is:
> {code:java}
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:312)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:206)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>     at 
> org.apache.beam.samples.ingest.amazon.IngestToS3.main(IngestToS3.java:82)
> Caused by: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at org.apache.beam.sdk.io.aws.s3.S3FileSystem.copy(S3FileSystem.java:563)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$copy$4(S3FileSystem.java:495)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$callTasks$8(S3FileSystem.java:642)
>     at 
> org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:100)
>     at 
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The XML you 
> provided was not well-formed or did not validate against our published schema 
> (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
>     at 
> 

Build failed in Jenkins: beam_PerformanceTests_Spark #1392

[Apache Jenkins Server]

See 


--
[...truncated 92.05 KB...]
'apache-beam-testing:bqjob_r2d88b74fa20c8ba6_0161c3e414e9_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-23 18:19:22,040 b47ebc1b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 18:19:39,960 b47ebc1b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 18:19:41,929 b47ebc1b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:01.96s,  CPU:0.25s,  MaxMemory:25432kb 
STDOUT: Upload complete.
Waiting on bqjob_r635d74d129a9e545_0161c3e46342_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r635d74d129a9e545_0161c3e46342_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r635d74d129a9e545_0161c3e46342_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-23 18:19:41,930 b47ebc1b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 18:19:57,413 b47ebc1b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 18:19:59,487 b47ebc1b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.06s,  CPU:0.24s,  MaxMemory:25424kb 
STDOUT: Upload complete.
Waiting on bqjob_r73b2c9ee89e3a6d5_0161c3e4a793_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r73b2c9ee89e3a6d5_0161c3e4a793_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r73b2c9ee89e3a6d5_0161c3e4a793_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-23 18:19:59,488 b47ebc1b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 18:20:25,752 b47ebc1b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 18:20:28,119 b47ebc1b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.36s,  CPU:0.27s,  MaxMemory:25140kb 
STDOUT: Upload complete.
Waiting on bqjob_r5537fbf1ecb2e659_0161c3e5163b_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r5537fbf1ecb2e659_0161c3e5163b_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job

Build failed in Jenkins: beam_PerformanceTests_Python #948

[Apache Jenkins Server]

See 


--
[...truncated 15 B...]
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0259636661998b8aabd3bf20eff8d519bcb9fb47 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0259636661998b8aabd3bf20eff8d519bcb9fb47
Commit message: "Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn 
ValidatesRunner tests in Flink Streaming Runner"
 > git rev-list --no-walk 0259636661998b8aabd3bf20eff8d519bcb9fb47 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5350414050609336361.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5582647355228599374.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7660244293180327252.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5782834365962723497.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3829433438117179660.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6229682268401588902.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6046

[Apache Jenkins Server]

See 

[beam] 12/12: Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner tests in Flink Streaming Runner

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0259636661998b8aabd3bf20eff8d519bcb9fb47
Merge: 51b793a 6148a6d
Author: Aljoscha Krettek 
AuthorDate: Fri Feb 23 11:59:36 2018 +0100

Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn ValidatesRunner 
tests in Flink Streaming Runner

 .../org/apache/beam/runners/core/DoFnRunners.java  |   4 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |   2 +-
 runners/flink/build.gradle |  24 ++--
 runners/flink/pom.xml  |   1 -
 .../beam/runners/flink/FlinkPipelineOptions.java   |  12 ++
 .../flink/FlinkStreamingTransformTranslators.java  |  35 +-
 .../apache/beam/runners/flink/TestFlinkRunner.java |   1 +
 .../wrappers/streaming/DoFnOperator.java   | 127 +
 .../wrappers/streaming/SplittableDoFnOperator.java |  16 +++
 .../wrappers/streaming/WindowDoFnOperator.java |  18 +++
 .../streaming/io/BoundedSourceWrapper.java |  28 +
 .../streaming/io/UnboundedSourceWrapper.java   |  56 -
 12 files changed, 234 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 02/12: Allow overriding DoFnRunners in subclasses of Flink DoFnOperator

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c40868fe2fc550ae97c7b2d9308dd8b58b20edab
Author: Aljoscha Krettek 
AuthorDate: Thu Jan 11 10:52:01 2018 +0100

Allow overriding DoFnRunners in subclasses of Flink DoFnOperator
---
 .../wrappers/streaming/DoFnOperator.java   | 69 +++---
 .../wrappers/streaming/WindowDoFnOperator.java | 18 ++
 2 files changed, 52 insertions(+), 35 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 8ccbd8f..2e7f741 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -218,6 +218,39 @@ public class DoFnOperator
 return doFn;
   }
 
+  // allow overriding this, for example SplittableDoFnOperator will not create 
a
+  // stateful DoFn runner because ProcessFn, which is used for executing a 
Splittable DoFn
+  // doesn't play by the normal DoFn rules and WindowDoFnOperator uses 
LateDataDroppingDoFnRunner
+  protected DoFnRunner createWrappingDoFnRunner(
+  DoFnRunner wrappedRunner) {
+
+if (keyCoder != null) {
+  StatefulDoFnRunner.CleanupTimer cleanupTimer =
+  new StatefulDoFnRunner.TimeInternalsCleanupTimer(
+  timerInternals, windowingStrategy);
+
+  // we don't know the window type
+  @SuppressWarnings({"unchecked", "rawtypes"})
+  Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
+
+  @SuppressWarnings({"unchecked", "rawtypes"})
+  StatefulDoFnRunner.StateCleaner stateCleaner =
+  new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+  doFn, keyedStateInternals, windowCoder);
+
+
+  return DoFnRunners.defaultStatefulDoFnRunner(
+  doFn,
+  wrappedRunner,
+  windowingStrategy,
+  cleanupTimer,
+  stateCleaner);
+
+} else {
+  return doFnRunner;
+}
+  }
+
   @Override
   public void setup(
   StreamTask containingTask,
@@ -304,41 +337,7 @@ public class DoFnOperator
 stepContext,
 windowingStrategy);
 
-if (doFn instanceof GroupAlsoByWindowViaWindowSetNewDoFn) {
-  // When the doFn is this, we know it came from WindowDoFnOperator and
-  //   InputT = KeyedWorkItem
-  //   OutputT = KV
-  //
-  // for some K, V
-
-
-  doFnRunner = DoFnRunners.lateDataDroppingRunner(
-  (DoFnRunner) doFnRunner,
-  timerInternals,
-  windowingStrategy);
-} else if (keyCoder != null) {
-  // It is a stateful DoFn
-
-  StatefulDoFnRunner.CleanupTimer cleanupTimer =
-  new StatefulDoFnRunner.TimeInternalsCleanupTimer(
-  stepContext.timerInternals(), windowingStrategy);
-
-  // we don't know the window type
-  @SuppressWarnings({"unchecked", "rawtypes"})
-  Coder windowCoder = windowingStrategy.getWindowFn().windowCoder();
-
-  @SuppressWarnings({"unchecked", "rawtypes"})
-  StatefulDoFnRunner.StateCleaner stateCleaner =
-  new StatefulDoFnRunner.StateInternalsStateCleaner<>(
-  doFn, stepContext.stateInternals(), windowCoder);
-
-  doFnRunner = DoFnRunners.defaultStatefulDoFnRunner(
-  doFn,
-  doFnRunner,
-  windowingStrategy,
-  cleanupTimer,
-  stateCleaner);
-}
+doFnRunner = createWrappingDoFnRunner(doFnRunner);
 
 if (options.getEnableMetrics()) {
   doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, 
getRuntimeContext());
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
index 7a04238..8447ade 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
@@ -23,6 +23,8 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
@@ -79,6 +81,22 @@ public class WindowDoFnOperator

Build failed in Jenkins: beam_PerformanceTests_Python #947

[Apache Jenkins Server]

See 


Changes:

[rangadi] `extractOutput()` ended up resetting underlying aggregation. This is 
due

[rangadi] review comments.

[pawel.pk.kaczmarczyk] [BEAM-3317] Use fixed system time for testing

[rangadi] Use TreeSet in place of PriorityQueue.

[rangadi] update checksWithMultipleMerges() to check for multiple merges by

[aljoscha.krettek] Make parameter of DoFnRunners.lateDataDroppingRunner() more 
specific

[aljoscha.krettek] Allow overriding DoFnRunners in subclasses of Flink 
DoFnOperator

[aljoscha.krettek] Invoke finishBundle() before teardown() in DoFnOperator

[aljoscha.krettek] [BEAM-2140] Ignore event-time timers in 
SplittableDoFnOperator

[aljoscha.krettek] [BEAM-2140] Block DoFnOperator.close() if we have pending 
timers

[aljoscha.krettek] [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in 
FlinkRunner

[aljoscha.krettek] Make ProcessFnRunner constructor public

[aljoscha.krettek] [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing 
SDF

[aljoscha.krettek] [BEAM-2140] Enable SDF tests for Flink Streaming Runner

[aljoscha.krettek] [BEAM-2140] Enable SDF tests in gradle for Flink Streaming 
Runner

[aljoscha.krettek] [BEAM-3727] Never shutdown sources in Flink Streaming 
execution mode

--
[...truncated 896 B...]
Checking out Revision 0259636661998b8aabd3bf20eff8d519bcb9fb47 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0259636661998b8aabd3bf20eff8d519bcb9fb47
Commit message: "Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn 
ValidatesRunner tests in Flink Streaming Runner"
 > git rev-list --no-walk 817cf972ddce7e1c7682c3379c2cceadf1f96317 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4440139217511866953.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8011973014920223897.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8143444178262184075.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8519347200784564090.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins522216024602313869.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins490523219888297232.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #979

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Make parameter of DoFnRunners.lateDataDroppingRunner() more 
specific

[aljoscha.krettek] Allow overriding DoFnRunners in subclasses of Flink 
DoFnOperator

[aljoscha.krettek] Invoke finishBundle() before teardown() in DoFnOperator

[aljoscha.krettek] [BEAM-2140] Ignore event-time timers in 
SplittableDoFnOperator

[aljoscha.krettek] [BEAM-2140] Block DoFnOperator.close() if we have pending 
timers

[aljoscha.krettek] [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in 
FlinkRunner

[aljoscha.krettek] Make ProcessFnRunner constructor public

[aljoscha.krettek] [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing 
SDF

[aljoscha.krettek] [BEAM-2140] Enable SDF tests for Flink Streaming Runner

[aljoscha.krettek] [BEAM-2140] Enable SDF tests in gradle for Flink Streaming 
Runner

[aljoscha.krettek] [BEAM-3727] Never shutdown sources in Flink Streaming 
execution mode

--
[...truncated 119.98 KB...]
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 484, in save_string
self.write(SHORT_BINSTRING + chr(n) + obj)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #5080

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1391

[Apache Jenkins Server]

See 


Changes:

[rangadi] `extractOutput()` ended up resetting underlying aggregation. This is 
due

[rangadi] review comments.

[pawel.pk.kaczmarczyk] [BEAM-3317] Use fixed system time for testing

[rangadi] Use TreeSet in place of PriorityQueue.

[rangadi] update checksWithMultipleMerges() to check for multiple merges by

[aljoscha.krettek] Make parameter of DoFnRunners.lateDataDroppingRunner() more 
specific

[aljoscha.krettek] Allow overriding DoFnRunners in subclasses of Flink 
DoFnOperator

[aljoscha.krettek] Invoke finishBundle() before teardown() in DoFnOperator

[aljoscha.krettek] [BEAM-2140] Ignore event-time timers in 
SplittableDoFnOperator

[aljoscha.krettek] [BEAM-2140] Block DoFnOperator.close() if we have pending 
timers

[aljoscha.krettek] [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in 
FlinkRunner

[aljoscha.krettek] Make ProcessFnRunner constructor public

[aljoscha.krettek] [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing 
SDF

[aljoscha.krettek] [BEAM-2140] Enable SDF tests for Flink Streaming Runner

[aljoscha.krettek] [BEAM-2140] Enable SDF tests in gradle for Flink Streaming 
Runner

[aljoscha.krettek] [BEAM-3727] Never shutdown sources in Flink Streaming 
execution mode

--
[...truncated 93.88 KB...]
'apache-beam-testing:bqjob_r28745172585fab78_0161c29a6501_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-23 12:19:15,709 ff6131fe MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 12:19:30,946 ff6131fe MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 12:19:33,432 ff6131fe MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.48s,  CPU:0.25s,  MaxMemory:25224kb 
STDOUT: Upload complete.
Waiting on bqjob_r3bb8edb45ef22c99_0161c29aa90f_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r3bb8edb45ef22c99_0161c29aa90f_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3bb8edb45ef22c99_0161c29aa90f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-23 12:19:33,433 ff6131fe MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 12:19:53,483 ff6131fe MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 12:19:55,585 ff6131fe MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.09s,  CPU:0.23s,  MaxMemory:25416kb 
STDOUT: Upload complete.
Waiting on bqjob_r32982b4a29d9d7e3_0161c29b011c_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r32982b4a29d9d7e3_0161c29b011c_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r32982b4a29d9d7e3_0161c29b011c_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for 

[beam] 09/12: [BEAM-2140] Enable SDF tests for Flink Streaming Runner

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c920a39b9781ac604f55059e597e327643d58205
Author: Aljoscha Krettek 
AuthorDate: Fri Jan 5 14:20:19 2018 +0100

[BEAM-2140] Enable SDF tests for Flink Streaming Runner
---
 runners/flink/pom.xml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 46917e9..cd4d8c3 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -94,7 +94,6 @@
 org.apache.beam.sdk.testing.LargeKeys$Above100MB,
 org.apache.beam.sdk.testing.UsesCommittedMetrics,
 org.apache.beam.sdk.testing.UsesTestStream,
-org.apache.beam.sdk.testing.UsesSplittableParDo
   
   none
   true

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 05/12: [BEAM-2140] Block DoFnOperator.close() if we have pending timers

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3eecb381f3fb412df6844cfc13b12bf265253926
Author: Aljoscha Krettek 
AuthorDate: Fri Jan 5 14:17:49 2018 +0100

[BEAM-2140] Block DoFnOperator.close() if we have pending timers

It can happen that the input operation finishes while we still have
pending processing-time timers, for example from processing a Splittable
DoFn. This change makes sure that we block as long as we have pending
timers.

This change also makes sure that we forward a +Inf watermark in close().
We have to do this because it can happen that we get a +Inf watermark on
input while we still have active watermark holds (which will get
resolved when all pending timers are gone). With this change we make
sure to send a +Inf watermark downstream once everything is resolved.
---
 .../wrappers/streaming/DoFnOperator.java   | 43 +-
 1 file changed, 33 insertions(+), 10 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index dd2f9c4..37f56f5 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -371,18 +371,41 @@ public class DoFnOperator
 
   @Override
   public void close() throws Exception {
-super.close();
-
-// sanity check: these should have been flushed out by +Inf watermarks
-if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
-  BagState pushedBack =
-  nonKeyedStateInternals.state(StateNamespaces.global(), 
pushedBackTag);
+try {
 
-  Iterable pushedBackContents = pushedBack.read();
-  if (pushedBackContents != null && 
!Iterables.isEmpty(pushedBackContents)) {
-String pushedBackString = Joiner.on(",").join(pushedBackContents);
+  // This is our last change to block shutdown of this operator while
+  // there are still remaining processing-time timers. Flink will ignore 
pending
+  // processing-time timers when upstream operators have shut down and 
will also
+  // shut down this operator with pending processing-time timers.
+  while (this.numProcessingTimeTimers() > 0) {
+getContainingTask().getCheckpointLock().wait(100);
+  }
+  if (this.numProcessingTimeTimers() > 0) {
 throw new RuntimeException(
-"Leftover pushed-back data: " + pushedBackString + ". This 
indicates a bug.");
+"There are still processing-time timers left, this indicates a 
bug");
+  }
+
+  // make sure we send a +Inf watermark downstream. It can happen that we 
receive +Inf
+  // in processWatermark*() but have holds, so we have to re-evaluate here.
+  processWatermark(new Watermark(Long.MAX_VALUE));
+  if (currentOutputWatermark < Long.MAX_VALUE) {
+throw new RuntimeException("There are still watermark holds. Watermark 
held at "
++ keyedStateInternals.watermarkHold().getMillis() + ".");
+  }
+} finally {
+  super.close();
+
+  // sanity check: these should have been flushed out by +Inf watermarks
+  if (!sideInputs.isEmpty() && nonKeyedStateInternals != null) {
+BagState pushedBack =
+nonKeyedStateInternals.state(StateNamespaces.global(), 
pushedBackTag);
+
+Iterable pushedBackContents = pushedBack.read();
+if (pushedBackContents != null && 
!Iterables.isEmpty(pushedBackContents)) {
+  String pushedBackString = Joiner.on(",").join(pushedBackContents);
+  throw new RuntimeException(
+  "Leftover pushed-back data: " + pushedBackString + ". This 
indicates a bug.");
+}
   }
 }
   }

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_Verify #4296

[Apache Jenkins Server]

See 


Changes:

[pawel.pk.kaczmarczyk] [BEAM-3317] Use fixed system time for testing

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 

[beam] 11/12: [BEAM-3727] Never shutdown sources in Flink Streaming execution mode

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6148a6d063a0503ee435ab5084fcba3fb864b26f
Author: Aljoscha Krettek 
AuthorDate: Thu Feb 22 14:26:16 2018 +0100

[BEAM-3727] Never shutdown sources in Flink Streaming execution mode

This adds an option that controls whether to shutdown sources or not in
case of reaching the +Inf watermark.

The reason for this is https://issues.apache.org/jira/browse/FLINK-2491,
which causes checkpointing to stop once some source is shut down.
---
 .../beam/runners/flink/FlinkPipelineOptions.java   | 12 +
 .../apache/beam/runners/flink/TestFlinkRunner.java |  1 +
 .../streaming/io/BoundedSourceWrapper.java | 28 +++
 .../streaming/io/UnboundedSourceWrapper.java   | 56 --
 4 files changed, 71 insertions(+), 26 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index 01f7847..b2cbefb 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -137,4 +137,16 @@ public interface FlinkPipelineOptions
   Long getMaxBundleTimeMills();
   void setMaxBundleTimeMills(Long time);
 
+  /**
+   * Whether to shutdown sources when their watermark reaches {@code +Inf}. 
For production use
+   * cases you want this to be disabled because Flink will currently (versions 
{@literal <=} 1.5)
+   * stop doing checkpoints when any operator (which includes sources) is 
finished.
+   *
+   * Please see https://issues.apache.org/jira/browse/FLINK-2491;>FLINK-2491 for
+   * progress on this issue.
+   */
+  @Description("If set, shutdown sources when their watermark reaches +Inf.")
+  @Default.Boolean(false)
+  Boolean isShutdownSourcesOnFinalWatermark();
+  void setShutdownSourcesOnFinalWatermark(Boolean shutdownOnFinalWatermark);
 }
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
index 01b67e5..47d4494 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
@@ -36,6 +36,7 @@ public class TestFlinkRunner extends 
PipelineRunner {
   private TestFlinkRunner(FlinkPipelineOptions options) {
 // We use [auto] for testing since this will make it pick up the Testing 
ExecutionEnvironment
 options.setFlinkMaster("[auto]");
+options.setShutdownSourcesOnFinalWatermark(true);
 this.delegate = FlinkRunner.fromOptions(options);
   }
 
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
index 5ddc46f..6db5426 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
@@ -21,6 +21,7 @@ import com.google.common.annotations.VisibleForTesting;
 import java.util.ArrayList;
 import java.util.List;
 import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.flink.FlinkPipelineOptions;
 import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
 import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
 import org.apache.beam.sdk.io.BoundedSource;
@@ -180,6 +181,33 @@ public class BoundedSourceWrapper
 
 // emit final Long.MAX_VALUE watermark, just to be sure
 ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
+
+FlinkPipelineOptions options = 
serializedOptions.get().as(FlinkPipelineOptions.class);
+if (!options.isShutdownSourcesOnFinalWatermark()) {
+  // do nothing, but still look busy ...
+  // we can't return here since Flink requires that all operators stay up,
+  // otherwise checkpointing would not work correctly anymore
+  //
+  // See https://issues.apache.org/jira/browse/FLINK-2491 for progress on 
this issue
+
+  // wait until this is canceled
+  final Object waitLock = new Object();
+  while (isRunning) {
+try {
+  // Flink will interrupt us at some point
+  //noinspection SynchronizationOnLocalVariableOrMethodParameter
+  synchronized (waitLock) {
+// don't wait indefinitely, in case something goes horribly wrong
+waitLock.wait(1000);
+  }
+} catch (InterruptedException 

[beam] 06/12: [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d15979f2cf0d6bc47d049a6ea157d9d7b2b97848
Author: Aljoscha Krettek 
AuthorDate: Wed Jan 10 06:48:49 2018 +0100

[BEAM-2140] Don't use StatefulDoFnRunner when running SDF in FlinkRunner
---
 .../translation/wrappers/streaming/SplittableDoFnOperator.java | 10 ++
 1 file changed, 10 insertions(+)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 44be5f3..e088b07 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -26,6 +26,7 @@ import java.util.Map;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
 import 
org.apache.beam.runners.core.OutputAndTimeBoundedSplittableProcessElementInvoker;
@@ -88,6 +89,15 @@ public class SplittableDoFnOperator<
   }
 
   @Override
+  protected DoFnRunner<
+  KeyedWorkItem>, OutputT> 
createWrappingDoFnRunner(
+  DoFnRunner>, OutputT> 
wrappedRunner) {
+// don't wrap in anything because we don't need state cleanup because 
ProcessFn does
+// all that
+return wrappedRunner;
+  }
+
+  @Override
   public void open() throws Exception {
 super.open();
 

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 08/12: [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 74ac703bbfa40d16e7d1115912768c7a63598d52
Author: Aljoscha Krettek 
AuthorDate: Thu Feb 8 13:02:26 2018 +0100

[BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing SDF

For this to work, we need to also change how we wrap values in
KeyedWorkItems, because ProcessFnRunner expects them to be in the
GlobalWindow.
---
 .../flink/FlinkStreamingTransformTranslators.java  | 35 ++
 .../wrappers/streaming/DoFnOperator.java   | 11 +--
 2 files changed, 39 insertions(+), 7 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index d39b5c1..f5dc3ce 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -1017,10 +1017,8 @@ class FlinkStreamingTransformTranslators {
 
 
   WindowedValue.
-  FullWindowedValueCoder> 
windowedWorkItemCoder =
-  WindowedValue.getFullCoder(
-  workItemCoder,
-  input.getWindowingStrategy().getWindowFn().windowCoder());
+  ValueOnlyWindowedValueCoder> 
windowedWorkItemCoder =
+  WindowedValue.getValueOnlyCoder(workItemCoder);
 
   CoderTypeInformation>> 
workItemTypeInfo =
   new CoderTypeInformation<>(windowedWorkItemCoder);
@@ -1029,7 +1027,7 @@ class FlinkStreamingTransformTranslators {
 
   DataStream>> 
workItemStream =
   inputDataStream
-  .flatMap(new ToKeyedWorkItem<>())
+  .flatMap(new ToKeyedWorkItemInGlobalWindow<>())
   .returns(workItemTypeInfo)
   .name("ToKeyedWorkItem");
 
@@ -1041,6 +1039,33 @@ class FlinkStreamingTransformTranslators {
 }
   }
 
+  private static class ToKeyedWorkItemInGlobalWindow
+  extends RichFlatMapFunction<
+  WindowedValue>,
+  WindowedValue>> {
+
+@Override
+public void flatMap(
+WindowedValue> inWithMultipleWindows,
+Collector>> out) 
throws Exception {
+
+  // we need to wrap each one work item per window for now
+  // since otherwise the PushbackSideInputRunner will not correctly
+  // determine whether side inputs are ready
+  //
+  // this is tracked as https://issues.apache.org/jira/browse/BEAM-1850
+  for (WindowedValue> in : 
inWithMultipleWindows.explodeWindows()) {
+SingletonKeyedWorkItem workItem =
+new SingletonKeyedWorkItem<>(
+in.getValue().getKey(),
+in.withValue(in.getValue().getValue()));
+
+out.collect(WindowedValue.valueInGlobalWindow(workItem));
+  }
+}
+  }
+
+
   private static class FlattenPCollectionTranslator
   extends FlinkStreamingPipelineTranslator.StreamTransformTranslator<
 PTransform> {
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 37f56f5..f9b4ee3 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -39,10 +39,12 @@ import javax.annotation.Nullable;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.core.NullSideInputReader;
+import org.apache.beam.runners.core.ProcessFnRunner;
 import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
 import org.apache.beam.runners.core.SideInputHandler;
 import org.apache.beam.runners.core.SideInputReader;
 import org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner;
+import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
 import org.apache.beam.runners.core.StateInternals;
 import org.apache.beam.runners.core.StateNamespace;
 import org.apache.beam.runners.core.StateNamespaces;
@@ -352,8 +354,13 @@ public class DoFnOperator
 .scheduleAtFixedRate(
 timestamp -> checkInvokeFinishBundleByTime(), 
bundleCheckPeriod, bundleCheckPeriod);
 
-pushbackDoFnRunner =
-   

[jira] [Updated] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-2140:
---
Fix Version/s: 2.4.0

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 04/12: [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2e0b192efbae72fe5c0ef61abfc7b4d1b3bb75f5
Author: Aljoscha Krettek 
AuthorDate: Fri Jan 5 14:16:27 2018 +0100

[BEAM-2140] Ignore event-time timers in SplittableDoFnOperator

SplittableDoFnOperator is only used for executing ProcessFn, which does
not use event-time timers. However, StatefulDoFnRunner does use
event-time timers for state cleanup so this change makes sure that they
don't end up being forwarded to the ProcessFn.
---
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java  | 1 -
 .../translation/wrappers/streaming/SplittableDoFnOperator.java  | 6 ++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 688a7cb..dd2f9c4 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -38,7 +38,6 @@ import java.util.concurrent.ScheduledFuture;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
 import org.apache.beam.runners.core.NullSideInputReader;
 import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
 import org.apache.beam.runners.core.SideInputHandler;
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index 1a418a0..44be5f3 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -37,6 +37,7 @@ import org.apache.beam.runners.core.TimerInternals;
 import org.apache.beam.runners.core.TimerInternalsFactory;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.state.TimeDomain;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -142,6 +143,11 @@ public class SplittableDoFnOperator<
 
   @Override
   public void fireTimer(InternalTimer timer) {
+if (timer.getNamespace().getDomain().equals(TimeDomain.EVENT_TIME)) {
+  // ignore this, it can only be a state cleanup timers from 
StatefulDoFnRunner and ProcessFn
+  // does its own state cleanup and should never set event-time timers.
+  return;
+}
 doFnRunner.processElement(
 WindowedValue.valueInGlobalWindow(
 KeyedWorkItems.timersWorkItem(

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 10/12: [BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6ba3fc1ff2798764756a7188980facc31eb1ac4c
Author: Aljoscha Krettek 
AuthorDate: Thu Feb 8 13:00:34 2018 +0100

[BEAM-2140] Enable SDF tests in gradle for Flink Streaming Runner
---
 runners/flink/build.gradle | 24 +---
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index dc77bcf..362288e 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -97,13 +97,23 @@ def createValidatesRunnerTask(Map m) {
 classpath = configurations.validatesRunner
 testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
 maxParallelForks 4
-useJUnit {
-  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
-  excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
-  excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
-  excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
-  excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
-  excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+if (config.streaming) {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+  }
+} else {
+  useJUnit {
+includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
+excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+  }
 }
   }
 }

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 07/12: Make ProcessFnRunner constructor public

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4ec4644952c05167f52ae0937e043140d26059ae
Author: Aljoscha Krettek 
AuthorDate: Thu Feb 8 13:01:22 2018 +0100

Make ProcessFnRunner constructor public

We need this to be able to instantiate with the constructor in the Flink
Runner.
---
 .../src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
index 88275d6..e4dfd13 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
@@ -43,7 +43,7 @@ public class ProcessFnRunner
   private final Collection views;
   private final ReadyCheckingSideInputReader sideInputReader;
 
-  ProcessFnRunner(
+  public ProcessFnRunner(
   DoFnRunner>, OutputT> 
underlying,
   Collection views,
   ReadyCheckingSideInputReader sideInputReader) {

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 03/12: Invoke finishBundle() before teardown() in DoFnOperator

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4029e23f3075365eddcdad5187b4fc8a6e590989
Author: Aljoscha Krettek 
AuthorDate: Fri Jan 5 14:22:40 2018 +0100

Invoke finishBundle() before teardown() in DoFnOperator
---
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java | 3 +++
 1 file changed, 3 insertions(+)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 2e7f741..688a7cb 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -363,6 +363,9 @@ public class DoFnOperator
   super.dispose();
   checkFinishBundleTimer.cancel(true);
 } finally {
+  if (bundleStarted) {
+invokeFinishBundle();
+  }
   doFnInvoker.invokeTeardown();
 }
   }

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 01/12: Make parameter of DoFnRunners.lateDataDroppingRunner() more specific

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 50c6d9316c8c5089065dd291b097e06b0c80980e
Author: Aljoscha Krettek 
AuthorDate: Thu Jan 11 10:56:49 2018 +0100

Make parameter of DoFnRunners.lateDataDroppingRunner() more specific
---
 .../src/main/java/org/apache/beam/runners/core/DoFnRunners.java   | 4 ++--
 .../runners/flink/translation/wrappers/streaming/DoFnOperator.java| 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 9d3e25d..80c830a 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -80,12 +80,12 @@ public class DoFnRunners {
   public static 
   DoFnRunner, KV> 
lateDataDroppingRunner(
   DoFnRunner, KV> wrappedRunner,
-  StepContext stepContext,
+  TimerInternals timerInternals,
   WindowingStrategy windowingStrategy) {
 return new LateDataDroppingDoFnRunner<>(
 wrappedRunner,
 windowingStrategy,
-stepContext.timerInternals());
+timerInternals);
   }
 
   /**
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 41a35ce..8ccbd8f 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -314,7 +314,7 @@ public class DoFnOperator
 
   doFnRunner = DoFnRunners.lateDataDroppingRunner(
   (DoFnRunner) doFnRunner,
-  stepContext,
+  timerInternals,
   windowingStrategy);
 } else if (keyCoder != null) {
   // It is a stateful DoFn

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[jira] [Closed] (BEAM-2140) Fix SplittableDoFn ValidatesRunner tests in FlinkRunner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2140.
--
Resolution: Fixed

> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> ---
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled 
> the tests to unblock the open PR for BEAM-1763.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (51b793a -> 0259636)

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 51b793a  Merge pull request #4723: [BEAM-3317] Use fixed system time 
for KinesisReaderTest
 new 50c6d93  Make parameter of DoFnRunners.lateDataDroppingRunner() more 
specific
 new c40868f  Allow overriding DoFnRunners in subclasses of Flink 
DoFnOperator
 new 4029e23f Invoke finishBundle() before teardown() in DoFnOperator
 new 2e0b192  [BEAM-2140] Ignore event-time timers in SplittableDoFnOperator
 new 3eecb38  [BEAM-2140] Block DoFnOperator.close() if we have pending 
timers
 new d15979f  [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in 
FlinkRunner
 new 4ec4644  Make ProcessFnRunner constructor public
 new 74ac703  [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing 
SDF
 new c920a39  [BEAM-2140] Enable SDF tests for Flink Streaming Runner
 new 6ba3fc1  [BEAM-2140] Enable SDF tests in gradle for Flink Streaming 
Runner
 new 6148a6d  [BEAM-3727] Never shutdown sources in Flink Streaming 
execution mode
 new 0259636  Merge pull request #4639: [BEAM-2140] Fix SplittableDoFn 
ValidatesRunner tests in Flink Streaming Runner

The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/core/DoFnRunners.java  |   4 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |   2 +-
 runners/flink/build.gradle |  24 ++--
 runners/flink/pom.xml  |   1 -
 .../beam/runners/flink/FlinkPipelineOptions.java   |  12 ++
 .../flink/FlinkStreamingTransformTranslators.java  |  35 +-
 .../apache/beam/runners/flink/TestFlinkRunner.java |   1 +
 .../wrappers/streaming/DoFnOperator.java   | 127 +
 .../wrappers/streaming/SplittableDoFnOperator.java |  16 +++
 .../wrappers/streaming/WindowDoFnOperator.java |  18 +++
 .../streaming/io/BoundedSourceWrapper.java |  28 +
 .../streaming/io/UnboundedSourceWrapper.java   |  56 -
 12 files changed, 234 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #5079

[Apache Jenkins Server]

See 


--
[...truncated 157.33 KB...]
2018-02-23T11:43:21.166 [INFO] Tests run: 243, Failures: 0, Errors: 0, Skipped: 0
2018-02-23T11:43:21.166 [INFO] 
[JENKINS] Recording test results
2018-02-23T11:43:21.678 [INFO] 
2018-02-23T11:43:21.678 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-core-java ---
2018-02-23T11:43:21.786 [INFO] 
2018-02-23T11:43:21.786 [INFO] --- jacoco-maven-plugin:0.7.9:report (report) @ 
beam-runners-core-java ---
2018-02-23T11:43:21.787 [INFO] Loading execution data file 

2018-02-23T11:43:21.826 [INFO] Analyzed bundle 'Apache Beam :: Runners :: Core 
Java' with 173 classes
2018-02-23T11:43:22.324 [INFO] 
2018-02-23T11:43:22.324 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-runners-core-java ---
2018-02-23T11:43:22.347 [INFO] Building jar: 

2018-02-23T11:43:22.480 [INFO] 
2018-02-23T11:43:22.481 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-core-java ---
2018-02-23T11:43:22.482 [INFO] Skipping because packaging 'jar' is not pom.
2018-02-23T11:43:22.695 [INFO] 
2018-02-23T11:43:22.695 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-runners-core-java ---
2018-02-23T11:43:22.707 [INFO] Building jar: 

2018-02-23T11:43:22.833 [INFO] 
2018-02-23T11:43:22.833 [INFO] --- maven-shade-plugin:3.1.0:shade 
(bundle-and-repackage) @ beam-runners-core-java ---
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.beam:beam-model-pipeline:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.9 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.25 from 
the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding net.bytebuddy:byte-buddy:jar:1.7.10 
from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.commons:commons-compress:jar:1.14 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.6 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.4.0-SNAPSHOT from the 
shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
org.apache.beam:beam-model-job-management:jar:2.4.0-SNAPSHOT from the shaded 
jar.
2018-02-23T11:43:22.836 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 
from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the 
shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from 
the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.3.0 from the shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the 
shaded jar.
2018-02-23T11:43:22.836 [INFO] Excluding 
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1 from the shaded 
jar.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #976

[Apache Jenkins Server]

See 


Changes:

[rangadi] `extractOutput()` ended up resetting underlying aggregation. This is 
due

[rangadi] review comments.

[rangadi] Use TreeSet in place of PriorityQueue.

[rangadi] update checksWithMultipleMerges() to check for multiple merges by

--
[...truncated 114.45 KB...]
errors.extend(self.options.view_as(cls).validate(self))
  File 
"
 line 379, in validate
def validate(self, validator):
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 

[beam] 01/01: Merge pull request #4723: [BEAM-3317] Use fixed system time for KinesisReaderTest

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 51b793aea70c333011123c3ffd58ea6f26c61b51
Merge: 0be4c54 d0f272f
Author: Ismaël Mejía 
AuthorDate: Fri Feb 23 10:21:36 2018 +0100

Merge pull request #4723: [BEAM-3317] Use fixed system time for 
KinesisReaderTest

 .../beam/sdk/io/kinesis/KinesisReaderTest.java | 65 --
 1 file changed, 37 insertions(+), 28 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (0be4c54 -> 51b793a)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0be4c54  Merge pull request #4688: [BEAM-3705] ApproximateUnique 
resets its accumulator with each firing.
 add d0f272f  [BEAM-3317] Use fixed system time for testing
 new 51b793a  Merge pull request #4723: [BEAM-3317] Use fixed system time 
for KinesisReaderTest

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/kinesis/KinesisReaderTest.java | 65 --
 1 file changed, 37 insertions(+), 28 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_Verify #4295

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #977

[Apache Jenkins Server]

See 


--
[...truncated 118.43 KB...]
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1057, in save_cell
pickler.save_reduce(_create_cell, (obj.cell_contents,), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 807, in save_code
pickler.save_reduce(CodeType, args, obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 448, in save_int
self.write("%c%c%c" % (BININT2, obj&0xff, obj>>8))
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 

[jira] [Resolved] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3317.

   Resolution: Fixed
Fix Version/s: 2.4.0

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
> Attachments: kinesisflakytests2.txt
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #978

[Apache Jenkins Server]

See 


Changes:

[pawel.pk.kaczmarczyk] [BEAM-3317] Use fixed system time for testing

--
[...truncated 115.45 KB...]
  File 
"
 line 555, in from_runner_api
context)
  File 
"
 line 846, in from_runner_api_parameter
return Read(SourceBase.from_runner_api(parameter.source, context))
  File 
"
 line 110, in from_runner_api
context)
  File 
"
 line 86, in 
lambda proto, unused_context: pickler.loads(proto.value))
  File 
"
 line 221, in loads
return dill.loads(s)
  File 
"
 line 277, in loads
return load(file)
  File 
"
 line 266, in load
obj = pik.load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
value = func(*args)
  File 
"
 line 527, in _create_type
return typeobj(*args)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 597, in from_runner_api
context.transforms.get_by_id(root_transform_id)]
  File 
"
 line 69, in get_by_id
self._id_to_proto[id], self._pipeline_context)
  File 
"
 line 842, in from_runner_api
part = context.transforms.get_by_id(transform_id)
  File 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4297

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Make parameter of DoFnRunners.lateDataDroppingRunner() more 
specific

[aljoscha.krettek] Allow overriding DoFnRunners in subclasses of Flink 
DoFnOperator

[aljoscha.krettek] Invoke finishBundle() before teardown() in DoFnOperator

[aljoscha.krettek] [BEAM-2140] Ignore event-time timers in 
SplittableDoFnOperator

[aljoscha.krettek] [BEAM-2140] Block DoFnOperator.close() if we have pending 
timers

[aljoscha.krettek] [BEAM-2140] Don't use StatefulDoFnRunner when running SDF in 
FlinkRunner

[aljoscha.krettek] Make ProcessFnRunner constructor public

[aljoscha.krettek] [BEAM-2140] Use ProcessFnRunner in DoFnRunner for executing 
SDF

[aljoscha.krettek] [BEAM-2140] Enable SDF tests for Flink Streaming Runner

[aljoscha.krettek] [BEAM-2140] Enable SDF tests in gradle for Flink Streaming 
Runner

[aljoscha.krettek] [BEAM-3727] Never shutdown sources in Flink Streaming 
execution mode

--
[...truncated 1.06 MB...]
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "pair_with_one.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "ref_AppliedPTransform_pair_with_one_5", 
"user_name": "pair_with_one"
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
 
  "component_encodings": []
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "group_and_sum/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "group_and_sum/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": 

[jira] [Closed] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs (mockito-all/hamcrest-all)

[PandaMonkey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

PandaMonkey closed BEAM-3690.
-
Resolution: Fixed

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs (mockito-all/hamcrest-all)
> 
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: beam_conflicts.txt
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)