[jira] [Assigned] (BEAM-3182) [Nexmark][SQL] Implement supported queries

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud reassigned BEAM-3182:


Assignee: Andrew Pilloud  (was: Anton Kedin)

> [Nexmark][SQL] Implement supported queries
> --
>
> Key: BEAM-3182
> URL: https://issues.apache.org/jira/browse/BEAM-3182
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Andrew Pilloud
>Priority: Major
>
> Implement all queries which can be run with current SQL features and Nexmark 
> infrastructure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4466: [BEAM-3476] [SQL] covariance aggregation functions

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d13c1feeaeeec4af079f2da523849c218cc50882
Merge: 2aca6ce 94fd067
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 11:10:36 2018 -0800

Merge pull request #4466: [BEAM-3476] [SQL] covariance aggregation functions

 sdks/java/extensions/sql/pom.xml   |   3 +
 .../impl/transform/BeamAggregationTransforms.java  |  90 ++--
 .../impl/transform/agg/CovarianceAccumulator.java  | 151 +
 .../sql/impl/transform/agg/CovarianceFn.java   | 120 
 .../agg => utils}/BigDecimalConverter.java |   2 +-
 .../sql/BeamSqlDslAggregationCovarianceTest.java   | 116 
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  50 ---
 .../agg => utils}/BigDecimalConverterTest.java |   2 +-
 8 files changed, 470 insertions(+), 64 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (2aca6ce -> d13c1fe)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2aca6ce  BEAM-3618 - Remove extraneous "return" statement
 add 94fd067  Covariance Functions
 new d13c1fe  Merge pull request #4466: [BEAM-3476] [SQL] covariance 
aggregation functions

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sql/pom.xml   |   3 +
 .../impl/transform/BeamAggregationTransforms.java  |  90 ++--
 .../impl/transform/agg/CovarianceAccumulator.java  | 151 +
 .../sql/impl/transform/agg/CovarianceFn.java   | 120 
 .../agg => utils}/BigDecimalConverter.java |   2 +-
 .../sql/BeamSqlDslAggregationCovarianceTest.java   | 116 
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  50 ---
 .../agg => utils}/BigDecimalConverterTest.java |   2 +-
 8 files changed, 470 insertions(+), 64 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceAccumulator.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
 rename 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{transform/agg
 => utils}/BigDecimalConverter.java (97%)
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
 rename 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/{transform/agg
 => utils}/BigDecimalConverterTest.java (96%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Commented] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352800#comment-16352800
 ] 

Jean-Baptiste Onofré commented on BEAM-3617:


I just ran nexmark ({{mvn exec:java 
-Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Dexec.args="--suite=SMOKE 
--streaming=true --manageResources=false --monitorJobs=true"}}):

* On Beam 2.2.0

{code}
Performance:
  Conf  Runtime(sec)(Baseline)  Events(/sec)(Baseline)   Results
(Baseline)
     8.1 12335.0  10
  0001   4.0 25106.7   92000
  0002   2.2 46082.9 351
  0003   4.6 21767.5 444
  0004   1.4  7092.2  40
  0005   5.9 16869.1  12
  0006   1.0 10111.2 401
  0007 153.2   653.0   1
  0008   3.3 30413.66000
  0009   0.9 1.1 298
  0010   5.2 19241.9   1
  0011   4.3 23153.51919
  0012   3.3 30712.51919
==
{code}

* On Beam 2.3.0 (release branch):

{code}
Performance:
  Conf  Runtime(sec)(Baseline)  Events(/sec)(Baseline)   Results
(Baseline)
    10.5  9554.7  10
  0001   7.2 13848.5   92000
  0002   3.9 25654.2 351
  0003   5.9 17059.0 444
  0004   1.7  6013.2  40
  0005   8.4 11899.1  12
  0006   1.4  7077.1 401
  00071019.098.1   1
  0008   5.0 19888.66000
  0009   1.3  7905.1 298
  0010   6.2 16186.5   1
  0011   9.0 11088.91919
  0012   6.4 15535.21919
==
{code}

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352808#comment-16352808
 ] 

Kenneth Knowles commented on BEAM-3414:
---

Those instructions are not quite right. The JStorm runner is a work in progress 
on a development branch, and not released with the nightly build either. You 
will need to check out that branch and install it locally. I should say that 
the JStorm runner has not been updated in a very long time.

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Aljoscha Krettek
>Priority: Major
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Match Maven Java SDK harness jar name format

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit cb383921f8c45cf88c123f893b881ae44ccc75bf
Merge: d13c1fe 17e41c3
Author: Lukasz Cwik 
AuthorDate: Mon Feb 5 11:36:44 2018 -0800

Match Maven Java SDK harness jar name format

 sdks/java/container/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (d13c1fe -> cb38392)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d13c1fe  Merge pull request #4466: [BEAM-3476] [SQL] covariance 
aggregation functions
 add 17e41c3  Fix gradle java sdk image build
 new cb38392  Match Maven Java SDK harness jar name format

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/container/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Commented] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352829#comment-16352829
 ] 

Kenneth Knowles commented on BEAM-3617:
---

Good idea to check this. I added it to the spreadsheet and signed you up JB. I 
only added direct runner since other runners should opt in.

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4590 [BEAM-3183] Allow a callable as input to runner.run().

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 79041126519ad64c4f357fb8ddcc7ae67a7b2203
Merge: cb38392 4d06e82
Author: Robert Bradshaw 
AuthorDate: Mon Feb 5 12:15:35 2018 -0800

Merge pull request #4590 [BEAM-3183] Allow a callable as input to 
runner.run().

 sdks/python/apache_beam/runners/runner.py  |  9 +++--
 sdks/python/apache_beam/runners/runner_test.py | 14 ++
 2 files changed, 21 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (cb38392 -> 7904112)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cb38392  Match Maven Java SDK harness jar name format
 add 4d06e82  [BEAM-3183] Allow a callable as input to runner.run().
 new 7904112  Merge pull request #4590 [BEAM-3183] Allow a callable as 
input to runner.run().

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/runner.py  |  9 +++--
 sdks/python/apache_beam/runners/runner_test.py | 14 ++
 2 files changed, 21 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4887

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3601) Switch to Java 8 futures

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3601:
--
Labels: portability  (was: )

> Switch to Java 8 futures
> 
>
> Key: BEAM-3601
> URL: https://issues.apache.org/jira/browse/BEAM-3601
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3543) Fn API metrics in Java SDK Harness

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3543:
--
Labels: portability  (was: )

> Fn API metrics in Java SDK Harness
> --
>
> Key: BEAM-3543
> URL: https://issues.apache.org/jira/browse/BEAM-3543
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3608) Pre-shade Guava for things we want to keep using

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3608:
--
Labels: portability  (was: )

> Pre-shade Guava for things we want to keep using
> 
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: portability
>
> Instead of shading as part of our build, we can shade before build so that it 
> is apparent when reading code, and in IDEs, that a particular class resides 
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs 
> unless they are annotated {{@Internal}}, and it makes sharing between our own 
> modules never get broken by shading again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3543) Fn API metrics in Java SDK Harness

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3543:
-

Assignee: (was: Kenneth Knowles)

> Fn API metrics in Java SDK Harness
> --
>
> Key: BEAM-3543
> URL: https://issues.apache.org/jira/browse/BEAM-3543
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3320) DataflowRunner for python streaming uses portable containers

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3320:
-

Assignee: Alan Myrvold  (was: Ahmet Altay)

> DataflowRunner for python streaming uses portable containers
> 
>
> Key: BEAM-3320
> URL: https://issues.apache.org/jira/browse/BEAM-3320
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: portability
>
> Python streaming uses Dataflow built containers to run on Dataflow. It should 
> start using the portable containers to take advantage of the changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (7904112 -> b812fa0)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7904112  Merge pull request #4590 [BEAM-3183] Allow a callable as 
input to runner.run().
 add 2de70ff  Moving User metrics to be in the PTransform proto for Fn API.
 new b812fa0  Merge pull request #4544: [BEAM-3563] Adding User metric 
proto to PTransform metrics proto.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 model/fn-execution/src/main/proto/beam_fn_api.proto |  9 +++--
 sdks/python/apache_beam/metrics/execution.py| 15 ++-
 sdks/python/apache_beam/metrics/metricbase.py   | 10 ++
 .../runners/portability/fn_api_runner.py| 21 -
 .../apache_beam/runners/worker/bundle_processor.py  |  5 +
 .../python/apache_beam/runners/worker/operations.py |  3 ++-
 6 files changed, 30 insertions(+), 33 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4544: [BEAM-3563] Adding User metric proto to PTransform metrics proto.

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b812fa061f0ca09f343ca2111444e585520594b5
Merge: 7904112 2de70ff
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 14:22:53 2018 -0800

Merge pull request #4544: [BEAM-3563] Adding User metric proto to 
PTransform metrics proto.

 model/fn-execution/src/main/proto/beam_fn_api.proto |  9 +++--
 sdks/python/apache_beam/metrics/execution.py| 15 ++-
 sdks/python/apache_beam/metrics/metricbase.py   | 10 ++
 .../runners/portability/fn_api_runner.py| 21 -
 .../apache_beam/runners/worker/bundle_processor.py  |  5 +
 .../python/apache_beam/runners/worker/operations.py |  3 ++-
 6 files changed, 30 insertions(+), 33 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (b812fa0 -> 8edc18e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b812fa0  Merge pull request #4544: [BEAM-3563] Adding User metric 
proto to PTransform metrics proto.
 add b5dd1c1  Add MoreFutures utility
 add e71baf0  Switch runners/java-fn-execution from Guava futures to Java 8 
futures
 add c757854  Switch DataflowRunner from Guava futures to Java 8 futures
 add 6bb6530  Switch gcp-core from Guava futures to Java 8 futures
 add f64a633  Switch runners/core-construction-java from Guava futures to 
Java 8 futures
 add 0f19c32  Switch AWS IO from Guava futures to Java 8 futures
 add b1fd123  Switch BigTableIO from Guava futures to Java 8 futures
 new 8edc18e  Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/ArtifactServiceStager.java   |  32 +--
 .../beam/runners/dataflow/util/PackageUtil.java|  60 ++---
 .../fnexecution/control/FnApiControlClient.java|  35 ++-
 .../fnexecution/control/SdkHarnessClient.java  |  34 ++-
 .../control/FnApiControlClientPoolServiceTest.java |   9 +-
 .../control/FnApiControlClientTest.java|  30 +--
 .../fnexecution/control/SdkHarnessClientTest.java  |  17 +-
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 251 +
 .../apache/beam/sdk/util/ThrowingRunnable.java}|   8 +-
 .../util/{Weighted.java => ThrowingSupplier.java}  |  14 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java |  21 +-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  10 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   |  32 +--
 .../beam/sdk/io/gcp/bigtable/BigtableService.java  |   4 +-
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  24 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   |  14 +-
 .../io/gcp/bigtable/BigtableServiceImplTest.java   |   9 +-
 17 files changed, 427 insertions(+), 177 deletions(-)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java
 copy 
sdks/java/{extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/package-info.java
 => core/src/main/java/org/apache/beam/sdk/util/ThrowingRunnable.java} (84%)
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/util/{Weighted.java => 
ThrowingSupplier.java} (81%)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8edc18e8a91c0b65f124ac33d4c3f67c9678fc98
Merge: b812fa0 b1fd123
Author: Kenn Knowles 
AuthorDate: Mon Feb 5 14:32:31 2018 -0800

Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures

 .../core/construction/ArtifactServiceStager.java   |  32 +--
 .../beam/runners/dataflow/util/PackageUtil.java|  60 ++---
 .../fnexecution/control/FnApiControlClient.java|  35 ++-
 .../fnexecution/control/SdkHarnessClient.java  |  34 ++-
 .../control/FnApiControlClientPoolServiceTest.java |   9 +-
 .../control/FnApiControlClientTest.java|  30 +--
 .../fnexecution/control/SdkHarnessClientTest.java  |  17 +-
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 251 +
 .../org/apache/beam/sdk/util/ThrowingRunnable.java |  24 ++
 .../org/apache/beam/sdk/util/ThrowingSupplier.java |  26 +++
 .../java/org/apache/beam/sdk/util/GcsUtil.java |  21 +-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  10 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   |  32 +--
 .../beam/sdk/io/gcp/bigtable/BigtableService.java  |   4 +-
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  24 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   |  14 +-
 .../io/gcp/bigtable/BigtableServiceImplTest.java   |   9 +-
 17 files changed, 465 insertions(+), 167 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4888

[Apache Jenkins Server]

See 

[beam] branch master updated (8edc18e -> 8e2092b)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8edc18e  Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures
 add 16e5fef  [BEAM-3566] Replace apply_* hooks in DirectRunner with 
PTransformOverrides
 add e1a96eb  Address reviewer comments
 new 8e2092b  Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/gcp/pubsub_test.py  |  58 -
 sdks/python/apache_beam/pipeline.py|  34 +--
 sdks/python/apache_beam/pipeline_test.py   |   9 +-
 .../runners/dataflow/dataflow_runner_test.py   |   4 +-
 .../runners/dataflow/ptransform_overrides.py   |   6 +-
 .../apache_beam/runners/direct/direct_runner.py| 234 -
 .../runners/direct/helper_transforms.py|   8 +-
 .../runners/direct/sdf_direct_runner.py|   9 +-
 sdks/python/apache_beam/runners/sdf_common.py  |  15 +-
 sdks/python/apache_beam/transforms/combiners.py|  39 ++--
 10 files changed, 253 insertions(+), 163 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Resolved] (BEAM-3564) Handle an Unpopulated "Environment" field in Environments

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3564.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Handle an Unpopulated "Environment" field in Environments
> -
>
> Key: BEAM-3564
> URL: https://issues.apache.org/jira/browse/BEAM-3564
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: Not applicable
>
>
> The field may be present and unset (for example, Window Into the Global 
> Windows need not set an environment, but is permitted to). This should 
> succeed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Merge: 8edc18e e1a96eb
Author: Robert Bradshaw 
AuthorDate: Mon Feb 5 15:32:09 2018 -0800

Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in 
DirectRunner with PTransformOverrides

[BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides

 sdks/python/apache_beam/io/gcp/pubsub_test.py  |  58 -
 sdks/python/apache_beam/pipeline.py|  34 +--
 sdks/python/apache_beam/pipeline_test.py   |   9 +-
 .../runners/dataflow/dataflow_runner_test.py   |   4 +-
 .../runners/dataflow/ptransform_overrides.py   |   6 +-
 .../apache_beam/runners/direct/direct_runner.py| 234 -
 .../runners/direct/helper_transforms.py|   8 +-
 .../runners/direct/sdf_direct_runner.py|   9 +-
 sdks/python/apache_beam/runners/sdf_common.py  |  15 +-
 sdks/python/apache_beam/transforms/combiners.py|  39 ++--
 10 files changed, 253 insertions(+), 163 deletions(-)

diff --cc sdks/python/apache_beam/pipeline.py
index 4c48428,c59a29a..f164bf5
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@@ -61,8 -61,8 +61,9 @@@ from apache_beam.options.pipeline_optio
  from apache_beam.options.pipeline_options import StandardOptions
  from apache_beam.options.pipeline_options import TypeOptions
  from apache_beam.options.pipeline_options_validator import 
PipelineOptionsValidator
 +from apache_beam.portability import common_urns
  from apache_beam.pvalue import PCollection
+ from apache_beam.pvalue import PDone
  from apache_beam.runners import PipelineRunner
  from apache_beam.runners import create_runner
  from apache_beam.transforms import ptransform

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Assigned] (BEAM-3528) Populate Environment in Side Input SDKFunctionSpecs in ParDoTranslation

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3528:
-

Assignee: (was: Thomas Groh)

> Populate Environment in Side Input SDKFunctionSpecs in ParDoTranslation
> ---
>
> Key: BEAM-3528
> URL: https://issues.apache.org/jira/browse/BEAM-3528
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3529) Side Input callbacks should fire after a window has expired in addition to when the trigger would fire

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353096#comment-16353096
 ] 

Thomas Groh commented on BEAM-3529:
---

Yes; note that the simple test implementation uses {{TestStream}}, so there's 
limited coverage by default, but it is certainly worth writing.

> Side Input callbacks should fire after a window has expired in addition to 
> when the trigger would fire
> --
>
> Key: BEAM-3529
> URL: https://issues.apache.org/jira/browse/BEAM-3529
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Otherwise a streaming pipeline where elements trying to access a window that 
> has expired will not progress, even though we can be certain that there will 
> never be inputs in that window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3318) ConcurrentModificationException in populating Display Data

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3318:
-

Assignee: (was: Thomas Groh)

> ConcurrentModificationException in populating Display Data
> --
>
> Key: BEAM-3318
> URL: https://issues.apache.org/jira/browse/BEAM-3318
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, sdk-java-core
>Reporter: Thomas Groh
>Priority: Major
>
> Example: 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/16088/testReport/junit/org.apache.beam.examples/WindowedWordCountIT/testWindowedWordCountInBatchDynamicSharding/
> (Link is to a successful execution, but there is a stacktrace present within 
> the build)
> 2017-12-07T20:12:36.484 [ERROR] 
> testWindowedWordCountInBatchDynamicSharding(org.apache.beam.examples.WindowedWordCountIT)
>   Time elapsed: 5.833 s  <<< ERROR!
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException:
>  Error while populating display data for component 
> 'org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData':
>  null
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:802)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.delegate(DisplayData.java:768)
>   at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:145)
>   at com.sun.proxy.$Proxy30.populateDisplayData(Unknown Source)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:794)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:733)
>   at 
> org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:81)
>   at 
> org.apache.beam.runners.direct.DisplayDataValidator.evaluateDisplayData(DisplayDataValidator.java:47)
>   at 
> org.apache.beam.runners.direct.DisplayDataValidator.validateOptions(DisplayDataValidator.java:39)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:180)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:63)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.examples.WindowedWordCount.main(WindowedWordCount.java:214)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:187)
>   at 
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchDynamicSharding(WindowedWordCountIT.java:94)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
>   at 
> org.apache.beam.sdk.options.ProxyInvocationHandler$PipelineOptionsDisplayData.populateDisplayData(ProxyInvocationHandler.java:

[jira] [Assigned] (BEAM-3408) PackageUtil should retry check for whether file is already staged

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3408:
-

Assignee: (was: Thomas Groh)

> PackageUtil should retry check for whether file is already staged
> -
>
> Key: BEAM-3408
> URL: https://issues.apache.org/jira/browse/BEAM-3408
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Priority: Major
>
> The code to stage files retries the upload, but the existence (and size) 
> check is attempted only once:
> https://github.com/apache/beam/blob/dc3a664e3b4b4f63500954aae25dce0cb5f8b314/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L193
> This causes post-commit tests to fail sometimes:
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4632/testReport/junit/org.apache.beam.sdk.io/BoundedReadFromUnboundedSourceTest/testDedup/
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4636/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyMultimapSideInput/
> We should retry this check as well. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3619) Add tests for Python DirectRunner implementation of PubSub I/O

[Charles Chen (JIRA)]

Charles Chen created BEAM-3619:
--

 Summary: Add tests for Python DirectRunner implementation of 
PubSub I/O
 Key: BEAM-3619
 URL: https://issues.apache.org/jira/browse/BEAM-3619
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


We should add behavior tests and mocks to the Python DirectRunner 
implementation of PubSub I/O.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2430) Java FnApiDoFnRunner to share across runners

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2430.
---
   Resolution: Done
Fix Version/s: 2.3.0

I claim that a {{FnApiDoFnRunner}} is the wrong abstraction. However, a similar 
concept exists in the SdkHarnessClient, BundleProcessor, and ActiveBundle 
abstractions introduced in [https://github.com/apache/beam/pull/4472,] which is 
what clients who wish to use the portability framework should use when sending 
data to a remote Sdk Harness.

> Java FnApiDoFnRunner to share across runners
> 
>
> Key: BEAM-2430
> URL: https://issues.apache.org/jira/browse/BEAM-2430
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.3.0
>
>
> As the portability framework comes into focus, let's fill out the support 
> code for making it easy to onboard a new runner.
> There is some amount of using the Fn API that has to do only with the fact 
> that a runner is implemented in Java, and is not specific to that runner. 
> This should be part of the runners-core library, and designed so that a 
> runner can set it up however it likes, and just pass elements without having 
> to explicitly manage things like requests, responses, protos, and coders.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-849) Redesign PipelineResult API

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353111#comment-16353111
 ] 

Thomas Groh commented on BEAM-849:
--

This should be strongly informed by and should in turn inform the structure of 
the Beam Job API, especially the APIs available for interacting with an 
existing job.

> Redesign PipelineResult API
> ---
>
> Key: BEAM-849
> URL: https://issues.apache.org/jira/browse/BEAM-849
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Thomas Groh
>Priority: Major
>
> Current state: 
> Jira https://issues.apache.org/jira/browse/BEAM-443 addresses 
> waitUntilFinish() and cancel(). 
> However, there are additional work around PipelineResult: 
> need clearly defined contract and verification across all runners 
> need to revisit how to handle metrics/aggregators 
> need to be able to get logs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2718) Add bundle retry logic to the DirectRunner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

María GH updated BEAM-2718:
---
Fix Version/s: (was: Not applicable)
   2.3.0

> Add bundle retry logic to the DirectRunner
> --
>
> Key: BEAM-2718
> URL: https://issues.apache.org/jira/browse/BEAM-2718
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: Not applicable
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 2.3.0
>
>
> When processing of a bundle fails, the bundle should be retried 3 times (for 
> a total of 4 attempts to process it).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-2769:
--
Summary: Java SDK support for submitting a Portable Pipeline  (was: Java 
SDK support for portable pipelines)

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-2769:
-

Assignee: (was: Thomas Groh)

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3146) Fn Execution APIs should not depend on the Java SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-3146:
--
Description: 
Runners should be able to interact with the Fn API with complete ignorance of 
the underlying SDK, and as such should not obtain a dependency on the Java SDK.

This dependency edge should be banned. This will also require moving shared 
model abstractions which require code execution to a module on which both fn 
execution APIs and the Java SDK can depend on.

NOTE: the current state of this change requires either factoring of Coder, 
BoundedWindow, and associated runner-relevant code to a common location. 
Currently, these dependency edges exist to reduce work required to encode and 
decode these elements and minimize duplication between the runner and SDK for 
model-level concepts.

  was:
Runners should be able to interact with the Fn API with complete ignorance of 
the underlying SDK, and as such should not obtain a dependency on the Java SDK.

This dependency edge should be banned. This will also require moving shared 
model abstractions which require code execution to a module on which both fn 
execution APIs and the Java SDK can depend on.


> Fn Execution APIs should not depend on the Java SDK
> ---
>
> Key: BEAM-3146
> URL: https://issues.apache.org/jira/browse/BEAM-3146
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Runners should be able to interact with the Fn API with complete ignorance of 
> the underlying SDK, and as such should not obtain a dependency on the Java 
> SDK.
> This dependency edge should be banned. This will also require moving shared 
> model abstractions which require code execution to a module on which both fn 
> execution APIs and the Java SDK can depend on.
> NOTE: the current state of this change requires either factoring of Coder, 
> BoundedWindow, and associated runner-relevant code to a common location. 
> Currently, these dependency edges exist to reduce work required to encode and 
> decode these elements and minimize duplication between the runner and SDK for 
> model-level concepts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3146) Fn Execution APIs should not depend on the Java SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3146:
-

Assignee: (was: Thomas Groh)

> Fn Execution APIs should not depend on the Java SDK
> ---
>
> Key: BEAM-3146
> URL: https://issues.apache.org/jira/browse/BEAM-3146
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Major
>
> Runners should be able to interact with the Fn API with complete ignorance of 
> the underlying SDK, and as such should not obtain a dependency on the Java 
> SDK.
> This dependency edge should be banned. This will also require moving shared 
> model abstractions which require code execution to a module on which both fn 
> execution APIs and the Java SDK can depend on.
> NOTE: the current state of this change requires either factoring of Coder, 
> BoundedWindow, and associated runner-relevant code to a common location. 
> Currently, these dependency edges exist to reduce work required to encode and 
> decode these elements and minimize duplication between the runner and SDK for 
> model-level concepts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #879

[Apache Jenkins Server]

See 


Changes:

[Pablo] Moving User metrics to be in the PTransform proto for Fn API.

[ccy] [BEAM-3566] Replace apply_* hooks in DirectRunner with

[ccy] Address reviewer comments

[jiangkai] Covariance Functions

[robertwb] [BEAM-3183] Allow a callable as input to runner.run().

[sidhom] Fix gradle java sdk image build

[kenn] Add MoreFutures utility

[kenn] Switch runners/java-fn-execution from Guava futures to Java 8 futures

[kenn] Switch DataflowRunner from Guava futures to Java 8 futures

[kenn] Switch gcp-core from Guava futures to Java 8 futures

[kenn] Switch runners/core-construction-java from Guava futures to Java 8

[kenn] Switch AWS IO from Guava futures to Java 8 futures

[kenn] Switch BigTableIO from Guava futures to Java 8 futures

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8e2092ba28ad31517d65b3085840e6e9a46c83bf (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Commit message: "Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides"
 > git rev-list 2aca6ce2b69d76b9c1f8127d8647f9b1c7589219 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8695652388871773062.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8584536492190967477.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6593059882232593754.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1460524177545159586.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6518771498913675996.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8705279625347698268.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) exte

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4889

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4152

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-3566] Replace apply_* hooks in DirectRunner with

[ccy] Address reviewer comments

--
[...truncated 1.52 MB...]
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Giving up after 4 attempts.
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented

[jira] [Closed] (BEAM-3526) Support for checkpointMark finalize in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi closed BEAM-3526.
--
Resolution: Fixed

> Support for checkpointMark finalize in KafkaIO
> --
>
> Key: BEAM-3526
> URL: https://issues.apache.org/jira/browse/BEAM-3526
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>
> Users have been asking for option to commit offsets back to Kafka inside 
> `CheckpointMark.finalizeCheckpoint()'. This option is some what better than 
> setting 'AUTO_COMMIT' in Kafka consumer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3526) Support for checkpointMark finalize in KafkaIO

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-3526:
---
Fix Version/s: 2.4.0

> Support for checkpointMark finalize in KafkaIO
> --
>
> Key: BEAM-3526
> URL: https://issues.apache.org/jira/browse/BEAM-3526
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>
> Users have been asking for option to commit offsets back to Kafka inside 
> `CheckpointMark.finalizeCheckpoint()'. This option is some what better than 
> setting 'AUTO_COMMIT' in Kafka consumer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3620) Deprecate older Kafka clients

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-3620:
--

 Summary: Deprecate older Kafka clients
 Key: BEAM-3620
 URL: https://issues.apache.org/jira/browse/BEAM-3620
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Affects Versions: 2.3.0
Reporter: Raghu Angadi
Assignee: Raghu Angadi
 Fix For: 2.4.0


Kafka versions 0.10.1 and above have better support for timestamps. We intend 
to deprecate older versions as a first step towards dropping the support 
entirely.

The older versions (0.9.x and 0.10.0) at run time still work, but KafkaIO 
itself does not support these at build time. It lets us remove some of the 
reflection based code used to support older versions at build time. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3397

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5876

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Priority: Critical  (was: Major)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Labels: flake  (was: )

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3598:
--
Priority: Critical  (was: Major)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3598:
--
Labels: flake  (was: )

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Summary: KinesisReaderTest is Flaky due to overadvanced watermarks  (was: 
KenisisReaderTest is Flaky due to overadvanced watermarks)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Priority: Critical  (was: Major)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3317:
--
Labels: flake sickbay  (was: sickbay)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Reuven Lax
>Priority: Major
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3599:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3317:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3605:
-

Assignee: Kenneth Knowles  (was: Paweł Kaczmarczyk)

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard i

[jira] [Assigned] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3598:
-

Assignee: Kenneth Knowles  (was: Reuven Lax)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353283#comment-16353283
 ] 

Kenneth Knowles commented on BEAM-3228:
---

I haven't seen this one in a bit.

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.co

[jira] [Resolved] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3228.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launc

[jira] [Updated] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3228:
--
Labels: flake  (was: )

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Priority: Major
>  Labels: flake
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>   at jenkins.maven3.agent.Maven33Main.launch(Maven33Main.java:

[jira] [Updated] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3228:
--
Priority: Critical  (was: Major)

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>   at jenkins.maven

[jira] [Assigned] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3228:
-

Assignee: Kenneth Knowles

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Laun

[jira] [Assigned] (BEAM-3228) org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3228:
-

Assignee: Ismaël Mejía  (was: Kenneth Knowles)

> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest is flaky
> ---
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Ismaël Mejía
>Priority: Critical
>  Labels: flake
> Fix For: 2.3.0
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.La

[jira] [Created] (BEAM-3621) HCatalog failing frequently on fetch of org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3621:
-

 Summary: HCatalog failing frequently on fetch of 
org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde
 Key: BEAM-3621
 URL: https://issues.apache.org/jira/browse/BEAM-3621
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Assignee: Reuven Lax


The download seems to fail. There are three aspects to this:

 - The Gradle build should have a local persistent cache so it isn't fetching. 
But as I understand it Gradle has a policy of still pinging the repo even when 
cached.
 - The Gradle build is probably re-fetching due to {{--rerun-tasks}} so we need 
to stop doing that.
 - The artifact is not in "Central" repository but in "Spring Plugins". That is 
probably why all the failures are on this artifact and no other artifacts cause 
failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #835

[Apache Jenkins Server]

See 


--
[...truncated 130.35 KB...]
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 807, in save_code
pickler.save_reduce(CodeType, args, obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 548, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 807, in save_code
pickler.save_reduce(CodeType, args, obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 489, in save_string
self.memoize(obj)
  File "/usr/lib/python2.7/pickle.py", line 246, in memoize
self.write(self.put(memo_len))
  File "/usr/lib/python2.7/pickle.py", line 250, in put
def put(self, i, pack=struct.pack):
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 364, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 377, in run
return self.runner.run_pipeline(self)
  File 
"
 line 283, in run_pipeline
proto_pipeline = pipeline.to_runner_api()
  File 
"
 line 569, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"

Build failed in Jenkins: beam_PostCommit_Python_Verify #4153

[Apache Jenkins Server]

See 


--
[...truncated 1.52 MB...]
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"

Build failed in Jenkins: beam_PerformanceTests_Python #880

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 8e2092ba28ad31517d65b3085840e6e9a46c83bf (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8e2092ba28ad31517d65b3085840e6e9a46c83bf
Commit message: "Merge pull request #4529 from [BEAM-3566] Replace apply_* 
hooks in DirectRunner with PTransformOverrides"
 > git rev-list 8e2092ba28ad31517d65b3085840e6e9a46c83bf # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8050750355974562083.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4583841911990687607.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8261732458401309699.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2058054755165199310.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/aa/64/ea9f0f022bb9660b8a3c549754bf9105e05284ec408e93a65644522c1ecc/setuptools-38.5.0-py2.py3-none-any.whl#md5=f9582f22c550e37a0aa30a0dd2f72661
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins682193667744957802.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6282038092927760416.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Req

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #107

[Apache Jenkins Server]

See 


--
[...truncated 15.51 KB...]
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 

 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5916002223588525032.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/107/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-06 06:00:35,281 4b1cd6db MainThread INFO Verbose logging to: 

2018-02-06 06:00:35,282 4b1cd6db MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-06 06:00:35,282 4b1cd6db MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/107/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-06 06:00:35,569 4b1cd6db MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-06 06:00:35,569 4b1cd6db MainThread INFO Initializ

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3398

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5877

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4892

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)

[Romain Manni-Bucau (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Romain Manni-Bucau updated BEAM-3616:
-
Component/s: sdk-py-core
 sdk-go

> Ensure coders are updatable partially in a cluster (aka add versions to 
> coders)
> ---
>
> Key: BEAM-3616
> URL: https://issues.apache.org/jira/browse/BEAM-3616
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go, sdk-java-core, sdk-py-core
>Reporter: Romain Manni-Bucau
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow up ticket for the following discussion: 
> https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E
> Long story short: seems being able to update a live stream is a constraint 
> beam must address. This implies no change in coder serialization. To do it 
> the only way is to put a marker (like a version) otherwise the live update 
> will be broken if part of a cluster is updated only.
> This task is about ensuring we support it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)

[Romain Manni-Bucau (JIRA)]

Romain Manni-Bucau created BEAM-3616:


 Summary: Ensure coders are updatable partially in a cluster (aka 
add versions to coders)
 Key: BEAM-3616
 URL: https://issues.apache.org/jira/browse/BEAM-3616
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Romain Manni-Bucau
Assignee: Kenneth Knowles
 Fix For: 3.0.0


Follow up ticket for the following discussion: 
https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E

Long story short: seems being able to update a live stream is a constraint beam 
must address. This implies no change in coder serialization. To do it the only 
way is to put a marker (like a version) otherwise the live update will be 
broken if part of a cluster is updated only.

This task is about ensuring we support it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 03/03: Merge pull request #4563: Fixes Flink timer service restore

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 93818abf901dd996aa30445457153811b13ece43
Merge: 24804e9 6808b88
Author: Aljoscha Krettek 
AuthorDate: Mon Feb 5 11:57:36 2018 +0100

Merge pull request #4563: Fixes Flink timer service restore

 .../flink/FlinkStreamingTransformTranslators.java  |   3 +-
 .../wrappers/streaming/DoFnOperator.java   |  20 ++--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 105 +
 3 files changed, 119 insertions(+), 9 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 01/03: Remove erroneous cast in FlinkStreamingTransformTranslators

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 01d74b3b8381c04ffba3879281897ea2ef6fbcc5
Author: Dawid Wysakowicz 
AuthorDate: Mon Feb 5 11:36:33 2018 +0100

Remove erroneous cast in FlinkStreamingTransformTranslators
---
 .../apache/beam/runners/flink/FlinkStreamingTransformTranslators.java  | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 811c159..a2923a9 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -58,7 +58,6 @@ import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn;
@@ -253,7 +252,7 @@ class FlinkStreamingTransformTranslators {
   if 
(context.getOutput(transform).isBounded().equals(PCollection.IsBounded.BOUNDED))
 {
 boundedTranslator.translateNode(transform, context);
   } else {
-unboundedTranslator.translateNode((Read.Unbounded) transform, 
context);
+unboundedTranslator.translateNode(transform, context);
   }
 }
   }

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] branch master updated (24804e9 -> 93818ab)

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 24804e9  Merge pull request #3978: [BEAM-230] Remove deprecated 
WindowedValue.valueInEmptyWindows
 new 01d74b3  Remove erroneous cast in FlinkStreamingTransformTranslators
 new 6808b88  [BEAM-3186] Correctly use deserialized timerService in Flink 
Runner
 new 93818ab  Merge pull request #4563: Fixes Flink timer service restore

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../flink/FlinkStreamingTransformTranslators.java  |   3 +-
 .../wrappers/streaming/DoFnOperator.java   |  20 ++--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 105 +
 3 files changed, 119 insertions(+), 9 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 02/03: [BEAM-3186] Correctly use deserialized timerService in Flink Runner

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6808b8855f90f69098bd37d2353da7d3ca329eb3
Author: Dawid Wysakowicz 
AuthorDate: Thu Feb 1 13:47:30 2018 +0100

[BEAM-3186] Correctly use deserialized timerService in Flink Runner

Before, DoFnOperator was always creating a new timer service and not
using the one that was checkpointed in state. This was manifesting as
lost timers after a checkpoint/restore cycle.
---
 .../wrappers/streaming/DoFnOperator.java   |  20 ++--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 105 +
 2 files changed, 118 insertions(+), 7 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index de3c054..830a718 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -276,8 +276,10 @@ public class DoFnOperator
   keyedStateInternals = new FlinkStateInternals<>((KeyedStateBackend) 
getKeyedStateBackend(),
   keyCoder);
 
-  timerService = (HeapInternalTimerService)
-  getInternalTimerService("beam-timer", new 
CoderTypeSerializer<>(timerCoder), this);
+  if (timerService == null) {
+timerService = (HeapInternalTimerService)
+getInternalTimerService("beam-timer", new 
CoderTypeSerializer<>(timerCoder), this);
+  }
 
   timerInternals = new FlinkTimerInternals();
 
@@ -730,11 +732,15 @@ public class DoFnOperator
 // We just initialize our timerService
 if (keyCoder != null) {
   if (timerService == null) {
-timerService = new HeapInternalTimerService<>(
-totalKeyGroups,
-localKeyGroupRange,
-this,
-getRuntimeContext().getProcessingTimeService());
+final HeapInternalTimerService localService =
+new HeapInternalTimerService<>(
+totalKeyGroups,
+localKeyGroupRange,
+this,
+getRuntimeContext().getProcessingTimeService());
+
localService.startTimerService(getKeyedStateBackend().getKeySerializer(),
+new CoderTypeSerializer<>(timerCoder), this);
+timerService = localService;
   }
   timerService.restoreTimersForKeyGroup(div, keyGroupIdx, 
getUserCodeClassloader());
 }
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
index 6c32047..73a0a08 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
@@ -58,6 +58,7 @@ import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
@@ -609,6 +610,110 @@ public class DoFnOperatorTest {
 
   }
 
+  @Test
+  public void testTimersRestore() throws Exception {
+final Instant timerTimestamp = new Instant(1000);
+final String outputMessage = "Timer fired";
+
+WindowingStrategy windowingStrategy =
+WindowingStrategy.of(FixedWindows.of(new Duration(10_000)));
+
+DoFn fn = new DoFn() {
+  private static final String EVENT_TIMER_ID = "eventTimer";
+
+  @TimerId(EVENT_TIMER_ID)
+  private final TimerSpec eventTimer = 
TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+  @ProcessElement
+  public void processElement(ProcessContext context, 
@TimerId(EVENT_TIMER_ID) Timer timer) {
+timer.set(timerTimestamp);
+  }
+
+  @OnTimer(EVENT_TIMER_ID)
+  public void onEventTime(OnTimerContext context) {
+assertEquals(
+"Timer timestamp must match set timestamp.", timerTimestamp, 
context.timestamp());
+context.outputWithTimestamp(outputMessage, context.timestamp());
+  }
+};
+
+WindowedValue.FullWindowedValueCoder inputCoder =
+WindowedValue.getFullCoder(
+VarIntCoder.of(),
+windowingStrategy.getWindowFn().windowCoder());
+
+WindowedValue.FullWindowedValueCoder outputCoder =
+WindowedValue.getFullCoder(
+

[jira] [Commented] (BEAM-3186) In-flight data loss when restoring from savepoint

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352248#comment-16352248
 ] 

Aljoscha Krettek commented on BEAM-3186:


[~jbonofre] This was fixed on master in these commits:
01d74b3b8381c04ffba3879281897ea2ef6fbcc5
6808b8855f90f69098bd37d2353da7d3ca329eb3

Should I cherry-pick them onto the 2.3.0 release branch or do you want to do 
that?

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_JDBC #228

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3589) Flink runner breaks with ClassCastException on UnboundedSource

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352250#comment-16352250
 ] 

Aljoscha Krettek commented on BEAM-3589:


[~jbonofre] This was now fixed as part of another fix in this commit:
01d74b3b8381c04ffba3879281897ea2ef6fbcc5

> Flink runner breaks with ClassCastException on UnboundedSource
> --
>
> Key: BEAM-3589
> URL: https://issues.apache.org/jira/browse/BEAM-3589
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Grzegorz Kołakowski
>Priority: Blocker
> Fix For: 2.3.0
>
>
> When you execute a pipeline tha uses an unbounded source and an empty 
> transform it produces a ClassCastException:
> {quote}[WARNING] 
> java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> Caused by: java.lang.ClassCastException: 
> org.apache.beam.runners.core.construction.AutoValue_PTransformTranslation_UnknownRawPTransform
>  cannot be cast to org.apache.beam.sdk.io.Read$Unbounded
>     at 
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode
>  (FlinkStreamingTransformTranslators.java:256)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform
>  (FlinkStreamingPipelineTranslator.java:139)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform
>  (FlinkStreamingPipelineTranslator.java:118)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:670)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:662)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600 
> (TransformHierarchy.java:311)
>     at org.apache.beam.sdk.runners.TransformHierarchy.visit 
> (TransformHierarchy.java:245)
>     at org.apache.beam.sdk.Pipeline.traverseTopologically (Pipeline.java:458)
>     at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate 
> (FlinkPipelineTranslator.java:38)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate 
> (FlinkStreamingPipelineTranslator.java:70)
>     at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate 
> (FlinkPipelineExecutionEnvironment.java:113)
>     at org.apache.beam.runners.flink.FlinkRunner.run (FlinkRunner.java:110)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297)
>     at org.apache.beam.sdk.nexmark.NexmarkLauncher.run 
> (NexmarkLauncher.java:1139)
>     at org.apache.beam.sdk.nexmark.Main.runAll (Main.java:69)
>     at org.apache.beam.sdk.nexmark.Main.main (Main.java:301)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> {quote}
> You can reproduce it quickly by running this command from the nexmark 
> directory:
> {quote}mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
> -Pflink-runner "-Dexec.args=--runner=FlinkRunner --query=2 --streaming=true 
> --manageResources=false --monitorJobs=true"
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3589) Flink runner breaks with ClassCastException on UnboundedSource

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-3589:
---
Fix Version/s: 2.3.0

> Flink runner breaks with ClassCastException on UnboundedSource
> --
>
> Key: BEAM-3589
> URL: https://issues.apache.org/jira/browse/BEAM-3589
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Grzegorz Kołakowski
>Priority: Blocker
> Fix For: 2.3.0
>
>
> When you execute a pipeline tha uses an unbounded source and an empty 
> transform it produces a ClassCastException:
> {quote}[WARNING] 
> java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> Caused by: java.lang.ClassCastException: 
> org.apache.beam.runners.core.construction.AutoValue_PTransformTranslation_UnknownRawPTransform
>  cannot be cast to org.apache.beam.sdk.io.Read$Unbounded
>     at 
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode
>  (FlinkStreamingTransformTranslators.java:256)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform
>  (FlinkStreamingPipelineTranslator.java:139)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform
>  (FlinkStreamingPipelineTranslator.java:118)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:670)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:662)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600 
> (TransformHierarchy.java:311)
>     at org.apache.beam.sdk.runners.TransformHierarchy.visit 
> (TransformHierarchy.java:245)
>     at org.apache.beam.sdk.Pipeline.traverseTopologically (Pipeline.java:458)
>     at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate 
> (FlinkPipelineTranslator.java:38)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate 
> (FlinkStreamingPipelineTranslator.java:70)
>     at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate 
> (FlinkPipelineExecutionEnvironment.java:113)
>     at org.apache.beam.runners.flink.FlinkRunner.run (FlinkRunner.java:110)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297)
>     at org.apache.beam.sdk.nexmark.NexmarkLauncher.run 
> (NexmarkLauncher.java:1139)
>     at org.apache.beam.sdk.nexmark.Main.runAll (Main.java:69)
>     at org.apache.beam.sdk.nexmark.Main.main (Main.java:301)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> {quote}
> You can reproduce it quickly by running this command from the nexmark 
> directory:
> {quote}mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
> -Pflink-runner "-Dexec.args=--runner=FlinkRunner --query=2 --streaming=true 
> --manageResources=false --monitorJobs=true"
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3589) Flink runner breaks with ClassCastException on UnboundedSource

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-3589.
--
Resolution: Fixed

> Flink runner breaks with ClassCastException on UnboundedSource
> --
>
> Key: BEAM-3589
> URL: https://issues.apache.org/jira/browse/BEAM-3589
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.3.0
>Reporter: Ismaël Mejía
>Assignee: Grzegorz Kołakowski
>Priority: Blocker
> Fix For: 2.3.0
>
>
> When you execute a pipeline tha uses an unbounded source and an empty 
> transform it produces a ClassCastException:
> {quote}[WARNING] 
> java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> Caused by: java.lang.ClassCastException: 
> org.apache.beam.runners.core.construction.AutoValue_PTransformTranslation_UnknownRawPTransform
>  cannot be cast to org.apache.beam.sdk.io.Read$Unbounded
>     at 
> org.apache.beam.runners.flink.FlinkStreamingTransformTranslators$ReadSourceTranslator.translateNode
>  (FlinkStreamingTransformTranslators.java:256)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.applyStreamingTransform
>  (FlinkStreamingPipelineTranslator.java:139)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform
>  (FlinkStreamingPipelineTranslator.java:118)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:670)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:623)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:647)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit 
> (TransformHierarchy.java:662)
>     at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600 
> (TransformHierarchy.java:311)
>     at org.apache.beam.sdk.runners.TransformHierarchy.visit 
> (TransformHierarchy.java:245)
>     at org.apache.beam.sdk.Pipeline.traverseTopologically (Pipeline.java:458)
>     at org.apache.beam.runners.flink.FlinkPipelineTranslator.translate 
> (FlinkPipelineTranslator.java:38)
>     at 
> org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate 
> (FlinkStreamingPipelineTranslator.java:70)
>     at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate 
> (FlinkPipelineExecutionEnvironment.java:113)
>     at org.apache.beam.runners.flink.FlinkRunner.run (FlinkRunner.java:110)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297)
>     at org.apache.beam.sdk.nexmark.NexmarkLauncher.run 
> (NexmarkLauncher.java:1139)
>     at org.apache.beam.sdk.nexmark.Main.runAll (Main.java:69)
>     at org.apache.beam.sdk.nexmark.Main.main (Main.java:301)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke (Method.java:498)
>     at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
>     at java.lang.Thread.run (Thread.java:748)
> {quote}
> You can reproduce it quickly by running this command from the nexmark 
> directory:
> {quote}mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
> -Pflink-runner "-Dexec.args=--runner=FlinkRunner --query=2 --streaming=true 
> --manageResources=false --monitorJobs=true"
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3186) In-flight data loss when restoring from savepoint

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-3186.
--
Resolution: Fixed

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #229

[Apache Jenkins Server]

See 


--
[...truncated 12.64 KB...]
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/bbf3bfde/beam/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/4340ba98/beam/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/b92f9be4/beam/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/de23011f/beam/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/028c3463/beam/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/594a7f7b/beam/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/594a7f7b/beam/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/594a7f7b/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/594a7f7b/beam/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_AvroIOIT/runs/594a7f7b/beam/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_Performance

Build failed in Jenkins: beam_PerformanceTests_JDBC #230

[Apache Jenkins Server]

See 


Changes:

[mingmxu] var_pop and var_samp

[robertwb] Fix (and test) FnApiRunner in grpc mode.

[lcwik] [BEAM-2862] Add PTransform overrides specific to execution of

[klk] Key StateTable off id, not full StateTag

[ekirpichov] Cleans up PTransform validation across Beam

[relax] Add ability to set job labels to BEAM Java SDK

[chamikara] [BEAM-2861] Fix tests which fail with google credential

[chamikara] Add first few BigtableWriteException to suppressed list when 
rethrowing

[mingmxu] [BEAM-2804] support TIMESTAMP in sort

[chamikara] Fix GqlQueryTranslateFn to pass localhost option to DatastoreFactory

[klk] Improve error message for bad DoFn URN in ParDoTranslation

[klk] Key DoFnInstanceManager cache on AppliedPTransform

[klk] Add fast path to ParDoTranslation for known ParDo class

[millsd] Mark AuctionOrBidWindowCoder as consistentWithEquals

[owenzhang1990] [BEAM-2632] Use Junit Paramaterized test suits in TextIOReadTest

[staslevin] [BEAM-2859] Fixed processing timers not being properly fired when

[millsd] Enforce correctness of consistentWithEquals and fix broken hashCode()

[millsd] Make AuctionOrBidWindowCoder use structuralValue instead of

[pei] flink-runner: constructs exception string only when neccessary, it

[joey.baruch] Add javadoc to ConsoleIO

[klk] Key FlinkRunner streaming translation off URN

[kirpichov] [BEAM-1637] Create Elasticsearch IO compatible with ES 5.x

[kirpichov] Piggyback: deflakes MongoDBGridFSIOTest which failed during merge

[aljoscha.krettek] [BEAM-2807] Fix NPE error on 
CoderTypeSerializerConfigSnapshot

[lcwik] [BEAM-2790] Use byte[] instead of ByteBuffer to read from

[lcwik] Fall back to inputStream.read(byteBuffer) in case of off-heap byteBuffer

[chamikara] Improve performance of bundle retry

[lcwik] [BEAM-2956] Attempt to correctly report the Dataflow distribution in GCP

[lcwik] [BEAM-2961] Add missing OnTimeBehavior proto translation to

[lcwik] [BEAM-2808] Improving error message in

[echauchot] [BEAM-2957] Fix flaky ElasticsearchIOTest.testSplit in

[kirpichov] Speeds up CompressedSourceTest

[klk] Move DoFnInfo to SDK util

[klk] Update Dataflow container version to 20170918

[aljoscha.krettek] [BEAM-2948] Register default FileSystems in 
StreamOperator.setup()

[robertwb] [BEAM-2964] Exclude incompatible six release.

[kirpichov] [BEAM-407] Fixes findbugs warnings in OffsetRangeTracker

[robertwb] Refactor fn api runner into universal local runner.

[robertwb] Implement ULR subprocess mode.

[robertwb] Allow worker to be started as a subprocess.

[robertwb] Streaming Job API.

[robertwb] Lint and documentation.

[robertwb] Allow subclasses of tuple, list, and dict as pvaluish inputs/outputs.

[robertwb] Support multiple materializations of the same pvalue.

[kirpichov] Updates grpc-google-pubsub-v1 to grpc-google-cloud-pubsub-v1

[kirpichov] Marks TikaIO as experimental to allow backward-incompatible changes

[aljoscha.krettek] [BEAM-2377] Allow cross compilation (2.10,2.11) for flink 
runner

[tgroh] Add PipelineOptionsTranslation

[tgroh] Add Nullable getters to MetricsContainerImpl

[robertwb] Exclude incompatible six release, part 2

[robertwb] Enable progress request handling in python SDK harness

[robertwb] Minor cleanup.

[robertwb] Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for flink

[robertwb] Execute windowing in Fn API runner.

[klk] Send portable ParDo protos to Dataflow instead of just DoFnInfo

[klk] Update Dataflow worker to 20170921

[tgroh] Clarify semantics of objects returned by state access

[klk] DataflowRunner depends on, and shades, protobuf

[klk] Shade generated Runner API classes in Dataflow runner

[tgroh] Initial set of pipeline jobs.

[relax] Make sure that we default to alwaysRetry instead of passing in a null

[relax] Add unit-test coverage.

[relax] Strip table decorators before creating tables.

[relax] Move stripping code into BigQueryHelpers and add better unit-test

[klk] Revert "This closes #3859: [BEAM-2884] Send portable protos for ParDo in

[relax] Fix type parameter in AvroIO.Write

[chamikara] Revert "Initial set of pipeline jobs."

[klk] Update Dataflow worker to 20170922-01

[tgroh] Artifact API Cleanup

[tgroh] Add a Local FS implementation of the Artifact Staging API

[chamikara] Included immediate results after CoGroupByKey for better 
readability in

[tgroh] [BEAM-2876] Add preliminary provision API

[chamikara] Updates WriteToBigQuery PTransform to get project id from

[tgroh] Use the SubscriptionProvider in PubsubUnboundedSource

[kirpichov] [BEAM-2467] Kinesis source watermark based on

[tgroh] Update the Dataflow v1b3 API version

[tgroh] Change checksum back to md5

[relax] Move file deletion into subsequent ParDo.

[tgroh] Move Runners Core Fn Api classes into a package

[chamikara] Add test to fix partial writouts after a bundle retry

[github] Catch subprocess error in proto gener

Build failed in Jenkins: beam_PerformanceTests_Python #877

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] Remove erroneous cast in FlinkStreamingTransformTranslators

[aljoscha.krettek] [BEAM-3186] Correctly use deserialized timerService in Flink 
Runner

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 93818abf901dd996aa30445457153811b13ece43 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 93818abf901dd996aa30445457153811b13ece43
Commit message: "Merge pull request #4563: Fixes Flink timer service restore"
 > git rev-list 24804e98d22180c1dc6603c8a437073ec2adde2d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins556338002970157487.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7139136910107254409.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1208891581456414687.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6510183300117871212.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/aa/64/ea9f0f022bb9660b8a3c549754bf9105e05284ec408e93a65644522c1ecc/setuptools-38.5.0-py2.py3-none-any.whl#md5=f9582f22c550e37a0aa30a0dd2f72661
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4816526190465977560.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2854794569014409044.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requir

Jenkins build is back to normal : beam_PerformanceTests_JDBC #231

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3186) In-flight data loss when restoring from savepoint

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352322#comment-16352322
 ] 

Jean-Baptiste Onofré commented on BEAM-3186:


[~aljoscha] thanks, I cherry-pick on release-2.3.0 branch.

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4884

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #232

[Apache Jenkins Server]

See 


--
GitHub pull request #4585 of commit 1e38bea87d0bd8d6c629f58410765cf3c2809216, 
no merge conflicts.
Setting status of 1e38bea87d0bd8d6c629f58410765cf3c2809216 to PENDING with url 
https://builds.apache.org/job/beam_PerformanceTests_JDBC/232/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Java JdbcIO Performance Test
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision b53be98449c43cf04b78dcc7881a3a98ef39c6c8 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53be98449c43cf04b78dcc7881a3a98ef39c6c8
Commit message: "Merge 1e38bea87d0bd8d6c629f58410765cf3c2809216 into 
93818abf901dd996aa30445457153811b13ece43"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6662170660726059490.sh
+ gcloud container clusters list
NAME   ZONE   MASTER_VERSION  MASTER_IP   MACHINE_TYPE   
NODE_VERSION  NUM_NODES  STATUS
io-datastores  us-central1-a  1.7.11-gke.1104.154.102.21  n1-standard-1  
1.7.11-gke.1  1  RUNNING
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8319512339861463527.sh
+ gcloud container clusters get-credentials io-datastores --verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, zone=None).
WARNING: Accessing a Container Engine cluster requires the kubernetes 
commandline
client [kubectl]. To install, run
  $ gcloud components install kubectl

DEBUG: (gcloud.container.clusters.get-credentials) The required property [zone] 
is not currently set.
It can be set on a per-command basis by re-running your command with the 
[--zone] flag.

You may set it for your current workspace by running:

  $ gcloud config set compute/zone VALUE

or it can be set temporarily by the environment variable [CLOUDSDK_COMPUTE_ZONE]
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 
632, in Execute
result = args.cmd_func(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 
1369, in Run
resources = command_instance.Run(args)
  File 
"/usr/lib/google-cloud-sdk/lib/surface/container/clusters/get_credentials.py", 
line 56, in Run
cluster_ref = adapter.ParseCluster(args.name)
  File 
"/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py",
 line 143, in ParseCluster
properties.VALUES.compute.zone.Get(required=True)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 
1069, in Get
required)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 
1352, in _GetProperty
prop, flag=flag_to_use)
RequiredPropertyError: The required property [zone] is not currently set.
It can be set on a per-command basis by re-running your command with the 
[--zone] flag.

You may set it for your current workspace by running:

  $ gcloud config set compute/zone VALUE

or it can be set temporarily by the environm

[beam] branch release-2.3.0 updated (7da24e6 -> befa911)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7da24e6  [BEAM-3551] Define compiler -parameters flag in the default 
options
 new 73f4d99  Remove erroneous cast in FlinkStreamingTransformTranslators
 new befa911  [BEAM-3186] Correctly use deserialized timerService in Flink 
Runner

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../flink/FlinkStreamingTransformTranslators.java  |   3 +-
 .../wrappers/streaming/DoFnOperator.java   |  20 ++--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 105 +
 3 files changed, 119 insertions(+), 9 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] 02/02: [BEAM-3186] Correctly use deserialized timerService in Flink Runner

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit befa91180a0935ac4ae667084908db8a43746279
Author: Dawid Wysakowicz 
AuthorDate: Thu Feb 1 13:47:30 2018 +0100

[BEAM-3186] Correctly use deserialized timerService in Flink Runner

Before, DoFnOperator was always creating a new timer service and not
using the one that was checkpointed in state. This was manifesting as
lost timers after a checkpoint/restore cycle.
---
 .../wrappers/streaming/DoFnOperator.java   |  20 ++--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 105 +
 2 files changed, 118 insertions(+), 7 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index de3c054..830a718 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -276,8 +276,10 @@ public class DoFnOperator
   keyedStateInternals = new FlinkStateInternals<>((KeyedStateBackend) 
getKeyedStateBackend(),
   keyCoder);
 
-  timerService = (HeapInternalTimerService)
-  getInternalTimerService("beam-timer", new 
CoderTypeSerializer<>(timerCoder), this);
+  if (timerService == null) {
+timerService = (HeapInternalTimerService)
+getInternalTimerService("beam-timer", new 
CoderTypeSerializer<>(timerCoder), this);
+  }
 
   timerInternals = new FlinkTimerInternals();
 
@@ -730,11 +732,15 @@ public class DoFnOperator
 // We just initialize our timerService
 if (keyCoder != null) {
   if (timerService == null) {
-timerService = new HeapInternalTimerService<>(
-totalKeyGroups,
-localKeyGroupRange,
-this,
-getRuntimeContext().getProcessingTimeService());
+final HeapInternalTimerService localService =
+new HeapInternalTimerService<>(
+totalKeyGroups,
+localKeyGroupRange,
+this,
+getRuntimeContext().getProcessingTimeService());
+
localService.startTimerService(getKeyedStateBackend().getKeySerializer(),
+new CoderTypeSerializer<>(timerCoder), this);
+timerService = localService;
   }
   timerService.restoreTimersForKeyGroup(div, keyGroupIdx, 
getUserCodeClassloader());
 }
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
index 6c32047..73a0a08 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
@@ -58,6 +58,7 @@ import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
@@ -609,6 +610,110 @@ public class DoFnOperatorTest {
 
   }
 
+  @Test
+  public void testTimersRestore() throws Exception {
+final Instant timerTimestamp = new Instant(1000);
+final String outputMessage = "Timer fired";
+
+WindowingStrategy windowingStrategy =
+WindowingStrategy.of(FixedWindows.of(new Duration(10_000)));
+
+DoFn fn = new DoFn() {
+  private static final String EVENT_TIMER_ID = "eventTimer";
+
+  @TimerId(EVENT_TIMER_ID)
+  private final TimerSpec eventTimer = 
TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+  @ProcessElement
+  public void processElement(ProcessContext context, 
@TimerId(EVENT_TIMER_ID) Timer timer) {
+timer.set(timerTimestamp);
+  }
+
+  @OnTimer(EVENT_TIMER_ID)
+  public void onEventTime(OnTimerContext context) {
+assertEquals(
+"Timer timestamp must match set timestamp.", timerTimestamp, 
context.timestamp());
+context.outputWithTimestamp(outputMessage, context.timestamp());
+  }
+};
+
+WindowedValue.FullWindowedValueCoder inputCoder =
+WindowedValue.getFullCoder(
+VarIntCoder.of(),
+windowingStrategy.getWindowFn().windowCoder());
+
+WindowedValue.FullWindowedValueCoder outputCoder =
+WindowedValue.getFullCoder(
+ 

[beam] 01/02: Remove erroneous cast in FlinkStreamingTransformTranslators

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 73f4d997b1489ec5e4c11cbed9538f8ed4e38caa
Author: Dawid Wysakowicz 
AuthorDate: Mon Feb 5 11:36:33 2018 +0100

Remove erroneous cast in FlinkStreamingTransformTranslators
---
 .../apache/beam/runners/flink/FlinkStreamingTransformTranslators.java  | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 811c159..a2923a9 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -58,7 +58,6 @@ import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.UnboundedSource;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn;
@@ -253,7 +252,7 @@ class FlinkStreamingTransformTranslators {
   if 
(context.getOutput(transform).isBounded().equals(PCollection.IsBounded.BOUNDED))
 {
 boundedTranslator.translateNode(transform, context);
   } else {
-unboundedTranslator.translateNode((Read.Unbounded) transform, 
context);
+unboundedTranslator.translateNode(transform, context);
   }
 }
   }

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[jira] [Commented] (BEAM-3186) In-flight data loss when restoring from savepoint

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352336#comment-16352336
 ] 

Jean-Baptiste Onofré commented on BEAM-3186:


Cherry-pick done on the release-2.3.0 branch. Thanks !

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Dawid Wysakowicz
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #233

[Apache Jenkins Server]

See 


--
[...truncated 282 B...]
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision cdd4ca35a9b4a8b415dab2c0df3b1f6f44c7492f 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f cdd4ca35a9b4a8b415dab2c0df3b1f6f44c7492f
Commit message: "Merge f521bcb07aa1097bc1cda7300a6e4a446e068a1c into 
93818abf901dd996aa30445457153811b13ece43"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1810647867324642778.sh
+ gcloud container clusters list
NAME   ZONE   MASTER_VERSION  MASTER_IP   MACHINE_TYPE   
NODE_VERSION  NUM_NODES  STATUS
io-datastores  us-central1-a  1.7.11-gke.1104.154.102.21  n1-standard-1  
1.7.11-gke.1  1  RUNNING
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7476215323738956507.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
WARNING: Accessing a Container Engine cluster requires the kubernetes 
commandline
client [kubectl]. To install, run
  $ gcloud components install kubectl

Fetching cluster endpoint and auth data.
DEBUG: unable to load default kubeconfig: [Errno 2] No such file or directory: 
'/home/jenkins/.kube/config'; recreating /home/jenkins/.kube/config
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7724032151790843545.sh
+ find / -regex '.*kube.*config.*'
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/hadoop/config-files/core-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/postgres/pkb-config-local.yml
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/.test-infra/kubernetes/postgres/pkb-config.yml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files/hdfs-site.xml
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/config-files/hdfs-envs
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TFRecordIOIT/src/.test-infra/kubernetes/hadoop/con

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5866

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352390#comment-16352390
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


Using 2.3.0-SNAPSHOT artifacts provided by Maven with the following pipeline 
works without problem with the Flink runner:

{code}
public static final void main(String args[]) throws Exception {
PipelineOptions options = 
PipelineOptionsFactory.fromArgs(args).create();
Pipeline pipeline = Pipeline.create(options);

pipeline
.apply(TextIO.read().from("/home/jbonofre/artists.csv"))
.apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext processContext) {
String element = processContext.element();
String[] split = element.split(",");
processContext.output(split[1]);
}
}))
.apply(Count.perElement())
.apply(MapElements.via(new SimpleFunction, 
String>() {
public String apply(KV element) {
return "{\"" + element.getKey() + "\": \"" + 
element.getValue() + "\"}";
}
}))
.apply(TextIO.write().to("/home/jbonofre/label.json"));

pipeline.run();
}
{code}

I'm now testing with artifacts built by Gradle.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4885

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #234

[Apache Jenkins Server]

See 


--
GitHub pull request #4585 of commit 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3, 
no merge conflicts.
Setting status of 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3 to PENDING with url 
https://builds.apache.org/job/beam_PerformanceTests_JDBC/234/ and message: 
'Build started sha1 is merged.'
Using context: Jenkins: Java JdbcIO Performance Test
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4585/*:refs/remotes/origin/pr/4585/*
 > git rev-parse refs/remotes/origin/pr/4585/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4585/merge^{commit} # timeout=10
Checking out Revision 93c244a4c3f5ef4dc392b2edfb977f68811f4e56 
(refs/remotes/origin/pr/4585/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 93c244a4c3f5ef4dc392b2edfb977f68811f4e56
Commit message: "Merge 4d2fed623eddef9f3a8bb1981e1911bdcb2001b3 into 
93818abf901dd996aa30445457153811b13ece43"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins538140067291400.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: unable to load default kubeconfig: [Errno 2] No such file or directory: 
'/home/jenkins/.kube/config'; recreating /home/jenkins/.kube/config
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: Checking for updates...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4589194026611297974.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4356914166225145917.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8814813696663637891.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5246839393785401843.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5115933430353196731.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4143112721671497997.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/re

[jira] [Commented] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352564#comment-16352564
 ] 

Jean-Baptiste Onofré commented on BEAM-3587:


I cloned the test repo: https://github.com/pelletier/beam-flink-example

Clearly the problem is not in the artifacts, but in the shadowJar. I'm testing 
the fix and will provide a PR to user. If it's what I think, I will close this 
Jira as it's not a Beam issue.

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3587:
--

Assignee: Jean-Baptiste Onofré  (was: Ben Sidhom)

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

Jean-Baptiste Onofré created BEAM-3617:
--

 Summary: Performance degradation on the direct runner
 Key: BEAM-3617
 URL: https://issues.apache.org/jira/browse/BEAM-3617
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Affects Versions: 2.3.0
Reporter: Jean-Baptiste Onofré
Assignee: Thomas Groh
 Fix For: 2.3.0


Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
shows a performance degradation:

{code}

 Beam 2.2.0   Beam 2.3.0
  Query  Runtime(sec) Runtime(sec)

     6.410.6
  0001   5.110.2
  0002   3.0 5.8
  0003   3.8 6.2
  0004   0.9 1.4
  0005   5.811.4
  0006   0.8 1.4
  0007 193.8  1249.1
  0008   3.9 6.9
  0009   0.9 1.3
  0010   6.4 8.2
  0011   5.0 9.4
  0012   4.7 9.1
{code}

We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)