[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=133903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133903
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 11/Aug/18 03:34
Start Date: 11/Aug/18 03:34
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6176: [[BEAM-5092] Row 
comparison should be faster when both are POJOs.
URL: https://github.com/apache/beam/pull/6176#issuecomment-412247847
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133903)
Time Spent: 7.5h  (was: 7h 20m)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (cb6ac22 -> 7c56118)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from cb6ac22  Prepare repository for deployment.
 add a292925  Adding Pablo Estrada as author for blog
 add 9925d87  Blog post for Beam 2.6.0 release
 add 4e8792f  Added Rafael Fernandez as author.
 add 9420874  This closes #533
 new 7c56118  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../2018/08/10/beam-2.6.0.html}|  88 +++-
 content/blog/index.html|  81 +++
 content/feed.xml   | 152 ++---
 content/index.html |  10 +-
 src/_data/authors.yml  |   7 +
 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md |  58 
 6 files changed, 278 insertions(+), 118 deletions(-)
 copy content/{beam/capability/2016/04/03/presentation-materials.html => 
blog/2018/08/10/beam-2.6.0.html} (75%)
 create mode 100644 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7c561185f26728de9e321c5cb80ef8751a6779d9
Author: Mergebot 
AuthorDate: Sat Aug 11 01:49:18 2018 +

Prepare repository for deployment.
---
 .../2018/08/10/beam-2.6.0.html}| 283 ++---
 content/blog/index.html|  81 ++
 content/feed.xml   | 152 ++-
 content/index.html |  10 +-
 4 files changed, 241 insertions(+), 285 deletions(-)

diff --git a/content/index.html b/content/blog/2018/08/10/beam-2.6.0.html
similarity index 56%
copy from content/index.html
copy to content/blog/2018/08/10/beam-2.6.0.html
index cc857a1..eb7c18c 100644
--- a/content/index.html
+++ b/content/blog/2018/08/10/beam-2.6.0.html
@@ -28,9 +28,8 @@
   
   
   
-  Apache Beam
-  
+  Apache Beam 2.6.0
+  
   https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
   
   https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
@@ -39,7 +38,7 @@
   
   
   
-  https://beam.apache.org/; data-proofer-ignore>
+  https://beam.apache.org/blog/2018/08/10/beam-2.6.0.html; 
data-proofer-ignore>
   
   https://beam.apache.org/feed.xml;>
   
@@ -52,7 +51,7 @@
   
 
 
-  
+  
 
+
+
+
+http://schema.org/BlogPosting;>
+
+  
+Apache Beam 2.6.0
+Aug 10, 2018 • 
-
-  
-
-  
-
-  
-Apache Beam: An advanced unified programming model
-  
-  
-Implement batch and streaming data processing jobs that run on any 
execution engine.
-  
-  
-Learn more
-Download Beam SDK 2.6.0
-  
-  
-Java 
Quickstart
-Python 
Quickstart
-   Go 
Quickstart
-  
-
-  
-  
-
-  
-The latest from the blog
-  
-  
-
-
-  Apache Beam 
2.5.0
-  Jun 26, 2018
-
-
-
-  Apache Beam 
2.3.0
-  Feb 19, 2018
-
-
-
-  Apache Beam: A Look 
Back at 2017
-  Jan 9, 2018
-
-
-  
-
-  
-
-  
-
+  
+
+  
 
-
-  
-All about Apache Beam
-  
-  
-
-
-  
-Unified
-  
-  
-Use a single programming model for both batch and streaming use cases.
-  
-
-
-
-  
-Portable
-  
-  
-Execute pipelines on multiple execution environments.
-  
-
-
-
-  
-Extensible
-  
-  
-Write and share new SDKs, IO connectors, and transformation libraries.
-  
-
-
-  
-
+  
+Apache Beam version 2.6.0
 
-
-
-
-
-
+We are glad to present the new 2.6.0 release of Beam. This release includes 
multiple fixes and new functionalities. Specifically, we spent a significant 
amount of time trying to automate the release, and fix the continuous 
integration. We also have significant new features in SQL, and portability. 
Please go ahead and take a look at the following highlights:
 
-
-  
-Works with
-  
-  
-
-
-  http://apex.apache.org;>
-
-
-
-  http://flink.apache.org;>
-
-
-
-  http://spark.apache.org/;>
-
-
-
-  https://cloud.google.com/dataflow/;>
-
-
-
-  http://gearpump.apache.org/;>
-
-
-
-  http://samza.apache.org/;>
-
-
-  
-
+For more information, please, check the detailed https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527version=12343392;>release
 notes.
 
-
-  
-
-  Testimonials
-
-
-  
-  
-
-  A framework that delivers the flexibility and advanced functionality 
our customers need.
-
-
-  
-  
-–Talend
-  
-
-  
-  
-  
-
-  Apache Beam has powerful semantics that solve real-world challenges 
of stream processing.
-
-
-  
-  
-–PayPal
-  
-
-  
-  
-  
-
-  Apache Beam represents a principled approach for analyzing data 
streams.
-
-
-  
-  
-–data Artisans
-  
-
-  
-  
-
-
-  Beam is an open source community and contributions are greatly 
appreciated!
-  If you’d like to contribute, please see the Contribute section.
-
-  
-
+New Features / Improvements
+
+gRPC/Protobuf shading
+
+
+  gRPC/protobuf is now shaded in the majority of Apache Beam Java modules. 
A few modules which expose gRPC/protobuf on the API surface still maintain a 
direct dependency.

[beam-site] 01/01: This closes #533

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9420874966e85721e2e8dc707cdee5b72c193790
Merge: cb6ac22 4e8792f
Author: Mergebot 
AuthorDate: Sat Aug 11 01:46:22 2018 +

This closes #533

 src/_data/authors.yml  |  7 
 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md | 58 ++
 2 files changed, 65 insertions(+)

[beam-site] branch mergebot updated (7ff1a24 -> 9420874)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 7ff1a24  This closes #529
 add cb6ac22  Prepare repository for deployment.
 add a292925  Adding Pablo Estrada as author for blog
 add 9925d87  Blog post for Beam 2.6.0 release
 add 4e8792f  Added Rafael Fernandez as author.
 new 9420874  This closes #533

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  | 28 ---
 src/_data/authors.yml  |  7 +++
 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md | 58 ++
 4 files changed, 77 insertions(+), 18 deletions(-)
 create mode 100644 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1229

[Apache Jenkins Server]

See 


--
[...truncated 17.70 MB...]
Aug 11, 2018 12:35:37 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0811003533-369d2d74/output/results/staging/
Aug 11, 2018 12:35:37 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <115993 bytes, hash fUOufKxQEqCUAPaKXmZqDQ> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0811003533-369d2d74/output/results/staging/pipeline-fUOufKxQEqCUAPaKXmZqDQ.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.7.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Aug 11, 2018 12:35:39 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-08-10_17_35_38-5677145682546382374?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-08-10_17_35_38-5677145682546382374

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Aug 11, 2018 12:35:39 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-08-10_17_35_38-5677145682546382374
Aug 11, 2018 12:35:39 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-08-10_17_35_38-5677145682546382374 with 0 
expected assertions.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:38.319Z: Autoscaling is enabled for job 
2018-08-10_17_35_38-5677145682546382374. The number of workers will be between 
1 and 1000.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:38.362Z: Autoscaling was automatically enabled for 
job 2018-08-10_17_35_38-5677145682546382374.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:41.608Z: Checking required Cloud APIs are enabled.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:41.920Z: Checking permissions granted to controller 
Service Account.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:46.594Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:47.320Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:47.907Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:47.954Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:48.607Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:48.672Z: Elided trivial flatten 
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:48.718Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:48.755Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 11, 2018 12:35:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-11T00:35:48.802Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 

Build failed in Jenkins: beam_PreCommit_Java_Cron #211

[Apache Jenkins Server]

See 


Changes:

[xiliu] [BEAM-4849] Support running Beam Samza jobs in Yarn

[yifanzou] [BEAM-5084] upgrade fastjson

[amaliujia] test conditional operators and functions at DSL level

[amaliujia] allow test null value

[amaliujia] test aggregation functions at DSL level

[pablo] Step in cleanup of create_operation function

[aaltay] Several improvements for consistency of Python2 / Python3 conversions.

[lcwik] Suppressing the error stack docker image pull

[pablo] Add post-commit tests status to README.md in root

--
[...truncated 17.41 MB...]
LogicalProject(auction=[$0], num=[$1])
  LogicalJoin(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
LogicalProject(auction=[$0], num=[$2], starttime=[$1])
  LogicalAggregate(group=[{0, 1}], num=[COUNT()])
LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 1)])
  BeamIOSourceRel(table=[[beam, Bid]])
LogicalProject(maxnum=[$1], starttime=[$0])
  LogicalAggregate(group=[{0}], maxnum=[MAX($1)])
LogicalProject(starttime=[$1], num=[$0])
  LogicalProject(num=[$2], starttime=[$1])
LogicalAggregate(group=[{0, 1}], num=[COUNT()])
  LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 1)])
BeamIOSourceRel(table=[[beam, Bid]])

Aug 11, 2018 12:22:05 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], proj#0..1=[{exprs}])
  BeamJoinRel(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
BeamCalcRel(expr#0..2=[{inputs}], auction=[$t0], num=[$t2], 
starttime=[$t1])
  BeamAggregationRel(group=[{0, 1}], num=[COUNT()])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[1], 
expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7])
  BeamIOSourceRel(table=[[beam, Bid]])
BeamCalcRel(expr#0..1=[{inputs}], maxnum=[$t1], starttime=[$t0])
  BeamAggregationRel(group=[{1}], maxnum=[MAX($0)])
BeamCalcRel(expr#0..2=[{inputs}], num=[$t2], starttime=[$t1])
  BeamAggregationRel(group=[{0, 1}], num=[COUNT()])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], 
expr#6=[1], expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7])
  BeamIOSourceRel(table=[[beam, Bid]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery3Test > 
testJoinsPeopleWithAuctions STANDARD_ERROR
Aug 11, 2018 12:22:06 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `P`.`name`, `P`.`city`, `P`.`state`, `A`.`id`
FROM `beam`.`Auction` AS `A`
INNER JOIN `beam`.`Person` AS `P` ON `A`.`seller` = `P`.`id`
WHERE `A`.`category` = 10 AND (`P`.`state` = 'OR' OR `P`.`state` = 'ID' OR 
`P`.`state` = 'CA')
Aug 11, 2018 12:22:06 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(name=[$11], city=[$14], state=[$15], id=[$0])
  LogicalFilter(condition=[AND(=($8, 10), OR(=($15, 'OR'), =($15, 'ID'), 
=($15, 'CA')))])
LogicalJoin(condition=[=($7, $10)], joinType=[inner])
  BeamIOSourceRel(table=[[beam, Auction]])
  BeamIOSourceRel(table=[[beam, Person]])

Aug 11, 2018 12:22:06 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..17=[{inputs}], name=[$t11], city=[$t14], state=[$t15], 
id=[$t0])
  BeamJoinRel(condition=[=($7, $10)], joinType=[inner])
BeamCalcRel(expr#0..9=[{inputs}], expr#10=[10], expr#11=[=($t8, $t10)], 
proj#0..9=[{exprs}], $condition=[$t11])
  BeamIOSourceRel(table=[[beam, Auction]])
BeamCalcRel(expr#0..7=[{inputs}], expr#8=['OR'], expr#9=[=($t5, $t8)], 
expr#10=['ID'], expr#11=[=($t5, $t10)], expr#12=['CA'], expr#13=[=($t5, $t12)], 
expr#14=[OR($t9, $t11, $t13)], proj#0..7=[{exprs}], $condition=[$t14])
  BeamIOSourceRel(table=[[beam, Person]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery7Test > testBids STANDARD_ERROR
Aug 11, 2018 12:22:06 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, `B`.`extra`
FROM (SELECT `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, 
`B`.`extra`, TUMBLE_START(`B`.`dateTime`, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B`
GROUP BY `B`.`auction`, `B`.`price`, `B`.`bidder`, `B`.`dateTime`, 
`B`.`extra`, TUMBLE(`B`.`dateTime`, INTERVAL '10' SECOND)) AS `B`
INNER JOIN (SELECT MAX(`B1`.`price`) AS `maxprice`, 
TUMBLE_START(`B1`.`dateTime`, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B1`
GROUP BY TUMBLE(`B1`.`dateTime`, INTERVAL '10' SECOND)) AS `B1` ON 

Build failed in Jenkins: beam_PerformanceTests_Python #1289

[Apache Jenkins Server]

See 


Changes:

[xiliu] [BEAM-4849] Support running Beam Samza jobs in Yarn

[yifanzou] [BEAM-5084] upgrade fastjson

[amaliujia] test conditional operators and functions at DSL level

[amaliujia] allow test null value

[amaliujia] test aggregation functions at DSL level

[pablo] Step in cleanup of create_operation function

[aaltay] Several improvements for consistency of Python2 / Python3 conversions.

[lcwik] Suppressing the error stack docker image pull

[pablo] Add post-commit tests status to README.md in root

--
[...truncated 48.69 KB...]
copying apache_beam/runners/interactive/interactive_runner_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/interactive
copying apache_beam/runners/interactive/pipeline_graph.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/interactive
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/flink_runner_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/local_job_service.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/local_job_service_main.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/portable_runner.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/portable_runner_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/portable_stager.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/portable_stager_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/stager.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/stager_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.7.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pxd 

[beam-site] branch pabloem-patch-3 updated (9925d87 -> 4e8792f)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch pabloem-patch-3
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 9925d87  Blog post for Beam 2.6.0 release
 add 4e8792f  Added Rafael Fernandez as author.

No new revisions were added by this update.

Summary of changes:
 src/_data/authors.yml | 3 +++
 1 file changed, 3 insertions(+)

[beam-site] branch pabloem-patch-3 updated: Blog post for Beam 2.6.0 release

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch pabloem-patch-3
in repository https://gitbox.apache.org/repos/asf/beam-site.git


The following commit(s) were added to refs/heads/pabloem-patch-3 by this push:
 new 9925d87  Blog post for Beam 2.6.0 release
9925d87 is described below

commit 9925d87d98d8fb0379f485be2f77fcb65e25daa6
Author: Pablo 
AuthorDate: Fri Aug 10 16:23:43 2018 -0700

Blog post for Beam 2.6.0 release
---
 src/_posts/src/_posts/2018-08-10-beam-2.6.0.md | 58 ++
 1 file changed, 58 insertions(+)

diff --git a/src/_posts/src/_posts/2018-08-10-beam-2.6.0.md 
b/src/_posts/src/_posts/2018-08-10-beam-2.6.0.md
new file mode 100644
index 000..3211985
--- /dev/null
+++ b/src/_posts/src/_posts/2018-08-10-beam-2.6.0.md
@@ -0,0 +1,58 @@
+---
+layout: post
+title:  "Apache Beam 2.6.0"
+date:   2018-08-10 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+  - pabloem, rfernand
+---
+# Apache Beam version 2.6.0
+
+We are glad to present the new 2.6.0 release of Beam. This release includes 
multiple fixes and new functionalities. Specifically, we spent a significant 
amount of time trying to automate the release, and fix the continuous 
integration. We also have significant new features in SQL, and portability. 
Please go ahead and take a look at the following highlights:
+
+For more information, please, check the detailed [release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12343392
+).
+
+## New Features / Improvements
+
+### gRPC/Protobuf shading
+
+* gRPC/protobuf is now shaded in the majority of Apache Beam Java modules. A 
few modules which expose gRPC/protobuf on the API surface still maintain a 
direct dependency.
+
+### Beam SQL 
+
+* Support for the EXISTS and LIKE operators
+* Implemented SUM() aggregations
+* Fixes to CASE expression
+* Support for date comparison
+* Support LIMIT on Unbounded Data
+
+### Portability
+
+* Shared libraries for supporting timers and user state available for runner 
integration
+* A Universal Local Runner has been added to Beam. This runner runs in a 
single machine using portability, and containerized SDK harnesses.
+* Flink Runner now accepts jobs using the Job API
+
+### IOs
+
+* Bounded SplittableDoFn support is now available in all runners (Note SDF is 
the new I/O connector API)
+* HBaseIO is the first I/O supporting Bounded SDF (using readAll)
+* Miscellaneous Fixes
+
+### SDKs
+
+* Faster Python Avro IO
+* Python Avro IO has a use_fastavro option that will use fastavro instead of 
apache/avro, for a [3-6x 
speedup!](https://gist.github.com/ryan-williams/ede5ae61605e7ba6aa655071858ef52b)
+
+### Other
+
+* Updates to various dependency versions
+* Improvements on stability, performance, and documentation
+
+
+# List of Contributors
+According to git shortlog, the following 39 people contributed to the 2.6.0 
release. Thank you to all contributors!
+
+Ahmet Altay, 
+Alan Myrvold, Alexey Romanenko, Andrew Pilloud, Ankur Goenka, Boyuan Zhang, 
Charles Chen, cclauss, Daniel Oliveira, Elliott Brossard, Eric Beach, Etienne 
Chauchot, Eugene Kirpichov, Henning Rohde, Ismaël Mejía, Kai Jiang, Kasia, 
Kenneth Knowles, Luis Osa, Lukasz Cwik, Maria Garcia Herrero, Mark Liu, 
Matthias Feys, Pablo Estrada, Rafael Fernandez, Reuven Lax, Robert Bradshaw, 
Robert Burke, Robin Qiu, Ryan Williams, Scott Wegner, Rui Weng, Sergei Lebedev, 
Sindy Li, Thomas Weise, Udi Meiri, [...]

[beam-site] branch pabloem-patch-1 deleted (was e25ab8e)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch pabloem-patch-1
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 was e25ab8e  Adding 'editable mode' flag installation for Python.

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[beam-site] branch pabloem-patch-2 deleted (was 113519e)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch pabloem-patch-2
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 was 113519e  Updating downloads page.

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[jira] [Created] (BEAM-5131) Support ANY (sub query)

[Rui Wang (JIRA)]

Rui Wang created BEAM-5131:
--

 Summary: Support ANY (sub query)
 Key: BEAM-5131
 URL: https://issues.apache.org/jira/browse/BEAM-5131
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang


SELECT order_id FROM order WHERE item_id = ANY (SELECT item_id FROM 
special_item_table).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5130) Support ALL (sub query)

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5130:
---
Description: 
Example:

 

SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM 
special_item_table).

  was:
Example:

 

SELECT order_id FROM order WHERE item_id = ANY (SELECT item_id FROM 
special_item_table).


> Support ALL (sub query)
> ---
>
> Key: BEAM-5130
> URL: https://issues.apache.org/jira/browse/BEAM-5130
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> Example:
>  
> SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM 
> special_item_table).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5130) Support ALL (sub query)

[Rui Wang (JIRA)]

Rui Wang created BEAM-5130:
--

 Summary: Support ALL (sub query)
 Key: BEAM-5130
 URL: https://issues.apache.org/jira/browse/BEAM-5130
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang


Example:

 

SELECT order_id FROM order WHERE item_id = ANY (SELECT item_id FROM 
special_item_table).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5129) Support Rank() OVER

[Rui Wang (JIRA)]

Rui Wang created BEAM-5129:
--

 Summary: Support Rank() OVER
 Key: BEAM-5129
 URL: https://issues.apache.org/jira/browse/BEAM-5129
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5101) Improve Floor and Ceil tests at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5101?focusedWorklogId=133853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133853
 ]

ASF GitHub Bot logged work on BEAM-5101:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:53
Start Date: 10/Aug/18 22:53
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6207: [BEAM-5101][SQL] 
Improve tests of Floor and Ceil functions
URL: https://github.com/apache/beam/pull/6207#issuecomment-412226519
 
 
   For example, 
[code](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/math/BeamSqlCeilExpression.java#L41)
 shows the conversion from original type to `Double`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133853)
Time Spent: 20m  (was: 10m)

> Improve Floor and Ceil tests at DSL level
> -
>
> Key: BEAM-5101
> URL: https://issues.apache.org/jira/browse/BEAM-5101
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5101) Improve Floor and Ceil tests at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5101?focusedWorklogId=133854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133854
 ]

ASF GitHub Bot logged work on BEAM-5101:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:53
Start Date: 10/Aug/18 22:53
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6207: [BEAM-5101][SQL] 
Improve tests of Floor and Ceil functions
URL: https://github.com/apache/beam/pull/6207#issuecomment-412226546
 
 
   R: @akedin @apilloud 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133854)
Time Spent: 0.5h  (was: 20m)

> Improve Floor and Ceil tests at DSL level
> -
>
> Key: BEAM-5101
> URL: https://issues.apache.org/jira/browse/BEAM-5101
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5101) Improve Floor and Ceil tests at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5101?focusedWorklogId=133851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133851
 ]

ASF GitHub Bot logged work on BEAM-5101:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:50
Start Date: 10/Aug/18 22:50
Worklog Time Spent: 10m 
  Work Description: amaliujia opened a new pull request #6207: 
[BEAM-5101][SQL] Improve tests of Floor and Ceil functions
URL: https://github.com/apache/beam/pull/6207
 
 
   Improve  tests of Floor and Ceil functions. 
   
   
   There is a bug for Floor and Ceil where Floor and Ceil convert some 
primitive types (e.g. int) to Double but the code 
[BeamCalcRel](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L127)
 still generates a ROW with initial schema (e.g. int32), so Row verification 
will fail because of mismatch between INT32 type and Double value. Add a 
`@Ingore` with created JIRA link to the unit test.
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133851)
Time Spent: 10m
Remaining Estimate: 0h

> Improve Floor and Ceil 

[jira] [Closed] (BEAM-5103) Test aggregation functions at DSL levels

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5103.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5106) Test conditional functions at DSL level

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5106.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133844
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:35
Start Date: 10/Aug/18 22:35
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6204: [BEAM-5125] Increase 
timeout on PortableExecutionTest to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412223807
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133844)
Time Spent: 1h 10m  (was: 1h)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133845
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:35
Start Date: 10/Aug/18 22:35
Worklog Time Spent: 10m 
  Work Description: Ardagan removed a comment on issue #6204: [BEAM-5125] 
Increase timeout on PortableExecutionTest to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412223807
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133845)
Time Spent: 1h 20m  (was: 1h 10m)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133842=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133842
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:31
Start Date: 10/Aug/18 22:31
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6205: 
[BEAM-4374] Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#discussion_r209398946
 
 

 ##
 File path: sdks/python/apache_beam/metrics/monitoring_infos.py
 ##
 @@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# cython: language_level=3
+# cython: profile=True
+
+from __future__ import absolute_import
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.metrics.cells import DistributionData
+from apache_beam.metrics.cells import DistributionResult
+from apache_beam.metrics.cells import GaugeResult
+from apache_beam.metrics.cells import GaugeData
+from google.protobuf import timestamp_pb2
+
+import time
+
+USER_COUNTER_URN_PREFIX = 'beam:metric:user:'
+ELEMENT_COUNT_URN = 'beam:metric:element_count:v1'
+START_BUNDLE_MSECS_URN = (
+'beam:metric:pardo_execution_time:start_bundle_msecs:v1')
+PROCESS_BUNDLE_MSECS_URN = (
+'beam:metric:pardo_execution_time:process_bundle_msecs:v1')
+FINISH_BUNDLE_MSECS_URN = (
+'beam:metric:pardo_execution_time:finish_bundle_msecs:v1')
+TOTAL_MSECS_URN = (
+'beam:metric:ptransform_execution_time:total_msecs:v1')
+
+# TODO(ajamato): Implement the remaining types, i.e. Double types
+# Extrema types, etc. See:
+# https://s.apache.org/beam-fn-api-metrics
+SUM_INT64_TYPE = 'beam:metrics:SumInt64'
 
 Review comment:
   Can we stick with the existing case of snake_case instead of CamelCase for 
all URNs?
   
   Also, Eugene was able to move the URNs into proto form 
(https://github.com/apache/beam/pull/4672) to share across languages. Can we 
adopt the same now? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133842)
Time Spent: 1h  (was: 50m)

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5128) Fix Floor and Ceil functions

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang reassigned BEAM-5128:
--

Assignee: (was: Xu Mingmin)

> Fix Floor and Ceil functions
> 
>
> Key: BEAM-5128
> URL: https://issues.apache.org/jira/browse/BEAM-5128
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> Fool and Ceil converts non decimal type to double type no matter what the 
> field type is. So the Row will fail the code once it detects the mismatch 
> between field type and value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5128) Fix Floor and Ceil functions

[Rui Wang (JIRA)]

Rui Wang created BEAM-5128:
--

 Summary: Fix Floor and Ceil functions
 Key: BEAM-5128
 URL: https://issues.apache.org/jira/browse/BEAM-5128
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Xu Mingmin


Fool and Ceil converts non decimal type to double type no matter what the field 
type is. So the Row will fail the code once it detects the mismatch between 
field type and value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3786) Back FileSystem by VFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3786?focusedWorklogId=133839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133839
 ]

ASF GitHub Bot logged work on BEAM-3786:


Author: ASF GitHub Bot
Created on: 10/Aug/18 22:20
Start Date: 10/Aug/18 22:20
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #4803: [BEAM-3786] basic vfs 
filesystem boostrap
URL: https://github.com/apache/beam/pull/4803#issuecomment-412221397
 
 
   Staleness on this PR is totally in my camp. I will try to get it back to 
track in the following weeks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133839)
Time Spent: 50m  (was: 40m)

> Back FileSystem by VFS
> --
>
> Key: BEAM-3786
> URL: https://issues.apache.org/jira/browse/BEAM-3786
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> A ticket to not forget the discussion about reusing commons-vfs to benefit 
> from its API and connectivity under the hood and automatically behind beam 
> file[system|io] API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4823) Create SNS IO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-4823:
--

Assignee: Ankit Jhalaria  (was: Jean-Baptiste Onofré)

> Create SNS IO 
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133829
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:42
Start Date: 10/Aug/18 21:42
Worklog Time Spent: 10m 
  Work Description: ajamato commented on issue #6205: [BEAM-4374] 
Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#issuecomment-412214244
 
 
   @pabloem @aaltay 
   Would you mind doing an initial review for me. I just want to get some 
feedback on the overall structure of the changes I have made and the testing 
method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133829)
Time Spent: 50m  (was: 40m)

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133827
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:41
Start Date: 10/Aug/18 21:41
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412213959
 
 
   @pabloem FYI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133827)
Time Spent: 2h 50m  (was: 2h 40m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133828
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:41
Start Date: 10/Aug/18 21:41
Worklog Time Spent: 10m 
  Work Description: ajamato commented on a change in pull request #6205: 
[BEAM-4374] Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#discussion_r209391337
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
 ##
 @@ -1304,17 +1351,25 @@ def query(self, filter=None):
 
 
 class RunnerResult(runner.PipelineResult):
-  def __init__(self, state, metrics_by_stage):
+  def __init__(self, state, metrics_by_stage, monitoring_infos_by_stage):
 super(RunnerResult, self).__init__(state)
 self._metrics_by_stage = metrics_by_stage
+self._monitoring_infos_by_stage = monitoring_infos_by_stage
 self._user_metrics = None
 
   def wait_until_finish(self, duration=None):
 return self._state
 
-  def metrics(self):
+  def metrics(self, use_monitoring_infos=False, user_metrics_only=True):
 
 Review comment:
   This type of change, is this reasonable? Adding these extra options on 
pulling out ht metrrics. This is currently the only way we have to test 
metrics, and I added the user_metrics_only, to allow testing the system metrics.
   
   Note: We can update this code to only use use_monitoring_infos and delete 
all the legacy metrics once we are happy with this.  But I have made use of 
Metrics/MonitoringInfos optional for now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133828)
Time Spent: 40m  (was: 0.5h)

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133826
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:40
Start Date: 10/Aug/18 21:40
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6204: [BEAM-5125] Increase 
timeout on PortableExecutionTest to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412213833
 
 
   @pabloem


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133826)
Time Spent: 1h  (was: 50m)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4823) Create SNS IO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=133825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133825
 ]

ASF GitHub Bot logged work on BEAM-4823:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:39
Start Date: 10/Aug/18 21:39
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #6202: 
[BEAM-4823] - Adds a Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#discussion_r209390115
 
 

 ##
 File path: 
sdks/java/io/sns/src/main/java/org/apache/beam/sdk/io/sns/AwsClientsProvider.java
 ##
 @@ -0,0 +1,17 @@
+package org.apache.beam.sdk.io.sns;
 
 Review comment:
   Missing Apache headers in this and other files (that's the reason the build 
broke)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133825)
Time Spent: 20m  (was: 10m)

> Create SNS IO 
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133823
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:38
Start Date: 10/Aug/18 21:38
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6204: [BEAM-5125] Increase 
timeout on test to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412213369
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133823)
Time Spent: 50m  (was: 40m)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Add post-commit tests status to README.md in root

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 0f94528  Add post-commit tests status to README.md in root
0f94528 is described below

commit 0f9452823750179f740a2ef4fe143c7448c1e4c0
Author: Mikhail Gryzykhin 
AuthorDate: Fri Aug 10 14:32:16 2018 -0700

Add post-commit tests status to README.md in root
---
 README.md | 8 
 1 file changed, 8 insertions(+)

diff --git a/README.md b/README.md
index 16ab6d4..2f1e2a9 100644
--- a/README.md
+++ b/README.md
@@ -28,6 +28,14 @@
 [![Build 
Status](https://builds.apache.org/buildStatus/icon?job=beam_PostCommit_Java_GradleBuild)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild)
 [![Coverage 
Status](https://coveralls.io/repos/github/apache/beam/badge.svg?branch=master)](https://coveralls.io/github/apache/beam?branch=master)
 
+### Post-commit tests status (on master branch)
+Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
+--- | --- | --- | --- | --- | --- | --- | ---
+Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
+Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build Status](https://builds.apache.org/job/beam_PostC [...]
+Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompleted 
[...]
+
+
 ## Overview
 
 Beam provides a general approach to expressing [embarrassingly 
parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) data 
processing pipelines and supports three categories of users, each of which have 
relatively disparate backgrounds and needs.

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133822
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:35
Start Date: 10/Aug/18 21:35
Worklog Time Spent: 10m 
  Work Description: ajamato commented on a change in pull request #6205: 
[BEAM-4374] Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#discussion_r209390191
 
 

 ##
 File path: sdks/python/apache_beam/metrics/monitoring_infos.py
 ##
 @@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# cython: language_level=3
+# cython: profile=True
+
+from __future__ import absolute_import
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.metrics.cells import DistributionData
+from apache_beam.metrics.cells import DistributionResult
+from apache_beam.metrics.cells import GaugeResult
+from apache_beam.metrics.cells import GaugeData
+from google.protobuf import timestamp_pb2
+
+import time
 
 Review comment:
   I am curious about your overall thoughts on the use of monitoring_infos and 
metrics in comments and naming.
   
   Note: A Metric is a specific type of MonitoringInfo.
   See: https://s.apache.org/beam-fn-api-metrics


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133822)
Time Spent: 0.5h  (was: 20m)

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133821
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:35
Start Date: 10/Aug/18 21:35
Worklog Time Spent: 10m 
  Work Description: ajamato commented on a change in pull request #6205: 
[BEAM-4374] Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#discussion_r209390191
 
 

 ##
 File path: sdks/python/apache_beam/metrics/monitoring_infos.py
 ##
 @@ -0,0 +1,216 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# cython: language_level=3
+# cython: profile=True
+
+from __future__ import absolute_import
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.metrics.cells import DistributionData
+from apache_beam.metrics.cells import DistributionResult
+from apache_beam.metrics.cells import GaugeResult
+from apache_beam.metrics.cells import GaugeData
+from google.protobuf import timestamp_pb2
+
+import time
 
 Review comment:
   I am curious about your overall thoughts on the use of monitoring_infos and 
metrics in comments and naming.
   
   Note: A Metric is a specific type of MonitoringInfo.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133821)
Time Spent: 20m  (was: 10m)

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=133819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133819
 ]

ASF GitHub Bot logged work on BEAM-4374:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:33
Start Date: 10/Aug/18 21:33
Worklog Time Spent: 10m 
  Work Description: ajamato commented on a change in pull request #6205: 
[BEAM-4374] Implementing a subset of the new metrics framework in python.
URL: https://github.com/apache/beam/pull/6205#discussion_r209389982
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/bundle_processor.py
 ##
 @@ -313,6 +313,14 @@ def metrics(self):
 self._fix_output_tags(transform_id, op.progress_metrics())
 for transform_id, op in self.ops.items()})
 
+  def monitoring_infos(self):
 
 Review comment:
   Curious about reviewer thoughts on how to test this best. Would it make 
sense to test BundleProcessor on its own? It might be a bit complex to set this 
up, not sure if this is the kind of testing approach that we want for beam.
   
   Currently I have tested this indirectly via fn_api_runner_test (this is how 
the existing user metrics were being tested, the 'system' element count and 
processing time metrics were not tested at all :( )


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133819)
Time Spent: 10m
Remaining Estimate: 0h

> Update existing metrics in the FN API to use new Metric Schema
> --
>
> Key: BEAM-4374
> URL: https://issues.apache.org/jira/browse/BEAM-4374
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Alex Amato
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Update existing metrics to use the new proto and cataloging schema defined in:
> [_https://s.apache.org/beam-fn-api-metrics_]
>  * Check in new protos
>  * Define catalog file for metrics
>  * Port existing metrics to use this new format, based on catalog 
> names+metadata



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133818
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 21:29
Start Date: 10/Aug/18 21:29
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412211588
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133818)
Time Spent: 2h 40m  (was: 2.5h)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5127) [SQL] Avoid String parsing in BeamTableUtils

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-5127:
-

 Summary: [SQL] Avoid String parsing in BeamTableUtils
 Key: BEAM-5127
 URL: https://issues.apache.org/jira/browse/BEAM-5127
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Reporter: Anton Kedin


Looks like we're going through DateTime parsing on each projection: 

https://github.com/apache/beam/blob/9319ce18ac625b239c8cdc1b32e2b697d83c2504/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L125

https://github.com/apache/beam/blob/9ef5e8690871737cccfe513ac00481320f662c18/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java#L110

We should avoid unnecessary casting and parsing





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #776

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5084) Beam Dependency Update Request: com.alibaba:fastjson 1.2.49

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5084?focusedWorklogId=133806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133806
 ]

ASF GitHub Bot logged work on BEAM-5084:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:27
Start Date: 10/Aug/18 20:27
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6172: [BEAM-5084] upgrade 
fastjson
URL: https://github.com/apache/beam/pull/6172#issuecomment-412196662
 
 
   Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133806)
Time Spent: 40m  (was: 0.5h)

> Beam Dependency Update Request: com.alibaba:fastjson 1.2.49
> ---
>
> Key: BEAM-5084
> URL: https://issues.apache.org/jira/browse/BEAM-5084
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 2018-08-06 12:10:16.162201
> Please review and upgrade the com.alibaba:fastjson to the latest 
> version 1.2.49 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5084) Beam Dependency Update Request: com.alibaba:fastjson 1.2.49

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5084.
---
   Resolution: Fixed
Fix Version/s: 2.7.0

> Beam Dependency Update Request: com.alibaba:fastjson 1.2.49
> ---
>
> Key: BEAM-5084
> URL: https://issues.apache.org/jira/browse/BEAM-5084
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 2018-08-06 12:10:16.162201
> Please review and upgrade the com.alibaba:fastjson to the latest 
> version 1.2.49 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #775

[Apache Jenkins Server]

See 


Changes:

[aaltay] Several improvements for consistency of Python2 / Python3 conversions.

--
[...truncated 162.62 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was 

[beam] branch master updated: Suppressing the error stack docker image pull

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 9c148c5  Suppressing the error stack docker image pull
9c148c5 is described below

commit 9c148c5d68e8d193d2c3f101dc03380a0df4eb37
Author: Ankur 
AuthorDate: Fri Aug 10 13:08:06 2018 -0700

Suppressing the error stack docker image pull

This is a not fatal error so suppressing the errorstack for better log 
readability
---
 .../apache/beam/runners/fnexecution/environment/DockerCommand.java  | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
index e869aa3..3162fc8 100644
--- 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
@@ -77,7 +77,11 @@ class DockerCommand {
   runShortCommand(
   
ImmutableList.builder().add(dockerExecutable).add("pull").add(imageTag).build());
 } catch (IOException | TimeoutException | InterruptedException e) {
-  LOG.warn(String.format("Unable to pull docker image %s", imageTag), e);
+  if (LOG.isDebugEnabled()) {
+LOG.debug("Unable to pull docker image {}", imageTag, e);
+  } else {
+LOG.warn("Unable to pull docker image {}, cause: {}", imageTag, 
e.getMessage());
+  }
 }
 // TODO: Validate args?
 return runShortCommand(

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133797
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:06
Start Date: 10/Aug/18 20:06
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6174: [BEAM-5106][SQL]test 
conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174#issuecomment-412191599
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133797)
Time Spent: 4h  (was: 3h 50m)

> Test conditional functions at DSL level
> ---
>
> Key: BEAM-5106
> URL: https://issues.apache.org/jira/browse/BEAM-5106
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5106) Test conditional functions at DSL level

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5106?focusedWorklogId=133798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133798
 ]

ASF GitHub Bot logged work on BEAM-5106:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:06
Start Date: 10/Aug/18 20:06
Worklog Time Spent: 10m 
  Work Description: akedin closed pull request #6174: [BEAM-5106][SQL]test 
conditional operators and functions at DSL level
URL: https://github.com/apache/beam/pull/6174
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
index 201059e92b4..f969f734af0 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
@@ -43,6 +43,7 @@
 import java.util.stream.Collectors;
 import 
org.apache.beam.sdk.extensions.sql.integrationtest.BeamSqlBuiltinFunctionsIntegrationTestBase;
 import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
 import org.apache.calcite.runtime.SqlFunctions;
 import org.apache.calcite.sql.SqlKind;
 import org.apache.calcite.sql.SqlOperator;
@@ -1170,4 +1171,32 @@ public void testTimestampMinusInterval() throws 
Exception {
 parseDate("1983-01-19 01:01:58"));
 checker.buildRunAndCheck();
   }
+
+  @Test
+  @SqlOperatorTest(name = "CASE", kind = "CASE")
+  @SqlOperatorTest(name = "NULLIF", kind = "NULLIF")
+  @SqlOperatorTest(name = "COALESCE", kind = "COALESCE")
+  public void testConditionalOperatorsAndFunctions() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("CASE 1 WHEN 1 THEN 'hello' ELSE 'world' END", "hello")
+.addExpr(
+"CASE 2 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"world")
+.addExpr(
+"CASE 3 " + "WHEN 1 THEN 'hello' " + "WHEN 3 THEN 'bond' " + 
"ELSE 'world' END",
+"bond")
+.addExpr("CASE " + "WHEN 1 = 1 THEN 'hello' " + "ELSE 'world' 
END", "hello")
+.addExpr("CASE " + "WHEN 1 > 1 THEN 'hello' " + "ELSE 'world' 
END", "world")
+.addExpr("NULLIF(5, 4) ", 5)
+.addExpr("NULLIF(4, 5) ", 4)
+.addExpr("NULLIF(5, 5)", null, FieldType.INT32)
+.addExpr("COALESCE(1, 5) ", 1)
+.addExpr("COALESCE(NULL, 5) ", 5)
+.addExpr("COALESCE(NULL, 4, 5) ", 4)
+.addExpr("COALESCE(NULL, NULL, 5) ", 5)
+.addExpr("COALESCE(5, NULL) ", 5);
+
+checker.buildRunAndCheck();
+  }
 }
diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
index 4380b0b7241..8ba5374ec8b 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
@@ -30,6 +30,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.extensions.sql.SqlTransform;
 import org.apache.beam.sdk.extensions.sql.TestUtils;
@@ -128,6 +129,7 @@ private static ExpressionTestCase of(
 
 abstract String sqlExpr();
 
+@Nullable
 abstract Object expectedResult();
 
 abstract FieldType resultFieldType();
@@ -176,7 +178,13 @@ public void buildRunAndCheck() {
 String expression = testCase.sqlExpr();
 Object expectedValue = testCase.expectedResult();
 String sql = String.format("SELECT %s FROM PCOLLECTION", expression);
-Schema schema = Schema.builder().addField(expression, 
testCase.resultFieldType()).build();
+Schema schema;
+if (expectedValue == null) {
+  schema =
+  Schema.builder().addNullableField(expression, 
testCase.resultFieldType()).build();
+} else {
+  schema = Schema.builder().addField(expression, 
testCase.resultFieldType()).build();
+}
 
 PCollection output =
 

[beam] 01/01: Merge pull request #6174 from amaliujia/rui_wang-test_conditional

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a5e88c92b35de7e52cfa57679e4ee27830e47a96
Merge: 26bbee5 9d6f892
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Fri Aug 10 13:06:40 2018 -0700

Merge pull request #6174 from amaliujia/rui_wang-test_conditional

[BEAM-5106][SQL]test conditional operators and functions at DSL level

 .../sql/BeamSqlDslSqlStdOperatorsTest.java | 29 +++
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 10 +-
 ...BeamSqlConditionalFunctionsIntegrationTest.java | 41 --
 3 files changed, 38 insertions(+), 42 deletions(-)

[beam] branch master updated (26bbee5 -> a5e88c9)

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 26bbee5  Merge pull request #6175 from 
amaliujia/rui_wang-test_aggregation_func_at_dsl_level
 add 4968515  test conditional operators and functions at DSL level
 add 9d6f892  allow test null value
 new a5e88c9  Merge pull request #6174 from 
amaliujia/rui_wang-test_conditional

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/BeamSqlDslSqlStdOperatorsTest.java | 29 +++
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 10 +-
 ...BeamSqlConditionalFunctionsIntegrationTest.java | 41 --
 3 files changed, 38 insertions(+), 42 deletions(-)
 delete mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlConditionalFunctionsIntegrationTest.java

[beam] branch master updated (b0b56a7 -> 26bbee5)

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b0b56a7  Merge pull request #6172 from 
yifanzou/BEAM-5084/upgrade_com_alibaba_fastjson
 add b924b01  test aggregation functions at DSL level
 new 26bbee5  Merge pull request #6175 from 
amaliujia/rui_wang-test_aggregation_func_at_dsl_level

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/BeamSqlDslSqlStdOperatorsTest.java | 127 +
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  89 +--
 .../beam/sdk/extensions/sql/utils/RowAsserts.java  |  11 ++
 3 files changed, 220 insertions(+), 7 deletions(-)

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133796
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:05
Start Date: 10/Aug/18 20:05
Worklog Time Spent: 10m 
  Work Description: akedin closed pull request #6175: [BEAM-5103][SQL]test 
aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
index 201059e92b4..7b035984b5d 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
@@ -568,6 +568,133 @@ public void testComparisonOperatorFunction() {
 checker.buildRunAndCheck();
   }
 
+  @Test
+  @SqlOperatorTest(name = "MAX", kind = "MAX")
+  public void testMax() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("MAX(c_tinyint)", (byte) 3)
+.addExpr("MAX(c_smallint)", (short) 3)
+.addExpr("MAX(c_integer)", 3)
+.addExpr("MAX(c_bigint)", 3L)
+.addExpr("MAX(c_float)", 3.0f)
+.addExpr("MAX(c_double)", 3.0)
+.addExpr("MAX(c_decimal)", BigDecimal.valueOf(3.0))
+.addExpr("MAX(ts)", parseDate("1986-04-15 11:35:26"));
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Test
+  @SqlOperatorTest(name = "MIN", kind = "MIN")
+  public void testMin() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("MIN(c_tinyint)", (byte) 1)
+.addExpr("MIN(c_smallint)", (short) 1)
+.addExpr("MIN(c_integer)", 1)
+.addExpr("MIN(c_bigint)", 1L)
+.addExpr("MIN(c_float)", 1.0f)
+.addExpr("MIN(c_double)", 1.0)
+.addExpr("MIN(c_decimal)", BigDecimal.valueOf(1.0))
+.addExpr("MIN(ts)", parseDate("1986-02-15 11:35:26"));
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Test
+  @SqlOperatorTest(name = "SUM", kind = "SUM")
+  public void testSum() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("SUM(c_tinyint)", (byte) 6)
+.addExpr("SUM(c_smallint)", (short) 6)
+.addExpr("SUM(c_integer)", 6)
+.addExpr("SUM(c_bigint)", 6L)
+.addExpr("SUM(c_float)", 6.0f)
+.addExpr("SUM(c_double)", 6.0)
+.addExpr("SUM(c_decimal)", BigDecimal.valueOf(6.0));
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Test
+  @SqlOperatorTest(name = "AVG", kind = "AVG")
+  public void testAvg() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("AVG(c_tinyint)", (byte) 2)
+.addExpr("AVG(c_smallint)", (short) 2)
+.addExpr("AVG(c_integer)", 2)
+.addExpr("AVG(c_bigint)", 2L)
+.addExpr("AVG(c_float)", 2.0f)
+.addExpr("AVG(c_double)", 2.0)
+.addExpr("AVG(c_decimal)", BigDecimal.valueOf(2.0));
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Ignore("https://issues.apache.org/jira/browse/BEAM-5111;)
+  @Test
+  @SqlOperatorTest(name = "$SUM0", kind = "SUM0")
+  public void testSUM0() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("$SUM0(c_tinyint)", (byte) 6)
+.addExpr("$SUM0(c_smallint)", (short) 6)
+.addExpr("$SUM0(c_integer)", 6)
+.addExpr("$SUM0(c_bigint)", 6L)
+.addExpr("$SUM0(c_float)", 6.0f)
+.addExpr("$SUM0(c_double)", 6.0)
+.addExpr("$SUM0(c_decimal)", BigDecimal.valueOf(6.0));
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Test
+  @SqlOperatorTest(name = "COUNT", kind = "COUNT")
+  public void testCount() {
+ExpressionChecker checker =
+new ExpressionChecker().addExpr("COUNT(*)", 4L).addExpr("COUNT(1)", 
4L);
+checker.buildRunAndCheck(getAggregationTestPCollection());
+  }
+
+  @Test
+  @SqlOperatorTest(name = "VAR_POP", kind = "VAR_POP")
+  public void testVARPOP() {
+ExpressionChecker checker =
+new ExpressionChecker()
+.addExpr("VAR_POP(c_integer)", 0)
+

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133795
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:05
Start Date: 10/Aug/18 20:05
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6175: [BEAM-5103][SQL]test 
aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-412191368
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133795)
Time Spent: 5.5h  (was: 5h 20m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6175 from amaliujia/rui_wang-test_aggregation_func_at_dsl_level

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 26bbee515a41a004de07e9b60c8b9689d61bbd0b
Merge: b0b56a7 b924b01
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Fri Aug 10 13:05:55 2018 -0700

Merge pull request #6175 from 
amaliujia/rui_wang-test_aggregation_func_at_dsl_level

[BEAM-5103][SQL]test aggregation functions at DSL level

 .../sql/BeamSqlDslSqlStdOperatorsTest.java | 127 +
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  89 +--
 .../beam/sdk/extensions/sql/utils/RowAsserts.java  |  11 ++
 3 files changed, 220 insertions(+), 7 deletions(-)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133793
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:03
Start Date: 10/Aug/18 20:03
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6204: [BEAM-5125] Increase 
timeout on test to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412190819
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133793)
Time Spent: 0.5h  (was: 20m)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5084) Beam Dependency Update Request: com.alibaba:fastjson 1.2.49

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5084?focusedWorklogId=133792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133792
 ]

ASF GitHub Bot logged work on BEAM-5084:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:03
Start Date: 10/Aug/18 20:03
Worklog Time Spent: 10m 
  Work Description: akedin closed pull request #6172: [BEAM-5084] upgrade 
fastjson
URL: https://github.com/apache/beam/pull/6172
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/extensions/sql/build.gradle 
b/sdks/java/extensions/sql/build.gradle
index 55c9d6663f3..1d5d4d27c3c 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -74,7 +74,7 @@ dependencies {
   shadow library.java.commons_lang3
   shadow library.java.jackson_databind
   shadow library.java.joda_time
-  shadow "com.alibaba:fastjson:1.2.12"
+  shadow "com.alibaba:fastjson:1.2.49"
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-kafka", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133792)
Time Spent: 0.5h  (was: 20m)

> Beam Dependency Update Request: com.alibaba:fastjson 1.2.49
> ---
>
> Key: BEAM-5084
> URL: https://issues.apache.org/jira/browse/BEAM-5084
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 2018-08-06 12:10:16.162201
> Please review and upgrade the com.alibaba:fastjson to the latest 
> version 1.2.49 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5084) Beam Dependency Update Request: com.alibaba:fastjson 1.2.49

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5084?focusedWorklogId=133791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133791
 ]

ASF GitHub Bot logged work on BEAM-5084:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:03
Start Date: 10/Aug/18 20:03
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6172: [BEAM-5084] upgrade 
fastjson
URL: https://github.com/apache/beam/pull/6172#issuecomment-412190673
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133791)
Time Spent: 20m  (was: 10m)

> Beam Dependency Update Request: com.alibaba:fastjson 1.2.49
> ---
>
> Key: BEAM-5084
> URL: https://issues.apache.org/jira/browse/BEAM-5084
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 2018-08-06 12:10:16.162201
> Please review and upgrade the com.alibaba:fastjson to the latest 
> version 1.2.49 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133794
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 20:03
Start Date: 10/Aug/18 20:03
Worklog Time Spent: 10m 
  Work Description: Ardagan removed a comment on issue #6204: [BEAM-5125] 
Increase timeout on test to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412190819
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133794)
Time Spent: 40m  (was: 0.5h)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (c7ce77d -> b0b56a7)

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c7ce77d  Several improvements for consistency of Python2 / Python3 
conversions. (#6144)
 add 2c12964  [BEAM-5084] upgrade fastjson
 new b0b56a7  Merge pull request #6172 from 
yifanzou/BEAM-5084/upgrade_com_alibaba_fastjson

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sql/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] 01/01: Merge pull request #6172 from yifanzou/BEAM-5084/upgrade_com_alibaba_fastjson

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b0b56a75875e08a94c98fd0ae360e77b72c13808
Merge: c7ce77d 2c12964
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Fri Aug 10 13:03:11 2018 -0700

Merge pull request #6172 from 
yifanzou/BEAM-5084/upgrade_com_alibaba_fastjson

[BEAM-5084] upgrade fastjson

 sdks/java/extensions/sql/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #5717

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-5126) PreCommit filtering broken based upon PR contents

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5126:
---

 Summary: PreCommit filtering broken based upon PR contents
 Key: BEAM-5126
 URL: https://issues.apache.org/jira/browse/BEAM-5126
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Luke Cwik
Assignee: Alan Myrvold


PR precommits used to be filtered by the contents of the PR.

 

Example PR that should have only spawned the Java PreCommit:

https://github.com/apache/beam/pull/6159

 

This broke work done in BEAM-4445



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #774

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5103) Test aggregation functions at DSL levels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5103?focusedWorklogId=133783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133783
 ]

ASF GitHub Bot logged work on BEAM-5103:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:42
Start Date: 10/Aug/18 19:42
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6175: 
[BEAM-5103][SQL]test aggregation functions at DSL level
URL: https://github.com/apache/beam/pull/6175#issuecomment-412185490
 
 
   Ping


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133783)
Time Spent: 5h 20m  (was: 5h 10m)

> Test aggregation functions at DSL levels
> 
>
> Key: BEAM-5103
> URL: https://issues.apache.org/jira/browse/BEAM-5103
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Typical aggregation functions include COUNT, SUM, MAX, MIN, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1225

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4475) Go precommit should include "go test ./..."

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4475?focusedWorklogId=133780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133780
 ]

ASF GitHub Bot logged work on BEAM-4475:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:34
Start Date: 10/Aug/18 19:34
Worklog Time Spent: 10m 
  Work Description: herohde closed pull request #6096: [BEAM-4475] Include 
Go regression tests in precommit
URL: https://github.com/apache/beam/pull/6096
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index d98ce9b3d71..cec9da7a0cd 100644
--- a/build.gradle
+++ b/build.gradle
@@ -196,9 +196,9 @@ task javaPostCommit() {
 task goPreCommit() {
   dependsOn ":rat"
   dependsOn ":beam-sdks-go:test"
+  dependsOn ":beam-sdks-go-test:test"
 
   dependsOn ":beam-sdks-go-examples:build"
-  dependsOn ":beam-sdks-go-test:build"
 
   // Ensure all container Go boot code builds as well.
   dependsOn ":beam-sdks-java-container:build"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133780)
Time Spent: 50m  (was: 40m)

> Go precommit should include "go test ./..."
> ---
>
> Key: BEAM-4475
> URL: https://issues.apache.org/jira/browse/BEAM-4475
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
> Fix For: 2.6.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> It would have prevented a recent break caused by a green PR: 
> https://github.com/apache/beam/pull/5558.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133781
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:34
Start Date: 10/Aug/18 19:34
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412183716
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133781)
Time Spent: 2.5h  (was: 2h 20m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133778
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:31
Start Date: 10/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin removed a comment on issue #6201: [BEAM-5122] 
Increase timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412168472
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133778)
Time Spent: 2h 10m  (was: 2h)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133777
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:31
Start Date: 10/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin removed a comment on issue #6201: [BEAM-5122] 
Increase timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412159497
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133777)
Time Spent: 2h  (was: 1h 50m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133776
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:31
Start Date: 10/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin removed a comment on issue #6201: [BEAM-5122] 
Increase timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412158140
 
 
   run java precommit
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133776)
Time Spent: 1h 50m  (was: 1h 40m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133775
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:31
Start Date: 10/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin removed a comment on issue #6201: [BEAM-5122] 
Increase timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412146432
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133775)
Time Spent: 1h 40m  (was: 1.5h)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133779
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:31
Start Date: 10/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin removed a comment on issue #6201: [BEAM-5122] 
Increase timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412168496
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133779)
Time Spent: 2h 20m  (was: 2h 10m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Several improvements for consistency of Python2 / Python3 conversions. (#6144)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new c7ce77d  Several improvements for consistency of Python2 / Python3 
conversions. (#6144)
c7ce77d is described below

commit c7ce77de32c99e7d4fdacfeae5709d8cdd8b8838
Author: tvalentyn 
AuthorDate: Fri Aug 10 12:24:33 2018 -0700

Several improvements for consistency of Python2 / Python3 conversions. 
(#6144)

* Use (str, past.builtins.unicode) where six.string_types was used in the 
past.

* Use past.builtins.unicode where six.text_type was used in the past.

* For Py2/Py3 compatibility, add complimentary __ne__ methods which are 
autogenerated in Py3.

* Use bytes where six.binary_type was used in the past.
---
 sdks/python/apache_beam/io/gcp/pubsub.py   | 15 +++
 sdks/python/apache_beam/io/gcp/pubsub_test.py  |  4 ++--
 sdks/python/apache_beam/runners/common.py  |  6 +++---
 sdks/python/apache_beam/testing/test_stream.py |  9 +
 4 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/sdks/python/apache_beam/io/gcp/pubsub.py 
b/sdks/python/apache_beam/io/gcp/pubsub.py
index d5c3f99..2414194 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub.py
@@ -27,8 +27,7 @@ from __future__ import absolute_import
 import re
 from builtins import object
 
-import six
-from past.builtins import basestring
+from past.builtins import unicode
 
 from apache_beam import coders
 from apache_beam.io.iobase import Read
@@ -58,7 +57,7 @@ class PubsubMessage(object):
   This interface is experimental. No backwards compatibility guarantees.
 
   Attributes:
-data: (six.binary_type) Message data. May be None.
+data: (bytes) Message data. May be None.
 attributes: (dict) Key-value map of str to str, containing both 
user-defined
   and service generated attributes (such as id_label and
   timestamp_attribute). May be None.
@@ -151,7 +150,7 @@ class ReadFromPubSub(PTransform):
 case, deduplication of the stream will be strictly best effort.
   with_attributes:
 True - output elements will be :class:`~PubsubMessage` objects.
-False - output elements will be of type ``six.binary_type`` (message
+False - output elements will be of type ``bytes`` (message
 data only).
   timestamp_attribute: Message value to use as element timestamp. If None,
 uses message publishing time as the timestamp.
@@ -176,7 +175,7 @@ class ReadFromPubSub(PTransform):
 
   def expand(self, pvalue):
 pcoll = pvalue.pipeline | Read(self._source)
-pcoll.element_type = six.binary_type
+pcoll.element_type = bytes
 if self.with_attributes:
   pcoll = pcoll | Map(PubsubMessage._from_proto_str)
   pcoll.element_type = PubsubMessage
@@ -207,7 +206,7 @@ class _ReadStringsFromPubSub(PTransform):
  | ReadFromPubSub(self.topic, self.subscription, self.id_label,
   with_attributes=False)
  | 'DecodeString' >> Map(lambda b: b.decode('utf-8')))
-p.element_type = basestring
+p.element_type = unicode
 return p
 
 
@@ -247,7 +246,7 @@ class WriteToPubSub(PTransform):
   topic: Cloud Pub/Sub topic in the form "/topics//".
   with_attributes:
 True - input elements will be :class:`~PubsubMessage` objects.
-False - input elements will be of type ``six.binary_type`` (message
+False - input elements will be of type ``bytes`` (message
 data only).
   id_label: If set, will set an attribute for each Cloud Pub/Sub message
 with the given name and a unique value. This attribute can then be used
@@ -276,7 +275,7 @@ class WriteToPubSub(PTransform):
 # Without attributes, message data is written as-is. With attributes,
 # message data + attributes are passed as a serialized protobuf string (see
 # ``PubsubMessage._to_proto_str`` for exact protobuf message type).
-pcoll.element_type = six.binary_type
+pcoll.element_type = bytes
 return pcoll | Write(self._sink)
 
   def to_runner_api_parameter(self, context):
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_test.py 
b/sdks/python/apache_beam/io/gcp/pubsub_test.py
index 914b72a..44024cd 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_test.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_test.py
@@ -127,7 +127,7 @@ class TestReadFromPubSubOverride(unittest.TestCase):
   None, 'a_label', with_attributes=False,
   timestamp_attribute=None)
  | beam.Map(lambda x: x))
-self.assertEqual(str, pcoll.element_type)
+self.assertEqual(bytes, pcoll.element_type)
 
 # Apply the necessary PTransformOverrides.
 overrides = _get_transform_overrides(p.options)
@@ -150,7 +150,7 @@ class 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #773

[Apache Jenkins Server]

See 


--
[...truncated 166.61 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.0.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5716

[Apache Jenkins Server]

See 


--
[...truncated 1.09 MB...]
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_invalid_ellipsis_type_param 

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133768
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 19:06
Start Date: 10/Aug/18 19:06
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412177149
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133768)
Time Spent: 1.5h  (was: 1h 20m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133765
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:58
Start Date: 10/Aug/18 18:58
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6204: [BEAM-5125] Increase 
timeout on test to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204#issuecomment-412175117
 
 
   R: @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133765)
Time Spent: 20m  (was: 10m)

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Test fails in both: post and precommit tests. Fails more often in pre-commits.
> Pre-commit history: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]
> Post-commit history: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]
> Sample job:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/
> Log:
> java.lang.AssertionError: job state expected: but was: at 
> org.junit.Assert.fail(Assert.java:88) at 
> org.junit.Assert.failNotEquals(Assert.java:834) at 
> org.junit.Assert.assertEquals(Assert.java:118) at 
> org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5125?focusedWorklogId=133764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133764
 ]

ASF GitHub Bot logged work on BEAM-5125:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:57
Start Date: 10/Aug/18 18:57
Worklog Time Spent: 10m 
  Work Description: Ardagan opened a new pull request #6204: [BEAM-5125] 
Increase timeout on test to reduce flakiness.
URL: https://github.com/apache/beam/pull/6204
 
 
   Test is failing often in pre-commit tests and post-commit tests. This looks 
like failing due to timeout. Setting less strict timeout might resolve 
flakiness.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133764)
Time Spent: 10m
Remaining Estimate: 0h

> beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink 
> PortableExecutionTest testExecution_1_ flaky
> ---
>
> Key: BEAM-5125
> URL: https://issues.apache.org/jira/browse/BEAM-5125
> Project: Beam
>  Issue Type: Bug
>  Components: 

[jira] [Work logged] (BEAM-5002) Beam Dependency Update Request: org.slf4j

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5002?focusedWorklogId=133763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133763
 ]

ASF GitHub Bot logged work on BEAM-5002:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:50
Start Date: 10/Aug/18 18:50
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6203: Do Not Merge 
[BEAM-5002] upgrade slf4j
URL: https://github.com/apache/beam/pull/6203#issuecomment-412173031
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133763)
Time Spent: 0.5h  (was: 20m)

> Beam Dependency Update Request: org.slf4j
> -
>
> Key: BEAM-5002
> URL: https://issues.apache.org/jira/browse/BEAM-5002
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 2018-07-25 20:33:53.630864
> Please review and upgrade the org.slf4j to the latest version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1224

[Apache Jenkins Server]

See 


--
[...truncated 19.76 MB...]
INFO: 2018-08-10T18:44:12.075Z: Autoscaling was automatically enabled for 
job 2018-08-10_11_44_11-14621371131318998056.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:14.733Z: Checking required Cloud APIs are enabled.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:14.929Z: Checking permissions granted to controller 
Service Account.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:18.968Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:19.544Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:19.829Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:19.878Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.160Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.208Z: Elided trivial flatten 
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.256Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.305Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.353Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.402Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.437Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.479Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-10T18:44:20.518Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Aug 10, 2018 6:44:25 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler 

[jira] [Created] (BEAM-5125) beam_PostCommit_Java_GradleBuild org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky

[Mikhail Gryzykhin (JIRA)]

Mikhail Gryzykhin created BEAM-5125:
---

 Summary: beam_PostCommit_Java_GradleBuild 
org.apache.beam.runners.flink PortableExecutionTest testExecution_1_ flaky
 Key: BEAM-5125
 URL: https://issues.apache.org/jira/browse/BEAM-5125
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Mikhail Gryzykhin


Test fails in both: post and precommit tests. Fails more often in pre-commits.

Pre-commit history: 
[https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/]

Post-commit history: 
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1223/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/history/?start=75]

Sample job:

https://builds.apache.org/job/beam_PreCommit_Java_Phrase/180/testReport/junit/org.apache.beam.runners.flink/PortableExecutionTest/testExecution_1_/

Log:

java.lang.AssertionError: job state expected: but was: at 
org.junit.Assert.fail(Assert.java:88) at 
org.junit.Assert.failNotEquals(Assert.java:834) at 
org.junit.Assert.assertEquals(Assert.java:118) at 
org.apache.beam.runners.flink.PortableExecutionTest.testExecution(PortableExecutionTest.java:177)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4849) Support running Beam Samza jobs in Yarn

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4849.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (b532b38 -> 95f611c)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b532b38  Step in cleanup of create_operation function
 add 3c75e6c  [BEAM-4849] Support running Beam Samza jobs in Yarn
 new 95f611c  [BEAM-4849] Support running Beam Samza jobs in Yarn

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../samza/container/BeamContainerRunner.java   | 24 --
 .../samza/container/ContainerCfgFactory.java   | 55 ++
 .../samza/{ => container}/package-info.java|  2 +-
 3 files changed, 66 insertions(+), 15 deletions(-)
 copy sdks/java/core/src/main/java/org/apache/beam/sdk/state/ValueState.java => 
runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamContainerRunner.java
 (65%)
 create mode 100644 
runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgFactory.java
 copy runners/samza/src/main/java/org/apache/beam/runners/samza/{ => 
container}/package-info.java (94%)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=133759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133759
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:39
Start Date: 10/Aug/18 18:39
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6029: [BEAM-4849] Support 
running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029#issuecomment-412170285
 
 
   Sorry for the delay, was out on vacation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133759)
Time Spent: 1h 20m  (was: 1h 10m)

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4849) Support running Beam Samza jobs in Yarn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?focusedWorklogId=133760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133760
 ]

ASF GitHub Bot logged work on BEAM-4849:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:39
Start Date: 10/Aug/18 18:39
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6029: [BEAM-4849] Support 
running Beam Samza jobs in Yarn
URL: https://github.com/apache/beam/pull/6029
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamContainerRunner.java
 
b/runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamContainerRunner.java
new file mode 100644
index 000..bdc135f4034
--- /dev/null
+++ 
b/runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamContainerRunner.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.samza.container;
+
+import org.apache.samza.config.Config;
+import org.apache.samza.config.ShellCommandConfig;
+import org.apache.samza.runtime.LocalContainerRunner;
+
+/** Runs the beam Yarn container, using the static global job model. */
+public class BeamContainerRunner extends LocalContainerRunner {
+
+  public BeamContainerRunner(Config config) {
+super(ContainerCfgFactory.jobModel, 
System.getenv(ShellCommandConfig.ENV_CONTAINER_ID()));
+  }
+}
diff --git 
a/runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgFactory.java
 
b/runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgFactory.java
new file mode 100644
index 000..751441ea73b
--- /dev/null
+++ 
b/runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgFactory.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.samza.container;
+
+import java.net.URI;
+import java.util.Random;
+import org.apache.samza.config.Config;
+import org.apache.samza.config.ConfigFactory;
+import org.apache.samza.config.ShellCommandConfig;
+import org.apache.samza.container.SamzaContainer;
+import org.apache.samza.job.model.JobModel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Factory for the Beam yarn container to load job model. */
+public class ContainerCfgFactory implements ConfigFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(ContainerCfgFactory.class);
+
+  private static final Object LOCK = new Object();
+  static volatile JobModel jobModel;
+
+  @Override
+  public Config getConfig(URI configUri) {
+if (jobModel == null) {
+  synchronized (LOCK) {
+if (jobModel == null) {
+  String containerId = 
System.getenv(ShellCommandConfig.ENV_CONTAINER_ID());
+  LOG.info(String.format("Got container ID: %s", containerId));
+  String coordinatorUrl = 
System.getenv(ShellCommandConfig.ENV_COORDINATOR_URL());
+  LOG.info(String.format("Got coordinator URL: %s", coordinatorUrl));
+  int delay = new 

[beam] 01/01: [BEAM-4849] Support running Beam Samza jobs in Yarn

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 95f611c27f4b338ab344d25e529993db2cb59915
Merge: b532b38 3c75e6c
Author: Lukasz Cwik 
AuthorDate: Fri Aug 10 11:39:48 2018 -0700

[BEAM-4849] Support running Beam Samza jobs in Yarn

 .../samza/container/BeamContainerRunner.java   | 31 
 .../samza/container/ContainerCfgFactory.java   | 55 ++
 .../beam/runners/samza/container/package-info.java | 20 
 3 files changed, 106 insertions(+)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=133756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133756
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:36
Start Date: 10/Aug/18 18:36
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6123: [BEAM-4028] Step in 
cleanup of create_operation function
URL: https://github.com/apache/beam/pull/6123
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/worker/operations.py 
b/sdks/python/apache_beam/runners/worker/operations.py
index ac2780a5d87..0488fe928d3 100644
--- a/sdks/python/apache_beam/runners/worker/operations.py
+++ b/sdks/python/apache_beam/runners/worker/operations.py
@@ -572,15 +572,14 @@ def process(self, o):
   self.output(o)
 
 
-def create_operation(name_context, spec, counter_factory, step_name,
- state_sampler, test_shuffle_source=None,
+def create_operation(name_context, spec, counter_factory, step_name=None,
+ state_sampler=None, test_shuffle_source=None,
  test_shuffle_sink=None, is_streaming=False):
   """Create Operation object for given operation specification."""
+
+  # TODO(pabloem): Document arguments to this function call.
   if not isinstance(name_context, common.NameContext):
-# TODO(BEAM-4028): Remove ad-hoc NameContext once all has been migrated.
-name_context = common.DataflowNameContext(step_name=name_context,
-  user_name=step_name,
-  system_name=None)
+name_context = common.NameContext(step_name=name_context)
 
   if isinstance(spec, operation_specs.WorkerRead):
 if isinstance(spec.source, iobase.SourceBundle):


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133756)
Time Spent: 5h 40m  (was: 5.5h)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Step in cleanup of create_operation function

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new b532b38  Step in cleanup of create_operation function
b532b38 is described below

commit b532b38958527529bf561c92d34b1f1230213395
Author: Pablo 
AuthorDate: Thu Aug 2 10:56:34 2018 -0700

Step in cleanup of create_operation function
---
 sdks/python/apache_beam/runners/worker/operations.py | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sdks/python/apache_beam/runners/worker/operations.py 
b/sdks/python/apache_beam/runners/worker/operations.py
index ac2780a..0488fe9 100644
--- a/sdks/python/apache_beam/runners/worker/operations.py
+++ b/sdks/python/apache_beam/runners/worker/operations.py
@@ -572,15 +572,14 @@ class FlattenOperation(Operation):
   self.output(o)
 
 
-def create_operation(name_context, spec, counter_factory, step_name,
- state_sampler, test_shuffle_source=None,
+def create_operation(name_context, spec, counter_factory, step_name=None,
+ state_sampler=None, test_shuffle_source=None,
  test_shuffle_sink=None, is_streaming=False):
   """Create Operation object for given operation specification."""
+
+  # TODO(pabloem): Document arguments to this function call.
   if not isinstance(name_context, common.NameContext):
-# TODO(BEAM-4028): Remove ad-hoc NameContext once all has been migrated.
-name_context = common.DataflowNameContext(step_name=name_context,
-  user_name=step_name,
-  system_name=None)
+name_context = common.NameContext(step_name=name_context)
 
   if isinstance(spec, operation_specs.WorkerRead):
 if isinstance(spec.source, iobase.SourceBundle):

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=133755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133755
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:35
Start Date: 10/Aug/18 18:35
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6123: [BEAM-4028] 
Step in cleanup of create_operation function
URL: https://github.com/apache/beam/pull/6123#issuecomment-412169242
 
 
   Thanks! This LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133755)
Time Spent: 5.5h  (was: 5h 20m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133753
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:32
Start Date: 10/Aug/18 18:32
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412168496
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133753)
Time Spent: 1h 20m  (was: 1h 10m)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?focusedWorklogId=133752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133752
 ]

ASF GitHub Bot logged work on BEAM-5122:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:32
Start Date: 10/Aug/18 18:32
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #6201: [BEAM-5122] Increase 
timeout in PubsubJsonIT:testUsesDlq to try workaround pubsub latency.
URL: https://github.com/apache/beam/pull/6201#issuecomment-412168472
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133752)
Time Spent: 1h 10m  (was: 1h)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #210

[Apache Jenkins Server]

See 


Changes:

[aljoscha.krettek] [BEAM-4798] Fix IndexOutOfBoundsException when Flink 
parallelism is more

[aljoscha.krettek] [BEAM-4798] Fix UnboundedSourceWrapperTest

--
[...truncated 17.62 MB...]
  LogicalJoin(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner])
LogicalProject(auction=[$0], price=[$1], bidder=[$2], dateTime=[$3], 
extra=[$4], starttime=[$5])
  LogicalAggregate(group=[{0, 1, 2, 3, 4, 5}])
LogicalProject(auction=[$0], price=[$2], bidder=[$1], 
dateTime=[$3], extra=[$4], $f5=[TUMBLE($3, 1)])
  BeamIOSourceRel(table=[[beam, Bid]])
LogicalProject(maxprice=[$1], starttime=[$0])
  LogicalAggregate(group=[{0}], maxprice=[MAX($1)])
LogicalProject($f0=[TUMBLE($3, 1)], price=[$2])
  BeamIOSourceRel(table=[[beam, Bid]])

Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..7=[{inputs}], proj#0..4=[{exprs}])
  BeamJoinRel(condition=[AND(=($5, $7), =($1, $6))], joinType=[inner])
BeamCalcRel(expr#0..5=[{inputs}], proj#0..5=[{exprs}])
  BeamAggregationRel(group=[{0, 1, 2, 3, 4, 5}])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[1], 
expr#6=[TUMBLE($t3, $t5)], auction=[$t0], price=[$t2], bidder=[$t1], 
dateTime=[$t3], extra=[$t4], $f5=[$t6])
  BeamIOSourceRel(table=[[beam, Bid]])
BeamCalcRel(expr#0..1=[{inputs}], maxprice=[$t1], starttime=[$t0])
  BeamAggregationRel(group=[{0}], maxprice=[MAX($1)])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[1], 
expr#6=[TUMBLE($t3, $t5)], $f0=[$t6], price=[$t2])
  BeamIOSourceRel(table=[[beam, Bid]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery2Test > 
testSkipsEverySecondElement STANDARD_ERROR
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`auction`, `PCOLLECTION`.`price`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
WHERE MOD(`PCOLLECTION`.`auction`, 2) = 0
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], price=[$2])
  LogicalFilter(condition=[=(MOD($0, 2), 0)])
BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[2], expr#6=[MOD($t0, $t5)], 
expr#7=[0], expr#8=[=($t6, $t7)], auction=[$t0], price=[$t2], $condition=[$t8])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery2Test > 
testSkipsEveryThirdElement STANDARD_ERROR
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`auction`, `PCOLLECTION`.`price`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
WHERE MOD(`PCOLLECTION`.`auction`, 3) = 0
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], price=[$2])
  LogicalFilter(condition=[=(MOD($0, 3), 0)])
BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[3], expr#6=[MOD($t0, $t5)], 
expr#7=[0], expr#8=[=($t6, $t7)], auction=[$t0], price=[$t2], $condition=[$t8])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery1Test > testConvertsPriceToEur 
STANDARD_ERROR
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`auction`, `PCOLLECTION`.`bidder`, 
`DolToEur`(`PCOLLECTION`.`price`) AS `price`, `PCOLLECTION`.`dateTime`, 
`PCOLLECTION`.`extra`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], bidder=[$1], price=[DolToEur($2)], 
dateTime=[$3], extra=[$4])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 10, 2018 6:23:40 PM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[DolToEur($t2)], 
proj#0..1=[{exprs}], price=[$t5], dateTime=[$t3], extra=[$t4])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Finished generating test XML results (0.002 secs) into: 

Generating HTML test report...

[jira] [Work logged] (BEAM-5002) Beam Dependency Update Request: org.slf4j

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5002?focusedWorklogId=133742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133742
 ]

ASF GitHub Bot logged work on BEAM-5002:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:16
Start Date: 10/Aug/18 18:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6203: Do Not Merge 
[BEAM-5002] upgrade slf4j
URL: https://github.com/apache/beam/pull/6203#issuecomment-412164174
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133742)
Time Spent: 20m  (was: 10m)

> Beam Dependency Update Request: org.slf4j
> -
>
> Key: BEAM-5002
> URL: https://issues.apache.org/jira/browse/BEAM-5002
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:33:53.630864
> Please review and upgrade the org.slf4j to the latest version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133739
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:15
Start Date: 10/Aug/18 18:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209342440
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
 ##
 @@ -1392,5 +1418,32 @@ artifactId=${project.name}
 args argsNeeded
   }
 }
+
+
+/** 
***/
+
+// Method to create the PortableValidatesRunnerTask.
+// Generated task expect :beam-sdks-java-core and :beam-runners-core-java 
in the evaluation dependency. More specifically the task expects 
evaluationDependsOn(":beam-sdks-java-core") and 
evaluationDependsOn(":beam-runners-core-java") in the calling build.gradle file.
+// The method takes PortableValidatesRunnerConfiguration as parameter.
+project.ext.createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfiguration : new 
PortableValidatesRunnerConfiguration()
+  def name = config.name
+  def beamTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+  ]
+  if(config.jobServerConfig){
 
 Review comment:
   nit: spacing around ){


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133739)
Time Spent: 13h  (was: 12h 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133741
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:15
Start Date: 10/Aug/18 18:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209343372
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -65,6 +66,7 @@ dependencies {
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.slf4j_api
+  shadow library.java.slf4j_simple
 
 Review comment:
   Why do you want to add slf4j_simple here?
   
   You typically only add SLF4J logging backends as test / application 
dependencies. (e..g add it to the validatesPortableRunner configuration)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133741)
Time Spent: 13h 10m  (was: 13h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133740
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:15
Start Date: 10/Aug/18 18:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209342955
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
 ##
 @@ -1394,5 +1420,31 @@ artifactId=${project.name}
 args argsNeeded
   }
 }
+
+
+/** 
***/
+
+// Method to create the PortableValidatesRunnerTask.
+// The method takes PortableValidatesRunnerConfiguration as parameter.
+project.ext.createPortableValidatesRunnerTask = {
+  def config = it ? it as PortableValidatesRunnerConfiguration : new 
PortableValidatesRunnerConfiguration()
+  def name = config.name
+  def beamTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+  ]
+  if(config.jobServerConfig){
+
beamTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+  }
+  project.tasks.create(name: name, type: Test) {
+group = "Verification"
+description = "Validates the PortableRunner with JobServer 
${config.jobServerDriver}"
+systemProperty "beamTestPipelineOptions", 
JsonOutput.toJson(beamTestPipelineOptions)
+classpath = config.testClasspathConfiguration
+testClassesDirs = 
project.files(project.project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
 project.project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
 
 Review comment:
   you should be able to add:
   ```
   project.evaluationDependsOn(":beam-sdks-java-core")
   project.evaluationDependsOn(":beam-runners-core-java")
   ```
   
   Don't forget to update the comment as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133740)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=133738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133738
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 10/Aug/18 18:15
Start Date: 10/Aug/18 18:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6073: 
[BEAM-4176] Validate Runner Tests generalization and enable for local reference 
runner
URL: https://github.com/apache/beam/pull/6073#discussion_r209343881
 
 

 ##
 File path: runners/flink/job-server/build.gradle
 ##
 @@ -40,15 +40,15 @@ apply plugin: "application"
 mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
 
 Review comment:
   we should be able to drop the evaluationDependsOn above because it was moved 
into the task.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 133738)
Time Spent: 12h 50m  (was: 12h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5053) CreateRequestsPerMinutePerUser quota limit exceeded

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576665#comment-16576665
 ] 

Mikhail Gryzykhin commented on BEAM-5053:
-

Lowering priority. If this issue does not resurface until 2018/08/18, consider 
this ticket resolved.

> CreateRequestsPerMinutePerUser quota limit exceeded
> ---
>
> Key: BEAM-5053
> URL: https://issues.apache.org/jira/browse/BEAM-5053
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures, testing
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull]
>  
> Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per 
> minute.
> Our Java Validate Runner tests spin up more than 250 jobs in a run.
> We should look into best way to fix the issue. Currently, I see following 
> options:
>  # Add retry logic. Although this limits us to 1 dataflow job start per 
> second for whole Jenkins. In long scale this can also block one test job if 
> other jobs take all the slots.
>  # Utilize different users to spin Dataflow jobs.
>  # Find way to rise quota limit on Dataflow. By default the field limits 
> value to 60 requests per minute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5122) beam_PostCommit_Java_GradleBuild org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5122:

Priority: Critical  (was: Major)

> beam_PostCommit_Java_GradleBuild 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq
> -
>
> Key: BEAM-5122
> URL: https://issues.apache.org/jira/browse/BEAM-5122
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Anton Kedin
>Priority: Critical
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/history/]
> Test flakes with timeout of getting update on pubsub:
> java.lang.AssertionError: Did not receive signal on 
> projects/apache-beam-testing/subscriptions/result-subscription--6677803195159868432
>  in 60s at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:269)
>  at 
> org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.waitForSuccess(TestPubsubSignal.java:237)
>  at 
> org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonIT.testUsesDlq(PubsubJsonIT.java:206)
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1216/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.pubsub/PubsubJsonIT/testUsesDlq/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5053) CreateRequestsPerMinutePerUser quota limit exceeded

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5053:

Priority: Minor  (was: Major)

> CreateRequestsPerMinutePerUser quota limit exceeded
> ---
>
> Key: BEAM-5053
> URL: https://issues.apache.org/jira/browse/BEAM-5053
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures, testing
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull]
>  
> Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per 
> minute.
> Our Java Validate Runner tests spin up more than 250 jobs in a run.
> We should look into best way to fix the issue. Currently, I see following 
> options:
>  # Add retry logic. Although this limits us to 1 dataflow job start per 
> second for whole Jenkins. In long scale this can also block one test job if 
> other jobs take all the slots.
>  # Utilize different users to spin Dataflow jobs.
>  # Find way to rise quota limit on Dataflow. By default the field limits 
> value to 60 requests per minute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5070) nexmark.sources.UnboundedEventSourceTest.resumeFromCheckpoint is flaky

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5070:

Priority: Critical  (was: Major)

> nexmark.sources.UnboundedEventSourceTest.resumeFromCheckpoint is flaky
> --
>
> Key: BEAM-5070
> URL: https://issues.apache.org/jira/browse/BEAM-5070
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Affects Versions: 2.5.0
>Reporter: Reuven Lax
>Assignee: Anton Kedin
>Priority: Critical
>
> This test fails fairly frequently.
> History: 
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1219/testReport/junit/org.apache.beam.sdk.nexmark.sources/UnboundedEventSourceTest/resumeFromCheckpoint/history/]
>  Sample job: 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1219/testReport/org.apache.beam.sdk.nexmark.sources/UnboundedEventSourceTest/resumeFromCheckpoint/
> Failure log:
> org.junit.ComparisonFailure: 
> expected:<...":"UTC"},"afterNow":[true,"beforeNow":fals]e,"equalNow":false},...>
>  but 
> was:<...":"UTC"},"afterNow":[false,"beforeNow":tru]e,"equalNow":false},...> 
> at org.junit.Assert.assertEquals(Assert.java:115) at 
> org.junit.Assert.assertEquals(Assert.java:144) at 
> org.apache.beam.sdk.nexmark.sources.UnboundedEventSourceTest$EventIdChecker.add(UnboundedEventSourceTest.java:71)
>  at 
> org.apache.beam.sdk.nexmark.sources.UnboundedEventSourceTest.resumeFromCheckpoint(UnboundedEventSourceTest.java:96)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)