date:20180621



 [ 
https://issues.apache.org/jira/browse/BEAM-4617?focusedWorklogId=114626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114626
 ]

ASF GitHub Bot logged work on BEAM-4617:


Author: ASF GitHub Bot
Created on: 22/Jun/18 05:58
Start Date: 22/Jun/18 05:58
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #475: [BEAM-4617] Adds a 
dependencies guide to Beam Website
URL: https://github.com/apache/beam-site/pull/475#issuecomment-399329056
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114626)
Time Spent: 1h  (was: 50m)

> Add a dependencies guide
> 
>
> Key: BEAM-4617
> URL: https://issues.apache.org/jira/browse/BEAM-4617
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Initial discussion: 
> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/8b9b3768adfc40d3527d1ce5e8a51d90e5782a348a3abfb9e5dc85ef@%3Cdev.beam.apache.org%3E
> Doc: 
> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #312

See 


Changes:

[robertwb] [BEAM-4150] Extract IO coders from the RemoteGrpcPort.

[robertwb] Add back old code path for dataflow.

--
[...truncated 273.67 KB...]
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO}]
2018/06/22 05:48:03 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/22 05:48:03 Pipeline:
2018/06/22 05:48:03 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/22 05:48:03 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/22 05:48:03 Pipeline:
2018/06/22 05:48:03 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/22 05:48:03 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/22 05:48:03 Pipeline:
2018/06/22 05:48:03 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/22 05:48:03 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/22 05:48:03 wait[5] unblocked w/ 1 [false]
2018/06/22 05:48:03 wait[5] done
2018/06/22 05:48:03 Pipeline:
2018/06/22 05:48:03 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes

[beam] branch master updated (b8f215f -> 0fbbf82)

2018-06-21 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b8f215f  Merge pull request #5730: [BEAM-4584] Remove 
build_rules.gradle
 add 0583702  [BEAM-4150] Extract IO coders from the RemoteGrpcPort.
 add caacf23  Add back old code path for dataflow.
 new 0fbbf82  Merge pull request #5700 from [BEAM-4150] Extract IO coders 
from the RemoteGrpcPort.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/portability/fn_api_runner.py | 10 ++
 sdks/python/apache_beam/runners/worker/bundle_processor.py   | 11 ---
 2 files changed, 14 insertions(+), 7 deletions(-)

[jira] [Commented] (BEAM-4604) Support Triggers for "GroupIntoBatches" Transform

2018-06-21 Thread Oleksandr Bushkovskyi (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520029#comment-16520029
 ] 

Oleksandr Bushkovskyi commented on BEAM-4604:
-

[~kenn], [~echauchot] thank you, yes I think that GroupByKey will resolve my 
case. Also I believe some note in "GroupIntoBatches" documentation would be 
really helpfull.

Regarding workflow, should I close or resolve this issue now?

> Support Triggers for "GroupIntoBatches" Transform
> -
>
> Key: BEAM-4604
> URL: https://issues.apache.org/jira/browse/BEAM-4604
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Oleksandr Bushkovskyi
>Priority: Major
>
> I think it makes sense to implement triggering support for "GroupIntoBatches" 
> transform.
> I've spent quite a long time trying to understand why my triggering behavior 
> doesn't work with "GroupIntoBatches".
> This transform has an exactly same signature and similar naming as 
> "GroupByKey" transform. It's confusing that this two similar from outer view 
> transforms works differently with triggers.
> At least it should be clearly documented with "GroupIntoBatches" that it 
> doesn't support triggers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4617) Add a dependencies guide



 [ 
https://issues.apache.org/jira/browse/BEAM-4617?focusedWorklogId=114618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114618
 ]

ASF GitHub Bot logged work on BEAM-4617:


Author: ASF GitHub Bot
Created on: 22/Jun/18 05:19
Start Date: 22/Jun/18 05:19
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #475: [BEAM-4617] Adds a 
dependencies guide to Beam Website
URL: https://github.com/apache/beam-site/pull/475#issuecomment-399323668
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114618)
Time Spent: 50m  (was: 40m)

> Add a dependencies guide
> 
>
> Key: BEAM-4617
> URL: https://issues.apache.org/jira/browse/BEAM-4617
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Initial discussion: 
> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/8b9b3768adfc40d3527d1ce5e8a51d90e5782a348a3abfb9e5dc85ef@%3Cdev.beam.apache.org%3E
> Doc: 
> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4575) Beam SQL should cleanly transform graph from Calcite



 [ 
https://issues.apache.org/jira/browse/BEAM-4575?focusedWorklogId=114611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114611
 ]

ASF GitHub Bot logged work on BEAM-4575:


Author: ASF GitHub Bot
Created on: 22/Jun/18 04:18
Start Date: 22/Jun/18 04:18
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5687: [BEAM-4575][SQL] 
Don't wait on Unbounded PCollections
URL: https://github.com/apache/beam/pull/5687#issuecomment-399316073
 
 
   LGTM and I think the quota issues are resolved. I'm taking advantage of 
"allow edits by maintainers" to fix this and that. Your actual code has been 
g2g the whole time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114611)
Time Spent: 5h 40m  (was: 5.5h)

> Beam SQL should cleanly transform graph from Calcite
> 
>
> Key: BEAM-4575
> URL: https://issues.apache.org/jira/browse/BEAM-4575
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> It would be nice if the Beam graph matched the Calcite graph in structure 
> with each node generating a PTransform that is applied onto the PCollection 
> of it's parent. We should also ensure that each Calcite node only appears in 
> the Beam graph one time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #332

See

[jira] [Work logged] (BEAM-4618) Jenkins perf tests broken by gradle refactor



 [ 
https://issues.apache.org/jira/browse/BEAM-4618?focusedWorklogId=114608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114608
 ]

ASF GitHub Bot logged work on BEAM-4618:


Author: ASF GitHub Bot
Created on: 22/Jun/18 03:40
Start Date: 22/Jun/18 03:40
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5732: [BEAM-4618] Fix 
errors in BeamModulePlugin perf test section
URL: https://github.com/apache/beam/pull/5732#issuecomment-399311523
 
 
   Run Java TextIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114608)
Time Spent: 40m  (was: 0.5h)

> Jenkins perf tests broken by gradle refactor
> 
>
> Key: BEAM-4618
> URL: https://issues.apache.org/jira/browse/BEAM-4618
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The move from {{build_rules.gradle}} to 
> {{buildSrc/.../BeamModulePlugin.groovy}} requires putting {{project.}} in 
> front of a bunch of variables. The perf test code path was not tested, and is 
> broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4618) Jenkins perf tests broken by gradle refactor



 [ 
https://issues.apache.org/jira/browse/BEAM-4618?focusedWorklogId=114607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114607
 ]

ASF GitHub Bot logged work on BEAM-4618:


Author: ASF GitHub Bot
Created on: 22/Jun/18 03:39
Start Date: 22/Jun/18 03:39
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5732: [BEAM-4618] Fix 
errors in BeamModulePlugin perf test section
URL: https://github.com/apache/beam/pull/5732#issuecomment-399311381
 
 
   Run Java AvroIO Performance Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114607)
Time Spent: 0.5h  (was: 20m)

> Jenkins perf tests broken by gradle refactor
> 
>
> Key: BEAM-4618
> URL: https://issues.apache.org/jira/browse/BEAM-4618
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The move from {{build_rules.gradle}} to 
> {{buildSrc/.../BeamModulePlugin.groovy}} requires putting {{project.}} in 
> front of a bunch of variables. The perf test code path was not tested, and is 
> broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4618) Jenkins perf tests broken by gradle refactor



 [ 
https://issues.apache.org/jira/browse/BEAM-4618?focusedWorklogId=114606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114606
 ]

ASF GitHub Bot logged work on BEAM-4618:


Author: ASF GitHub Bot
Created on: 22/Jun/18 03:38
Start Date: 22/Jun/18 03:38
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5732: [BEAM-4618] Fix 
errors in BeamModulePlugin perf test section
URL: https://github.com/apache/beam/pull/5732#issuecomment-399311183
 
 
   R: @lgajowy 
   
   FYI this is where the code moved, and what I had to do to make it work in 
Gradle plugin code. It is very slightly different, in boring ways, than the 
Gradle DSL.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114606)
Time Spent: 20m  (was: 10m)

> Jenkins perf tests broken by gradle refactor
> 
>
> Key: BEAM-4618
> URL: https://issues.apache.org/jira/browse/BEAM-4618
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The move from {{build_rules.gradle}} to 
> {{buildSrc/.../BeamModulePlugin.groovy}} requires putting {{project.}} in 
> front of a bunch of variables. The perf test code path was not tested, and is 
> broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4618) Jenkins perf tests broken by gradle refactor



 [ 
https://issues.apache.org/jira/browse/BEAM-4618?focusedWorklogId=114605=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114605
 ]

ASF GitHub Bot logged work on BEAM-4618:


Author: ASF GitHub Bot
Created on: 22/Jun/18 03:37
Start Date: 22/Jun/18 03:37
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5732: 
[BEAM-4618] Fix errors in BeamModulePlugin perf test section
URL: https://github.com/apache/beam/pull/5732
 
 
   The `BeamModulePlugin` section on perf tests was not migrated adequately 
from `build_rules.gradle`
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114605)
Time Spent: 10m
Remaining Estimate: 0h

> Jenkins perf tests broken by gradle refactor
> 
>
> Key: BEAM-4618
> URL: https://issues.apache.org/jira/browse/BEAM-4618
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The move from {{build_rules.gradle}} to 
> {{buildSrc/.../BeamModulePlugin.groovy}} requires putting {{project.}} in 
> front of a bunch of variables. The perf test code path was not tested, and is 
> broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4618) Jenkins perf tests broken by gradle refactor

2018-06-21 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-4618:
-

 Summary: Jenkins perf tests broken by gradle refactor
 Key: BEAM-4618
 URL: https://issues.apache.org/jira/browse/BEAM-4618
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The move from {{build_rules.gradle}} to 
{{buildSrc/.../BeamModulePlugin.groovy}} requires putting {{project.}} in front 
of a bunch of variables. The perf test code path was not tested, and is broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4617) Add a dependencies guide



 [ 
https://issues.apache.org/jira/browse/BEAM-4617?focusedWorklogId=114595=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114595
 ]

ASF GitHub Bot logged work on BEAM-4617:


Author: ASF GitHub Bot
Created on: 22/Jun/18 02:06
Start Date: 22/Jun/18 02:06
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #475: [BEAM-4617] Adds a 
dependencies guide to Beam Website
URL: https://github.com/apache/beam-site/pull/475#issuecomment-399298305
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114595)
Time Spent: 40m  (was: 0.5h)

> Add a dependencies guide
> 
>
> Key: BEAM-4617
> URL: https://issues.apache.org/jira/browse/BEAM-4617
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Initial discussion: 
> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E
> Vote: 
> https://lists.apache.org/thread.html/8b9b3768adfc40d3527d1ce5e8a51d90e5782a348a3abfb9e5dc85ef@%3Cdev.beam.apache.org%3E
> Doc: 
> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4292) Add streaming wordcount example

2018-06-21 Thread Henning Rohde (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519908#comment-16519908
 ] 

Henning Rohde commented on BEAM-4292:
-

I tried to run a simplified version of your example and it fails with:

```
java.lang.RuntimeException: java.lang.ClassCastException: 
com.google.cloud.dataflow.worker.BatchModeExecutionContext cannot be cast to 
com.google.cloud.dataflow.worker.StreamingModeExecutionContext
at 
com.google.cloud.dataflow.worker.BeamFnMapTaskExecutorFactory$4.typedApply(BeamFnMapTaskExecutorFactory.java:442)
at 
com.google.cloud.dataflow.worker.BeamFnMapTaskExecutorFactory$4.typedApply(BeamFnMapTaskExecutorFactory.java:413)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at 
com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at 
com.google.cloud.dataflow.worker.BeamFnMapTaskExecutorFactory.create(BeamFnMapTaskExecutorFactory.java:175)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:336)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:290)
at 
com.google.cloud.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:179)
at 
com.google.cloud.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:107)
Caused by: java.lang.ClassCastException: 
com.google.cloud.dataflow.worker.BatchModeExecutionContext cannot be cast to 
com.google.cloud.dataflow.worker.StreamingModeExecutionContext
at 
com.google.cloud.dataflow.worker.PubsubSink$Factory.create(PubsubSink.java:111)
at 
com.google.cloud.dataflow.worker.PubsubSink$Factory.create(PubsubSink.java:84)
at 
com.google.cloud.dataflow.worker.SinkRegistry.create(SinkRegistry.java:103)
at 
com.google.cloud.dataflow.worker.SinkRegistry.create(SinkRegistry.java:37)
at 
com.google.cloud.dataflow.worker.BeamFnMapTaskExecutorFactory.createWriteOperation(BeamFnMapTaskExecutorFactory.java:483)
at 
com.google.cloud.dataflow.worker.BeamFnMapTaskExecutorFactory$4.typedApply(BeamFnMapTaskExecutorFactory.java:429)
... 9 more
```

It looks like Dataflow supports pubsub only in streaming mode. I would say that 
the simplest path forward is just not write #1 as a pipeline, but just read 
using the Beam filesystem and publish to pubsub using the API similarly to how 
streaming_wordcap works.

> Add streaming wordcount example
> ---
>
> Key: BEAM-4292
> URL: https://issues.apache.org/jira/browse/BEAM-4292
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: James Wilson
>Priority: Major
> Attachments: publish_wordline.go
>
>
> It is referenced on the Beam website as part of the Wordcount progression:
> https://beam.apache.org/get-started/wordcount-example/#streamingwordcount-example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4617) Add a dependencies guide

[
https://issues.apache.org/jira/browse/BEAM-4617?focusedWorklogId=114592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114592
]

ASF GitHub Bot logged work on BEAM-4617:

Author: ASF GitHub Bot
Created on: 22/Jun/18 01:25
Start Date: 22/Jun/18 01:25
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request #475:
[BEAM-4617] Adds a dependencies guide to Beam Website
URL: https://github.com/apache/beam-site/pull/475#discussion_r197318969

##
File path: src/contribute/dependencies.md
##
@@ -0,0 +1,70 @@
+---
+layout: section
+title: 'Dependencies Guide'
+section_menu: section-menu/contribute.html
+permalink: /contribute/dependencies/
+---
+
+# Dependencies Guide
+
+This document describes policies for keeping Beam dependencies up to date.
+
+Old dependencies cause user pain and can result in a system being unusable for
some users. Many users do not use Beam in isolation and bundle other
dependencies in the same deployment. These additional dependencies might pull
in incompatible dependencies to user’s environment which can again result in
broken Beam pipelines, sometimes with undefined behavior. To prevent this,
users will have to update their deployment environment or worse yet may end up
not being able to use Beam along with some of the other dependencies at all.
+
+Beam Java SDK’s Gradle build defines a set of top level
[dependencies](https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy)
and various components (runners, IO connectors, etc) can choose to include
these dependencies. Components usually use the versions defined at the top
level but may choose to override these versions.
+
+If a component _X_ chooses to override the version of a dependency _D_ from
_a_ to _b_ and another component _Y_ is incompatible with version _b_ of _D_,
deployment of a user that uses both components _X_ and _Y_ will end up in a
broken state.
+
+A similar issue could arise if two dependencies of Beam depend on a common
library but use incompatible versions of that library.
+
+Also, users might not use Beam in isolation, a user that depends on Beam as
well as other libraries in the same environment might run into similar issues
if Beam and the other library share a dependency while using incompatible
versions.
+
+Beam Python SDK handles dependencies slightly differently, all dependencies
are defined in a single
[setup.py](https://github.com/apache/beam/blob/master/sdks/python/setup.py)
file and are grouped. One of the groups describes required dependencies while
other groups are for defining dependencies for various optional features. All
Python modules have to use the versions of dependencies defined in
[setup.py](https://github.com/apache/beam/blob/master/sdks/python/setup.py)
file. Additionally, for most of the dependencies, Python SDK allows automatic
upgrades upto next major version. Because of this setup, Python SDK currently
does not run into component conflicts but other two forms of dependency
conflicts described above can still occur.
+
+This picture can become even more complicated during runtime. Runner specific
code might be incompatible with dependencies included by certain modules and if
these dependencies leak into runtime, a pipeline might end up in a broken state.
+
+The overall issue is not common to Beam and well known in the industry as the
Diamond [Dependency problem \(or Dependency
Hell\)](https://en.wikipedia.org/wiki/Dependency_hell).
+
+One common solution for the diamond dependency problem is [semantic
versioning](https://semver.org/). The basic idea is that dependencies will be
versioned in the form _x.y.z_ where _x_ is the _major version_, _y_ is the
_minor version_, and _z_ is the _patch version_. A major version change may be
backwards incompatible and is expected to be rare. Minor and patch versions may
be released more regularly but are expected to be backwards compatible. But in
practice, important fixes (such as security patches) might get released in the
form of minor or patch version updates and it will be healthy for the Beam
project to depend on recently released minor versions of dependencies.
+
+## Identifying outdated dependencies
+
+A big part of keeping dependencies up to date involves identifying outdated
dependencies of Beam that the community should try to upgrade.
+
+Beam currently executes a weekly Jenkins job that tries to identify outdated
dependencies for various SDKs. This Jenkins job generates a weekly report that
is shared in Beam dev list. In the future we hope to automatically create JIRAs
based on this report.
+
+In addition to this, Beam community members might identify other critical
dependency updates that have to be manually performed. For example,
+* A minor release of a

[jira] [Work logged] (BEAM-4617) Add a dependencies guide

[
https://issues.apache.org/jira/browse/BEAM-4617?focusedWorklogId=114590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114590
]

ASF GitHub Bot logged work on BEAM-4617:

Author: ASF GitHub Bot
Created on: 22/Jun/18 01:13
Start Date: 22/Jun/18 01:13
Worklog Time Spent: 10m
Work Description: aaltay commented on a change in pull request #475:
[BEAM-4617] Adds a dependencies guide to Beam Website
URL: https://github.com/apache/beam-site/pull/475#discussion_r197317764

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute



 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=114587=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114587
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 22/Jun/18 00:57
Start Date: 22/Jun/18 00:57
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5700: [BEAM-4150] Extract 
IO coders from the RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/5700#issuecomment-399287926
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114587)
Time Spent: 1h 50m  (was: 1h 40m)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #555

See

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114576
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 22/Jun/18 00:22
Start Date: 22/Jun/18 00:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5730: [BEAM-4584] Remove 
build_rules.gradle
URL: https://github.com/apache/beam/pull/5730#issuecomment-399282954
 
 
   There is a current backlog of updating statuses, if you wait 30 mins to an 
hour you'll see it get through eventually.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114576)
Time Spent: 5.5h  (was: 5h 20m)

> Use a "binary plugin" in buildSrc instead of "script plugin" in 
> build_rules.gradle
> --
>
> Key: BEAM-4584
> URL: https://issues.apache.org/jira/browse/BEAM-4584
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> This is a best practice recommended by the Gradle documentation. It is 
> expected to speed things up, and makes dependency management and testing 
> slightly better.
> In terms of code, it is not much different, though some things that were 
> previously implicit in the top-level groovy DSL will need to be made explicit 
> in the plugin definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #891

See

[jira] [Created] (BEAM-4617) Add a dependencies guide

2018-06-21 Thread Chamikara Jayalath (JIRA)

Chamikara Jayalath created BEAM-4617:


 Summary: Add a dependencies guide
 Key: BEAM-4617
 URL: https://issues.apache.org/jira/browse/BEAM-4617
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Initial discussion: 
https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E

Vote: 
https://lists.apache.org/thread.html/8b9b3768adfc40d3527d1ce5e8a51d90e5782a348a3abfb9e5dc85ef@%3Cdev.beam.apache.org%3E

Doc: 
https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #20

See 


--
[...truncated 12.32 MB...]
at 
org.apache.gearpump.streaming.dsl.plan.functions.FlatMapper.process(FunctionRunner.scala:84)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.process(FunctionRunner.scala:52)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:57)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Count/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:64)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #890

See 


Changes:

[kenn] Remove build_rules.gradle

--
[...truncated 17.15 MB...]
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-06-21_16_42_07-13321182350370133787
Jun 21, 2018 11:42:08 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-06-21_16_42_07-13321182350370133787 with 0 
expected assertions.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:07.672Z: Autoscaling is enabled for job 
2018-06-21_16_42_07-13321182350370133787. The number of workers will be between 
1 and 1000.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:07.693Z: Autoscaling was automatically enabled for 
job 2018-06-21_16_42_07-13321182350370133787.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:10.394Z: Checking required Cloud APIs are enabled.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:10.577Z: Checking permissions granted to controller 
Service Account.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:14.482Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:14.885Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.063Z: Expanding GroupByKey operations into 
optimizable parts.
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.101Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.282Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.311Z: Elided trivial flatten 
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.351Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.382Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.415Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.442Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.470Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jun 21, 2018 11:42:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T23:42:15.494Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #19

See 


Changes:

[kenn] Remove build_rules.gradle

--
[...truncated 12.39 MB...]
at 
org.apache.beam.runners.gearpump.translators.functions.DoFnFunction.flatMap(DoFnFunction.java:52)
at 
org.apache.gearpump.streaming.dsl.scalaapi.functions.FlatMapFunction$$anon$1.flatMap(FlatMapFunction.scala:35)
at 
org.apache.gearpump.streaming.dsl.plan.functions.FlatMapper.process(FunctionRunner.scala:84)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.process(FunctionRunner.scala:52)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:57)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Min/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:69)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #323

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 66.32 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 8,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 8,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 8,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 8,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 8,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 8,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #331

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 65.60 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.002 secs.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.002 secs.
:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project 
':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library',

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT_HDFS #42

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 65.25 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
completed. Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 10,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 10,5,main]) 
completed. Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 10,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 10,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 10,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 10,5,main]) completed. Took 
0.001 secs.
:check (Thread[Task worker for ':buildSrc' Thread 10,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 10,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #311

See 


--
[...truncated 273.55 KB...]
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO}]
2018/06/21 23:24:24 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/21 23:24:24 Pipeline:
2018/06/21 23:24:24 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/21 23:24:24 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/21 23:24:24 Pipeline:
2018/06/21 23:24:24 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/21 23:24:24 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/21 23:24:24 Pipeline:
2018/06/21 23:24:24 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 23:24:24 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/21 23:24:24 wait[5] unblocked w/ 1 [false]
2018/06/21 23:24:24 wait[5] done
2018/06/21 23:24:24 Pipeline:
2018/06/21 23:24:24 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T ->

Build failed in Jenkins: beam_PostCommit_Py_ValCont #72

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b8f215fdbae55084b05c21bcc2fa3f3c7eef7456 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b8f215fdbae55084b05c21bcc2fa3f3c7eef7456
Commit message: "Merge pull request #5730: [BEAM-4584] Remove 
build_rules.gradle"
 > git rev-list --no-walk b8f215fdbae55084b05c21bcc2fa3f3c7eef7456 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Py_ValCont] $ /bin/bash -xe /tmp/jenkins2023714384250999704.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution is an incubating feature.
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check
> Task :buildSrc:build
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying BeamModulePlugin to beam
createPerformanceTestHarness with default configuration for project beam
Adding 48 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying BeamModulePlugin to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container

> Configure project :beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container
Project evaluation failed including an error in afterEvaluate {}. Run with 
--stacktrace for details of the afterEvaluate {} error.

FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 61

* What went wrong:
A problem occurred evaluating project ':beam-sdks-python-container'.
> No such property: rootProject for class: 
> org.apache.beam.gradle.BeamModulePlugin

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #323

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 61.82 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.002 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 13,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 13,5,main]) 
completed. Took 0.002 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. Took 
0.005 secs.
:check (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project 
':beam-sdks-java-core', project

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #337

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 64.48 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 5,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 5,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 5,5,main]) 
completed. Took 0.002 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 5,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 5,5,main]) 
completed. Took 0.002 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 5,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 5,5,main]) 
completed. Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 5,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 5,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 13,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 13,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 14,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 14,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project

Build failed in Jenkins: beam_PerformanceTests_JDBC #751

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 62.99 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 6,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 6,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 6,5,main]) 
completed. Took 0.003 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 6,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 6,5,main]) 
completed. Took 0.002 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 6,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 6,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project 
':beam-sdks-java-core', project

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #418

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 54.95 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #647

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 55.16 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.001 secs.
:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools',

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114558=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114558
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 23:20
Start Date: 21/Jun/18 23:20
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197304282
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
+  main = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
+  args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+}
+
+task jobServerShadowJar(type: 
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
 
 Review comment:
   @angoenka With the new layout, to create the uber jar, you run `./gradlew -p 
runners/flink/job-server shadowJar`. It will output the uber jar at 
`runners/flink/job-server/build/libs/flink-job-server.jar`. Note that if we 
plan to actually publish this, we may want to go back to a versioned artifact 
name.
   
   In order to debug the job server through gradle: `./gradlew -p 
runners/flink/job-server runShadow -PjobHost= 
-PartifactsDir=`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114558)
Time Spent: 3h 10m  (was: 3h)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #666

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 54.83 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 
secs.
:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.002 secs.
:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project 
':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #423

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 60.71 KB...]
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #652

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 55.55 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 12,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.001 secs.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 12,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools', project

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #147

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 56.10 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.001 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.002 secs.
:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 5,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 5,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-go-test', project 
':beam-sdks-java-build-tools',

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #649

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[lcwik] Revert "Revert #5689 to fix build"

[github] Add migration notice to apache/beam website README.me

[robertwb] Always consider Read to be a primitive for dataflow pipelines.

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[kenn] Migrate build_rules.gradle to buildSrc plugin

[amaliujia] support unbounded limit.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

[robertwb] Don't require a staged SDK for the Python portable worker.

[kenn] Remove build_rules.gradle

--
[...truncated 55.64 KB...]
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 11,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.001 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.002 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.001 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 11,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 11,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 11,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 11,5,main]) completed. Took 
0.003 secs.
:check (Thread[Task worker for ':buildSrc' Thread 11,5,main]) started.

> Task :buildSrc:check UP-TO-DATE
Skipping task ':buildSrc:check' as it has no actions.
:check (Thread[Task worker for ':buildSrc' Thread 11,5,main]) completed. Took 
0.0 secs.
:build (Thread[Task worker for ':buildSrc' Thread 11,5,main]) started.

> Task :buildSrc:build UP-TO-DATE
Skipping task ':buildSrc:build' as it has no actions.
:build (Thread[Task worker for ':buildSrc' Thread 11,5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-model-fn-execution', project ':beam-model-job-management', project 
':beam-model-pipeline', project ':beam-runners-apex', project 
':beam-runners-core-construction-java', project ':beam-runners-core-java', 
project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #888

See

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #18

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

--
[...truncated 12.26 MB...]
at 
org.apache.beam.runners.gearpump.translators.functions.DoFnFunction.flatMap(DoFnFunction.java:52)
at 
org.apache.gearpump.streaming.dsl.scalaapi.functions.FlatMapFunction$$anon$1.flatMap(FlatMapFunction.scala:35)
at 
org.apache.gearpump.streaming.dsl.plan.functions.FlatMapper.process(FunctionRunner.scala:84)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.process(FunctionRunner.scala:52)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:57)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Count/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:64)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at

[jira] [Closed] (BEAM-4194) [SQL] Support LIMIT on Unbounded Data

2018-06-21 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-4194.
--
   Resolution: Done
Fix Version/s: Not applicable

> [SQL] Support LIMIT on Unbounded Data
> -
>
> Key: BEAM-4194
> URL: https://issues.apache.org/jira/browse/BEAM-4194
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> We need to support queries with "LIMIT xxx".
> Problem is that we don't know when aggregates will trigger, they can 
> potentially accumulate values in global window and never trigger.
> If we have some trigger syntax (BEAM-4193), then the use case becomes similar 
> to what we have at the moment, where the user defines the trigger upstream 
> for all inputs. In this case LIMIT probably can be implemented as 
> sample.any(5) with trigger at count.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2937) Fn API combiner support w/ lifting to PGBK



 [ 
https://issues.apache.org/jira/browse/BEAM-2937?focusedWorklogId=114555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114555
 ]

ASF GitHub Bot logged work on BEAM-2937:


Author: ASF GitHub Bot
Created on: 21/Jun/18 23:06
Start Date: 21/Jun/18 23:06
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #5536: [BEAM-2937] Add 
Combine Grouped Values URN.
URL: https://github.com/apache/beam/pull/5536#issuecomment-399270819
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114555)
Time Spent: 3h 20m  (was: 3h 10m)

> Fn API combiner support w/ lifting to PGBK
> --
>
> Key: BEAM-2937
> URL: https://issues.apache.org/jira/browse/BEAM-2937
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The FnAPI should support this optimization. Detailed design: 
> https://s.apache.org/beam-runner-api-combine-model
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #437

See

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #310

See 


Changes:

[kenn] Remove build_rules.gradle

--
[...truncated 273.68 KB...]
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO}]
2018/06/21 23:03:11 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/21 23:03:11 Pipeline:
2018/06/21 23:03:11 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/21 23:03:11 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/21 23:03:11 Pipeline:
2018/06/21 23:03:11 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/21 23:03:11 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/21 23:03:11 Pipeline:
2018/06/21 23:03:11 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 23:03:11 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/21 23:03:11 wait[5] unblocked w/ 1 [false]
2018/06/21 23:03:11 wait[5] done
2018/06/21 23:03:11 Pipeline:
2018/06/21 23:03:11 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz]

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114553
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 21/Jun/18 23:00
Start Date: 21/Jun/18 23:00
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5730: [BEAM-4584] 
Remove build_rules.gradle
URL: https://github.com/apache/beam/pull/5730
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index 5fc2ee5b731..b24879b1e64 100644
--- a/build.gradle
+++ b/build.gradle
@@ -21,7 +21,7 @@
 // that can be applied to configure a project for certain common
 // tasks.
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 
 // Add performanceTest task to this build.gradle file
 // so that running Performance tests using PerfKitBenchmarker is possible.
diff --git a/build_rules.gradle b/build_rules.gradle
deleted file mode 100644
index e24364d4c1d..000
--- a/build_rules.gradle
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-// DEPRECATED: this "script plugin" just applies the "binary plugin" 
BeamModulePlugin (in buildSrc)
-//
-// Please directly apply BeamModulePlugin instead.
-
-println "Applying build_rules.gradle to $project.name"
-
-apply plugin: org.apache.beam.gradle.BeamModulePlugin
diff --git a/examples/java/build.gradle b/examples/java/build.gradle
index c3d6ec0f723..b882b7ef557 100644
--- a/examples/java/build.gradle
+++ b/examples/java/build.gradle
@@ -18,7 +18,7 @@
 
 import groovy.json.JsonOutput
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(failOnWarning: true)
 
 description = "Apache Beam :: Examples :: Java"
diff --git a/model/fn-execution/build.gradle b/model/fn-execution/build.gradle
index 15770dc5ffb..ca6c806e438 100644
--- a/model/fn-execution/build.gradle
+++ b/model/fn-execution/build.gradle
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(enableFindbugs: false, enableErrorProne: false)
 applyGrpcNature()
 
diff --git a/model/job-management/build.gradle 
b/model/job-management/build.gradle
index 2b475376a37..66ce25b9cda 100644
--- a/model/job-management/build.gradle
+++ b/model/job-management/build.gradle
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(enableFindbugs: false, enableErrorProne: false)
 applyGrpcNature()
 
diff --git a/model/pipeline/build.gradle b/model/pipeline/build.gradle
index c3ea10490e3..9e1c13aeffd 100644
--- a/model/pipeline/build.gradle
+++ b/model/pipeline/build.gradle
@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(enableFindbugs: false, enableErrorProne: false)
 applyGrpcNature()
 
diff --git a/runners/apex/build.gradle b/runners/apex/build.gradle
index 40c1506be4f..816eee3df82 100644
--- a/runners/apex/build.gradle
+++ b/runners/apex/build.gradle
@@ -18,7 +18,7 @@
 
 import groovy.json.JsonOutput
 
-apply from: project(":").file("build_rules.gradle")
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
 applyJavaNature(failOnWarning: true)
 
 description = "Apache Beam :: Runners :: Apex"
diff --git a/runners/core-construction-java/build.gradle 
b/runners/core-construction-java/build.gradle
index 09727d24a20..774de1cc548 100644
--- a/runners/core-construction-java/build.gradle
+++ b/runners/core-construction-java/build.gradle
@@ -16,7 +16,7 @@
  * limitations under the

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114552
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 21/Jun/18 23:00
Start Date: 21/Jun/18 23:00
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5730: [BEAM-4584] 
Remove build_rules.gradle
URL: https://github.com/apache/beam/pull/5730#issuecomment-399269568
 
 
   All the builds claim success though none updated the status here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114552)
Time Spent: 5h 10m  (was: 5h)

> Use a "binary plugin" in buildSrc instead of "script plugin" in 
> build_rules.gradle
> --
>
> Key: BEAM-4584
> URL: https://issues.apache.org/jira/browse/BEAM-4584
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> This is a best practice recommended by the Gradle documentation. It is 
> expected to speed things up, and makes dependency management and testing 
> slightly better.
> In terms of code, it is not much different, though some things that were 
> previously implicit in the top-level groovy DSL will need to be made explicit 
> in the plugin definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_ValCont #71

See 


Changes:

[kenn] Remove build_rules.gradle

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b8f215fdbae55084b05c21bcc2fa3f3c7eef7456 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b8f215fdbae55084b05c21bcc2fa3f3c7eef7456
Commit message: "Merge pull request #5730: [BEAM-4584] Remove 
build_rules.gradle"
 > git rev-list --no-walk 91977a2ef1d080021a6de1e3870625b483aaa979 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Py_ValCont] $ /bin/bash -xe /tmp/jenkins2250923489928844514.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution is an incubating feature.
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check
> Task :buildSrc:build
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying BeamModulePlugin to beam
createPerformanceTestHarness with default configuration for project beam
Adding 48 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying BeamModulePlugin to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container
Project evaluation failed including an error in afterEvaluate {}. Run with 
--stacktrace for details of the afterEvaluate {} error.

FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 61

* What went wrong:
A problem occurred evaluating project ':beam-sdks-python-container'.
> No such property: rootProject for class: 
> org.apache.beam.gradle.BeamModulePlugin

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 12s

Publishing build scan...
https://gradle.com/s/l6agrnykffs4c

Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user

[beam] 01/01: Merge pull request #5730: [BEAM-4584] Remove build_rules.gradle

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b8f215fdbae55084b05c21bcc2fa3f3c7eef7456
Merge: 91977a2 8d88f03
Author: Kenn Knowles 
AuthorDate: Thu Jun 21 16:00:30 2018 -0700

Merge pull request #5730: [BEAM-4584] Remove build_rules.gradle

 build.gradle   |  2 +-
 build_rules.gradle | 25 --
 examples/java/build.gradle |  2 +-
 model/fn-execution/build.gradle|  2 +-
 model/job-management/build.gradle  |  2 +-
 model/pipeline/build.gradle|  2 +-
 runners/apex/build.gradle  |  2 +-
 runners/core-construction-java/build.gradle|  2 +-
 runners/core-java/build.gradle |  2 +-
 runners/direct-java/build.gradle   |  2 +-
 runners/extensions-java/metrics/build.gradle   |  2 +-
 runners/flink/build.gradle |  2 +-
 runners/gcp/gcemd/build.gradle |  2 +-
 runners/gcp/gcsproxy/build.gradle  |  2 +-
 runners/gearpump/build.gradle  |  2 +-
 runners/google-cloud-dataflow-java/build.gradle|  2 +-
 runners/java-fn-execution/build.gradle |  2 +-
 runners/local-java/build.gradle|  2 +-
 runners/reference/java/build.gradle|  2 +-
 runners/spark/build.gradle |  2 +-
 sdks/go/build.gradle   |  2 +-
 sdks/go/container/build.gradle |  2 +-
 sdks/go/examples/build.gradle  |  2 +-
 sdks/go/test/build.gradle  |  2 +-
 sdks/java/build-tools/build.gradle |  2 +-
 sdks/java/container/build.gradle   |  2 +-
 sdks/java/core/build.gradle|  2 +-
 .../google-cloud-platform-core/build.gradle|  2 +-
 sdks/java/extensions/jackson/build.gradle  |  2 +-
 sdks/java/extensions/join-library/build.gradle |  2 +-
 sdks/java/extensions/protobuf/build.gradle |  2 +-
 sdks/java/extensions/sketching/build.gradle|  2 +-
 sdks/java/extensions/sorter/build.gradle   |  2 +-
 sdks/java/extensions/sql/build.gradle  |  2 +-
 sdks/java/extensions/sql/jdbc/build.gradle |  2 +-
 sdks/java/fn-execution/build.gradle|  2 +-
 sdks/java/harness/build.gradle |  2 +-
 sdks/java/io/amazon-web-services/build.gradle  |  2 +-
 sdks/java/io/amqp/build.gradle |  2 +-
 sdks/java/io/cassandra/build.gradle|  2 +-
 sdks/java/io/common/build.gradle   |  2 +-
 .../elasticsearch-tests-2/build.gradle |  2 +-
 .../elasticsearch-tests-5/build.gradle |  2 +-
 .../elasticsearch-tests-common/build.gradle|  2 +-
 sdks/java/io/elasticsearch/build.gradle|  2 +-
 sdks/java/io/file-based-io-tests/build.gradle  |  2 +-
 sdks/java/io/google-cloud-platform/build.gradle|  2 +-
 sdks/java/io/hadoop-common/build.gradle|  2 +-
 sdks/java/io/hadoop-file-system/build.gradle   |  2 +-
 sdks/java/io/hadoop-input-format/build.gradle  |  2 +-
 sdks/java/io/hbase/build.gradle|  2 +-
 sdks/java/io/hcatalog/build.gradle |  2 +-
 sdks/java/io/jdbc/build.gradle |  2 +-
 sdks/java/io/jms/build.gradle  |  2 +-
 sdks/java/io/kafka/build.gradle|  2 +-
 sdks/java/io/kinesis/build.gradle  |  2 +-
 sdks/java/io/mongodb/build.gradle  |  2 +-
 sdks/java/io/mqtt/build.gradle |  2 +-
 sdks/java/io/parquet/build.gradle  |  2 +-
 sdks/java/io/redis/build.gradle|  2 +-
 sdks/java/io/solr/build.gradle |  2 +-
 sdks/java/io/tika/build.gradle |  2 +-
 sdks/java/io/xml/build.gradle  |  2 +-
 sdks/java/maven-archetypes/examples/build.gradle   |  2 +-
 sdks/java/maven-archetypes/starter/build.gradle|  2 +-
 sdks/java/nexmark/build.gradle |  2 +-
 sdks/python/container/build.gradle |  2 +-
 67 files changed, 66 insertions(+), 91 deletions(-)

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114554
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 23:00
Start Date: 21/Jun/18 23:00
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197301476
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
 
 Review comment:
   Do you still recommend a distinct configuration now that we actually export 
the artifact in a new project?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114554)
Time Spent: 3h  (was: 2h 50m)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (91977a2 -> b8f215f)

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 91977a2  Use the Current Users Container as the Environment
 add 8d88f03  Remove build_rules.gradle
 new b8f215f  Merge pull request #5730: [BEAM-4584] Remove 
build_rules.gradle

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build.gradle   |  2 +-
 build_rules.gradle | 25 --
 examples/java/build.gradle |  2 +-
 model/fn-execution/build.gradle|  2 +-
 model/job-management/build.gradle  |  2 +-
 model/pipeline/build.gradle|  2 +-
 runners/apex/build.gradle  |  2 +-
 runners/core-construction-java/build.gradle|  2 +-
 runners/core-java/build.gradle |  2 +-
 runners/direct-java/build.gradle   |  2 +-
 runners/extensions-java/metrics/build.gradle   |  2 +-
 runners/flink/build.gradle |  2 +-
 runners/gcp/gcemd/build.gradle |  2 +-
 runners/gcp/gcsproxy/build.gradle  |  2 +-
 runners/gearpump/build.gradle  |  2 +-
 runners/google-cloud-dataflow-java/build.gradle|  2 +-
 runners/java-fn-execution/build.gradle |  2 +-
 runners/local-java/build.gradle|  2 +-
 runners/reference/java/build.gradle|  2 +-
 runners/spark/build.gradle |  2 +-
 sdks/go/build.gradle   |  2 +-
 sdks/go/container/build.gradle |  2 +-
 sdks/go/examples/build.gradle  |  2 +-
 sdks/go/test/build.gradle  |  2 +-
 sdks/java/build-tools/build.gradle |  2 +-
 sdks/java/container/build.gradle   |  2 +-
 sdks/java/core/build.gradle|  2 +-
 .../google-cloud-platform-core/build.gradle|  2 +-
 sdks/java/extensions/jackson/build.gradle  |  2 +-
 sdks/java/extensions/join-library/build.gradle |  2 +-
 sdks/java/extensions/protobuf/build.gradle |  2 +-
 sdks/java/extensions/sketching/build.gradle|  2 +-
 sdks/java/extensions/sorter/build.gradle   |  2 +-
 sdks/java/extensions/sql/build.gradle  |  2 +-
 sdks/java/extensions/sql/jdbc/build.gradle |  2 +-
 sdks/java/fn-execution/build.gradle|  2 +-
 sdks/java/harness/build.gradle |  2 +-
 sdks/java/io/amazon-web-services/build.gradle  |  2 +-
 sdks/java/io/amqp/build.gradle |  2 +-
 sdks/java/io/cassandra/build.gradle|  2 +-
 sdks/java/io/common/build.gradle   |  2 +-
 .../elasticsearch-tests-2/build.gradle |  2 +-
 .../elasticsearch-tests-5/build.gradle |  2 +-
 .../elasticsearch-tests-common/build.gradle|  2 +-
 sdks/java/io/elasticsearch/build.gradle|  2 +-
 sdks/java/io/file-based-io-tests/build.gradle  |  2 +-
 sdks/java/io/google-cloud-platform/build.gradle|  2 +-
 sdks/java/io/hadoop-common/build.gradle|  2 +-
 sdks/java/io/hadoop-file-system/build.gradle   |  2 +-
 sdks/java/io/hadoop-input-format/build.gradle  |  2 +-
 sdks/java/io/hbase/build.gradle|  2 +-
 sdks/java/io/hcatalog/build.gradle |  2 +-
 sdks/java/io/jdbc/build.gradle |  2 +-
 sdks/java/io/jms/build.gradle  |  2 +-
 sdks/java/io/kafka/build.gradle|  2 +-
 sdks/java/io/kinesis/build.gradle  |  2 +-
 sdks/java/io/mongodb/build.gradle  |  2 +-
 sdks/java/io/mqtt/build.gradle |  2 +-
 sdks/java/io/parquet/build.gradle  |  2 +-
 sdks/java/io/redis/build.gradle|  2 +-
 sdks/java/io/solr/build.gradle |  2 +-
 sdks/java/io/tika/build.gradle |  2 +-
 sdks/java/io/xml/build.gradle  |  2 +-
 sdks/java/maven-archetypes/examples/build.gradle   |  2 +-
 sdks/java/maven-archetypes/starter/build.gradle|  2 +-
 sdks/java/nexmark/build.gradle |  2 +-
 sdks/python/container/build.gradle |  2 +-
 67 files changed, 66 insertions(+), 91 deletions(-)
 delete mode 100644 build_rules.gradle

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114551
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:59
Start Date: 21/Jun/18 22:59
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5728: [BEAM-4614] 
Enabling gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399269320
 
 
   One thing I believe is possible, but way more work that Pablo's change, is 
to build and publish the `buildSrc` directory as though it is a standalone 
plugin, so that sort of build will receive the initscript. It will need an id, 
etc. So I like just having the convenience of the property.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114551)
Time Spent: 1h 20m  (was: 1h 10m)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114549
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:58
Start Date: 21/Jun/18 22:58
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197301089
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
+  main = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
+  args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+}
+
+task jobServerShadowJar(type: 
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
 
 Review comment:
   I ended up doing it in a different way. Let me know what you think.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114549)
Time Spent: 2h 40m  (was: 2.5h)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114550
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:58
Start Date: 21/Jun/18 22:58
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197301187
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
+  main = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
+  args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+}
+
+task jobServerShadowJar(type: 
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
 
 Review comment:
   The main downside of my approach is that it requires an evaluation 
dependency.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114550)
Time Spent: 2h 50m  (was: 2h 40m)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114548
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:57
Start Date: 21/Jun/18 22:57
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5728: [BEAM-4614] 
Enabling gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399269038
 
 
   Previously we were in this state: 
https://github.com/gradle/gradle/issues/1323 (wontfix)
   Now we are in this state: https://github.com/gradle/gradle/issues/1055 
(maybe will fix?)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114548)
Time Spent: 1h 10m  (was: 1h)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114547
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:56
Start Date: 21/Jun/18 22:56
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5728: [BEAM-4614] 
Enabling gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399268817
 
 
   Most recent comment on https://issues.apache.org/jira/browse/BEAM-4512


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114547)
Time Spent: 1h  (was: 50m)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #17

See 


Changes:

[amaliujia] support unbounded limit.

--
[...truncated 12.35 MB...]
at 
org.apache.beam.runners.gearpump.translators.functions.DoFnFunction.flatMap(DoFnFunction.java:52)
at 
org.apache.gearpump.streaming.dsl.scalaapi.functions.FlatMapFunction$$anon$1.flatMap(FlatMapFunction.scala:35)
at 
org.apache.gearpump.streaming.dsl.plan.functions.FlatMapper.process(FunctionRunner.scala:84)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.process(FunctionRunner.scala:52)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:57)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Min/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:69)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #309

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

--
[...truncated 274.37 KB...]
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO}]
2018/06/21 22:51:08 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/21 22:51:08 Pipeline:
2018/06/21 22:51:08 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/21 22:51:08 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/21 22:51:08 Pipeline:
2018/06/21 22:51:08 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/21 22:51:08 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/21 22:51:08 Pipeline:
2018/06/21 22:51:08 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 22:51:08 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/21 22:51:08 wait[5] unblocked w/ 1 [false]
2018/06/21 22:51:08 wait[5] done
2018/06/21 22:51:08 Pipeline:
2018/06/21 22:51:08 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter):

Build failed in Jenkins: beam_PostCommit_Py_ValCont #70

See 


Changes:

[tgroh] Use the Current Users Container as the Environment

--
Started by GitHub push by lukecwik
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 91977a2ef1d080021a6de1e3870625b483aaa979 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 91977a2ef1d080021a6de1e3870625b483aaa979
Commit message: "Use the Current Users Container as the Environment"
 > git rev-list --no-walk 0a92663ade742b0af9cd316fde658c1d39a4196b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Py_ValCont] $ /bin/bash -xe /tmp/jenkins17511810274945036.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution is an incubating feature.
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check
> Task :buildSrc:build
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
Applying BeamModulePlugin to beam
createPerformanceTestHarness with default configuration for project beam
Adding 48 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
Applying BeamModulePlugin to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container
Project evaluation failed including an error in afterEvaluate {}. Run with 
--stacktrace for details of the afterEvaluate {} error.

FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 61

* What went wrong:
A problem occurred evaluating project ':beam-sdks-python-container'.
> No such property: rootProject for class: 
> org.apache.beam.gradle.BeamModulePlugin

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 10s

Publishing build scan...
https://gradle.com/s/hquikvmdhkrhm

Build step 'Execute shell' marked build as failure

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #436

See 


Changes:

[amaliujia] support unbounded limit.

--
[...truncated 201.56 KB...]
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s30"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s32", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s31"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Match"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-06-21_15_34_09-13638563469120977061]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-06-21_15_34_09-13638563469120977061?project=apache-beam-testing
root: INFO: Job 2018-06-21_15_34_09-13638563469120977061 is in state 
JOB_STATE_RUNNING
root: INFO: 2018-06-21T22:34:09.141Z: JOB_MESSAGE_WARNING: Job 
2018-06-21_15_34_09-13638563469120977061 might autoscale up to 1000 workers.
root: INFO: 2018-06-21T22:34:09.169Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-06-21_15_34_09-13638563469120977061. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-06-21T22:34:09.197Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-06-21_15_34_09-13638563469120977061.
root: INFO: 2018-06-21T22:34:12.715Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-06-21T22:34:12.882Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root:

[beam] 01/01: Use the Current Users Container as the Environment

2018-06-21 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 91977a2ef1d080021a6de1e3870625b483aaa979
Merge: 0a92663 e8f3b5e
Author: Lukasz Cwik 
AuthorDate: Thu Jun 21 15:48:28 2018 -0700

Use the Current Users Container as the Environment

 .../org/apache/beam/runners/core/construction/Environments.java| 7 +++
 1 file changed, 7 insertions(+)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #268

See

[beam] branch master updated (0a92663 -> 91977a2)

2018-06-21 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0a92663  Merge pull request #5682: [BEAM-4194] support unbounded limit
 add e8f3b5e  Use the Current Users Container as the Environment
 new 91977a2  Use the Current Users Container as the Environment

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/core/construction/Environments.java| 7 +++
 1 file changed, 7 insertions(+)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #887

See 


Changes:

[kenn] Migrate build_rules.gradle to buildSrc plugin

--
[...truncated 17.16 MB...]
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:19.354Z: Autoscaling is enabled for job 
2018-06-21_15_42_19-14915314985247801744. The number of workers will be between 
1 and 1000.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:19.384Z: Autoscaling was automatically enabled for 
job 2018-06-21_15_42_19-14915314985247801744.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:21.914Z: Checking required Cloud APIs are enabled.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:22.030Z: Checking permissions granted to controller 
Service Account.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:25.831Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.232Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.408Z: Expanding GroupByKey operations into 
optimizable parts.
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.444Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.643Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.667Z: Elided trivial flatten 
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.693Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.706Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.728Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.753Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.775Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-21T22:42:26.806Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jun 21, 2018 10:42:29 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO:

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #554

See 


Changes:

[kirpichov] Add an AddHarnessIdInterceptor

[kirpichov] Remove the runners/reference InProcessManagedChannelFactory

[kirpichov] Allow ManagedChannelFactories to attach Interceptors

[kirpichov] Populate and Use Harness IDs in the Java Harness

[kirpichov] Fixes checkstyle

[robertwb] [BEAM-4605] Remove subtransform of GBK.

[aaltay] Fix backlog bytes reported by CountingSource. (#5719)

[mairbek] Fixed utf8 encoding in Cloud Spanner mutation encoder

--
[...truncated 16.71 MB...]
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
Jun 21, 2018 10:41:48 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
Jun 21, 2018 10:41:48 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute



 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=114544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114544
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:45
Start Date: 21/Jun/18 22:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5700: 
[BEAM-4150] Extract IO coders from the RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/5700#discussion_r197298974
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
 ##
 @@ -881,18 +881,20 @@ def extract_endpoints(stage):
   if transform.spec.urn == bundle_processor.DATA_INPUT_URN:
 target = transform.unique_name, only_element(transform.outputs)
 data_input[target] = pcoll_buffers[pcoll_id]
+coder_id = pipeline_components.pcollections[
+only_element(transform.outputs.values())].coder_id
   elif transform.spec.urn == bundle_processor.DATA_OUTPUT_URN:
 target = transform.unique_name, only_element(transform.inputs)
 data_output[target] = pcoll_id
+coder_id = pipeline_components.pcollections[
+only_element(transform.inputs.values())].coder_id
   else:
 raise NotImplementedError
+  data_spec = beam_fn_api_pb2.RemoteGrpcPort(coder_id=coder_id)
 
 Review comment:
   Hmm, in that case. LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114544)
Time Spent: 1h 40m  (was: 1.5h)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4601) BigQuery reads from pure SQL

2018-06-21 Thread Rui Wang (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519824#comment-16519824
 ] 

Rui Wang commented on BEAM-4601:


Another potential blocker is how we deal with MAP type. 

 

Right now we don't convert Beam MAP type: 
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L60

> BigQuery reads from pure SQL
> 
>
> Key: BEAM-4601
> URL: https://issues.apache.org/jira/browse/BEAM-4601
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now Beam SQL can created a BigQuery table, however, read from BigQuery 
> table is not supported yet. We want to support reading from BigQuery table.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114540
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:37
Start Date: 21/Jun/18 22:37
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5728: [BEAM-4614] Enabling 
gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399264848
 
 
   Kenn, can you link the issue to init scripts not applying to buildscript 
plugins so that it can be added as a comment for why this exists.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114540)
Time Spent: 50m  (was: 40m)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #308

See 


Changes:

[amaliujia] support unbounded limit.

--
[...truncated 274.86 KB...]
2018/06/21 22:36:36 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/21 22:36:36 Pipeline:
2018/06/21 22:36:36 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/21 22:36:36 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/21 22:36:36 Pipeline:
2018/06/21 22:36:36 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/21 22:36:36 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/21 22:36:36 Pipeline:
2018/06/21 22:36:36 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 22:36:36 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/21 22:36:36 wait[5] unblocked w/ 1 [false]
2018/06/21 22:36:36 wait[5] done
2018/06/21 22:36:36 Pipeline:
2018/06/21 22:36:36 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 22:36:36 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1:

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114539
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:35
Start Date: 21/Jun/18 22:35
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197297233
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
+  main = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
+  args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+}
+
+task jobServerShadowJar(type: 
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
 
 Review comment:
   I would think that this would add the project and all its dependencies to 
the current project's compile scope dependency list:
   ```
   configurations {
 compile project(path: "beam-runners-flink", configuration: "shadow")
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114539)
Time Spent: 2.5h  (was: 2h 20m)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114538
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:34
Start Date: 21/Jun/18 22:34
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197297233
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
+  main = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
+  args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+}
+
+task jobServerShadowJar(type: 
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
 
 Review comment:
   I would think that this would add the project and all its dependencies to 
the project dependency list:
   ```
   configurations {
 compile project(path: "beam-runners-flink", configuration: "shadow")
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114538)
Time Spent: 2h 20m  (was: 2h 10m)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_ValCont #69

See 


Changes:

[amaliujia] support unbounded limit.

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0a92663ade742b0af9cd316fde658c1d39a4196b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0a92663ade742b0af9cd316fde658c1d39a4196b
Commit message: "Merge pull request #5682: [BEAM-4194] support unbounded limit"
 > git rev-list --no-walk 16748b18a97d2c5ac5f208a10bbd4ecf854236c5 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Py_ValCont] $ /bin/bash -xe /tmp/jenkins1361390540091042141.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution is an incubating feature.
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy FROM-CACHE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check
> Task :buildSrc:build
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
Applying BeamModulePlugin to beam
createPerformanceTestHarness with default configuration for project beam
Adding 48 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
Applying BeamModulePlugin to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container
Project evaluation failed including an error in afterEvaluate {}. Run with 
--stacktrace for details of the afterEvaluate {} error.

FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 61

* What went wrong:
A problem occurred evaluating project ':beam-sdks-python-container'.
> No such property: rootProject for class: 
> org.apache.beam.gradle.BeamModulePlugin

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 11s

Publishing build scan...
https://gradle.com/s/dyur425brjsyq

Build step 'Execute shell' marked build as failure
Not

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114536
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:32
Start Date: 21/Jun/18 22:32
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197296876
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
 
 Review comment:
   You don't need to duplicate every dependency.
   
   `
   configurations {
 jobServer
   }
   
   dependencies {
 ...
 jobServer project(path: project.path, configuration: "shadow")
   }
   
   task runJobServer {
 classpath = configurations.jobServer
 ...
   }
   ```
   should work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114536)
Time Spent: 2h  (was: 1h 50m)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4194) [SQL] Support LIMIT on Unbounded Data



 [ 
https://issues.apache.org/jira/browse/BEAM-4194?focusedWorklogId=114535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114535
 ]

ASF GitHub Bot logged work on BEAM-4194:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:32
Start Date: 21/Jun/18 22:32
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5682: [BEAM-4194] 
support unbounded limit
URL: https://github.com/apache/beam/pull/5682
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
index cbbafc72a93..eb7d9d2f109 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
@@ -80,6 +80,18 @@ public Calc copy(RelTraitSet traitSet, RelNode input, 
RexProgram program) {
 }
   }
 
+  public int getLimitCountOfSortRel() {
+if (input instanceof BeamSortRel) {
+  return ((BeamSortRel) input).getCount();
+}
+
+throw new RuntimeException("Could not get the limit count from a non 
BeamSortRel input.");
+  }
+
+  public boolean isInputSortRelAndLimitOnly() {
+return (input instanceof BeamSortRel) && ((BeamSortRel) 
input).isLimitOnly();
+  }
+
   /** {@code CalcFn} is the executor for a {@link BeamCalcRel} step. */
   public static class CalcFn extends DoFn {
 private BeamSqlExpressionExecutor executor;
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
index 2fbcb5b1e72..8e32a6aa2de 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
@@ -19,14 +19,19 @@
 
 import static com.google.common.base.Preconditions.checkArgument;
 
+import java.io.IOException;
+import java.io.Serializable;
 import java.util.List;
 import java.util.Map;
 import java.util.Queue;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentLinkedQueue;
 import javax.annotation.Nullable;
+import org.apache.beam.runners.direct.DirectOptions;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.coders.VarIntCoder;
 import org.apache.beam.sdk.metrics.Counter;
 import org.apache.beam.sdk.metrics.MetricNameFilter;
 import org.apache.beam.sdk.metrics.MetricQueryResults;
@@ -35,8 +40,12 @@
 import org.apache.beam.sdk.options.ApplicationNameOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.state.ValueState;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.Row;
 import org.apache.calcite.adapter.enumerable.EnumerableRel;
 import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
@@ -56,9 +65,13 @@
 import org.apache.calcite.rel.convert.ConverterImpl;
 import org.apache.calcite.rel.metadata.RelMetadataQuery;
 import org.apache.calcite.rel.type.RelDataType;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** BeamRelNode to replace a {@code Enumerable} node. */
 public class BeamEnumerableConverter extends ConverterImpl implements 
EnumerableRel {
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamEnumerableConverter.class);
 
   public BeamEnumerableConverter(RelOptCluster cluster, RelTraitSet traits, 
RelNode input) {
 super(cluster, ConventionTraitDef.INSTANCE, traits, input);
@@ -108,13 +121,37 @@ public static PipelineOptions 
createPipelineOptions(Map map) {
   
Thread.currentThread().setContextClassLoader(BeamEnumerableConverter.class.getClassLoader());
   if (node instanceof BeamIOSinkRel) {
 return count(options, node);
+  } else if (isLimitQuery(node)) {
+return limitCollect(options, node);
   }
+
   return collect(options, node);
 } finally {

[jira] [Work logged] (BEAM-4194) [SQL] Support LIMIT on Unbounded Data



 [ 
https://issues.apache.org/jira/browse/BEAM-4194?focusedWorklogId=114534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114534
 ]

ASF GitHub Bot logged work on BEAM-4194:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:32
Start Date: 21/Jun/18 22:32
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5682: [BEAM-4194] 
support unbounded limit
URL: https://github.com/apache/beam/pull/5682#issuecomment-399263935
 
 
   The tests actually part of this PR passed. We have some other quota issues 
but just for Dataflow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114534)
Time Spent: 10h 10m  (was: 10h)

> [SQL] Support LIMIT on Unbounded Data
> -
>
> Key: BEAM-4194
> URL: https://issues.apache.org/jira/browse/BEAM-4194
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> We need to support queries with "LIMIT xxx".
> Problem is that we don't know when aggregates will trigger, they can 
> potentially accumulate values in global window and never trigger.
> If we have some trigger syntax (BEAM-4193), then the use case becomes similar 
> to what we have at the moment, where the user defines the trigger upstream 
> for all inputs. In this case LIMIT probably can be implemented as 
> sample.any(5) with trigger at count.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4615) Flink job server driver wrapper



 [ 
https://issues.apache.org/jira/browse/BEAM-4615?focusedWorklogId=114537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114537
 ]

ASF GitHub Bot logged work on BEAM-4615:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:32
Start Date: 21/Jun/18 22:32
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5726: 
[BEAM-4615] Flink job server wrapper and shadow jar
URL: https://github.com/apache/beam/pull/5726#discussion_r197296876
 
 

 ##
 File path: runners/flink/build.gradle
 ##
 @@ -134,5 +134,31 @@ task validatesRunner {
   dependsOn validatesRunnerStreaming
 }
 
+task runJobServer(type: JavaExec) {
+  classpath = sourceSets.main.runtimeClasspath
 
 Review comment:
   You don't need to duplicate every dependency.
   
   ```
   configurations {
 jobServer
   }
   
   dependencies {
 ...
 jobServer project(path: project.path, configuration: "shadow")
   }
   
   task runJobServer {
 classpath = configurations.jobServer
 ...
   }
   ```
   should work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114537)
Time Spent: 2h 10m  (was: 2h)

> Flink job server driver wrapper
> ---
>
> Key: BEAM-4615
> URL: https://issues.apache.org/jira/browse/BEAM-4615
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This includes:
>  * A gradle wrapper that can execute the Flink job server driver so that it 
> can be easily run locally for testing/debugging.
>  * A shadow ("uber") target that packages all portable Flink runner 
> dependencies into a runnable jar. This jar can then be submitted to Flink 
> clusters via `flink run`.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (16748b1 -> 0a92663)

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 16748b1  Merge pull request #5683: [BEAM-4584] Migrate 
build_rules.gradle to buildSrc plugin
 add 60c8baa  support unbounded limit.
 new 0a92663  Merge pull request #5682: [BEAM-4194] support unbounded limit

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |  12 ++
 .../sql/impl/rel/BeamEnumerableConverter.java  | 177 -
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  92 +--
 .../sdk/extensions/sql/impl/JdbcDriverTest.java|  71 +
 .../sql/meta/provider/pubsub/PubsubJsonIT.java | 106 +++-
 5 files changed, 440 insertions(+), 18 deletions(-)

[beam] 01/01: Merge pull request #5682: [BEAM-4194] support unbounded limit

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0a92663ade742b0af9cd316fde658c1d39a4196b
Merge: 16748b1 60c8baa
Author: Kenn Knowles 
AuthorDate: Thu Jun 21 15:32:38 2018 -0700

Merge pull request #5682: [BEAM-4194] support unbounded limit

 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |  12 ++
 .../sql/impl/rel/BeamEnumerableConverter.java  | 177 -
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  92 +--
 .../sdk/extensions/sql/impl/JdbcDriverTest.java|  71 +
 .../sql/meta/provider/pubsub/PubsubJsonIT.java | 106 +++-
 5 files changed, 440 insertions(+), 18 deletions(-)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #16

See 


Changes:

[lcwik] Revert "Revert #5689 to fix build"

[kenn] Migrate build_rules.gradle to buildSrc plugin

--
[...truncated 12.35 MB...]
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.process(FunctionRunner.scala:52)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen$$anonfun$1.apply(FunctionRunner.scala:56)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:57)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Count/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:64)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114530
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:29
Start Date: 21/Jun/18 22:29
Worklog Time Spent: 10m 
  Work Description: kennknowles edited a comment on issue #5728: 
[BEAM-4614] Enabling gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399263188
 
 
   You should also do this to `buildSrc/build.gradle`. Technically, that is the 
only place it is truly required, because the normal `repositories { ... }` 
block in an initscript does affect the main build just fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114530)
Time Spent: 40m  (was: 0.5h)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #885

See

[jira] [Work logged] (BEAM-4614) Allow gradle build to take extra list of repositories through an init file



 [ 
https://issues.apache.org/jira/browse/BEAM-4614?focusedWorklogId=114528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114528
 ]

ASF GitHub Bot logged work on BEAM-4614:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:28
Start Date: 21/Jun/18 22:28
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5728: [BEAM-4614] 
Enabling gradle build to receive extra buildscript repositories
URL: https://github.com/apache/beam/pull/5728#issuecomment-399263188
 
 
   You should also do this to `buildSrc/build.gradle`. Technically, that is the 
only place it is truly required.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114528)
Time Spent: 0.5h  (was: 20m)

> Allow gradle build to take extra list of repositories through an init file
> --
>
> Key: BEAM-4614
> URL: https://issues.apache.org/jira/browse/BEAM-4614
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4007) Futurize and fix python 2 compatibility for typehints subpackage



 [ 
https://issues.apache.org/jira/browse/BEAM-4007?focusedWorklogId=114529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114529
 ]

ASF GitHub Bot logged work on BEAM-4007:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:28
Start Date: 21/Jun/18 22:28
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on a change in pull request 
#5337: [BEAM-4007] Futurize typehints subpackage
URL: https://github.com/apache/beam/pull/5337#discussion_r197296128
 
 

 ##
 File path: sdks/python/apache_beam/typehints/typecheck.py
 ##
 @@ -87,14 +92,14 @@ def wrapper(self, method, args, kwargs):
 except TypeCheckError as e:
   error_msg = ('Runtime type violation detected within ParDo(%s): '
'%s' % (self.full_label, e))
-  six.raise_from(TypeCheckError(error_msg), sys.exc_info()[2])
+  raise_with_traceback(TypeCheckError(error_msg))
 else:
   return self._check_type(result)
 
   def _check_type(self, output):
 if output is None:
   return output
-elif isinstance(output, (dict,) + six.string_types):
+elif isinstance(output, (dict, str, unicode)):
 
 Review comment:
   My understanding is that in Python 2, `bytes` is equivalent to `str`, but 
this is not the case in Python 3, so by leaving out `bytes` in Python 3, this 
code path would not be triggered if `output` is of type `bytes`, so that we 
should add this to the list.  Is that correct?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114529)
Time Spent: 3h 10m  (was: 3h)

> Futurize and fix python 2 compatibility for typehints subpackage
> 
>
> Key: BEAM-4007
> URL: https://issues.apache.org/jira/browse/BEAM-4007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114521
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:21
Start Date: 21/Jun/18 22:21
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #5730: 
[BEAM-4584] Remove build_rules.gradle
URL: https://github.com/apache/beam/pull/5730
 
 
   This followup removes `build_rules.gradle` which is just an unneeded proxy 
for `BeamModulePlugin`
   
   (I didn't do it in the prior PR because I thought it would be more difficult 
than it was)
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114521)
Time Spent: 5h  (was: 4h 50m)

> Use a "binary plugin" in buildSrc instead of "script plugin" in 
> build_rules.gradle
> --
>
> Key: BEAM-4584
> URL: https://issues.apache.org/jira/browse/BEAM-4584
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> This is a best practice recommended by the Gradle documentation. It is 
> expected to speed things up, and makes dependency management and testing 
> slightly better.
> In terms of code, it is not much different, though some things that were 
> previously implicit in the top-level groovy DSL will need to be made explicit 
> in the plugin definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-3883) Python SDK stages artifacts when talking to job server

2018-06-21 Thread Robert Bradshaw (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw reopened BEAM-3883:
---

Staging code is written, but not yet hooked up.

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_ValCont #68

See 


Changes:

[lcwik] Revert "Revert #5689 to fix build"

[kenn] Migrate build_rules.gradle to buildSrc plugin

--
Started by GitHub push by lukecwik
Started by GitHub push by lukecwik
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 16748b18a97d2c5ac5f208a10bbd4ecf854236c5 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 16748b18a97d2c5ac5f208a10bbd4ecf854236c5
Commit message: "Merge pull request #5683: [BEAM-4584] Migrate 
build_rules.gradle to buildSrc plugin"
 > git rev-list --no-walk d097f159f5cb7c75b7740ceec6c3d2a8cc023237 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Py_ValCont] $ /bin/bash -xe /tmp/jenkins635425483407543532.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution is an incubating feature.
Download 
https://plugins.gradle.org/m2/com/diffplug/gradle/spotless/com.diffplug.gradle.spotless.gradle.plugin/3.6.0/com.diffplug.gradle.spotless.gradle.plugin-3.6.0.pom
Download 
https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-plugin-gradle/3.6.0/spotless-plugin-gradle-3.6.0.pom
Download 
https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar
Download 
https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-ext-greclipse/2.3.0/spotless-ext-greclipse-2.3.0.pom
Download 
https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-ext-greclipse/2.3.0/spotless-ext-greclipse-2.3.0.jar
Generating JAR file 'gradle-api-4.8.jar'
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:spotlessGroovy
> Task :buildSrc:spotlessGroovyCheck
> Task :buildSrc:spotlessGroovyGradle
> Task :buildSrc:spotlessGroovyGradleCheck
> Task :buildSrc:spotlessCheck
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check
> Task :buildSrc:build
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
Applying BeamModulePlugin to beam
createPerformanceTestHarness with default configuration for project beam
Adding 48 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
Applying BeamModulePlugin to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container
Project evaluation failed including an error in afterEvaluate {}. Run with 
--stacktrace for details of the afterEvaluate {} error.

FAILURE: Build failed with an exception.

* Where:
Build file

Build failed in Jenkins: beam_PostCommit_Py_ValCont #67

See 


Changes:

[github] Add migration notice to apache/beam website README.me

--
[...truncated 151.93 KB...]
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pxd -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/util.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/util_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/testing
copying apache_beam/testing/data/standard_coders.yaml -> 
apache-beam-2.6.0.dev0/apache_beam/testing/data
copying apache_beam/testing/data/trigger_transcripts.yaml -> 
apache-beam-2.6.0.dev0/apache_beam/testing/data
copying apache_beam/tools/__init__.py -> 
apache-beam-2.6.0.dev0/apache_beam/tools
copying apache_beam/tools/distribution_counter_microbenchmark.py -> 
apache-beam-2.6.0.dev0/apache_beam/tools
copying apache_beam/tools/map_fn_microbenchmark.py -> 
apache-beam-2.6.0.dev0/apache_beam/tools
copying apache_beam/tools/sideinput_microbenchmark.py -> 
apache-beam-2.6.0.dev0/apache_beam/tools
copying apache_beam/tools/utils.py -> apache-beam-2.6.0.dev0/apache_beam/tools
copying apache_beam/transforms/__init__.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/combiners_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/core.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_source.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/create_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.pxd -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_combiners.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_dataflow_distribution_counter.pxd -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/cy_dataflow_distribution_counter.pyx -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/dataflow_distribution_counter_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/display_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/ptransform_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/py_dataflow_distribution_counter.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/sideinputs_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/timeutil.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/trigger_test.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util.py -> 
apache-beam-2.6.0.dev0/apache_beam/transforms
copying apache_beam/transforms/util_test.py ->

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle #15

See 


Changes:

[github] Add migration notice to apache/beam website README.me

--
[...truncated 12.28 MB...]
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.plan.functions.AndThen.finish(FunctionRunner.scala:58)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.onTrigger$1(WindowRunner.scala:128)
at 
org.apache.gearpump.streaming.dsl.window.impl.DefaultWindowRunner.trigger(WindowRunner.scala:152)
at 
org.apache.gearpump.streaming.task.TaskUtil$.trigger(TaskUtil.scala:41)
at 
org.apache.gearpump.streaming.dsl.task.TransformTask.onWatermarkProgress(TransformTask.scala:44)
at 
org.apache.gearpump.streaming.task.TaskWrapper.onWatermarkProgress(TaskWrapper.scala:140)
at 
org.apache.gearpump.streaming.task.TaskActor.org$apache$gearpump$streaming$task$TaskActor$$onUpstreamMinClock(TaskActor.scala:330)
at 
org.apache.gearpump.streaming.task.TaskActor$$anonfun$handleMessages$1.applyOrElse(TaskActor.scala:193)
at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
at 
org.apache.gearpump.streaming.task.TaskActor.aroundReceive(TaskActor.scala:47)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.AssertionError: Count/Flatten.PCollections.out: 
org/hamcrest/Matchers
at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:418)
at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:408)
at 
org.apache.beam.sdk.io.CountingSourceTest.addCountingAsserts(CountingSourceTest.java:64)
at 
org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits(CountingSourceTest.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage



 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=114516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114516
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:05
Start Date: 21/Jun/18 22:05
Worklog Time Spent: 10m 
  Work Description: Fematich opened a new pull request #5729: [BEAM-4006] 
Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729
 
 
   This pull request prepares the transforms subpackage for Python 3 support. 
This PR is part of a series in which all subpackages will be updated using the 
same approach.
   This approach has been documented [here 
](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit)and
 the first pull request in the series (Futurize coders subpackage) 
demonstrating this approach can be found at #5053.
   
   R: @aaltay @tvalentyn @RobbeSneyders


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114516)
Time Spent: 10m
Remaining Estimate: 0h

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #307

See 


Changes:

[kenn] Migrate build_rules.gradle to buildSrc plugin

--
[...truncated 273.46 KB...]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO}]
2018/06/21 22:04:15 Plan[plan]:
4: Impulse[0]
1: Discard
2: ParDo[beam.partitionFn] Out:[1]
3: ParDo[beam.createFn] Out:[2]
2018/06/21 22:04:15 Pipeline:
2018/06/21 22:04:15 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO} Out: int -> {8: int/int[varintz] GLO}]
2018/06/21 22:04:15 Plan[plan]:
9: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: Discard
7: ParDo[beam.partitionFn] Out:[1 2 3 4 5 6]
8: ParDo[beam.createFn] Out:[7]
2018/06/21 22:04:15 Pipeline:
2018/06/21 22:04:15 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: int/int[varintz] GLO}
{7: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO} Out: int -> {5: 
int/int[varintz] GLO} Out: int -> {6: int/int[varintz] GLO} Out: int -> {7: 
int/int[varintz] GLO}]
2018/06/21 22:04:15 Plan[plan]:
8: Impulse[0]
1: Discard
2: Discard
3: Discard
4: Discard
5: Discard
6: ParDo[beam.partitionFn] Out:[1 2 3 4 5]
7: ParDo[beam.createFn] Out:[6]
--- PASS: TestPartitionFailures (0.00s)
=== RUN   TestPartitionFlattenIdentity
2018/06/21 22:04:15 Pipeline:
2018/06/21 22:04:15 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo [In(Main): X <- {9: int/int[varintz] GLO}] -> []
2018/06/21 22:04:15 Plan[plan]:
12: Impulse[0]
13: Impulse[0]
1: ParDo[passert.failFn] Out:[]
2: Discard
3: ParDo[passert.failFn] Out:[]
4: ParDo[passert.diffFn] Out:[1 2 3]
5: wait[2] Out:4
6: buffer[6]. wait:5 Out:4
7: buffer[7]. wait:5 Out:4
8: Flatten[2]. Out:buffer[6]. wait:5 Out:4
9: ParDo[beam.partitionFn] Out:[8 8]
10: Multiplex. Out:[9 7]
11: ParDo[beam.createFn] Out:[10]
2018/06/21 22:04:15 wait[5] unblocked w/ 1 [false]
2018/06/21 22:04:15 wait[5] done
2018/06/21 22:04:15 Pipeline:
2018/06/21 22:04:15 Nodes: {1: []uint8/bytes GLO}
{2: int/int[varintz] GLO}
{3: int/int[varintz] GLO}
{4: int/int[varintz] GLO}
{5: int/int[varintz] GLO}
{6: []uint8/bytes GLO}
{7: int/int[varintz] GLO}
{8: int/int[varintz] GLO}
{9: int/int[varintz] GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T -> {2: 
int/int[varintz] GLO}]
3: ParDo [In(Main): int <- {2: int/int[varintz] GLO}] -> [Out: int -> {3: 
int/int[varintz] GLO} Out: int -> {4: int/int[varintz] GLO}]
4: Flatten [In(Main): int <- {3: int/int[varintz] GLO} In(Main): int <- {4: 
int/int[varintz] GLO}] -> [Out: int -> {5: int/int[varintz] GLO}]
5: Impulse [] -> [Out: []uint8 -> {6: []uint8/bytes GLO}]
6: ParDo [In(Main): []uint8 <- {6: []uint8/bytes GLO} In(Iter): T <- {5: 
int/int[varintz] GLO} In(Iter): T <- {2: int/int[varintz] GLO}] -> [Out: T -> 
{7: int/int[varintz] GLO} Out: T -> {8: int/int[varintz] GLO} Out: T -> {9: 
int/int[varintz] GLO}]
7: ParDo [In(Main): X <- {7: int/int[varintz] GLO}] -> []
8: ParDo

[jira] [Resolved] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle

2018-06-21 Thread Kenneth Knowles (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-4584.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Use a "binary plugin" in buildSrc instead of "script plugin" in 
> build_rules.gradle
> --
>
> Key: BEAM-4584
> URL: https://issues.apache.org/jira/browse/BEAM-4584
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> This is a best practice recommended by the Gradle documentation. It is 
> expected to speed things up, and makes dependency management and testing 
> slightly better.
> In terms of code, it is not much different, though some things that were 
> previously implicit in the top-level groovy DSL will need to be made explicit 
> in the plugin definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3921) Scripting extension based on Java Scripting API (JSR-223)



 [ 
https://issues.apache.org/jira/browse/BEAM-3921?focusedWorklogId=114515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114515
 ]

ASF GitHub Bot logged work on BEAM-3921:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:03
Start Date: 21/Jun/18 22:03
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #4944:  
[BEAM-3921] Scripting extension based on Java Scripting API (JSR-223)
URL: https://github.com/apache/beam/pull/4944#discussion_r197291347
 
 

 ##
 File path: 
sdks/java/extensions/scripting/src/main/java/org/apache/beam/sdk/extensions/scripting/ScriptingParDo.java
 ##
 @@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.scripting;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.Serializable;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import javax.script.Bindings;
+import javax.script.Compilable;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineManager;
+import javax.script.ScriptException;
+import javax.script.SimpleScriptContext;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * A transform to write simple ParDo transformations with a scripting language
+ * supported by Java JSR-223.
+ */
+@Experimental
+public abstract class ScriptingParDo
+extends PTransform, PCollection> {
+
+  private ValueProvider language = StaticValueProvider.of("js");
+  private ValueProvider script = StaticValueProvider.of("");
+  private Coder coder;
+
+  public static  ScriptingParDo of(final 
Coder coder) {
+final ScriptingParDo scripting = new 
ScriptingParDo() {};
+scripting.coder = coder;
+return scripting;
+  }
+
+  public ScriptingParDo withLanguage(final String language) {
+checkArgument(language != null, "language can not be null");
+return withLanguage(ValueProvider.StaticValueProvider.of(language));
+  }
+
+  public ScriptingParDo withLanguage(final 
ValueProvider language) {
+checkArgument(language != null, "language can not be null");
+this.language = language;
+return this;
+  }
+
+  public ScriptingParDo withScript(final String script) {
+checkArgument(script != null, "script can not be null");
+return withScript(ValueProvider.StaticValueProvider.of(script));
+  }
+
+  public ScriptingParDo withScript(final 
ValueProvider script) {
+checkArgument(script != null, "script can not be null");
+this.script = script;
+return this;
+  }
+
+  @Override
+  public PCollection expand(final PCollection apCollection) {
+if (language == null || script == null || script.get().isEmpty()) {
+  throw new IllegalArgumentException("Language and Script must be set");
+}
+return apCollection.apply(ParDo.of(new ScriptingDoFn<>(language.get(), 
script.get(;
+  }
+
+  @Override // ensure we don't always need to set the coder
+  public  Coder getDefaultOutputCoder(
+  final PCollection input, final PCollection output)
+  throws CannotProvideCoderException {
+if (coder != null) {
+  return (Coder) coder;
+}
+final Type superclass = getClass().getGenericSuperclass();
+if (ParameterizedType.class.isInstance(superclass)) {
+  final Type type = 
ParameterizedType.class.cast(superclass).getActualTypeArguments()[1];
+  return (Coder) 
output.getPipeline().getCoderRegistry().getCoder(TypeDescriptor.of(type));
+}
+return (Coder)

[jira] [Work logged] (BEAM-3921) Scripting extension based on Java Scripting API (JSR-223)



 [ 
https://issues.apache.org/jira/browse/BEAM-3921?focusedWorklogId=114514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114514
 ]

ASF GitHub Bot logged work on BEAM-3921:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:01
Start Date: 21/Jun/18 22:01
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #4944:  
[BEAM-3921] Scripting extension based on Java Scripting API (JSR-223)
URL: https://github.com/apache/beam/pull/4944#discussion_r197290694
 
 

 ##
 File path: 
sdks/java/extensions/scripting/src/main/java/org/apache/beam/sdk/extensions/scripting/ScriptingParDo.java
 ##
 @@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.scripting;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.Serializable;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import javax.script.Bindings;
+import javax.script.Compilable;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineManager;
+import javax.script.ScriptException;
+import javax.script.SimpleScriptContext;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * A transform to write simple ParDo transformations with a scripting language
+ * supported by Java JSR-223.
+ */
+@Experimental
+public abstract class ScriptingParDo
+extends PTransform, PCollection> {
+
+  private ValueProvider language = StaticValueProvider.of("js");
+  private ValueProvider script = StaticValueProvider.of("");
+  private Coder coder;
+
+  public static  ScriptingParDo of(final 
Coder coder) {
+final ScriptingParDo scripting = new 
ScriptingParDo() {};
+scripting.coder = coder;
+return scripting;
+  }
+
+  public ScriptingParDo withLanguage(final String language) {
+checkArgument(language != null, "language can not be null");
+return withLanguage(ValueProvider.StaticValueProvider.of(language));
+  }
+
+  public ScriptingParDo withLanguage(final 
ValueProvider language) {
+checkArgument(language != null, "language can not be null");
+this.language = language;
+return this;
+  }
+
+  public ScriptingParDo withScript(final String script) {
+checkArgument(script != null, "script can not be null");
+return withScript(ValueProvider.StaticValueProvider.of(script));
+  }
+
+  public ScriptingParDo withScript(final 
ValueProvider script) {
+checkArgument(script != null, "script can not be null");
+this.script = script;
+return this;
+  }
+
+  @Override
+  public PCollection expand(final PCollection apCollection) {
+if (language == null || script == null || script.get().isEmpty()) {
+  throw new IllegalArgumentException("Language and Script must be set");
+}
+return apCollection.apply(ParDo.of(new ScriptingDoFn<>(language.get(), 
script.get(;
+  }
+
+  @Override // ensure we don't always need to set the coder
+  public  Coder getDefaultOutputCoder(
+  final PCollection input, final PCollection output)
+  throws CannotProvideCoderException {
+if (coder != null) {
+  return (Coder) coder;
+}
+final Type superclass = getClass().getGenericSuperclass();
+if (ParameterizedType.class.isInstance(superclass)) {
+  final Type type = 
ParameterizedType.class.cast(superclass).getActualTypeArguments()[1];
+  return (Coder) 
output.getPipeline().getCoderRegistry().getCoder(TypeDescriptor.of(type));
+}
+return (Coder)

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114513
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:00
Start Date: 21/Jun/18 22:00
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5683: [BEAM-4584] 
Migrate build_rules.gradle to buildSrc plugin
URL: https://github.com/apache/beam/pull/5683
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
new file mode 100644
index 000..fa8b452b353
--- /dev/null
+++ b/buildSrc/build.gradle
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Plugins for configuring _this build_ of the module
+plugins {
+  id 'groovy'
+  id 'com.diffplug.gradle.spotless' version "3.6.0"
+}
+
+// Dependencies on other plugins used when this plugin
+// is invoked
+repositories { jcenter() }
+
+dependencies {
+  compile gradleApi()
+  compile localGroovy()
+  compile 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+}
+
+// Because buildSrc is built and tested automatically _before_ gradle
+// does anything else, it is not possible to spotlessApply because
+// spotlessCheck fails before that. So this hack allows disabling
+// the check for the moment of application.
+//
+// ./gradlew :buildSrc:spotlessApply -PdisableSpotlessCheck=true
+def disableSpotlessCheck = project.hasProperty('disableSpotlessCheck') &&
+project.disableSpotlessCheck == 'true'
+
+spotless {
+  enforceCheck !disableSpotlessCheck
+  groovy {
+excludeJava()
+greclipse().configFile('greclipse.properties')
+  }
+  groovyGradle {
+greclipse().configFile('greclipse.properties')
+  }
+}
diff --git a/buildSrc/greclipse.properties b/buildSrc/greclipse.properties
new file mode 100644
index 000..f2e56a9eb29
--- /dev/null
+++ b/buildSrc/greclipse.properties
@@ -0,0 +1,21 @@
+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+
+org.eclipse.jdt.core.formatter.tabulation.char=space
+org.eclipse.jdt.core.formatter.tabulation.size=2
+org.eclipse.jdt.core.formatter.indent_empty_lines=false
+groovy.formatter.multiline.indentation=4
diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
new file mode 100644
index 000..70ab9c3ff75
--- /dev/null
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -0,0 +1,1315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #434

See

[jira] [Work logged] (BEAM-4584) Use a "binary plugin" in buildSrc instead of "script plugin" in build_rules.gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-4584?focusedWorklogId=114512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114512
 ]

ASF GitHub Bot logged work on BEAM-4584:


Author: ASF GitHub Bot
Created on: 21/Jun/18 22:00
Start Date: 21/Jun/18 22:00
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5683: [BEAM-4584] 
Migrate build_rules.gradle to buildSrc plugin
URL: https://github.com/apache/beam/pull/5683#issuecomment-399256997
 
 
   We've established that Dataflow is having an outage for the 
`apache-beam-testing` project. We will sickbay those tests and move to fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 114512)
Time Spent: 4h 40m  (was: 4.5h)

> Use a "binary plugin" in buildSrc instead of "script plugin" in 
> build_rules.gradle
> --
>
> Key: BEAM-4584
> URL: https://issues.apache.org/jira/browse/BEAM-4584
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> This is a best practice recommended by the Gradle documentation. It is 
> expected to speed things up, and makes dependency management and testing 
> slightly better.
> In terms of code, it is not much different, though some things that were 
> previously implicit in the top-level groovy DSL will need to be made explicit 
> in the plugin definition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5683: [BEAM-4584] Migrate build_rules.gradle to buildSrc plugin

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 16748b18a97d2c5ac5f208a10bbd4ecf854236c5
Merge: 1ab7087 220b180
Author: Kenn Knowles 
AuthorDate: Thu Jun 21 15:00:23 2018 -0700

Merge pull request #5683: [BEAM-4584] Migrate build_rules.gradle to 
buildSrc plugin

 buildSrc/build.gradle  |   53 +
 buildSrc/greclipse.properties  |   21 +
 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 1315 +++
 build_rules.gradle | 1331 +---
 4 files changed, 1392 insertions(+), 1328 deletions(-)

[jira] [Work logged] (BEAM-3921) Scripting extension based on Java Scripting API (JSR-223)



 [ 
https://issues.apache.org/jira/browse/BEAM-3921?focusedWorklogId=114511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-114511
 ]

ASF GitHub Bot logged work on BEAM-3921:


Author: ASF GitHub Bot
Created on: 21/Jun/18 21:59
Start Date: 21/Jun/18 21:59
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #4944:  
[BEAM-3921] Scripting extension based on Java Scripting API (JSR-223)
URL: https://github.com/apache/beam/pull/4944#discussion_r197290313
 
 

 ##
 File path: 
sdks/java/extensions/scripting/src/main/java/org/apache/beam/sdk/extensions/scripting/ScriptingParDo.java
 ##
 @@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.scripting;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.Serializable;
+import java.lang.reflect.ParameterizedType;
+import java.lang.reflect.Type;
+import javax.script.Bindings;
+import javax.script.Compilable;
+import javax.script.CompiledScript;
+import javax.script.ScriptContext;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineManager;
+import javax.script.ScriptException;
+import javax.script.SimpleScriptContext;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/**
+ * A transform to write simple ParDo transformations with a scripting language
+ * supported by Java JSR-223.
+ */
+@Experimental
+public abstract class ScriptingParDo
+extends PTransform, PCollection> {
+
+  private ValueProvider language = StaticValueProvider.of("js");
+  private ValueProvider script = StaticValueProvider.of("");
+  private Coder coder;
+
+  public static  ScriptingParDo of(final 
Coder coder) {
+final ScriptingParDo scripting = new 
ScriptingParDo() {};
+scripting.coder = coder;
+return scripting;
+  }
+
+  public ScriptingParDo withLanguage(final String language) {
+checkArgument(language != null, "language can not be null");
+return withLanguage(ValueProvider.StaticValueProvider.of(language));
+  }
+
+  public ScriptingParDo withLanguage(final 
ValueProvider language) {
+checkArgument(language != null, "language can not be null");
+this.language = language;
+return this;
+  }
+
+  public ScriptingParDo withScript(final String script) {
+checkArgument(script != null, "script can not be null");
+return withScript(ValueProvider.StaticValueProvider.of(script));
+  }
+
+  public ScriptingParDo withScript(final 
ValueProvider script) {
+checkArgument(script != null, "script can not be null");
+this.script = script;
+return this;
+  }
+
+  @Override
+  public PCollection expand(final PCollection apCollection) {
+if (language == null || script == null || script.get().isEmpty()) {
+  throw new IllegalArgumentException("Language and Script must be set");
+}
+return apCollection.apply(ParDo.of(new ScriptingDoFn<>(language.get(), 
script.get(;
+  }
+
+  @Override // ensure we don't always need to set the coder
+  public  Coder getDefaultOutputCoder(
+  final PCollection input, final PCollection output)
+  throws CannotProvideCoderException {
+if (coder != null) {
+  return (Coder) coder;
+}
+final Type superclass = getClass().getGenericSuperclass();
+if (ParameterizedType.class.isInstance(superclass)) {
+  final Type type = 
ParameterizedType.class.cast(superclass).getActualTypeArguments()[1];
+  return (Coder) 
output.getPipeline().getCoderRegistry().getCoder(TypeDescriptor.of(type));
+}
+return (Coder)

[beam] branch master updated (1ab7087 -> 16748b1)