[beam] branch master updated (79697ed -> 2a70f0c)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 79697ed  [BEAM-4276] Surface graph.Fn encoding functions.
 add 855d758  Explain experimental/deprecated from users perspective.
 new 2a70f0c  Merge pull request #5625 from aaltay/expdep

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/utils/annotations.py | 11 +++
 1 file changed, 11 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #5625 from aaltay/expdep

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2a70f0cd1b4a567cd06c0ef9f32c3d118f762902
Merge: 79697ed 855d758
Author: Ahmet Altay 
AuthorDate: Tue Jun 12 22:48:39 2018 -0700

Merge pull request #5625 from aaltay/expdep

Explain experimental/deprecated from users perspective.

 sdks/python/apache_beam/utils/annotations.py | 11 +++
 1 file changed, 11 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Created] (BEAM-4550) Add information on API support

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-4550:
-

 Summary: Add information on API support
 Key: BEAM-4550
 URL: https://issues.apache.org/jira/browse/BEAM-4550
 Project: Beam
  Issue Type: New Feature
  Components: website
Reporter: Ahmet Altay


Add information to the web site about API and version support, including the 
meaning of experimental and deprecated annotations. Currently this information 
only exists in API docs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4007) Futurize and fix python 2 compatibility for typehints subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4007?focusedWorklogId=111401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111401
 ]

ASF GitHub Bot logged work on BEAM-4007:


Author: ASF GitHub Bot
Created on: 13/Jun/18 05:44
Start Date: 13/Jun/18 05:44
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5337: 
[BEAM-4007] Futurize typehints subpackage
URL: https://github.com/apache/beam/pull/5337#discussion_r194953113
 
 

 ##
 File path: sdks/python/apache_beam/typehints/typehints.py
 ##
 @@ -411,17 +416,23 @@ def __eq__(self, other):
   def __repr__(self):
 return 'Any'
 
+  def __hash__(self):
+return hash(id(self))
 
 Review comment:
   Can we make this `hash(type(self))` or are we running into something like: 
https://issues.apache.org/jira/browse/BEAM-3730 here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111401)
Time Spent: 20m  (was: 10m)

> Futurize and fix python 2 compatibility for typehints subpackage
> 
>
> Key: BEAM-4007
> URL: https://issues.apache.org/jira/browse/BEAM-4007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4007) Futurize and fix python 2 compatibility for typehints subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4007?focusedWorklogId=111402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111402
 ]

ASF GitHub Bot logged work on BEAM-4007:


Author: ASF GitHub Bot
Created on: 13/Jun/18 05:44
Start Date: 13/Jun/18 05:44
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5337: 
[BEAM-4007] Futurize typehints subpackage
URL: https://github.com/apache/beam/pull/5337#discussion_r194953249
 
 

 ##
 File path: sdks/python/apache_beam/typehints/typehints.py
 ##
 @@ -411,17 +416,23 @@ def __eq__(self, other):
   def __repr__(self):
 return 'Any'
 
+  def __hash__(self):
+return hash(id(self))
+
   def type_check(self, instance):
 pass
 
 
 class TypeVariable(AnyTypeConstraint):
 
+  def __init__(self, name):
+self.name = name
+
   def __eq__(self, other):
 return type(self) == type(other) and self.name == other.name
 
-  def __init__(self, name):
-self.name = name
+  def __hash__(self):
 
 Review comment:
   Let's change the implementation `hash(self.name)` to fulfilll the contract 
between `__hash__` and `__eq__`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111402)
Time Spent: 0.5h  (was: 20m)

> Futurize and fix python 2 compatibility for typehints subpackage
> 
>
> Key: BEAM-4007
> URL: https://issues.apache.org/jira/browse/BEAM-4007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4007) Futurize and fix python 2 compatibility for typehints subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4007?focusedWorklogId=111400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111400
 ]

ASF GitHub Bot logged work on BEAM-4007:


Author: ASF GitHub Bot
Created on: 13/Jun/18 05:44
Start Date: 13/Jun/18 05:44
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5337: 
[BEAM-4007] Futurize typehints subpackage
URL: https://github.com/apache/beam/pull/5337#discussion_r194954778
 
 

 ##
 File path: sdks/python/apache_beam/typehints/typehints_test.py
 ##
 @@ -1034,8 +1040,14 @@ class DecoratorHelpers(TypeHintTestCase):
   def test_hint_helper(self):
 self.assertTrue(is_consistent_with(Any, int))
 self.assertTrue(is_consistent_with(int, Any))
-self.assertTrue(is_consistent_with(str, object))
-self.assertFalse(is_consistent_with(object, str))
+# object builtin is shadowed by object imported from future.builtins on
 
 Review comment:
   Trying to understand why we need to use native_object in the test, given 
that we import object from builtins both in the test and in the typehints.py. 
Does the test not pass without this?
   Is this condition 
(https://github.com/RobbeSneyders/beam/blob/4fc7965ba67fd22a7a7c4a3935ec295aca2811be/sdks/python/apache_beam/typehints/typehints.py#L1110)
  evaluated differently without using native_object?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111400)
Time Spent: 10m
Remaining Estimate: 0h

> Futurize and fix python 2 compatibility for typehints subpackage
> 
>
> Key: BEAM-4007
> URL: https://issues.apache.org/jira/browse/BEAM-4007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510648#comment-16510648
 ] 

Ahmet Altay commented on BEAM-4543:
---

I do not know how widely it is used. I am aware of customers using it, but my 
guess is it is not heavily used. It is also marked experimental.

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111399
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 13/Jun/18 05:10
Start Date: 13/Jun/18 05:10
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396815219
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111399)
Time Spent: 52h 40m  (was: 52.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 52h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111398
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 13/Jun/18 05:05
Start Date: 13/Jun/18 05:05
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396814607
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111398)
Time Spent: 52.5h  (was: 52h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 52.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4537) CASE expression output type mismatch

[Kai Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Jiang resolved BEAM-4537.
-
   Resolution: Not A Bug
Fix Version/s: 2.6.0

It should not be problem after 
[PR#5544|https://github.com/apache/beam/pull/5544] (BEAM-4449)

> CASE expression output type mismatch
> 
>
> Key: BEAM-4537
> URL: https://issues.apache.org/jira/browse/BEAM-4537
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Assignee: Kai Jiang
>Priority: Major
> Fix For: 2.6.0
>
>
> TPC-DS query 84 involves with keyword coalesce(). coalesce will expand into 
> case expression.
> output type of CASE expression should match its family type with input types.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4476) Syntax Features Unsupported

[Kai Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510610#comment-16510610
 ] 

Kai Jiang commented on BEAM-4476:
-

part of BEAM-2281

> Syntax Features Unsupported
> ---
>
> Key: BEAM-4476
> URL: https://issues.apache.org/jira/browse/BEAM-4476
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Priority: Major
>
> Based on current version (56dc4cf), a coverage test was done with TPC-DS 
> queries and TPC-H queries. All the tests were running on DirectRunner. We 
> noticed that there are some features Beam SQL is now not supported.  
> This issue is used for an umbrella ticket to keep track of features we need 
> to implement and support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2281) call SqlFunctions in operator implementation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2281?focusedWorklogId=111397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111397
 ]

ASF GitHub Bot logged work on BEAM-2281:


Author: ASF GitHub Bot
Created on: 13/Jun/18 04:39
Start Date: 13/Jun/18 04:39
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #5626: [BEAM-2281] 
Refactoring literal expression type conversion 
URL: https://github.com/apache/beam/pull/5626#issuecomment-396811124
 
 
   R: @kennknowles 
   cc: @apilloud @akedin @XuMingmin


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111397)
Time Spent: 3h 40m  (was: 3.5h)

> call SqlFunctions in operator implementation
> 
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Calcite has a collections of functions in 
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to 
> leverage when adding operators as {{BeamSqlExpression}}. 
> [~xumingming] [~app-tarush], any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2281) call SqlFunctions in operator implementation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2281?focusedWorklogId=111396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111396
 ]

ASF GitHub Bot logged work on BEAM-2281:


Author: ASF GitHub Bot
Created on: 13/Jun/18 04:37
Start Date: 13/Jun/18 04:37
Worklog Time Spent: 10m 
  Work Description: vectorijk opened a new pull request #5626: [BEAM-2281] 
Refactoring literal expression type conversion 
URL: https://github.com/apache/beam/pull/5626
 
 
   call `SqlFunctions` in `BeamSqlFnExecutor` for refactoring literal 
expression type conversion.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111396)
Time Spent: 3.5h  (was: 3h 20m)

> call SqlFunctions in operator implementation
> 
>
> Key: BEAM-2281
> URL: https://issues.apache.org/jira/browse/BEAM-2281
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Calcite has a collections of functions in 
> {{org.apache.calcite.runtime.SqlFunctions}}. It sounds a good source to 
> leverage when adding operators as {{BeamSqlExpression}}. 
> [~xumingming] [~app-tarush], any comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=111394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111394
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 13/Jun/18 04:02
Start Date: 13/Jun/18 04:02
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r194950717
 
 

 ##
 File path: sdks/python/apache_beam/utils/windowed_value.py
 ##
 @@ -178,34 +182,22 @@ def __repr__(self):
 self.windows,
 self.pane_info)
 
+  def _key(self):
+return self.value, self.timestamp_micros, self.windows, self.pane_info
+
+  def __eq__(self, other):
+return (type(self) == type(other)
+and self.timestamp_micros == other.timestamp_micros
+and self.value == self.value
 
 Review comment:
   s/self/other in right hand side of the equality.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111394)
Time Spent: 1h 40m  (was: 1.5h)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=111395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111395
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 13/Jun/18 04:02
Start Date: 13/Jun/18 04:02
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5336: 
[BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r194950463
 
 

 ##
 File path: sdks/python/apache_beam/utils/windowed_value.py
 ##
 @@ -178,34 +182,22 @@ def __repr__(self):
 self.windows,
 self.pane_info)
 
+  def _key(self):
 
 Review comment:
   We don't need this anymore.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111395)
Time Spent: 1h 40m  (was: 1.5h)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111391
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 13/Jun/18 04:00
Start Date: 13/Jun/18 04:00
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#discussion_r194950555
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -398,6 +398,40 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+
+### Build python wheels and upload it to test.pypi.org
+
+1. Create your test.pypi.org account in 
https://test.pypi.org/account/register/ .
+   * Note: Please ping dev@ thread to ask for permissions accessing 
apache_beam in test.pypi.org.
+1. Fork https://github.com/robertwb/apache-beam-wheels .
 
 Review comment:
   From what I know so far, it's better to keep it as a separated repo. How 
about you @robertwb ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111391)
Time Spent: 1h 10m  (was: 1h)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111387
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 13/Jun/18 02:48
Start Date: 13/Jun/18 02:48
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396796728
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111387)
Time Spent: 52h 20m  (was: 52h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 52h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111386
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 13/Jun/18 02:44
Start Date: 13/Jun/18 02:44
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396795994
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111386)
Time Spent: 52h 10m  (was: 52h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 52h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=111385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111385
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 13/Jun/18 02:43
Start Date: 13/Jun/18 02:43
Worklog Time Spent: 10m 
  Work Description: herohde closed pull request #5620: [BEAM-4276] Surface 
graph.Fn encoding functions.
URL: https://github.com/apache/beam/pull/5620
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/graphx/user.go 
b/sdks/go/pkg/beam/core/runtime/graphx/user.go
index ab75b3d9fea..969a513f491 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/user.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/user.go
@@ -23,6 +23,7 @@ import (
"encoding/base64"
 
"github.com/apache/beam/sdks/go/pkg/beam/core/funcx"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
"github.com/apache/beam/sdks/go/pkg/beam/core/util/protox"
@@ -80,6 +81,28 @@ func DecodeFn(data string) (reflectx.Func, error) {
return fn.Fn, nil
 }
 
+// EncodeGraphFn encodes a *graph.Fn as a string.
+func EncodeGraphFn(u *graph.Fn) (string, error) {
+   ref, err := encodeFn(u)
+   if err != nil {
+   return "", err
+   }
+   return protox.EncodeBase64(ref)
+}
+
+// DecodeGraphFn decodes an encoded *graph.Fn.
+func DecodeGraphFn(data string) (*graph.Fn, error) {
+   var ref v1.Fn
+   if err := protox.DecodeBase64(data, ); err != nil {
+   return nil, err
+   }
+   fn, err := decodeFn()
+   if err != nil {
+   return nil, err
+   }
+   return fn, nil
+}
+
 // EncodeCoder encodes a coder as a string. Any custom coder function
 // symbol must be resolvable via the runtime.GlobalSymbolResolver. The types 
must
 // be encodable.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111385)
Time Spent: 8h 40m  (was: 8.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111376=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111376
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 13/Jun/18 01:19
Start Date: 13/Jun/18 01:19
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#discussion_r194932908
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -398,6 +398,40 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+
+### Build python wheels and upload it to test.pypi.org
+
+1. Create your test.pypi.org account in 
https://test.pypi.org/account/register/ .
+   * Note: Please ping dev@ thread to ask for permissions accessing 
apache_beam in test.pypi.org.
+1. Fork https://github.com/robertwb/apache-beam-wheels .
 
 Review comment:
   I was thinking that perhaps it would make sense to move necessary tooling to 
build wheels into some folder in Beam Github repository that contains release 
tools? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111376)
Time Spent: 1h  (was: 50m)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111374=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111374
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 13/Jun/18 01:12
Start Date: 13/Jun/18 01:12
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#discussion_r194932215
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -398,6 +398,40 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+
+### Build python wheels and upload it to test.pypi.org
+
+1. Create your test.pypi.org account in 
https://test.pypi.org/account/register/ .
+   * Note: Please ping dev@ thread to ask for permissions accessing 
apache_beam in test.pypi.org.
+1. Fork https://github.com/robertwb/apache-beam-wheels .
 
 Review comment:
   Hey Valentyn, could you please explain more about your ideas? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111374)
Time Spent: 50m  (was: 40m)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510484#comment-16510484
 ] 

Valentyn Tymofieiev commented on BEAM-4543:
---

I don't have a good answer yet about what it takes to do the migration, 
supporting datastore only in python 2 mode can be a fallback plan. Do we know 
how widely is this IO used?

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111369
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 13/Jun/18 00:58
Start Date: 13/Jun/18 00:58
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#discussion_r194930352
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -398,6 +398,40 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+
+### Build python wheels and upload it to test.pypi.org
+
+1. Create your test.pypi.org account in 
https://test.pypi.org/account/register/ .
+   * Note: Please ping dev@ thread to ask for permissions accessing 
apache_beam in test.pypi.org.
+1. Fork https://github.com/robertwb/apache-beam-wheels .
 
 Review comment:
   Perhaps this can live in https://github.com/apache/beam/tree/master/release 
or something like that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111369)
Time Spent: 40m  (was: 0.5h)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111368
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 13/Jun/18 00:57
Start Date: 13/Jun/18 00:57
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#discussion_r194930352
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -398,6 +398,40 @@ Staging binaries
 
 Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
 
+
+### Build python wheels and upload it to test.pypi.org
+
+1. Create your test.pypi.org account in 
https://test.pypi.org/account/register/ .
+   * Note: Please ping dev@ thread to ask for permissions accessing 
apache_beam in test.pypi.org.
+1. Fork https://github.com/robertwb/apache-beam-wheels .
 
 Review comment:
   Perhaps this can live in beam/release-tools or something like that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111368)
Time Spent: 0.5h  (was: 20m)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #752

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_MongoDBIO_IT #302

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510423#comment-16510423
 ] 

Ahmet Altay commented on BEAM-4543:
---

Do you have an idea of what would it take to this migration? Is it possible to 
support datastore only in python 2 mode until we can figure out the migration 
plan.

Valentyn, I am assigning this to you for the above questions.

cc: [~chamikara]

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-4543:
-

Assignee: Valentyn Tymofieiev

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4535) Python tests are failing for Windows

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4535?focusedWorklogId=111353=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111353
 ]

ASF GitHub Bot logged work on BEAM-4535:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:49
Start Date: 12/Jun/18 23:49
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5624: [BEAM-4535] Add a custom 
_url_dirname for local filesystems.
URL: https://github.com/apache/beam/pull/5624#issuecomment-396769737
 
 
   CC: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111353)
Time Spent: 20m  (was: 10m)

> Python tests are failing for Windows
> 
>
> Key: BEAM-4535
> URL: https://issues.apache.org/jira/browse/BEAM-4535
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Error is:
> Traceback (most recent call last):
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource_test.py", line 532, in test_read_auto_pattern
>     compression_type=CompressionTypes.AUTO))
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 119, in __init__
>     self._validate()
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\options\v
> alue_provider.py", line 133, in _f
>     return fnc(self, *args, **kwargs)
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 179, in _validate
>     'No files found based on the file pattern %s' % pattern)
> IOError: No files found based on the file pattern 
> c:\windows\temp\tmpwon5_g\mytemp*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4535) Python tests are failing for Windows

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4535?focusedWorklogId=111352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111352
 ]

ASF GitHub Bot logged work on BEAM-4535:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:49
Start Date: 12/Jun/18 23:49
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5624: [BEAM-4535] Add a custom 
_url_dirname for local filesystems.
URL: https://github.com/apache/beam/pull/5624#issuecomment-396769651
 
 
   R: @jbonofre 
   
   Added this cherrypick as well, which Cham mentioned in the voting thread 
yesterday.
   If fixes https://issues.apache.org/jira/browse/BEAM-4535


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111352)
Time Spent: 10m
Remaining Estimate: 0h

> Python tests are failing for Windows
> 
>
> Key: BEAM-4535
> URL: https://issues.apache.org/jira/browse/BEAM-4535
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Error is:
> Traceback (most recent call last):
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource_test.py", line 532, in test_read_auto_pattern
>     compression_type=CompressionTypes.AUTO))
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 119, in __init__
>     self._validate()
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\options\v
> alue_provider.py", line 133, in _f
>     return fnc(self, *args, **kwargs)
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 179, in _validate
>     'No files found based on the file pattern %s' % pattern)
> IOError: No files found based on the file pattern 
> c:\windows\temp\tmpwon5_g\mytemp*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=111351=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111351
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:49
Start Date: 12/Jun/18 23:49
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5620: [BEAM-4276] Surface 
graph.Fn encoding functions.
URL: https://github.com/apache/beam/pull/5620#issuecomment-396769557
 
 
   R: @herohde PTAL Comments addressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111351)
Time Spent: 8.5h  (was: 8h 20m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=111350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111350
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:48
Start Date: 12/Jun/18 23:48
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #5620: 
[BEAM-4276] Surface graph.Fn encoding functions.
URL: https://github.com/apache/beam/pull/5620#discussion_r194921365
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/user.go
 ##
 @@ -80,6 +81,28 @@ func DecodeFn(data string) (reflectx.Func, error) {
return fn.Fn, nil
 }
 
+// EncodeGraphFn encodes a *graph.Fn as a string.
+func EncodeGraphFn(u *graph.Fn) (string, error) {
 
 Review comment:
   It's related to Combiner Lifting, so I've used that jira tag.
   Since it can help with writing a bundle level precombine, using existing 
combineFns, which is what I'm using it for at present, since the beam model/SDK 
doesn't yet support lifting combines through a CoGroupByKey. That's the root of 
it anyway. I like thinking of it as enabling the writing of MetaStructs.
   
   The main goal is for me to avoid needing to write my precombine helper one 
way now and then re-writing later when there is official support for such a 
thing. If that feature does show up, I can simply use the same CombineFns over 
again.
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111350)
Time Spent: 8h 20m  (was: 8h 10m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4535) Python tests are failing for Windows

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510401#comment-16510401
 ] 

Udi Meiri commented on BEAM-4535:
-

Fixed in https://github.com/apache/beam/pull/5476

> Python tests are failing for Windows
> 
>
> Key: BEAM-4535
> URL: https://issues.apache.org/jira/browse/BEAM-4535
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Udi Meiri
>Priority: Major
>
> Error is:
> Traceback (most recent call last):
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource_test.py", line 532, in test_read_auto_pattern
>     compression_type=CompressionTypes.AUTO))
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 119, in __init__
>     self._validate()
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\options\v
> alue_provider.py", line 133, in _f
>     return fnc(self, *args, **kwargs)
>   File 
> "C:\Users\deft-testing-integra\python_sdk_download\apache_beam\io\fileba
> sedsource.py", line 179, in _validate
>     'No files found based on the file pattern %s' % pattern)
> IOError: No files found based on the file pattern 
> c:\windows\temp\tmpwon5_g\mytemp*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4549) Streaming pipelines with multiple side inputs fail on DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4549?focusedWorklogId=111343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111343
 ]

ASF GitHub Bot logged work on BEAM-4549:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:39
Start Date: 12/Jun/18 23:39
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5623: [BEAM-4549] 
Use per-pipeline unique ids for side inputs in DataflowRunner
URL: https://github.com/apache/beam/pull/5623#issuecomment-396767964
 
 
   Thanks, PTAL.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111343)
Time Spent: 40m  (was: 0.5h)

> Streaming pipelines with multiple side inputs fail on DataflowRunner
> 
>
> Key: BEAM-4549
> URL: https://issues.apache.org/jira/browse/BEAM-4549
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Streaming pipelines with multiple side inputs currently fail on 
> DataflowRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2873) Detect number of shards for file sink in Flink Streaming Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2873?focusedWorklogId=111342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111342
 ]

ASF GitHub Bot logged work on BEAM-2873:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:39
Start Date: 12/Jun/18 23:39
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #4760: [BEAM-2873] 
Setting number of shards for writes with runner determined sharding
URL: https://github.com/apache/beam/pull/4760#issuecomment-396767963
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111342)
Time Spent: 3h 20m  (was: 3h 10m)

> Detect number of shards for file sink in Flink Streaming Runner
> ---
>
> Key: BEAM-2873
> URL: https://issues.apache.org/jira/browse/BEAM-2873
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> [~reuvenlax] mentioned that this is done for the Dataflow Runner and the 
> default behaviour on Flink can be somewhat surprising for users.
> ML entry: https://www.mail-archive.com/dev@beam.apache.org/msg02665.html:
> This is how the file sink has always worked in Beam. If no sharding is 
> specified, then this means runner-determined sharding, and by default that is 
> one file per bundle. If Flink has small bundles, then I suggest using the 
> withNumShards method to explicitly pick the number of output shards.
> The Flink runner can detect that runner-determined sharding has been chosen, 
> and override it with a specific number of shards. For example, the Dataflow 
> streaming runner (which as you mentioned also has small bundles) detects this 
> case and sets the number of out files shards based on the number of workers 
> in the worker pool 
> [Here|https://github.com/apache/beam/blob/9e6530adb00669b7cf0f01cb8b128be0a21fd721/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L354]
>  is the code that does this; it should be quite simple to do something 
> similar for Flink, and then there will be no need for users to explicitly 
> call withNumShards themselves.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510381#comment-16510381
 ] 

Scott Wegner commented on BEAM-4494:


Prototype script based on [PR#5556|https://github.com/apache/beam/pull/5556]: 
https://gist.github.com/swegner/233ede6e4764009a93bcaf0091bf02d2

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111338
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:25
Start Date: 12/Jun/18 23:25
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396765720
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111338)
Time Spent: 52h  (was: 51h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 52h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4549) Streaming pipelines with multiple side inputs fail on DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4549?focusedWorklogId=111335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111335
 ]

ASF GitHub Bot logged work on BEAM-4549:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:24
Start Date: 12/Jun/18 23:24
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5623: 
[BEAM-4549] Use per-pipeline unique ids for side inputs in DataflowRunner
URL: https://github.com/apache/beam/pull/5623#discussion_r194917486
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
 ##
 @@ -587,10 +592,10 @@ def run_ParDo(self, transform_node):
 si_labels = {}
 full_label_counts = defaultdict(int)
 lookup_label = lambda side_pval: si_labels[side_pval]
-for ix, side_pval in enumerate(transform_node.side_inputs):
+for side_pval in transform_node.side_inputs:
   assert isinstance(side_pval, AsSideInput)
   step_name = 'SideInput-' + self._get_unique_step_name()
-  si_label = 'side%d' % ix
+  si_label = self._get_unique_side_input_name()
 
 Review comment:
   As I mentioned, it'd be cleaner to let this be step name + ix rather than 
introducing global state. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111335)
Time Spent: 0.5h  (was: 20m)

> Streaming pipelines with multiple side inputs fail on DataflowRunner
> 
>
> Key: BEAM-4549
> URL: https://issues.apache.org/jira/browse/BEAM-4549
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Streaming pipelines with multiple side inputs currently fail on 
> DataflowRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

[Robert Bradshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510373#comment-16510373
 ] 

Robert Bradshaw commented on BEAM-4494:
---

Is there any advantage to having a separate branch? There are lots of 
downsides...

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4549) Streaming pipelines with multiple side inputs fail on DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4549?focusedWorklogId=111334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111334
 ]

ASF GitHub Bot logged work on BEAM-4549:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:22
Start Date: 12/Jun/18 23:22
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5623: [BEAM-4549] 
Use per-pipeline unique ids for side inputs in DataflowRunner
URL: https://github.com/apache/beam/pull/5623#issuecomment-396765068
 
 
   R: @robertwb 
   CC: @aaltay @boyuanzz 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111334)
Time Spent: 20m  (was: 10m)

> Streaming pipelines with multiple side inputs fail on DataflowRunner
> 
>
> Key: BEAM-4549
> URL: https://issues.apache.org/jira/browse/BEAM-4549
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Streaming pipelines with multiple side inputs currently fail on 
> DataflowRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4549) Streaming pipelines with multiple side inputs fail on DataflowRunner

[Charles Chen (JIRA)]

Charles Chen created BEAM-4549:
--

 Summary: Streaming pipelines with multiple side inputs fail on 
DataflowRunner
 Key: BEAM-4549
 URL: https://issues.apache.org/jira/browse/BEAM-4549
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


Streaming pipelines with multiple side inputs currently fail on DataflowRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4549) Streaming pipelines with multiple side inputs fail on DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4549?focusedWorklogId=111333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111333
 ]

ASF GitHub Bot logged work on BEAM-4549:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:21
Start Date: 12/Jun/18 23:21
Worklog Time Spent: 10m 
  Work Description: charlesccychen opened a new pull request #5623: 
[BEAM-4549] Use per-pipeline unique ids for side inputs in DataflowRunner
URL: https://github.com/apache/beam/pull/5623
 
 
   This change fixes an issue where streaming pipelines with multiple side 
inputs fail on the DataflowRunner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111333)
Time Spent: 10m
Remaining Estimate: 0h

> Streaming pipelines with multiple side inputs fail on DataflowRunner
> 
>
> Key: BEAM-4549
> URL: https://issues.apache.org/jira/browse/BEAM-4549
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Streaming pipelines with multiple side inputs currently fail on 
> DataflowRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111332
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:20
Start Date: 12/Jun/18 23:20
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396764756
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111332)
Time Spent: 51h 50m  (was: 51h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111330
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:16
Start Date: 12/Jun/18 23:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396763951
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111330)
Time Spent: 51h 40m  (was: 51.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111329
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:16
Start Date: 12/Jun/18 23:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396763320
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111329)
Time Spent: 51.5h  (was: 51h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111328
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:16
Start Date: 12/Jun/18 23:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396762498
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111328)
Time Spent: 51h 20m  (was: 51h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111327
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:16
Start Date: 12/Jun/18 23:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396763951
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111327)
Time Spent: 51h 10m  (was: 51h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111324=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111324
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:12
Start Date: 12/Jun/18 23:12
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396763320
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111324)
Time Spent: 51h  (was: 50h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 51h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3708) Implement the portable lifted Combiner transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3708?focusedWorklogId=111320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111320
 ]

ASF GitHub Bot logged work on BEAM-3708:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:09
Start Date: 12/Jun/18 23:09
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #5622: [BEAM-3708] 
Adding Combine component implementations to Java SDK
URL: https://github.com/apache/beam/pull/5622
 
 
   Adding implementations for the components represented by the Combine
   URNs described in the portable combines doc:
   https://s.apache.org/beam-runner-api-combine-model
   
   This is just an initial implementation so it's lacking one major feature
   which is an optimization in the Precombine. Planning to add that
   right away, but want to get this in first.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111320)
Time Spent: 10m
Remaining Estimate: 0h

> Implement the portable lifted Combiner transforms in Java SDK
> -
>
> Key: BEAM-3708
> URL: https://issues.apache.org/jira/browse/BEAM-3708
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Java SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3708) Implement the portable lifted Combiner transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3708?focusedWorklogId=111323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111323
 ]

ASF GitHub Bot logged work on BEAM-3708:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:09
Start Date: 12/Jun/18 23:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #5622: [BEAM-3708] Adding 
Combine component implementations to Java SDK
URL: https://github.com/apache/beam/pull/5622#issuecomment-396762786
 
 
   R: @lukecwik


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111323)
Time Spent: 0.5h  (was: 20m)

> Implement the portable lifted Combiner transforms in Java SDK
> -
>
> Key: BEAM-3708
> URL: https://issues.apache.org/jira/browse/BEAM-3708
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Java SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3708) Implement the portable lifted Combiner transforms in Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3708?focusedWorklogId=111322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111322
 ]

ASF GitHub Bot logged work on BEAM-3708:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:09
Start Date: 12/Jun/18 23:09
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #5622: [BEAM-3708] Adding 
Combine component implementations to Java SDK
URL: https://github.com/apache/beam/pull/5622#issuecomment-396762786
 
 
   @lukecwik


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111322)
Time Spent: 20m  (was: 10m)

> Implement the portable lifted Combiner transforms in Java SDK
> -
>
> Key: BEAM-3708
> URL: https://issues.apache.org/jira/browse/BEAM-3708
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Java SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111319
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:08
Start Date: 12/Jun/18 23:08
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396762498
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111319)
Time Spent: 50h 50m  (was: 50h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111316
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 23:04
Start Date: 12/Jun/18 23:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396761902
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111316)
Time Spent: 50h 40m  (was: 50.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111315
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 22:59
Start Date: 12/Jun/18 22:59
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396761025
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111315)
Time Spent: 50.5h  (was: 50h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #751

[Apache Jenkins Server]

See 


Changes:

[swegner] [BEAM-925] Fix FindBugs violations in Dataflow Runner and re-enable

[swegner] [BEAM-4313] Fix ErrorProne warnings and enable validation in Dataflow

--
[...truncated 17.53 MB...]
Jun 12, 2018 10:49:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-06-12_15_49_08-17818686313465256589
Jun 12, 2018 10:49:09 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-06-12_15_49_08-17818686313465256589 with 0 
expected assertions.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:09.149Z: Autoscaling is enabled for job 
2018-06-12_15_49_08-17818686313465256589. The number of workers will be between 
1 and 1000.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:09.164Z: Autoscaling was automatically enabled for 
job 2018-06-12_15_49_08-17818686313465256589.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:11.769Z: Checking required Cloud APIs are enabled.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:12.004Z: Checking permissions granted to controller 
Service Account.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:15.483Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:15.793Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:15.925Z: Expanding GroupByKey operations into 
optimizable parts.
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:15.947Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.089Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.114Z: Elided trivial flatten 
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.146Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.166Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.187Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.210Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jun 12, 2018 10:49:18 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-12T22:49:16.232Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jun 12, 2018 10:49:18 PM 

[jira] [Updated] (BEAM-4548) Long execution delay when using DirectRunner to read from BigQuery Table

[Brian Foo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Foo updated BEAM-4548:

Description: 
When using DirectRunner to execute a simple select query against a BigQuery 
table that contains 100 rows, the pipeline stalls for over 3 minutes. The 
BigQuery UI can run the same query in under 2 seconds.

A similar issue was reported here: 
[https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read|https://www.google.com/url?q=https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read=D=hangouts=1528912448506000=AFQjCNHp9JWHFJOnJlBJmLODU1cGBIeXtg]

I ran a thread dump using Visual M seems like the main thread was in a state of 
backoff: 

java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at com.google.api.client.util.Sleeper$1.sleep(Sleeper.java:43)
 at com.google.api.client.util.BackOffUtils.next(BackOffUtils.java:50)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.nextBackOff(BigQueryServicesImpl.java:870)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$500(BigQueryServicesImpl.java:79)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:273)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:247)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.executeQuery(BigQueryQuerySource.java:191)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.getTableToExtract(BigQueryQuerySource.java:136)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:103)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:134)
 at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210)
 at 
org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
 at 
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)
 at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:144)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)

  was:
When using DirectRunner to execute a simple select query against a BigQuery 
table that contains 100 rows, the pipeline stalls for about 5 minutes. The 
BigQuery UI can run the same query in under 2 seconds.

A similar issue was reported here: 
[https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read|https://www.google.com/url?q=https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read=D=hangouts=1528912448506000=AFQjCNHp9JWHFJOnJlBJmLODU1cGBIeXtg]

I ran a thread dump using Visual M seems like the main thread was in a state of 
backoff: 

java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at com.google.api.client.util.Sleeper$1.sleep(Sleeper.java:43)
 at com.google.api.client.util.BackOffUtils.next(BackOffUtils.java:50)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.nextBackOff(BigQueryServicesImpl.java:870)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$500(BigQueryServicesImpl.java:79)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:273)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:247)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.executeQuery(BigQueryQuerySource.java:191)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.getTableToExtract(BigQueryQuerySource.java:136)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:103)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:134)
 at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210)
 at 
org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
 at 
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)
 at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:144)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
 at 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111310
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 22:47
Start Date: 12/Jun/18 22:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396758660
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111310)
Time Spent: 50h 20m  (was: 50h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=111308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111308
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 12/Jun/18 22:40
Start Date: 12/Jun/18 22:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-396757342
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111308)
Time Spent: 50h 10m  (was: 50h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 50h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4548) Long execution delay when using DirectRunner to read from BigQuery Table

[Brian Foo (JIRA)]

Brian Foo created BEAM-4548:
---

 Summary: Long execution delay when using DirectRunner to read from 
BigQuery Table
 Key: BEAM-4548
 URL: https://issues.apache.org/jira/browse/BEAM-4548
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp, runner-direct
Affects Versions: 2.4.0
Reporter: Brian Foo
Assignee: Chamikara Jayalath


When using DirectRunner to execute a simple select query against a BigQuery 
table that contains 100 rows, the pipeline stalls for about 5 minutes. The 
BigQuery UI can run the same query in under 2 seconds.

A similar issue was reported here: 
[https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read|https://www.google.com/url?q=https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read=D=hangouts=1528912448506000=AFQjCNHp9JWHFJOnJlBJmLODU1cGBIeXtg]

I ran a thread dump using Visual M seems like the main thread was in a state of 
backoff: 

java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at com.google.api.client.util.Sleeper$1.sleep(Sleeper.java:43)
 at com.google.api.client.util.BackOffUtils.next(BackOffUtils.java:50)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.nextBackOff(BigQueryServicesImpl.java:870)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$500(BigQueryServicesImpl.java:79)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:273)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:247)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.executeQuery(BigQueryQuerySource.java:191)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.getTableToExtract(BigQueryQuerySource.java:136)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:103)
 at 
org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:134)
 at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210)
 at 
org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87)
 at 
org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62)
 at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:144)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201)
 at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4385) Support LIKE operator

[Kai Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Jiang resolved BEAM-4385.
-
   Resolution: Fixed
Fix Version/s: 2.6.0

> Support LIKE operator
> -
>
> Key: BEAM-4385
> URL: https://issues.apache.org/jira/browse/BEAM-4385
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Rui Wang
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Currently the LIKE operator is not supported. It is pretty important.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4547) implement sum0 aggregation function

[Kai Jiang (JIRA)]

Kai Jiang created BEAM-4547:
---

 Summary: implement sum0 aggregation function
 Key: BEAM-4547
 URL: https://issues.apache.org/jira/browse/BEAM-4547
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Kai Jiang
Assignee: Kai Jiang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510298#comment-16510298
 ] 

Scott Wegner commented on BEAM-4494:


Starting to look at this now. The plan is to first move code into a separate 
branch. I can do this on my own fork for now, but eventually we'll want to push 
to apache/beam in order to develop Jenkins tests. 

Does anybody know the process for pushing a new branch? /cc [~robertwb] 
[~herohde]

> Migrate website source code to apache/beam [website-migration] branch
> -
>
> Key: BEAM-4494
> URL: https://issues.apache.org/jira/browse/BEAM-4494
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=111304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111304
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 12/Jun/18 22:07
Start Date: 12/Jun/18 22:07
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5540: [BEAM-4313] Fix 
and enforce FindBugs and ErrorProne violations in Dataflow Runner
URL: https://github.com/apache/beam/pull/5540
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
index 295e7c86fe2..ff258030635 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
@@ -218,7 +218,7 @@ private MetricsContainerStepMapMetricResults(
 }
 
 @Override
-public MetricQueryResults queryMetrics(MetricsFilter filter) {
+public MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
   return new QueryResults(filter);
 }
 
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
index 85d9acc7cc0..42d72ebcbac 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
@@ -29,6 +29,7 @@
 import java.util.concurrent.Executor;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
 import javax.annotation.concurrent.GuardedBy;
 import org.apache.beam.runners.core.construction.metrics.MetricFiltering;
 import org.apache.beam.runners.core.construction.metrics.MetricKey;
@@ -257,7 +258,7 @@ public static MetricQueryResults create(
   }
 
   @Override
-  public MetricQueryResults queryMetrics(MetricsFilter filter) {
+  public MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
 ImmutableList.Builder> counterResults = 
ImmutableList.builder();
 for (Entry> counter : 
counters.entries()) {
   maybeExtractResult(filter, counterResults, counter);
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
index f8f7cbd95bf..cb217e26e39 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
@@ -31,6 +31,7 @@
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
 import javax.annotation.concurrent.GuardedBy;
 import org.apache.beam.runners.core.construction.metrics.MetricFiltering;
 import org.apache.beam.runners.core.construction.metrics.MetricKey;
@@ -258,7 +259,7 @@ public static MetricQueryResults create(
   }
 
   @Override
-  public MetricQueryResults queryMetrics(MetricsFilter filter) {
+  public MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
 ImmutableList.Builder> counterResults = 
ImmutableList.builder();
 for (Entry> counter : 
counters.entries()) {
   maybeExtractResult(filter, counterResults, counter);
diff --git a/runners/google-cloud-dataflow-java/build.gradle 
b/runners/google-cloud-dataflow-java/build.gradle
index c5deb9c0a90..9195ccec42c 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -19,7 +19,7 @@
 import groovy.json.JsonOutput
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature(enableFindbugs: false /* BEAM-925 */)
+applyJavaNature(failOnWarning: true)
 
 description = "Apache Beam :: Runners :: Google Cloud Dataflow"
 
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
index 16087bdb56c..145a6a23e1b 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
+++ 

[beam] branch master updated (6a327f2 -> 9d3ee79)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6a327f2  Merge pull request #5610: Followup PR for #5591
 add cd2ad5e  [BEAM-925] Fix FindBugs violations in Dataflow Runner and 
re-enable
 add 1210c21  [BEAM-4313] Fix ErrorProne warnings and enable validation in 
Dataflow Runner
 new 9d3ee79  Merge pull request #5540: [BEAM-4313] Fix and enforce 
FindBugs and ErrorProne violations in Dataflow Runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/metrics/MetricsContainerStepMap.java  |  2 +-
 .../apache/beam/runners/direct/DirectMetrics.java  |  3 +-
 .../runners/direct/portable/DirectMetrics.java |  3 +-
 runners/google-cloud-dataflow-java/build.gradle|  2 +-
 .../beam/runners/dataflow/BatchViewOverrides.java  | 21 
 .../beam/runners/dataflow/DataflowMetrics.java |  9 +++-
 .../dataflow/DataflowPipelineTranslator.java   |  8 +--
 .../beam/runners/dataflow/DataflowRunner.java  | 10 ++--
 .../beam/runners/dataflow/DataflowRunnerInfo.java  |  1 +
 .../beam/runners/dataflow/TestDataflowRunner.java  |  5 +-
 .../beam/runners/dataflow/internal/IsmFormat.java  |  8 +--
 .../beam/runners/dataflow/util/CloudKnownType.java |  7 +--
 .../beam/runners/dataflow/util/CloudObject.java|  2 +-
 .../beam/runners/dataflow/util/MonitoringUtil.java |  3 +-
 .../beam/runners/dataflow/util/PackageUtil.java|  7 ++-
 .../runners/dataflow/util/RandomAccessData.java|  8 ++-
 .../dataflow/BatchStatefulParDoOverridesTest.java  |  4 +-
 .../dataflow/DataflowPipelineTranslatorTest.java   |  3 +-
 .../beam/runners/dataflow/DataflowRunnerTest.java  |  3 +-
 .../dataflow/PrimitiveParDoSingleFactoryTest.java  |  5 +-
 .../runners/dataflow/TestDataflowRunnerTest.java   |  2 +-
 .../options/DataflowPipelineOptionsTest.java   | 62 +++---
 .../runners/dataflow/util/MonitoringUtilTest.java  |  4 +-
 .../runners/dataflow/util/PackageUtilTest.java | 17 +++---
 .../beam/runners/dataflow/util/StructsTest.java|  2 +-
 .../src/main/resources/beam/findbugs-filter.xml| 27 ++
 .../org/apache/beam/sdk/metrics/MetricResults.java |  3 +-
 27 files changed, 146 insertions(+), 85 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5540: [BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow Runner

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9d3ee79454a6065a71580730dbc2021f228266a2
Merge: 6a327f2 1210c21
Author: Kenn Knowles 
AuthorDate: Tue Jun 12 15:07:30 2018 -0700

Merge pull request #5540: [BEAM-4313] Fix and enforce FindBugs and 
ErrorProne violations in Dataflow Runner

 .../core/metrics/MetricsContainerStepMap.java  |  2 +-
 .../apache/beam/runners/direct/DirectMetrics.java  |  3 +-
 .../runners/direct/portable/DirectMetrics.java |  3 +-
 runners/google-cloud-dataflow-java/build.gradle|  2 +-
 .../beam/runners/dataflow/BatchViewOverrides.java  | 21 
 .../beam/runners/dataflow/DataflowMetrics.java |  9 +++-
 .../dataflow/DataflowPipelineTranslator.java   |  8 +--
 .../beam/runners/dataflow/DataflowRunner.java  | 10 ++--
 .../beam/runners/dataflow/DataflowRunnerInfo.java  |  1 +
 .../beam/runners/dataflow/TestDataflowRunner.java  |  5 +-
 .../beam/runners/dataflow/internal/IsmFormat.java  |  8 +--
 .../beam/runners/dataflow/util/CloudKnownType.java |  7 +--
 .../beam/runners/dataflow/util/CloudObject.java|  2 +-
 .../beam/runners/dataflow/util/MonitoringUtil.java |  3 +-
 .../beam/runners/dataflow/util/PackageUtil.java|  7 ++-
 .../runners/dataflow/util/RandomAccessData.java|  8 ++-
 .../dataflow/BatchStatefulParDoOverridesTest.java  |  4 +-
 .../dataflow/DataflowPipelineTranslatorTest.java   |  3 +-
 .../beam/runners/dataflow/DataflowRunnerTest.java  |  3 +-
 .../dataflow/PrimitiveParDoSingleFactoryTest.java  |  5 +-
 .../runners/dataflow/TestDataflowRunnerTest.java   |  2 +-
 .../options/DataflowPipelineOptionsTest.java   | 62 +++---
 .../runners/dataflow/util/MonitoringUtilTest.java  |  4 +-
 .../runners/dataflow/util/PackageUtilTest.java | 17 +++---
 .../beam/runners/dataflow/util/StructsTest.java|  2 +-
 .../src/main/resources/beam/findbugs-filter.xml| 27 ++
 .../org/apache/beam/sdk/metrics/MetricResults.java |  3 +-
 27 files changed, 146 insertions(+), 85 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=111302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111302
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 12/Jun/18 22:02
Start Date: 12/Jun/18 22:02
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r194902611
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   I am trying to remove the GPL dependency in 
https://github.com/apache/beam/pull/5609. It's broken the build, so I will 
investigate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111302)
Time Spent: 3.5h  (was: 3h 20m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111299
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:56
Start Date: 12/Jun/18 21:56
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396747608
 
 
   R: @iemejia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111299)
Time Spent: 1h 20m  (was: 1h 10m)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111298
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:55
Start Date: 12/Jun/18 21:55
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396747330
 
 
   I've run the Seed job to apply these changes, and opened PR's to test 
triggering on various paths:
   
   * #5612: top-level build file, runs [go,java,python]
   * #5613: examples/, runs java only
   * #5614: sdks/go/, runs go only
   * #5615: sdks/java/, runs java only
   * #5616: model/, runs [go,java,python]
   * #5617 sdks/python, runs python only
   * #5618 release/, runs [go,java,python]
   * #5619 runners/, runs [go,java,python]


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111298)
Time Spent: 1h 10m  (was: 1h)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4546) Implement with hot key fanout for combiners

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-4546:
-

 Summary: Implement with hot key fanout for combiners
 Key: BEAM-4546
 URL: https://issues.apache.org/jira/browse/BEAM-4546
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py-core
Reporter: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=111296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111296
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:43
Start Date: 12/Jun/18 21:43
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r194898428
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   Excellent point @swegner, it seems that as part of the migration to gradle 
we forgot to use the correct library for findbugs-annotations, we should be 
using this one https://github.com/stephenc/findbugs-annotations that is Apache 
licensed as the maven build does.
   But gradle shows the one you mention and we cannot add LGPL code for this, 
so this is a release blocker.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111296)
Time Spent: 3h 20m  (was: 3h 10m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=111294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111294
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:43
Start Date: 12/Jun/18 21:43
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r194898428
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   Excellent point @swegner, it seems that as part of the migration to gradle 
we forgot to use the correct library for findbugs-annotations, we should be 
using this one https://github.com/stephenc/findbugs-annotations as the maven 
build does.
   But gradle shows the one you mention and we cannot add LGPL code for this, 
so this is a release blocker.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111294)
Time Spent: 3h 10m  (was: 3h)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111291
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:42
Start Date: 12/Jun/18 21:42
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396744111
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111291)
Time Spent: 40m  (was: 0.5h)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111293
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:42
Start Date: 12/Jun/18 21:42
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396737595
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111293)
Time Spent: 1h  (was: 50m)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111292
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:42
Start Date: 12/Jun/18 21:42
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396719936
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111292)
Time Spent: 50m  (was: 40m)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=111288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111288
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:35
Start Date: 12/Jun/18 21:35
Worklog Time Spent: 10m 
  Work Description: swegner commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r194896497
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   I see in 
[`build_rules.gradle`](https://github.com/apache/beam/blob/6a327f222bb6f7fedb694405fc945caf4671dbbb/build_rules.gradle#L196)
 that we have a `findbugs_annotations` dependency which looks new. It points to 
[`com.google.code.findbugs:findbugs-annotations:3.0.1`](https://mvnrepository.com/artifact/com.google.code.findbugs/findbugs-annotations/3.0.1).
 Looking in the jar I see the `@SuppressFBWarmings` annotation, so this might 
work, although it's also listed as LGPL. Is LGPL ok to use?
   
   I don't see any existing usages of `@SuppressFBWarnings` in our code base, 
which is why I didn't tackle it here. As you mentioned above, let's save it for 
another day.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111288)
Time Spent: 3h  (was: 2h 50m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111286
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:27
Start Date: 12/Jun/18 21:27
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #467: [BEAM-4545]Update 
release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467#issuecomment-396739932
 
 
   I'm not sure whether Release Manager is supposed to have himself own pypi 
account. If it's better to let all Release Manager use 'apache-beam-travis', 
I'll update. @robertwb @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111286)
Time Spent: 20m  (was: 10m)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111285
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:19
Start Date: 12/Jun/18 21:19
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396737595
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111285)
Time Spent: 0.5h  (was: 20m)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4545) Release guide should contain how to build python wheels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?focusedWorklogId=111284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111284
 ]

ASF GitHub Bot logged work on BEAM-4545:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:18
Start Date: 12/Jun/18 21:18
Worklog Time Spent: 10m 
  Work Description: boyuanzz opened a new pull request #467: 
[BEAM-4545]Update release guide with how to build python wheel
URL: https://github.com/apache/beam-site/pull/467
 
 
   r: @robertwb @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111284)
Time Spent: 10m
Remaining Estimate: 0h

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4545) Release guide should contain how to build python wheels

[Boyuan Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang updated BEAM-4545:
---
Summary: Release guide should contain how to build python wheels  (was: 
Release guide should contains how to build python wheels)

> Release guide should contain how to build python wheels
> ---
>
> Key: BEAM-4545
> URL: https://issues.apache.org/jira/browse/BEAM-4545
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4545) Release guide should contains how to build python wheels

[Boyuan Zhang (JIRA)]

Boyuan Zhang created BEAM-4545:
--

 Summary: Release guide should contains how to build python wheels
 Key: BEAM-4545
 URL: https://issues.apache.org/jira/browse/BEAM-4545
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Boyuan Zhang
Assignee: Boyuan Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=111280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111280
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:04
Start Date: 12/Jun/18 21:04
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r194887837
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.io.InvalidObjectException;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
+extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+implements ArtifactRetrievalService {
+
+  private static int artifactChunkSizeBytes = 200; // 2MB
+
+  public static DfsArtifactRetrievalService create() {
+return new DfsArtifactRetrievalService();
+  }
+
+  @Override
+  public void getManifest(
+  ArtifactApi.GetManifestRequest request,
+  StreamObserver responseObserver) {
+try {
+  ArtifactApi.ProxyManifest proxyManifest = 
loadManifestFrom(request.getRetrievalToken());
+  ArtifactApi.GetManifestResponse response =
+  ArtifactApi.GetManifestResponse
+  .newBuilder()
+  .setManifest(proxyManifest.getManifest())
+  .build();
+  responseObserver.onNext(response);
+  responseObserver.onCompleted();
+} catch (Exception e) {
+  responseObserver.onError(e);
+}
+  }
+
+  @Override
+  public void getArtifact(
+  ArtifactApi.GetArtifactRequest request,
+  StreamObserver responseObserver) {
+try {
+  ArtifactApi.ProxyManifest proxyManifest = 
loadManifestFrom(request.getRetrievalToken());
+  // validate that name is contained in manifest and location list
+  boolean containsArtifactName =
+  ImmutableList.copyOf(proxyManifest.getManifest().getArtifactList())
+  .stream()
+  .anyMatch(metadata -> 
metadata.getName().equals(request.getName()));
+  if (!containsArtifactName)  {
+throw new ArtifactNotFoundException(request.getName());
+  }
+  // look for file at URI specified by proxy manifest location
+  ImmutableList locationList =
+  ImmutableList.copyOf(proxyManifest.getLocationList());
+  ArtifactApi.ProxyManifest.Location location =
+  locationList
+  .stream()
+  .filter(loc -> loc.getName().equals(request.getName()))
+  .findFirst()
+  .orElseThrow(() -> new 
ArtifactNotFoundException(request.getName()));
+  ResourceId artifactResourceId =
+  FileSystems.matchNewResource(location.getUri(), false /* is 
directory */);
+  ReadableByteChannel artifactByteChannel = 
FileSystems.open(artifactResourceId);
+  ByteBuffer byteBuffer = ByteBuffer.allocate(artifactChunkSizeBytes);
+  while (artifactByteChannel.read(byteBuffer) > -1) {
+ByteString data = ByteString.copyFrom(byteBuffer);
 
 Review comment:
   Huh, yes I do.  Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go 

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=111279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111279
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:04
Start Date: 12/Jun/18 21:04
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r194886942
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.io.InvalidObjectException;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
 
 Review comment:
   I like that name better.  Will fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111279)
Time Spent: 4h 20m  (was: 4h 10m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=111278=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111278
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 12/Jun/18 21:04
Start Date: 12/Jun/18 21:04
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r194888360
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactNotFoundException.java
 ##
 @@ -0,0 +1,19 @@
+/*
 
 Review comment:
   Yep.  I ended up making it a nested class inside 
DfsArtifactRetrievalService, but looks like I never cleaned up the file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111278)
Time Spent: 4h 10m  (was: 4h)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4008) Futurize and fix python 2 compatibility for utils subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4008?focusedWorklogId=111265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111265
 ]

ASF GitHub Bot logged work on BEAM-4008:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:34
Start Date: 12/Jun/18 20:34
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5336: [BEAM-4008] Futurize utils subpackage
URL: https://github.com/apache/beam/pull/5336#discussion_r194879406
 
 

 ##
 File path: sdks/python/apache_beam/utils/windowed_value.py
 ##
 @@ -178,33 +182,14 @@ def __repr__(self):
 self.windows,
 self.pane_info)
 
-  def __hash__(self):
-return (hash(self.value) +
-3 * self.timestamp_micros +
-7 * hash(self.windows) +
-11 * hash(self.pane_info))
-
-  # We'd rather implement __eq__, but Cython supports that via __richcmp__
-  # instead.  Fortunately __cmp__ is understood by both (but not by Python 3).
-  def __cmp__(left, right):  # pylint: disable=no-self-argument
-"""Compares left and right for equality.
-
-For performance reasons, doesn't actually impose an ordering
-on unequal values (always returning 1).
-"""
-if type(left) is not type(right):
-  return cmp(type(left), type(right))
+  def _key(self):
+return self.value, self.timestamp_micros, self.windows, self.pane_info
 
-# TODO(robertwb): Avoid the type checks?
-# Returns False (0) if equal, and True (1) if not.
-return not WindowedValue._typed_eq(left, right)
+  def __eq__(self, other):
+return type(self) == type(other) and self._key() == other._key()
 
-  @staticmethod
-  def _typed_eq(left, right):
-return (left.timestamp_micros == right.timestamp_micros
-and left.value == right.value
-and left.windows == right.windows
-and left.pane_info == right.pane_info)
+  def __hash__(self):
 
 Review comment:
   Thanks for adding the benchmarking, this is very useful!
   I just ran the benchmark myself and got a performance regression of 6% for 
reading and 15% for adding to a dict.
   
   I will revert the changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111265)
Time Spent: 1.5h  (was: 1h 20m)

> Futurize and fix python 2 compatibility for utils subpackage
> 
>
> Key: BEAM-4008
> URL: https://issues.apache.org/jira/browse/BEAM-4008
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=111261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111261
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:27
Start Date: 12/Jun/18 20:27
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-396722360
 
 
   For the record, https://github.com/apache/beam/pull/5586 improves the 
performance of `dict.items()` with a Cython directive to use Python 3 
interpretation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111261)
Time Spent: 20.5h  (was: 20h 20m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20.5h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111255
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:19
Start Date: 12/Jun/18 20:19
Worklog Time Spent: 10m 
  Work Description: swegner opened a new pull request #5611: [BEAM-4445] 
Filter pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611
 
 
   This will filter pre-commit job triggered on PR's based on which files are 
touched. The impact is that most PRs will only run one set of relevant tests, 
rather than all three. This will decrease test overhead and the impact of flaky 
tests.
   
   This is discussed in the [Beam-Site Automation 
Reliability](https://s.apache.org/beam-site-automation) design, under 
"Pre-Commit Job Filtering"
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111255)
Time Spent: 10m
Remaining Estimate: 0h

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4445) Filter pre-commit triggering based on touched files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4445?focusedWorklogId=111256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111256
 ]

ASF GitHub Bot logged work on BEAM-4445:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:19
Start Date: 12/Jun/18 20:19
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5611: [BEAM-4445] Filter 
pre-commit triggering based on touched files
URL: https://github.com/apache/beam/pull/5611#issuecomment-396719936
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111256)
Time Spent: 20m  (was: 10m)

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510146#comment-16510146
 ] 

Valentyn Tymofieiev commented on BEAM-4543:
---

Possibly.  [https://pypi.org/project/googledatastore/] does not officially 
support Python 3.

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111251
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:15
Start Date: 12/Jun/18 20:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r194861219
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -1253,6 +1254,125 @@ ext.applyGrpcNature = {
 
 
/*/
 
+ext.applyPortabilityNature = {
+  println "applyPortabilityNature with " + (it ? "$it" : "default 
configuration") + " for project $project.name"
+  applyJavaNature(enableFindbugs: false, enableErrorProne: false, 
shadowClosure: {
+// guava uses the com.google.common and com.google.thirdparty package 
namespaces
+relocate "com.google.common", 
"org.apache.beam.vendor.guava.v20.com.google.common"
+relocate "com.google.thirdparty", 
"org.apache.beam.vendor.guava.v20.com.google.thirdparty"
+
+relocate "com.google.protobuf", 
"org.apache.beam.vendor.protobuf.v3.com.google.protobuf"
+relocate "com.google.gson", 
"org.apache.beam.vendor.gson.v2.com.google.gson"
+relocate "io.grpc", "org.apache.beam.vendor.grpc.v1.io.grpc"
+relocate "com.google.auth", 
"org.apache.beam.vendor.google_auth_library_credentials.v0_9_1.com.google.auth"
+relocate "com.google.api", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.api"
+relocate "com.google.cloud", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.cloud"
+relocate "com.google.logging", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.logging"
+relocate "com.google.longrunning", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.longrunning"
+relocate "com.google.rpc", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.rpc"
+relocate "com.google.type", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.type"
+relocate "io.opencensus", 
"org.apache.beam.vendor.opencensus.v0_11.io.opencensus"
+
+// Adapted from 
https://github.com/grpc/grpc-java/blob/e283f70ad91f99c7fee8b31b605ef12a4f9b1690/netty/shaded/build.gradle#L41
+relocate "io.netty", "org.apache.beam.vendor.netty.v4.io.netty"
+// We have to be careful with these replacements as they must not match any
+// string in NativeLibraryLoader, else they cause corruption. Note that
+// this includes concatenation of string literals and constants.
+relocate 'META-INF/native/libnetty', 
'META-INF/native/liborg_apache_beam_vendor_netty_v4_netty'
+relocate 'META-INF/native/netty', 
'META-INF/native/org_apache_beam_vendor_netty_v4_netty'
+
+// Don't include errorprone, JDK8 annotations, objenesis, junit, and 
mockito in the bundled jar
+exclude "com/google/errorprone/**"
+exclude "com/google/instrumentation/**"
+exclude "javax/annotation/**"
+exclude "junit/**"
+exclude "org/hamcrest/**"
+exclude "org/junit/**"
+exclude "org/mockito/**"
+exclude "org/objenesis/**"
+  })
+
+  // Don't force modules here because we don't want to take the shared 
declarations in build_rules.gradle
+  // because we would like to have the freedom to choose which versions of 
dependencies we
+  // are using for the portability APIs separate from what is being used 
inside other modules such as GCP.
+  configurations.all { config ->
+config.resolutionStrategy {
+  forcedModules = []
+}
+  }
+
+  apply plugin: "com.google.protobuf"
+  protobuf {
+protoc {
+  // The artifact spec for the Protobuf Compiler
+  artifact = "com.google.protobuf:protoc:3.5.1"
+}
+
+// Configure the codegen plugins
+plugins {
+  // An artifact spec for a protoc plugin, with "grpc" as
+  // the identifier, which can be referred to in the "plugins"
+  // container of the "generateProtoTasks" closure.
+  grpc {
+artifact = "io.grpc:protoc-gen-grpc-java:1.12.0"
+  }
+}
+
+generateProtoTasks {
+  ofSourceSet("main")*.plugins {
+// Apply the "grpc" plugin whose spec is defined above, without
+// options.  Note the braces cannot be omitted, otherwise the
+// plugin will not be added. This is because of the implicit way
+// NamedDomainObjectContainer binds the methods.
+grpc { }
+  }
+}
+  }
+
+  dependencies {
+compile 'com.google.guava:guava:20.0'
 
 Review comment:
   With vendoring, version drift only leads to larger jars since we will 
possibly be packaging multiple versions of the same dependency, 

[jira] [Commented] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510139#comment-16510139
 ] 

Ahmet Altay commented on BEAM-4543:
---

Is this blocking python 3 support in Beam?

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-4543:
-

Assignee: (was: Ahmet Altay)

> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111241=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111241
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:00
Start Date: 12/Jun/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r194860296
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -1253,6 +1254,125 @@ ext.applyGrpcNature = {
 
 
/*/
 
+ext.applyPortabilityNature = {
+  println "applyPortabilityNature with " + (it ? "$it" : "default 
configuration") + " for project $project.name"
+  applyJavaNature(enableFindbugs: false, enableErrorProne: false, 
shadowClosure: {
+// guava uses the com.google.common and com.google.thirdparty package 
namespaces
+relocate "com.google.common", 
"org.apache.beam.vendor.guava.v20.com.google.common"
 
 Review comment:
   Added comment explaining the process.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111241)
Time Spent: 4.5h  (was: 4h 20m)

> Remove duplicate dependency declarations from runners/direct-java
> -
>
> Key: BEAM-4481
> URL: https://issues.apache.org/jira/browse/BEAM-4481
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> beam-model-pipeline and others are duplicated in the dependency list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111243
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:00
Start Date: 12/Jun/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r194861219
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -1253,6 +1254,125 @@ ext.applyGrpcNature = {
 
 
/*/
 
+ext.applyPortabilityNature = {
+  println "applyPortabilityNature with " + (it ? "$it" : "default 
configuration") + " for project $project.name"
+  applyJavaNature(enableFindbugs: false, enableErrorProne: false, 
shadowClosure: {
+// guava uses the com.google.common and com.google.thirdparty package 
namespaces
+relocate "com.google.common", 
"org.apache.beam.vendor.guava.v20.com.google.common"
+relocate "com.google.thirdparty", 
"org.apache.beam.vendor.guava.v20.com.google.thirdparty"
+
+relocate "com.google.protobuf", 
"org.apache.beam.vendor.protobuf.v3.com.google.protobuf"
+relocate "com.google.gson", 
"org.apache.beam.vendor.gson.v2.com.google.gson"
+relocate "io.grpc", "org.apache.beam.vendor.grpc.v1.io.grpc"
+relocate "com.google.auth", 
"org.apache.beam.vendor.google_auth_library_credentials.v0_9_1.com.google.auth"
+relocate "com.google.api", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.api"
+relocate "com.google.cloud", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.cloud"
+relocate "com.google.logging", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.logging"
+relocate "com.google.longrunning", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.longrunning"
+relocate "com.google.rpc", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.rpc"
+relocate "com.google.type", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.type"
+relocate "io.opencensus", 
"org.apache.beam.vendor.opencensus.v0_11.io.opencensus"
+
+// Adapted from 
https://github.com/grpc/grpc-java/blob/e283f70ad91f99c7fee8b31b605ef12a4f9b1690/netty/shaded/build.gradle#L41
+relocate "io.netty", "org.apache.beam.vendor.netty.v4.io.netty"
+// We have to be careful with these replacements as they must not match any
+// string in NativeLibraryLoader, else they cause corruption. Note that
+// this includes concatenation of string literals and constants.
+relocate 'META-INF/native/libnetty', 
'META-INF/native/liborg_apache_beam_vendor_netty_v4_netty'
+relocate 'META-INF/native/netty', 
'META-INF/native/org_apache_beam_vendor_netty_v4_netty'
+
+// Don't include errorprone, JDK8 annotations, objenesis, junit, and 
mockito in the bundled jar
+exclude "com/google/errorprone/**"
+exclude "com/google/instrumentation/**"
+exclude "javax/annotation/**"
+exclude "junit/**"
+exclude "org/hamcrest/**"
+exclude "org/junit/**"
+exclude "org/mockito/**"
+exclude "org/objenesis/**"
+  })
+
+  // Don't force modules here because we don't want to take the shared 
declarations in build_rules.gradle
+  // because we would like to have the freedom to choose which versions of 
dependencies we
+  // are using for the portability APIs separate from what is being used 
inside other modules such as GCP.
+  configurations.all { config ->
+config.resolutionStrategy {
+  forcedModules = []
+}
+  }
+
+  apply plugin: "com.google.protobuf"
+  protobuf {
+protoc {
+  // The artifact spec for the Protobuf Compiler
+  artifact = "com.google.protobuf:protoc:3.5.1"
+}
+
+// Configure the codegen plugins
+plugins {
+  // An artifact spec for a protoc plugin, with "grpc" as
+  // the identifier, which can be referred to in the "plugins"
+  // container of the "generateProtoTasks" closure.
+  grpc {
+artifact = "io.grpc:protoc-gen-grpc-java:1.12.0"
+  }
+}
+
+generateProtoTasks {
+  ofSourceSet("main")*.plugins {
+// Apply the "grpc" plugin whose spec is defined above, without
+// options.  Note the braces cannot be omitted, otherwise the
+// plugin will not be added. This is because of the implicit way
+// NamedDomainObjectContainer binds the methods.
+grpc { }
+  }
+}
+  }
+
+  dependencies {
+compile 'com.google.guava:guava:20.0'
 
 Review comment:
   With vendoring, version drift only leads to larger jars since we will 
possibly be packaging multiple versions of the same dependency, 

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111244
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:00
Start Date: 12/Jun/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r194863255
 
 

 ##
 File path: model/fn-execution/build.gradle
 ##
 @@ -17,17 +17,14 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature(enableFindbugs: false, enableErrorProne: false)
-applyGrpcNature()
+applyPortabilityNature()
 
 description = "Apache Beam :: Model :: Fn Execution"
 ext.summary = "Portable definitions for execution user-defined functions."
 
 dependencies {
-  compile library.java.guava
-  shadow project(path: ":beam-model-pipeline", configuration: "shadow")
-  shadow library.java.protobuf_java
-  shadow library.java.grpc_core
-  shadow library.java.grpc_protobuf
-  shadow library.java.grpc_stub
+  // We purposely depend on the unshaded classes for compilation and
 
 Review comment:
   I couldn't figure out a way to have the generated source use a vendored 
version of grpc/protobuf. So the generated code uses imports like 
`com.google.protobuf.Message` and expects classes from `beam-model-pipeline` to 
extend non-vendored gRPC/protobuf classes.
   
   The flow is:
   1) generate code using non-vendored imports
   2) compile code using non-vendored dependencies
   3) produce shaded classes using the same shading rules as 
`beam-model-pipeline`
   
   This works because `beam-model-pipeline`, `beam-model-job-management`, and 
`beam-model-fn-execution` all use the same relocation rules so when the jar is 
loaded by java, java will see that they all extend the same vendored code 
locations.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111244)
Time Spent: 4h 50m  (was: 4h 40m)

> Remove duplicate dependency declarations from runners/direct-java
> -
>
> Key: BEAM-4481
> URL: https://issues.apache.org/jira/browse/BEAM-4481
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> beam-model-pipeline and others are duplicated in the dependency list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111242
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 20:00
Start Date: 12/Jun/18 20:00
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r194861371
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -1253,6 +1254,125 @@ ext.applyGrpcNature = {
 
 
/*/
 
+ext.applyPortabilityNature = {
+  println "applyPortabilityNature with " + (it ? "$it" : "default 
configuration") + " for project $project.name"
+  applyJavaNature(enableFindbugs: false, enableErrorProne: false, 
shadowClosure: {
+// guava uses the com.google.common and com.google.thirdparty package 
namespaces
+relocate "com.google.common", 
"org.apache.beam.vendor.guava.v20.com.google.common"
+relocate "com.google.thirdparty", 
"org.apache.beam.vendor.guava.v20.com.google.thirdparty"
+
+relocate "com.google.protobuf", 
"org.apache.beam.vendor.protobuf.v3.com.google.protobuf"
+relocate "com.google.gson", 
"org.apache.beam.vendor.gson.v2.com.google.gson"
+relocate "io.grpc", "org.apache.beam.vendor.grpc.v1.io.grpc"
+relocate "com.google.auth", 
"org.apache.beam.vendor.google_auth_library_credentials.v0_9_1.com.google.auth"
+relocate "com.google.api", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.api"
+relocate "com.google.cloud", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.cloud"
+relocate "com.google.logging", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.logging"
+relocate "com.google.longrunning", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.longrunning"
+relocate "com.google.rpc", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.rpc"
+relocate "com.google.type", 
"org.apache.beam.vendor.proto_google_common_protos.v1.com.google.type"
+relocate "io.opencensus", 
"org.apache.beam.vendor.opencensus.v0_11.io.opencensus"
+
+// Adapted from 
https://github.com/grpc/grpc-java/blob/e283f70ad91f99c7fee8b31b605ef12a4f9b1690/netty/shaded/build.gradle#L41
+relocate "io.netty", "org.apache.beam.vendor.netty.v4.io.netty"
+// We have to be careful with these replacements as they must not match any
+// string in NativeLibraryLoader, else they cause corruption. Note that
+// this includes concatenation of string literals and constants.
+relocate 'META-INF/native/libnetty', 
'META-INF/native/liborg_apache_beam_vendor_netty_v4_netty'
+relocate 'META-INF/native/netty', 
'META-INF/native/org_apache_beam_vendor_netty_v4_netty'
+
+// Don't include errorprone, JDK8 annotations, objenesis, junit, and 
mockito in the bundled jar
+exclude "com/google/errorprone/**"
+exclude "com/google/instrumentation/**"
+exclude "javax/annotation/**"
+exclude "junit/**"
+exclude "org/hamcrest/**"
+exclude "org/junit/**"
+exclude "org/mockito/**"
+exclude "org/objenesis/**"
+  })
+
+  // Don't force modules here because we don't want to take the shared 
declarations in build_rules.gradle
+  // because we would like to have the freedom to choose which versions of 
dependencies we
+  // are using for the portability APIs separate from what is being used 
inside other modules such as GCP.
+  configurations.all { config ->
+config.resolutionStrategy {
+  forcedModules = []
+}
+  }
+
+  apply plugin: "com.google.protobuf"
+  protobuf {
+protoc {
+  // The artifact spec for the Protobuf Compiler
+  artifact = "com.google.protobuf:protoc:3.5.1"
+}
+
+// Configure the codegen plugins
+plugins {
+  // An artifact spec for a protoc plugin, with "grpc" as
+  // the identifier, which can be referred to in the "plugins"
+  // container of the "generateProtoTasks" closure.
+  grpc {
+artifact = "io.grpc:protoc-gen-grpc-java:1.12.0"
+  }
+}
+
+generateProtoTasks {
+  ofSourceSet("main")*.plugins {
+// Apply the "grpc" plugin whose spec is defined above, without
+// options.  Note the braces cannot be omitted, otherwise the
+// plugin will not be added. This is because of the implicit way
+// NamedDomainObjectContainer binds the methods.
+grpc { }
+  }
+}
+  }
+
+  dependencies {
+compile 'com.google.guava:guava:20.0'
+compile 'com.google.protobuf:protobuf-java:3.5.1'
+compile 'com.google.protobuf:protobuf-java-util:3.5.1'
+compile 'com.google.code.gson:gson:2.7'
+ 

[jira] [Created] (BEAM-4544) Add support for intellij to work with vendored artifacts generated as part of the Apache Beam build process

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-4544:
---

 Summary: Add support for intellij to work with vendored artifacts 
generated as part of the Apache Beam build process
 Key: BEAM-4544
 URL: https://issues.apache.org/jira/browse/BEAM-4544
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Luke Cwik


Intellij has difficulty resolving vendored artifacts such as 
*org.apache.beam.vendor.protobuf.v3.com.google.protobuf.Message*. This prevents 
the code editor for making reasonable substitutions, intellisense from working, 
and tracing through source code during debugging.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-4543:
-

 Summary: Remove dependency on googledatastore in favor of 
google-cloud-datastore.
 Key: BEAM-4543
 URL: https://issues.apache.org/jira/browse/BEAM-4543
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay


apache-beam[gcp] package depends [1] on googledatastore package [2]. We should 
replace this dependency with google-cloud-datastore [3] which is officially 
supported, has better release cadence and also has Python 3 support.

[[1]https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126|https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126]

[[2] 
[https://pypi.org/project/googledatastore/]|https://pypi.org/project/googledatastore/]
 

[[3] 
[https://pypi.org/project/google-cloud-datastore/]|https://pypi.org/project/google-cloud-datastore/]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4543) Remove dependency on googledatastore in favor of google-cloud-datastore.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-4543:
--
Description: 
apache-beam[gcp] package depends [1] on googledatastore package [2]. We should 
replace this dependency with google-cloud-datastore [3] which is officially 
supported, has better release cadence and also has Python 3 support.

[1] 
https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126

[2] [https://pypi.org/project/googledatastore/]

[3] [https://pypi.org/project/google-cloud-datastore/]

 

  was:
apache-beam[gcp] package depends [1] on googledatastore package [2]. We should 
replace this dependency with google-cloud-datastore [3] which is officially 
supported, has better release cadence and also has Python 3 support.

[[1]https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126|https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126]

[[2] 
[https://pypi.org/project/googledatastore/]|https://pypi.org/project/googledatastore/]
 

[[3] 
[https://pypi.org/project/google-cloud-datastore/]|https://pypi.org/project/google-cloud-datastore/]

 


> Remove dependency on googledatastore in favor of google-cloud-datastore.
> 
>
> Key: BEAM-4543
> URL: https://issues.apache.org/jira/browse/BEAM-4543
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ahmet Altay
>Priority: Minor
>
> apache-beam[gcp] package depends [1] on googledatastore package [2]. We 
> should replace this dependency with google-cloud-datastore [3] which is 
> officially supported, has better release cadence and also has Python 3 
> support.
> [1] 
> https://github.com/apache/beam/blob/fad655462f8fadfdfaab0b7a09cab538f076f94e/sdks/python/setup.py#L126
> [2] [https://pypi.org/project/googledatastore/]
> [3] [https://pypi.org/project/google-cloud-datastore/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=111233=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111233
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 12/Jun/18 19:24
Start Date: 12/Jun/18 19:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5594: 
[BEAM-4481, BEAM-4484] Start vendoring portability dependencies to not have 
dependency conflicts
URL: https://github.com/apache/beam/pull/5594#discussion_r19486
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -1253,6 +1254,125 @@ ext.applyGrpcNature = {
 
 
/*/
 
+ext.applyPortabilityNature = {
+  println "applyPortabilityNature with " + (it ? "$it" : "default 
configuration") + " for project $project.name"
+  applyJavaNature(enableFindbugs: false, enableErrorProne: false, 
shadowClosure: {
+// guava uses the com.google.common and com.google.thirdparty package 
namespaces
+relocate "com.google.common", 
"org.apache.beam.vendor.guava.v20.com.google.common"
+relocate "com.google.thirdparty", 
"org.apache.beam.vendor.guava.v20.com.google.thirdparty"
+
+relocate "com.google.protobuf", 
"org.apache.beam.vendor.protobuf.v3.com.google.protobuf"
 
 Review comment:
   That is already being done by the 
`validateShadedJarDoesntLeakNonOrgApacheBeamClasses` task.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 111233)
Time Spent: 4h 20m  (was: 4h 10m)

> Remove duplicate dependency declarations from runners/direct-java
> -
>
> Key: BEAM-4481
> URL: https://issues.apache.org/jira/browse/BEAM-4481
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> beam-model-pipeline and others are duplicated in the dependency list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4050) Move pom generation metadata logic out of build_rules.gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4050?focusedWorklogId=111235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-111235
 ]

ASF GitHub Bot logged work on BEAM-4050:


Author: ASF GitHub Bot
Created on: 12/Jun/18 19:37
Start Date: 12/Jun/18 19:37
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5608: [BEAM-4050] Use new 
DSL methods for generated POM metadata
URL: https://github.com/apache/beam/pull/5608
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build_rules.gradle b/build_rules.gradle
index 50d18c7bdc6..09a2a66d9b8 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -740,58 +740,65 @@ artifactId=${project.name}
 classifier "javadoc"
   }
 
-  pom.withXml {
-def root = asNode()
-root.appendNode('name', project.description)
+  pom {
+name = project.description
 if (project.hasProperty("summary")) {
-root.appendNode('description', project.summary)
+description = project.summary
+}
+url = "http://beam.apache.org;
+inceptionYear = "2016"
+licenses {
+  license {
+name = "Apache License, Version 2.0"
+url = "http://www.apache.org/licenses/LICENSE-2.0.txt;
+distribution = "repo"
+  }
+}
+scm {
+  connection = 
"scm:git:https://gitbox.apache.org/repos/asf/beam.git;
+  developerConnection = 
"scm:git:https://gitbox.apache.org/repos/asf/beam.git;
+  url = "https://gitbox.apache.org/repos/asf?p=beam.git;a=summary;
+}
+issueManagement {
+  system = "jira"
+  url = "https://issues.apache.org/jira/browse/BEAM;
+}
+mailingLists {
+  mailingList {
+name = "Beam Dev"
+subscribe = "dev-subscr...@beam.apache.org"
+unsubscribe = "dev-unsubscr...@beam.apache.org"
+post = "d...@beam.apache.org"
+archive = "http://www.mail-archive.com/dev%beam.apache.org;
+  }
+  mailingList {
+name = "Beam User"
+subscribe = "user-subscr...@beam.apache.org"
+unsubscribe = "user-unsubscr...@beam.apache.org"
+post = "u...@beam.apache.org"
+archive = "http://www.mail-archive.com/user%beam.apache.org;
+  }
+  mailingList {
+name = "Beam Commits"
+subscribe = "commits-subscr...@beam.apache.org"
+unsubscribe = "commits-unsubscr...@beam.apache.org"
+post = "commits@beam.apache.org"
+archive = "http://www.mail-archive.com/commits%beam.apache.org;
+  }
+}
+developers {
+  developer {
+name = "The Apache Beam Team"
+email = "d...@beam.apache.org"
+url = "http://beam.apache.org;
+organization = "Apache Software Foundation"
+organizationUrl = "http://www.apache.org;
+  }
 }
-root.appendNode('url', "http://beam.apache.org;)
-root.appendNode('inceptionYear', "2016")
-
-def licenseNode = root.appendNode('licenses').appendNode('license')
-licenseNode.appendNode('name', "Apache License, Version 2.0")
-licenseNode.appendNode('url', 
"http://www.apache.org/licenses/LICENSE-2.0.txt;)
-licenseNode.appendNode('distribution', "repo")
-
-def scmNode = root.appendNode('scm')
-scmNode.appendNode('connection', 
"scm:git:https://gitbox.apache.org/repos/asf/beam.git;)
-scmNode.appendNode('developerConnection', 
"scm:git:https://gitbox.apache.org/repos/asf/beam.git;)
-scmNode.appendNode('url', 
"https://gitbox.apache.org/repos/asf?p=beam.git;a=summary;)
-
-def issueMgmtNode = root.appendNode('issueManagement')
-issueMgmtNode.appendNode('system', "jira")
-issueMgmtNode.appendNode('url', 
"https://issues.apache.org/jira/browse/BEAM;)
-
-def mailingListsNode = root.appendNode('mailingLists')
-def devListNode = mailingListsNode.appendNode('mailingList')
-devListNode.appendNode('name', "Beam Dev")
-devListNode.appendNode('subscribe', 
"dev-subscr...@beam.apache.org")
-devListNode.appendNode('unsubscribe', 
"dev-unsubscr...@beam.apache.org")
-