[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127072
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:56
Start Date: 25/Jul/18 04:56
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407633271
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127072)
Time Spent: 1h 40m  (was: 1.5h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127069
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:55
Start Date: 25/Jul/18 04:55
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407630480
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127069)
Time Spent: 1.5h  (was: 1h 20m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127065
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:51
Start Date: 25/Jul/18 04:51
Worklog Time Spent: 10m 
  Work Description: markflyhigh edited a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407632387
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127065)
Time Spent: 1h  (was: 50m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127067
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:51
Start Date: 25/Jul/18 04:51
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407632565
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127067)
Time Spent: 1h 20m  (was: 1h 10m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127066
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:51
Start Date: 25/Jul/18 04:51
Worklog Time Spent: 10m 
  Work Description: markflyhigh removed a comment on issue #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407632387
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127066)
Time Spent: 1h 10m  (was: 1h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127063
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:50
Start Date: 25/Jul/18 04:50
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407632387
 
 
   Run Seedjob


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127063)
Time Spent: 50m  (was: 40m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127060
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:35
Start Date: 25/Jul/18 04:35
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407630480
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127060)
Time Spent: 40m  (was: 0.5h)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=127059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127059
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 04:34
Start Date: 25/Jul/18 04:34
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407630401
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127059)
Time Spent: 0.5h  (was: 20m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555040#comment-16555040
 ] 

Thomas Weise commented on BEAM-4847:


BTW can the changes be made effective immediately, w/o waiting for the seed job?

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555034#comment-16555034
 ] 

Thomas Weise commented on BEAM-4847:


Yep, 3g/8g works at least for single build runs, based on latest precommits. So 
we don't know for sure if that setting had worked for concurrent builds also. 
Let's observe concurrent builds running for a day with 2g/4g.

 

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4860) com/google/thirdparty/publicsuffix is not shaded along with guava

[Fred Wulff (JIRA)]

Fred Wulff created BEAM-4860:


 Summary: com/google/thirdparty/publicsuffix is not shaded along 
with guava
 Key: BEAM-4860
 URL: https://issues.apache.org/jira/browse/BEAM-4860
 Project: Beam
  Issue Type: Bug
  Components: build-system
Affects Versions: 2.5.0
Reporter: Fred Wulff
Assignee: Luke Cwik


com.google.common is shaded, but com.google.thirdparty.publicsuffix (which is 
part of the guava library) is not, so attempting to use a more recent version 
of Guava while importing the beam libraries leads to conflicts



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=127012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127012
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 02:18
Start Date: 25/Jul/18 02:18
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6054: [BEAM-4847] 
Reduce Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054#issuecomment-407611386
 
 
   The 2 builds in the bug had the old values of --max-workers=48 
-Dorg.gradle.jvmargs=-Xmx1109m, so the -Xmx=8g may not have been the problem 
for these.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127012)
Time Spent: 3h 20m  (was: 3h 10m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555025#comment-16555025
 ] 

Alan Myrvold commented on BEAM-4847:


Both [https://builds.apache.org/job/beam_PreCommit_Java_Commit/528/consoleFull] 
and [https://builds.apache.org/job/beam_PreCommit_Java_Phrase/81/consoleFull] 
show the old, broken, values of --max-workers=48 -Dorg.gradle.jvmargs=-Xmx1109m 
 since the seed job had not updated to the new settings, so these are not 
evidence that -Xmx8g was broken, but 4gb should be ok.

 

 

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4809) Java preCommit and postCommit should build javadoc to check it builds ok

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4809?focusedWorklogId=127005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127005
 ]

ASF GitHub Bot logged work on BEAM-4809:


Author: ASF GitHub Bot
Created on: 25/Jul/18 02:10
Start Date: 25/Jul/18 02:10
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #5970: 
[BEAM-4809] Build javadoc on :javaPreCommit and :javaPostCommit
URL: https://github.com/apache/beam/pull/5970#issuecomment-407544800
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127005)
Time Spent: 3h 20m  (was: 3h 10m)

> Java preCommit and postCommit should build javadoc to check it builds ok
> 
>
> Key: BEAM-4809
> URL: https://issues.apache.org/jira/browse/BEAM-4809
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Java preCommit and postCommit should build javadoc to check it builds ok



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4809) Java preCommit and postCommit should build javadoc to check it builds ok

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4809?focusedWorklogId=127003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127003
 ]

ASF GitHub Bot logged work on BEAM-4809:


Author: ASF GitHub Bot
Created on: 25/Jul/18 02:10
Start Date: 25/Jul/18 02:10
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #5970: [BEAM-4809] Build 
javadoc on :javaPreCommit and :javaPostCommit
URL: https://github.com/apache/beam/pull/5970#issuecomment-407610100
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127003)
Time Spent: 3h  (was: 2h 50m)

> Java preCommit and postCommit should build javadoc to check it builds ok
> 
>
> Key: BEAM-4809
> URL: https://issues.apache.org/jira/browse/BEAM-4809
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Java preCommit and postCommit should build javadoc to check it builds ok



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4809) Java preCommit and postCommit should build javadoc to check it builds ok

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4809?focusedWorklogId=127006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127006
 ]

ASF GitHub Bot logged work on BEAM-4809:


Author: ASF GitHub Bot
Created on: 25/Jul/18 02:10
Start Date: 25/Jul/18 02:10
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #5970: 
[BEAM-4809] Build javadoc on :javaPreCommit and :javaPostCommit
URL: https://github.com/apache/beam/pull/5970#issuecomment-406656952
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127006)
Time Spent: 3.5h  (was: 3h 20m)

> Java preCommit and postCommit should build javadoc to check it builds ok
> 
>
> Key: BEAM-4809
> URL: https://issues.apache.org/jira/browse/BEAM-4809
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Java preCommit and postCommit should build javadoc to check it builds ok



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4809) Java preCommit and postCommit should build javadoc to check it builds ok

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4809?focusedWorklogId=127004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127004
 ]

ASF GitHub Bot logged work on BEAM-4809:


Author: ASF GitHub Bot
Created on: 25/Jul/18 02:10
Start Date: 25/Jul/18 02:10
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #5970: 
[BEAM-4809] Build javadoc on :javaPreCommit and :javaPostCommit
URL: https://github.com/apache/beam/pull/5970#issuecomment-407544663
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127004)
Time Spent: 3h 10m  (was: 3h)

> Java preCommit and postCommit should build javadoc to check it builds ok
> 
>
> Key: BEAM-4809
> URL: https://issues.apache.org/jira/browse/BEAM-4809
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Affects Versions: 2.5.0
>Reporter: Alan Myrvold
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Java preCommit and postCommit should build javadoc to check it builds ok



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=127000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-127000
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:49
Start Date: 25/Jul/18 01:49
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5935: [BEAM-4176] Initial 
implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#issuecomment-407606694
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 127000)
Time Spent: 4h 40m  (was: 4.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554968#comment-16554968
 ] 

Thomas Weise edited comment on BEAM-4847 at 7/25/18 1:48 AM:
-

It appears that when only a single build runs, Java PreCommit passes.

How about reducing to -Xmx4g ?

 


was (Author: thw):
It appears that when only a single build runs, Java PreCommit passes.

How about reducing to -Xmx8g ?

 

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126999
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:24
Start Date: 25/Jul/18 01:24
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6054: [BEAM-4847] Reduce 
Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/CommonJobProperties.groovy 
b/.test-infra/jenkins/CommonJobProperties.groovy
index 4373e1a4728..8ca6a4ed84a 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -189,9 +189,9 @@ class CommonJobProperties {
 
 // Ensure that parallel workers don't exceed total available memory.
 
-// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (3g/8g).
-context.switches("-Dorg.gradle.jvmargs=-Xms3g")
-context.switches("-Dorg.gradle.jvmargs=-Xmx8g")
+// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (2g/4g).
+context.switches("-Dorg.gradle.jvmargs=-Xms2g")
+context.switches("-Dorg.gradle.jvmargs=-Xmx4g")
   }
 
   // Sets common config for PreCommit jobs.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126999)
Time Spent: 3h 10m  (was: 3h)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Automate 'Start a snapshot build' step in beam release guide (#6042)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 8b35ae3  Automate 'Start a snapshot build' step in beam release guide 
(#6042)
8b35ae3 is described below

commit 8b35ae39fff3318284e632e31438df39a4fd5431
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Tue Jul 24 18:24:50 2018 -0700

Automate 'Start a snapshot build' step in beam release guide (#6042)

* Automate 'Start a snapshot build' step in beam release guide
---
 release/src/main/scripts/start_snapshot_build.sh | 85 
 1 file changed, 85 insertions(+)

diff --git a/release/src/main/scripts/start_snapshot_build.sh 
b/release/src/main/scripts/start_snapshot_build.sh
new file mode 100755
index 000..f425f43
--- /dev/null
+++ b/release/src/main/scripts/start_snapshot_build.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will create a empty PR and start gradle publish job.
+
+set -e
+
+LOCAL_BEAM_DIR=beam_snapshot_build
+HUB_VERSION=2.5.0
+HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
+GIT_REPO_URL=https://github.com/apache/beam.git
+USER_REMOTE_URL=
+USER_REMOTE_NAME=remote_repo
+BEAM_ROOT_DIR=beam
+BRANCH_NAME=snapshot_build
+
+echo "Please enter your repo URL forked from apache beam:"
+read USER_REMOTE_URL
+
+echo "=Environment Variables==="
+echo "Your beam repo URL: ${USER_REMOTE_URL}"
+echo "Beam repo will be cloned into: ~/${LOCAL_BEAM_DIR}/"
+echo "New PR will be created on branch: ${BRANCH_NAME}"
+
+cd ~
+
+echo "=Checking hub"
+if [[ -z `which hub` ]]; then
+  echo "There is no hub installed on your machine."
+  echo "Would you like to install hub with root permission? [y|N]"
+  read confirmation
+  if [[ $confirmation != "y"  ]]; then
+echo "Refused to install hub. Cannot proceed into next setp."
+exit
+  fi
+  echo "=Installing hub==="
+  wget 
https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
+  tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
+  sudo ./${HUB_ARTIFACTS_NAME}/install
+  echo "eval "$(hub alias -s)"" >> ~/.bashrc
+  rm -rf ${HUB_ARTIFACTS_NAME}*
+fi
+hub version
+
+echo "===Starting creating empty PR==="
+mkdir ${LOCAL_BEAM_DIR}
+cd ${LOCAL_BEAM_DIR}
+git clone ${GIT_REPO_URL}
+cd ${BEAM_ROOT_DIR}
+git remote add ${USER_REMOTE_NAME} ${USER_REMOTE_URL}
+git checkout -b ${BRANCH_NAME}
+touch empty_file.txt
+git add -A
+git commit -m "Add empty file in order to create PR"
+git push -f ${USER_REMOTE_NAME}
+
+cd ~/${LOCAL_BEAM_DIR}/${BEAM_ROOT_DIR}
+hub pull-request -F- <<<"[DO NOT MERGE]Start snapshot build for release process
+
+
+Run Gradle Publish"
+
+echo "NOTE: If there is no jenkins job started, please comment generated PR 
with: Run Gradle Publish"
+
+echo "===Cleaning up==="
+cd ~
+rm -rf ${LOCAL_BEAM_DIR}
+echo "Things remained you need to do manually after build successful:"
+echo "1. Close this generated PR in github website."
+echo "2. Delete your remote branch ${BRANCH_NAME} form your beam repo in 
github website."

[beam] 01/01: Merge pull request #6054: [BEAM-4847] Reduce Gradle JVM Xmx to 4g to fix Jenkins build failures.

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c34d072e01ba66b931957d8043e975681ddbf529
Merge: b87aa60 6c0e1a2
Author: Thomas Weise 
AuthorDate: Tue Jul 24 18:23:59 2018 -0700

Merge pull request #6054: [BEAM-4847] Reduce Gradle JVM Xmx to 4g to fix 
Jenkins build failures.

 .test-infra/jenkins/CommonJobProperties.groovy | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

[beam] branch master updated (b87aa60 -> c34d072)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b87aa60  Merge pull request #5955: [BEAM-4076] Enable schemas for more 
runners
 add 6c0e1a2  [BEAM-4847] Reduce Gradle JVM Xmx to 4g to fix Jenkins build 
failures.
 new c34d072  Merge pull request #6054: [BEAM-4847] Reduce Gradle JVM Xmx 
to 4g to fix Jenkins build failures.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/CommonJobProperties.groovy | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126996
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:10
Start Date: 25/Jul/18 01:10
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6054: [BEAM-4847] Reduce Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054#discussion_r204956902
 
 

 ##
 File path: .test-infra/jenkins/CommonJobProperties.groovy
 ##
 @@ -189,9 +189,9 @@ class CommonJobProperties {
 
 // Ensure that parallel workers don't exceed total available memory.
 
-// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (3g/8g).
+// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (3g/4g).
 context.switches("-Dorg.gradle.jvmargs=-Xms3g")
 
 Review comment:
   Maybe reduce this to 2g too?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126996)
Time Spent: 3h  (was: 2h 50m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126995
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:06
Start Date: 25/Jul/18 01:06
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6054: [BEAM-4847] Reduce 
Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054#issuecomment-407599869
 
 
   R: @alanmyrvold 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126995)
Time Spent: 2h 50m  (was: 2h 40m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126992
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:05
Start Date: 25/Jul/18 01:05
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6054: [BEAM-4847] Reduce 
Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054#issuecomment-407599683
 
 
   Run Java PreCommit
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126992)
Time Spent: 2h 40m  (was: 2.5h)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126990
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:04
Start Date: 25/Jul/18 01:04
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6054: [BEAM-4847] Reduce 
Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054#issuecomment-407599650
 
 
   Run Seed Job
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126990)
Time Spent: 2.5h  (was: 2h 20m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126987
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:04
Start Date: 25/Jul/18 01:04
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #6054: [BEAM-4847] 
Reduce Gradle JVM Xmx to 4g to fix Jenkins build failures.
URL: https://github.com/apache/beam/pull/6054
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126987)
Time Spent: 2h 20m  (was: 2h 10m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> 

[jira] [Work logged] (BEAM-4842) Update Flink Runner to Flink 1.5.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4842?focusedWorklogId=126985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126985
 ]

ASF GitHub Bot logged work on BEAM-4842:


Author: ASF GitHub Bot
Created on: 25/Jul/18 01:03
Start Date: 25/Jul/18 01:03
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6017: [BEAM-4842] Update 
Flink Runner to Flink 1.5.1
URL: https://github.com/apache/beam/pull/6017#issuecomment-407599471
 
 
   @iemejia build failures were due to CI setup, PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126985)
Time Spent: 1h 40m  (was: 1.5h)

> Update Flink Runner to Flink 1.5.1
> --
>
> Key: BEAM-4842
> URL: https://issues.apache.org/jira/browse/BEAM-4842
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554968#comment-16554968
 ] 

Thomas Weise commented on BEAM-4847:


It appears that when only a single build runs, Java PreCommit passes.

How about reducing to -Xmx8g ?

 

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1098

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4778) Less wasteful ArtifactStagingService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4778?focusedWorklogId=126982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126982
 ]

ASF GitHub Bot logged work on BEAM-4778:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:48
Start Date: 25/Jul/18 00:48
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #5958: [BEAM-4778] add 
option to flink job server to clean staged artifacts per-job
URL: https://github.com/apache/beam/pull/5958#issuecomment-407597236
 
 
   whew, finally passed precommit. lmk if you want me to squash+rebase 
@angoenka (or if there are any other comments I missed! I think I got 
everything)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126982)
Time Spent: 4.5h  (was: 4h 20m)

> Less wasteful ArtifactStagingService
> 
>
> Key: BEAM-4778
> URL: https://issues.apache.org/jira/browse/BEAM-4778
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Ryan Williams
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java]
>  is the main implementation of ArtifactStagingService.
> It stages artifacts into a directory; and in practice the passed staging 
> session token is such that the directory is different for every job. This 
> leads to 2 issues:
>  * It doesn't get cleaned up when the job finishes or even when the 
> JobService shuts down, so we have disk space leaks if running a lot of jobs 
> (e.g. a suite of ValidatesRunner tests)
>  * We repeatedly re-stage the same artifacts. Instead, ideally, we should 
> identify that some artifacts don't need to be staged - based on knowing their 
> md5. The artifact staging protocol has rudimentary support for this but may 
> need to be modified.
> CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4842) Update Flink Runner to Flink 1.5.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4842?focusedWorklogId=126980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126980
 ]

ASF GitHub Bot logged work on BEAM-4842:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:35
Start Date: 25/Jul/18 00:35
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6017: [BEAM-4842] Update 
Flink Runner to Flink 1.5.1
URL: https://github.com/apache/beam/pull/6017#issuecomment-407595144
 
 
   Run Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126980)
Time Spent: 1.5h  (was: 1h 20m)

> Update Flink Runner to Flink 1.5.1
> --
>
> Key: BEAM-4842
> URL: https://issues.apache.org/jira/browse/BEAM-4842
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4855) Microbenchmarks for windowinto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?focusedWorklogId=126979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126979
 ]

ASF GitHub Bot logged work on BEAM-4855:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:23
Start Date: 25/Jul/18 00:23
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6051: [BEAM-4855] Added 
windowinto_microbenchmark
URL: https://github.com/apache/beam/pull/6051#issuecomment-407593495
 
 
   Also, if we are specifically intrested in testing performance of hashes and 
comparators, we can have a microbenchmark that does just that for the types you 
mentioned by doing something similar to 
https://github.com/tvalentyn/beam/blob/3f67f63377dd85a241a48b1108a618ebfbe891cb/sdks/python/apache_beam/tools/windowed_value_microbenchmark.py#L35.
 That code lives on my branch and I did not megre it to Beam, but you are more 
than welcome to use it if it helps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126979)
Time Spent: 50m  (was: 40m)

> Microbenchmarks for windowinto
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the windowinto transform:
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #143

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-4845] Make BigQueryServices and FakeDatasetService public 
to fix

[amyrvold] Fix [BEAM-4847] by manually setting max workers and jvm memory and 
add

[relax] Enable schemas for Apex runner.

[relax] Enable Schemas on samza runner.

[relax] Enable schemas for Flink runner.

[relax] Enable schemas for Spark.

[relax] Fix bugs.

[relax] Fix Apex breakage.

[ehudm] Remove CODEOWNERs.

--
[...truncated 16.25 MB...]
INFO: 2018-07-25T00:19:17.462Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.503Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.551Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.589Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.641Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.688Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.734Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.780Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.817Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.862Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.913Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Jul 25, 2018 12:19:20 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-25T00:19:17.957Z: Unzipping flatten s13-u58 for input 
s14.org.apache.beam.sdk.values.PCollection.:364#f0cbc4d341b04049-c56
Jul 25, 2018 12:19:20 AM 

Jenkins build is back to normal : beam_PostCommit_Java_Nexmark_Flink #115

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4842) Update Flink Runner to Flink 1.5.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4842?focusedWorklogId=126977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126977
 ]

ASF GitHub Bot logged work on BEAM-4842:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:11
Start Date: 25/Jul/18 00:11
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6017: [BEAM-4842] Update 
Flink Runner to Flink 1.5.1
URL: https://github.com/apache/beam/pull/6017#issuecomment-407591549
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126977)
Time Spent: 1h 20m  (was: 1h 10m)

> Update Flink Runner to Flink 1.5.1
> --
>
> Key: BEAM-4842
> URL: https://issues.apache.org/jira/browse/BEAM-4842
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_Nexmark_Direct #118

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=126976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126976
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:10
Start Date: 25/Jul/18 00:10
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #6053: [BEAM-4859] 
Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053#issuecomment-407591496
 
 
   +R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126976)
Time Spent: 20m  (was: 10m)

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Automated way to run existing Python ValidatesRunner tests in streaming. 
> Gradle should also be able to run them, so Jenkins automation will be much 
> simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4859?focusedWorklogId=126973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126973
 ]

ASF GitHub Bot logged work on BEAM-4859:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:09
Start Date: 25/Jul/18 00:09
Worklog Time Spent: 10m 
  Work Description: markflyhigh opened a new pull request #6053: 
[BEAM-4859] Enable Python VR tests in streaming in postcommit task
URL: https://github.com/apache/beam/pull/6053
 
 
   - Enable existing Python VR tests in streaming in an automatic way. Add 
Gradle task so that we can automate this test on Jenkins in next step.
   - Fix incorrect task names.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126973)
Time Spent: 10m
Remaining Estimate: 0h

> Run Python ValidatesRunner tests in streaming
> -
>
> Key: BEAM-4859
> URL: https://issues.apache.org/jira/browse/BEAM-4859
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
> 

Jenkins build is back to normal : beam_PostCommit_Java_Nexmark_Spark #114

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4855) Microbenchmarks for windowinto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?focusedWorklogId=126971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126971
 ]

ASF GitHub Bot logged work on BEAM-4855:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:02
Start Date: 25/Jul/18 00:02
Worklog Time Spent: 10m 
  Work Description: tvalentyn edited a comment on issue #6051: [BEAM-4855] 
Added windowinto_microbenchmark
URL: https://github.com/apache/beam/pull/6051#issuecomment-407590033
 
 
   Awesome, thank you! Did you see helpers in 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/utils.py?
 Consider using them to structure your microbenchmark to avoid code duplication 
related to measurements.
   You could take a look at 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/coders_microbenchmark.py
 as an example.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126971)
Time Spent: 40m  (was: 0.5h)

> Microbenchmarks for windowinto
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the windowinto transform:
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4855) Microbenchmarks for windowinto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?focusedWorklogId=126970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126970
 ]

ASF GitHub Bot logged work on BEAM-4855:


Author: ASF GitHub Bot
Created on: 25/Jul/18 00:02
Start Date: 25/Jul/18 00:02
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6051: [BEAM-4855] Added 
windowinto_microbenchmark
URL: https://github.com/apache/beam/pull/6051#issuecomment-407590033
 
 
   Awesome. Did you see helpers in 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/utils.py?
 Consider using them to structure your microbenchmark to avoid code duplication 
related to measurements.
   You could take a look at 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/tools/coders_microbenchmark.py
 as an example.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126970)
Time Spent: 0.5h  (was: 20m)

> Microbenchmarks for windowinto
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the windowinto transform:
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4855) Microbenchmarks for windowinto

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys updated BEAM-4855:

Description: 
Add microbenchmarks for the windowinto transform:

R: [~tvalentyn]

  was:
Add microbenchmarks for the transforms subpackage:

Currently working on WindowInto microbenchmark

R: [~tvalentyn]


> Microbenchmarks for windowinto
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the windowinto transform:
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4855) Microbenchmarks for transforms package

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys updated BEAM-4855:

Description: 
Add microbenchmarks for the transforms subpackage:

Currently working on WindowInto microbenchmark

R: [~tvalentyn]

  was:
Add microbenchmarks for the transforms subpackage:

Currently working on TimestampedValue microbenchmark

R: [~tvalentyn]


> Microbenchmarks for transforms package
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the transforms subpackage:
> Currently working on WindowInto microbenchmark
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4859) Run Python ValidatesRunner tests in streaming

[Mark Liu (JIRA)]

Mark Liu created BEAM-4859:
--

 Summary: Run Python ValidatesRunner tests in streaming
 Key: BEAM-4859
 URL: https://issues.apache.org/jira/browse/BEAM-4859
 Project: Beam
  Issue Type: Test
  Components: testing
Reporter: Mark Liu
Assignee: Mark Liu


Automated way to run existing Python ValidatesRunner tests in streaming. Gradle 
should also be able to run them, so Jenkins automation will be much simpler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4855) Microbenchmarks for windowinto

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys updated BEAM-4855:

Summary: Microbenchmarks for windowinto  (was: Microbenchmarks for 
transforms package)

> Microbenchmarks for windowinto
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the transforms subpackage:
> Currently working on WindowInto microbenchmark
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4855) Microbenchmarks for transforms package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?focusedWorklogId=126965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126965
 ]

ASF GitHub Bot logged work on BEAM-4855:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:51
Start Date: 24/Jul/18 23:51
Worklog Time Spent: 10m 
  Work Description: Fematich commented on issue #6051: [BEAM-4855] Added 
timestamped_value_microbenchmark
URL: https://github.com/apache/beam/pull/6051#issuecomment-407588256
 
 
   I will reformat this microbenchmark to `windowinto_microbenchmark` that will 
test `FixedWindow`, `SlidingWindow` and `Sessions` similar to the used code 
now, I plan to update it tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126965)
Time Spent: 20m  (was: 10m)

> Microbenchmarks for transforms package
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the transforms subpackage:
> Currently working on TimestampedValue microbenchmark
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-4858:
--
Description: 
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division consistently [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
  negatively affects the performance of a custom pipeline I was using to 
benchmark these changes. The performance impact likely comes from changes in 
the logic that depends on  how division is evaluated, not from the performance 
of division operation itself.

In terms of Python 3 conversion the best course of action that avoids 
regression seems to be to preserve the existing Python 2 behavior using 
{{old_div}} from {{past.utils.division}}, in the medium term we should clean up 
the logic. We may want to add a targeted microbenchmark to evaluate performance 
of this code, and maybe cythonize the code, since it seems to be 
performance-sensitive.


  was:
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division consistently [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
  negatively affects the performance of a custom pipeline I was using to 
benchmark these changes.

In terms of Python 3 conversion the best course of action that avoids 
regression seems to be to preserve the existing Python 2 behavior using 
{{old_div}} from {{past.utils.division}}, in the medium term we should clean up 
the logic. We may want to add a targeted microbenchmark to evaluate performance 
of this code, and maybe cythonize the code, since it seems to be 
performance-sensitive.



> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division consistently [in the 
> 

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126964
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:48
Start Date: 24/Jul/18 23:48
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r204945776
 
 

 ##
 File path: sdks/python/apache_beam/transforms/util.py
 ##
 @@ -273,7 +279,7 @@ def div_keys(kv1_kv2):
 pairs = sorted(zip(sorted_data[::2], sorted_data[1::2]),
key=div_keys)
 # Keep the top 1/3 most different pairs, average the top 2/3 most similar.
-threshold = 2 * len(pairs) / 3
+threshold = 2 * len(pairs) // 3
 
 Review comment:
   Thanks for your patience with this investigation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126964)
Time Spent: 10h 20m  (was: 10h 10m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126963
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:46
Start Date: 24/Jul/18 23:46
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #499: [BEAM-4802] Update the 
contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/contribute/dependencies.md b/src/contribute/dependencies.md
index 11484cc02..99ec6e690 100644
--- a/src/contribute/dependencies.md
+++ b/src/contribute/dependencies.md
@@ -44,7 +44,7 @@ One common solution for the diamond dependency problem is 
[semantic versioning](
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list. In the future we hope to automatically create JIRAs 
based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 * A minor release of a dependency due to a critical security vulnerability. 
@@ -52,6 +52,17 @@ In addition to this, Beam community members might identify 
other critical depend
 
 These kind of urgently required upgrades might not get automatically picked up 
by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+## JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be created 
per significant outdated dependency based on the report. A bot named *Beam Jira 
Bot* was created for managing JIRA issues. Beam community agrees on the 
following policies that creates and updates issues.
+* Issues will be named as "Beam Dependency Update Request:  
".
+* Issues will be created under the component *"dependencies"*
+* Issues will be assigned to the primary owner of the dependencies, who are 
mentioned in the dependency ownership files. ([Java Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml)
 and [Python Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml))
+* If more than one owners found for a dependency, the first owner will be 
picked as the primary owner, the others will be pinged in the issue's 
description.
+* If no owners found, leave the assignee empty. The component lead is 
responsible for triaging the issue.
+* Avoid creating duplicate issues. Updating the descriptions of the open 
issues created by the previous dependency check.
+* The dependency sometimes is not able to be upgraded, the issue should be 
closed as *"won't fix"*. And, the bot should avoid recreating issues with 
"won't fix".
+
 ## Upgrading identified outdated dependencies
 
 After outdated dependencies are identified, Beam community has to act to 
upgrade the dependencies regularly. Beam community has agreed on following 
policies regarding upgrading dependencies.
@@ -72,7 +83,7 @@ For manually identified critical dependency updates, Beam 
community members shou
 
 __Dependency declarations may identify owners that are responsible for 
upgrading respective dependencies.__
 
-Owners can be mentioned in a comment. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
+Owners can be mentioned in the yaml files. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
 
 __Dependencies of Java SDK components that may cause issues to other 
components if leaked should be vendored.__
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8a7108dddbb79882c4a5909d41a6c1574ba5eb4e
Author: Mergebot 
AuthorDate: Tue Jul 24 23:46:13 2018 +

Prepare repository for deployment.
---
 content/contribute/dependencies/index.html | 18 --
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/content/contribute/dependencies/index.html 
b/content/contribute/dependencies/index.html
index 9cb343b..57c2c13 100644
--- a/content/contribute/dependencies/index.html
+++ b/content/contribute/dependencies/index.html
@@ -190,6 +190,7 @@
 
 
   Identifying outdated 
dependencies
+  JIRA Automation
   Upgrading 
identified outdated dependencies
   Dependency 
updates and backwards compatibility
 
@@ -238,7 +239,7 @@ limitations under the License.
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify 
outdated dependencies for various SDKs. This Jenkins job generates a weekly 
report that is shared in Beam dev list. In the future we hope to automatically 
create JIRAs based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify 
outdated dependencies for various SDKs. This Jenkins job generates a weekly 
report that is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 
@@ -248,6 +249,19 @@ limitations under the License.
 
 These kind of urgently required upgrades might not get automatically picked 
up by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be 
created per significant outdated dependency based on the report. A bot named 
Beam Jira Bot was created for managing JIRA issues. Beam community 
agrees on the following policies that creates and updates issues.
+
+  Issues will be named as “Beam Dependency Update Request:  
".
+  Issues will be created under the component “dependencies”
+  Issues will be assigned to the primary owner of the dependencies, who 
are mentioned in the dependency ownership files. (https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml;>Java
 Dependency Owners and https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml;>Python
 Dependency Owners)
+  If more than one owners found for a dependency, the first owner will be 
picked as the primary owner, the others will be pinged in the issue’s 
description.
+  If no owners found, leave the assignee empty. The component lead is 
responsible for triaging the issue.
+  Avoid creating duplicate issues. Updating the descriptions of the open 
issues created by the previous dependency check.
+  The dependency sometimes is not able to be upgraded, the issue should be 
closed as “won’t fix”. And, the bot should avoid recreating issues 
with “won’t fix”.
+
+
 Upgrading identified 
outdated dependencies
 
 After outdated dependencies are identified, Beam community has to act to 
upgrade the dependencies regularly. Beam community has agreed on following 
policies regarding upgrading dependencies.
@@ -268,7 +282,7 @@ limitations under the License.
 
 Dependency declarations may identify owners that are responsible 
for upgrading respective dependencies.
 
-Owners can be mentioned in a comment. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
+Owners can be mentioned in the yaml files. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
 
 Dependencies of Java SDK components that may cause issues to other 
components if leaked should be vendored.
 

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=126962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126962
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:45
Start Date: 24/Jul/18 23:45
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #5955: [BEAM-4076] Enable 
schemas for more runners
URL: https://github.com/apache/beam/pull/5955
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/apex/build.gradle b/runners/apex/build.gradle
index dbe19e8efa4..a2bfdec355a 100644
--- a/runners/apex/build.gradle
+++ b/runners/apex/build.gradle
@@ -93,7 +93,6 @@ task validatesRunnerBatch(type: Test) {
 excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
 excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
 excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
-excludeCategories 'org.apache.beam.sdk.testing.UsesSchema'
 excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
 excludeCategories 'org.apache.beam.sdk.testing.UsesTimersInParDo'
 excludeCategories 'org.apache.beam.sdk.testing.UsesMetricsPusher'
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
index 32113a97630..d44d18c849c 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
@@ -28,9 +28,11 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.stream.Collectors;
 import org.apache.beam.runners.apex.ApexRunner;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import 
org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems.ProcessElements;
+import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
@@ -76,6 +78,13 @@ public void translate(ParDo.MultiOutput 
transform, TranslationC
 PCollection input = context.getInput();
 List> sideInputs = transform.getSideInputs();
 
+Map, Coder> outputCoders =
+outputs
+.entrySet()
+.stream()
+.filter(e -> e.getValue() instanceof PCollection)
+.collect(
+Collectors.toMap(e -> e.getKey(), e -> ((PCollection) 
e.getValue()).getCoder()));
 ApexParDoOperator operator =
 new ApexParDoOperator<>(
 context.getPipelineOptions(),
@@ -85,6 +94,7 @@ public void translate(ParDo.MultiOutput 
transform, TranslationC
 input.getWindowingStrategy(),
 sideInputs,
 input.getCoder(),
+outputCoders,
 context.getStateBackend());
 
 Map, OutputPort> ports = 
Maps.newHashMapWithExpectedSize(outputs.size());
@@ -130,6 +140,14 @@ public void translate(
   PCollection input = context.getInput();
   List> sideInputs = transform.getSideInputs();
 
+  Map, Coder> outputCoders =
+  outputs
+  .entrySet()
+  .stream()
+  .filter(e -> e.getValue() instanceof PCollection)
+  .collect(
+  Collectors.toMap(e -> e.getKey(), e -> ((PCollection) 
e.getValue()).getCoder()));
+
   @SuppressWarnings({"rawtypes", "unchecked"})
   DoFn doFn = (DoFn) 
transform.newProcessFn(transform.getFn());
   ApexParDoOperator operator =
@@ -140,7 +158,8 @@ public void translate(
   transform.getAdditionalOutputTags().getAll(),
   input.getWindowingStrategy(),
   sideInputs,
-  null,
+  input.getCoder(),
+  outputCoders,
   context.getStateBackend());
 
   Map, OutputPort> ports = 
Maps.newHashMapWithExpectedSize(outputs.size());
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index f9d20520e2b..577835238e4 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -117,7 +117,13 @@
   private final List> sideInputs;
 
   @Bind(JavaSerializer.class)
-  private final Coder> 

[beam-site] branch asf-site updated (6a757d6 -> 8a7108d)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 6a757d6  Prepare repository for deployment.
 add 9a4887b  [BEAM-4802] Update the contribution guide (Dependency section)
 add f2ea645  This closes #499
 new 8a7108d  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/dependencies/index.html | 18 --
 src/contribute/dependencies.md | 15 +--
 2 files changed, 29 insertions(+), 4 deletions(-)

[beam] branch master updated (d2ac08d -> b87aa60)

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d2ac08d  [BEAM-4790] Remove CODEOWNERs.
 add 8564216  Enable schemas for Apex runner.
 add 53794ec  Enable Schemas on samza runner.
 add fdd1738  Enable schemas for Flink runner.
 add 91ec0cb  Enable schemas for Spark.
 add 966d6ab  Fix bugs.
 add 0964b1f  Fix Apex breakage.
 new b87aa60  Merge pull request #5955: [BEAM-4076] Enable schemas for more 
runners

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/apex/build.gradle  |  1 -
 .../runners/apex/translation/ParDoTranslator.java  | 21 +++-
 .../translation/operators/ApexParDoOperator.java   | 21 
 .../apex/translation/ParDoTranslatorTest.java  |  1 +
 .../core/construction/ParDoTranslation.java| 10 
 runners/flink/build.gradle |  1 -
 .../flink/FlinkBatchTransformTranslators.java  | 17 -
 .../flink/FlinkBatchTranslationContext.java| 10 
 .../FlinkStreamingPortablePipelineTranslator.java  | 10 +---
 .../flink/FlinkStreamingTransformTranslators.java  | 29 +-
 .../flink/FlinkStreamingTranslationContext.java| 16 +++-
 .../translation/functions/FlinkDoFnFunction.java   | 14 ---
 .../functions/FlinkStatefulDoFnFunction.java   | 14 ---
 .../wrappers/streaming/DoFnOperator.java   | 21 ++--
 .../streaming/ExecutableStageDoFnOperator.java |  6 -
 .../wrappers/streaming/SplittableDoFnOperator.java |  6 -
 .../wrappers/streaming/WindowDoFnOperator.java |  6 +++--
 .../beam/runners/flink/PipelineOptionsTest.java|  4 +++
 .../runners/flink/streaming/DoFnOperatorTest.java  | 26 +++
 .../streaming/ExecutableStageDoFnOperatorTest.java |  4 +++
 runners/gearpump/build.gradle  |  1 +
 runners/samza/build.gradle |  1 -
 .../apache/beam/runners/samza/runtime/DoFnOp.java  |  8 ++
 .../runtime/DoFnRunnerWithKeyedInternals.java  |  9 ---
 .../translation/ParDoBoundMultiTranslator.java | 10 
 runners/spark/build.gradle |  2 --
 .../spark/translation/EvaluationContext.java   | 10 
 .../spark/translation/MultiDoFnFunction.java   | 13 --
 .../spark/translation/TransformTranslator.java |  5 +++-
 .../streaming/StreamingTransformTranslator.java|  4 +++
 30 files changed, 247 insertions(+), 54 deletions(-)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=126960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126960
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:44
Start Date: 24/Jul/18 23:44
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5955: [BEAM-4076] Enable 
schemas for more runners
URL: https://github.com/apache/beam/pull/5955#issuecomment-407587091
 
 
   @echauchot all comments addressed. You were correct about the Apex failure - 
it was caused by a typo between inputCoder and windowedInputCoder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126960)
Time Spent: 15h 40m  (was: 15.5h)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5955: [BEAM-4076] Enable schemas for more runners

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b87aa6037b37aab95702fd1ace40b835f0f66f55
Merge: d2ac08d 0964b1f
Author: reuvenlax 
AuthorDate: Tue Jul 24 16:45:03 2018 -0700

Merge pull request #5955: [BEAM-4076] Enable schemas for more runners

 runners/apex/build.gradle  |  1 -
 .../runners/apex/translation/ParDoTranslator.java  | 21 +++-
 .../translation/operators/ApexParDoOperator.java   | 21 
 .../apex/translation/ParDoTranslatorTest.java  |  1 +
 .../core/construction/ParDoTranslation.java| 10 
 runners/flink/build.gradle |  1 -
 .../flink/FlinkBatchTransformTranslators.java  | 17 -
 .../flink/FlinkBatchTranslationContext.java| 10 
 .../FlinkStreamingPortablePipelineTranslator.java  | 10 +---
 .../flink/FlinkStreamingTransformTranslators.java  | 29 +-
 .../flink/FlinkStreamingTranslationContext.java| 16 +++-
 .../translation/functions/FlinkDoFnFunction.java   | 14 ---
 .../functions/FlinkStatefulDoFnFunction.java   | 14 ---
 .../wrappers/streaming/DoFnOperator.java   | 21 ++--
 .../streaming/ExecutableStageDoFnOperator.java |  6 -
 .../wrappers/streaming/SplittableDoFnOperator.java |  6 -
 .../wrappers/streaming/WindowDoFnOperator.java |  6 +++--
 .../beam/runners/flink/PipelineOptionsTest.java|  4 +++
 .../runners/flink/streaming/DoFnOperatorTest.java  | 26 +++
 .../streaming/ExecutableStageDoFnOperatorTest.java |  4 +++
 runners/gearpump/build.gradle  |  1 +
 runners/samza/build.gradle |  1 -
 .../apache/beam/runners/samza/runtime/DoFnOp.java  |  8 ++
 .../runtime/DoFnRunnerWithKeyedInternals.java  |  9 ---
 .../translation/ParDoBoundMultiTranslator.java | 10 
 runners/spark/build.gradle |  2 --
 .../spark/translation/EvaluationContext.java   | 10 
 .../spark/translation/MultiDoFnFunction.java   | 13 --
 .../spark/translation/TransformTranslator.java |  5 +++-
 .../streaming/StreamingTransformTranslator.java|  4 +++
 30 files changed, 247 insertions(+), 54 deletions(-)

[beam-site] 01/02: [BEAM-4802] Update the contribution guide (Dependency section)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9a4887b3d7595a367866fbf6cb2a0ca8aec8c876
Author: Yifan Zou 
AuthorDate: Wed Jul 18 20:18:37 2018 -0700

[BEAM-4802] Update the contribution guide (Dependency section)
---
 src/contribute/dependencies.md | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/src/contribute/dependencies.md b/src/contribute/dependencies.md
index 11484cc..99ec6e6 100644
--- a/src/contribute/dependencies.md
+++ b/src/contribute/dependencies.md
@@ -44,7 +44,7 @@ One common solution for the diamond dependency problem is 
[semantic versioning](
 
 A big part of keeping dependencies up to date involves identifying outdated 
dependencies of Beam that the community should try to upgrade.
 
-Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list. In the future we hope to automatically create JIRAs 
based on this report.
+Beam currently executes a weekly Jenkins job that tries to identify outdated 
dependencies for various SDKs. This Jenkins job generates a weekly report that 
is shared in Beam dev list.
 
 In addition to this, Beam community members might identify other critical 
dependency updates that have to be manually performed. For example,
 * A minor release of a dependency due to a critical security vulnerability. 
@@ -52,6 +52,17 @@ In addition to this, Beam community members might identify 
other critical depend
 
 These kind of urgently required upgrades might not get automatically picked up 
by the Jenkins job for few months. So Beam community has to act to identify 
such issues and perform upgrades early.
 
+## JIRA Automation
+
+In order to track the dependency upgrade process, JIRA tickets will be created 
per significant outdated dependency based on the report. A bot named *Beam Jira 
Bot* was created for managing JIRA issues. Beam community agrees on the 
following policies that creates and updates issues.
+* Issues will be named as "Beam Dependency Update Request:  
".
+* Issues will be created under the component *"dependencies"*
+* Issues will be assigned to the primary owner of the dependencies, who are 
mentioned in the dependency ownership files. ([Java Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml)
 and [Python Dependency 
Owners](https://github.com/apache/beam/blob/master/ownership/PYTHON_DEPENDENCY_OWNERS.yaml))
+* If more than one owners found for a dependency, the first owner will be 
picked as the primary owner, the others will be pinged in the issue's 
description.
+* If no owners found, leave the assignee empty. The component lead is 
responsible for triaging the issue.
+* Avoid creating duplicate issues. Updating the descriptions of the open 
issues created by the previous dependency check.
+* The dependency sometimes is not able to be upgraded, the issue should be 
closed as *"won't fix"*. And, the bot should avoid recreating issues with 
"won't fix".
+
 ## Upgrading identified outdated dependencies
 
 After outdated dependencies are identified, Beam community has to act to 
upgrade the dependencies regularly. Beam community has agreed on following 
policies regarding upgrading dependencies.
@@ -72,7 +83,7 @@ For manually identified critical dependency updates, Beam 
community members shou
 
 __Dependency declarations may identify owners that are responsible for 
upgrading respective dependencies.__
 
-Owners can be mentioned in a comment. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
+Owners can be mentioned in the yaml files. Blocking JIRAs will be initially 
assigned to these owners (if available). Release manager may choose to 
re-assign these JIRAs. A dependency may have more than one declared owner and 
in this case the JIRA will be assigned to one of the owners mentioned.
 
 __Dependencies of Java SDK components that may cause issues to other 
components if leaked should be vendored.__
 

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126958
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:43
Start Date: 24/Jul/18 23:43
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r204945047
 
 

 ##
 File path: sdks/python/apache_beam/transforms/util.py
 ##
 @@ -273,7 +279,7 @@ def div_keys(kv1_kv2):
 pairs = sorted(zip(sorted_data[::2], sorted_data[1::2]),
key=div_keys)
 # Keep the top 1/3 most different pairs, average the top 2/3 most similar.
-threshold = 2 * len(pairs) / 3
+threshold = 2 * len(pairs) // 3
 
 Review comment:
   Perfect! I will update the PR, thx!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126958)
Time Spent: 10h 10m  (was: 10h)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (4b85fc2 -> f2ea645)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4b85fc2  This closes #508
 add 6a757d6  Prepare repository for deployment.
 new 9a4887b  [BEAM-4802] Update the contribution guide (Dependency section)
 new f2ea645  This closes #499

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html |  5 -
 src/contribute/dependencies.md | 15 +--
 2 files changed, 13 insertions(+), 7 deletions(-)

[beam-site] 02/02: This closes #499

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f2ea645988e1132115363e7d272d58ac09c4a750
Merge: 6a757d6 9a4887b
Author: Mergebot 
AuthorDate: Tue Jul 24 23:43:10 2018 +

This closes #499

 src/contribute/dependencies.md | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

[jira] [Work logged] (BEAM-4802) Update "Dependency" section of the Contribution Guide

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4802?focusedWorklogId=126957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126957
 ]

ASF GitHub Bot logged work on BEAM-4802:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:42
Start Date: 24/Jul/18 23:42
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #499: [BEAM-4802] 
Update the contribution guide (Dependency section)
URL: https://github.com/apache/beam-site/pull/499#issuecomment-407586664
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126957)
Time Spent: 2h 40m  (was: 2.5h)

> Update "Dependency" section of the Contribution Guide
> -
>
> Key: BEAM-4802
> URL: https://issues.apache.org/jira/browse/BEAM-4802
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126956
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:40
Start Date: 24/Jul/18 23:40
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r204944489
 
 

 ##
 File path: sdks/python/apache_beam/transforms/util.py
 ##
 @@ -273,7 +279,7 @@ def div_keys(kv1_kv2):
 pairs = sorted(zip(sorted_data[::2], sorted_data[1::2]),
key=div_keys)
 # Keep the top 1/3 most different pairs, average the top 2/3 most similar.
-threshold = 2 * len(pairs) / 3
+threshold = 2 * len(pairs) // 3
 
 Review comment:
   I have confirmed that this change brings performance back to the same 
ballpark.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126956)
Time Spent: 10h  (was: 9h 50m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126955
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:34
Start Date: 24/Jul/18 23:34
Worklog Time Spent: 10m 
  Work Description: tvalentyn edited a comment on issue #5729: [BEAM-4006] 
Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#issuecomment-407585250
 
 
   @Fematich I'm taking a look at c270644. I don't believe `@total_ordering` 
was an issue, but I'll see if the change makes a difference, I also started 
working on a microbenchmark but stopped pursuing that direction once I saw that 
 window.py changes were not the main offender. I'll take a look at your 
microbenchmark as well. 
   
   Since we now know how to make a have Py3-compatible version of this change 
that performs comparably well, the rest of performance testing won't take much 
time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126955)
Time Spent: 9h 50m  (was: 9h 40m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126954
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:34
Start Date: 24/Jul/18 23:34
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5729: [BEAM-4006] 
Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#issuecomment-407585250
 
 
   I'm taking a look at c270644. I don't believe `@total_ordering` was an 
issue, but I'll see if the change makes a difference, I also started working on 
a microbenchmark but stopped pursuing that direction once I saw that  window.py 
changes were not the main offender. I'll take a look at your microbenchmark as 
well. 
   
   Since we now know how to make a have Py3-compatible version of this change 
that performs comparably well, the rest of performance testing won't take much 
time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126954)
Time Spent: 9h 40m  (was: 9.5h)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=126952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126952
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:24
Start Date: 24/Jul/18 23:24
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6052: [BEAM-4852] Only 
read symbol table when required.
URL: https://github.com/apache/beam/pull/6052#issuecomment-407583597
 
 
   Run Go PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126952)
Time Spent: 0.5h  (was: 20m)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4857) Beam Dependency Update Request: groupA:artifactA 4.0

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-4857:

Description: 


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 15:54:13.532422

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 16:14:13.608317

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

  was:


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 15:54:13.532422

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 


> Beam Dependency Update Request: groupA:artifactA 4.0
> 
>
> Key: BEAM-4857
> URL: https://issues.apache.org/jira/browse/BEAM-4857
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-24 15:50:08.854551
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-24 15:54:13.532422
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-24 16:14:13.608317
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4853) Beam Dependency Update Request: groupA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4853.
---
   Resolution: Fixed
Fix Version/s: 3.0.0
   Not applicable

Test Auto Jira issue reopening 

> Beam Dependency Update Request: groupA
> --
>
> Key: BEAM-4853
> URL: https://issues.apache.org/jira/browse/BEAM-4853
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable, 3.0.0
>
>
> 2018-07-24 14:49:21.295726
> Please review and upgrade the groupA to the latest version None 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-4853) Beam Dependency Update Request: groupA

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot reopened BEAM-4853:
-

> Beam Dependency Update Request: groupA
> --
>
> Key: BEAM-4853
> URL: https://issues.apache.org/jira/browse/BEAM-4853
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable, 3.0.0
>
>
> 2018-07-24 14:49:21.295726
> Please review and upgrade the groupA to the latest version None 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126950
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:13
Start Date: 24/Jul/18 23:13
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r204940139
 
 

 ##
 File path: sdks/python/apache_beam/transforms/util.py
 ##
 @@ -273,7 +279,7 @@ def div_keys(kv1_kv2):
 pairs = sorted(zip(sorted_data[::2], sorted_data[1::2]),
key=div_keys)
 # Keep the top 1/3 most different pairs, average the top 2/3 most similar.
-threshold = 2 * len(pairs) / 3
+threshold = 2 * len(pairs) // 3
 
 Review comment:
   Let's use past.utils.division.old_div in line 280 as an exception, and add a 
TODO(BEAM-4858) comment to clean this up.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126950)
Time Spent: 9.5h  (was: 9h 20m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=126948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126948
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:11
Start Date: 24/Jul/18 23:11
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #6052: [BEAM-4852] 
Only read symbol table when required.
URL: https://github.com/apache/beam/pull/6052
 
 
   Beam Go currently always reads in the symbol table on init, but it's not 
always necessary. Changing to initialized it on first use.
   
   -
   
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126948)
Time Spent: 10m
Remaining Estimate: 0h

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires 

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=126949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126949
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:11
Start Date: 24/Jul/18 23:11
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6052: [BEAM-4852] Only 
read symbol table when required.
URL: https://github.com/apache/beam/pull/6052#issuecomment-407581145
 
 
   R: @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126949)
Time Spent: 20m  (was: 10m)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126946
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 23:04
Start Date: 24/Jul/18 23:04
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5729: [BEAM-4006] 
Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#issuecomment-407579682
 
 
   With latest round of experiments, we finally got to the bottom of this 
performance regression, see: https://issues.apache.org/jira/browse/BEAM-4858. I 
will also put some details inline in util.py. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126946)
Time Spent: 9h 20m  (was: 9h 10m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-4858:
--
Description: 
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division consistently [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
  negatively affects the performance of a custom pipeline I was using to 
benchmark these changes.

In terms of Python 3 conversion the best course of action that avoids 
regression seems to be to preserve the existing Python 2 behavior using 
{{old_div}} from {{past.utils.division}}, in the medium term we should clean up 
the logic. We may want to add a targeted microbenchmark to evaluate performance 
of this code, and maybe cythonize the code, since it seems to be 
performance-sensitive.


  was:
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
 consistently negatively affects the performance of a custom pipeline I was 
using to benchmark these changes.

In terms of Python 3 conversion the best course of action that avoids 
regression seems to be to preserve the existing Python 2 behavior using 
{{old_div}} from {{past.utils.division}}, in the medium term we should clean up 
the logic. We may want to add a targeted microbenchmark to evaluate performance 
of this code, and maybe cythonize the code, since it seems to be 
performance-sensitive.



> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division consistently [in the 
> comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
>   negatively affects the performance of 

[jira] [Updated] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-4858:
--
Description: 
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
 consistently negatively affects the performance of a custom pipeline I was 
using to benchmark these changes.

In terms of Python 3 conversion the best course of action that avoids 
regression seems to be to preserve the existing Python 2 behavior using 
{{old_div}} from {{past.utils.division}}, in the medium term we should clean up 
the logic. We may want to add a targeted microbenchmark to evaluate performance 
of this code, and maybe cythonize the code, since it seems to be 
performance-sensitive.


  was:
Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/3971] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
 consistently negatively affects the performance of a custom pipeline I was 
using to benchmark these changes.

In terms of Python 3 conversion the best course of action seems to be to 
preserve the existing Python 2 behavior using old_div from `past.division`, in 
long term we may want to add microbenchmark that we can use to target this code 
and evaluate it's performance, clean up the logic, and maybe cythonize since 
this code seems to be performance-critical.



> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division [in the 
> comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
>  consistently negatively affects the performance of a custom pipeline I was 
> using to 

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=126944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126944
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:56
Start Date: 24/Jul/18 22:56
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5952: 
[BEAM-3744] Python PubSub API Fixes and Tests
URL: https://github.com/apache/beam/pull/5952#discussion_r204936379
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
 ##
 @@ -841,11 +841,12 @@ def run_Read(self, transform_node):
   transform.source.id_label)
   if transform.source.with_attributes:
 # Setting this property signals Dataflow runner to return full
-# PubsubMessages instead of just the payload.
+# PubsubMessages instead of just data.
 step.add_property(PropertyNames.PUBSUB_SERIALIZED_ATTRIBUTES_FN, '')
   if transform.source.timestamp_attribute is not None:
 step.add_property(PropertyNames.PUBSUB_TIMESTAMP_ATTRIBUTE,
   transform.source.timestamp_attribute)
+  logging.info('pubsub source')
 
 Review comment:
   Oops. Removed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126944)
Time Spent: 10h 10m  (was: 10h)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=126943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126943
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:56
Start Date: 24/Jul/18 22:56
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5952: 
[BEAM-3744] Python PubSub API Fixes and Tests
URL: https://github.com/apache/beam/pull/5952#discussion_r204936751
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
 ##
 @@ -841,11 +841,12 @@ def run_Read(self, transform_node):
   transform.source.id_label)
   if transform.source.with_attributes:
 # Setting this property signals Dataflow runner to return full
-# PubsubMessages instead of just the payload.
+# PubsubMessages instead of just data.
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126943)
Time Spent: 10h  (was: 9h 50m)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4857) Beam Dependency Update Request: groupA:artifactA 4.0

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-4857:

Description: 


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

2018-07-24 15:54:13.532422

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 

  was:


2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 


> Beam Dependency Update Request: groupA:artifactA 4.0
> 
>
> Key: BEAM-4857
> URL: https://issues.apache.org/jira/browse/BEAM-4857
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-24 15:50:08.854551
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 
> 2018-07-24 15:54:13.532422
> Please review and upgrade the groupA:artifactA to the latest version 
> 4.0 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4855) Microbenchmarks for transforms package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4855?focusedWorklogId=126941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126941
 ]

ASF GitHub Bot logged work on BEAM-4855:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:54
Start Date: 24/Jul/18 22:54
Worklog Time Spent: 10m 
  Work Description: Fematich opened a new pull request #6051: [BEAM-4855] 
Added timestamped_value_microbenchmark
URL: https://github.com/apache/beam/pull/6051
 
 
   Added microbenchmark for timestamped_value (as mentioned in #5729)
   
   R: @tvalentyn 
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126941)
Time Spent: 10m
Remaining Estimate: 0h

> Microbenchmarks for transforms package
> --
>
> Key: BEAM-4855
> URL: https://issues.apache.org/jira/browse/BEAM-4855
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add microbenchmarks for the transforms subpackage:
> Currently working on TimestampedValue microbenchmark
> R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-4858:
-

 Summary: Clean up _BatchSizeEstimator in element-batching 
transform.
 Key: BEAM-4858
 URL: https://issues.apache.org/jira/browse/BEAM-4858
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Robert Bradshaw


Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/3971] 
non-trivial performance-sensitive logic in element-batching transform. Let's 
take a look at 
[util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
 

Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
the type of the keys - whether they are integers or floats. 

The keys of key-value pairs contained in {{self._data}} are added as integers 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
 however, when we 'thin' the collected entries 
[here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
 the keys will become floats. Surprisingly, using either integer or float 
division [in the 
comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
 consistently negatively affects the performance of a custom pipeline I was 
using to benchmark these changes.

In terms of Python 3 conversion the best course of action seems to be to 
preserve the existing Python 2 behavior using old_div from `past.division`, in 
long term we may want to add microbenchmark that we can use to target this code 
and evaluate it's performance, clean up the logic, and maybe cythonize since 
this code seems to be performance-critical.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4854) Beam Dependency Update Request: groupA:artifactA 2.0

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4854.
---
   Resolution: Fixed
Fix Version/s: Not applicable

Test complete. Close this issue.

> Beam Dependency Update Request: groupA:artifactA 2.0
> 
>
> Key: BEAM-4854
> URL: https://issues.apache.org/jira/browse/BEAM-4854
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> 2018-07-24 14:49:22.764361
> Please review and upgrade the groupA:artifactA to the latest version 
> 2.0 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4856) Beam Dependency Update Request: groupA:artifactA 3.0

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4856.
---
   Resolution: Fixed
Fix Version/s: Not applicable

Test complete. Close this issue

> Beam Dependency Update Request: groupA:artifactA 3.0
> 
>
> Key: BEAM-4856
> URL: https://issues.apache.org/jira/browse/BEAM-4856
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> 2018-07-24 15:48:32.656795
> Please review and upgrade the groupA:artifactA to the latest version 
> 3.0 
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4857) Beam Dependency Update Request: groupA:artifactA 4.0

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4857:
---

 Summary: Beam Dependency Update Request: groupA:artifactA 4.0
 Key: BEAM-4857
 URL: https://issues.apache.org/jira/browse/BEAM-4857
 Project: Beam
  Issue Type: Sub-task
  Components: dependencies
Reporter: Beam JIRA Bot
Assignee: yifan zou




2018-07-24 15:50:08.854551

Please review and upgrade the groupA:artifactA to the latest version 
4.0 

cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4856) Beam Dependency Update Request: groupA:artifactA 3.0

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4856:
---

 Summary: Beam Dependency Update Request: groupA:artifactA 3.0
 Key: BEAM-4856
 URL: https://issues.apache.org/jira/browse/BEAM-4856
 Project: Beam
  Issue Type: Bug
  Components: dependencies
Reporter: Beam JIRA Bot
Assignee: yifan zou




2018-07-24 15:48:32.656795

Please review and upgrade the groupA:artifactA to the latest version 
3.0 

cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=126933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126933
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:28
Start Date: 24/Jul/18 22:28
Worklog Time Spent: 10m 
  Work Description: Fematich commented on issue #5729: [BEAM-4006] Futurize 
transforms subpackage
URL: https://github.com/apache/beam/pull/5729#issuecomment-407572571
 
 
   @tvalentyn is there anything I can help with? Are you planning to benchmark 
the changes in 
https://github.com/apache/beam/commit/c2706447a6a602614c2f9bf36db1a666fa938819? 
Or do you first want to add more microbenchmarks? I am working on a 
microbenchmark for TimestampedValue #BEAM-4855


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126933)
Time Spent: 9h 10m  (was: 9h)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4855) Microbenchmarks for transforms package

[Matthias Feys (JIRA)]

Matthias Feys created BEAM-4855:
---

 Summary: Microbenchmarks for transforms package
 Key: BEAM-4855
 URL: https://issues.apache.org/jira/browse/BEAM-4855
 Project: Beam
  Issue Type: Task
  Components: sdk-py-core
Reporter: Matthias Feys
Assignee: Matthias Feys


Add microbenchmarks for the transforms subpackage:

Currently working on TimestampedValue microbenchmark

R: [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=126928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126928
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:15
Start Date: 24/Jul/18 22:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6050: [BEAM-4658] Update 
pipeline representation in runner support libraries to handle timers.
URL: https://github.com/apache/beam/pull/6050#issuecomment-407569883
 
 
   R: @tweise @angoenka 
   CC: @charlesccychen @bsidhom


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126928)
Time Spent: 0.5h  (was: 20m)

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=126927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126927
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:14
Start Date: 24/Jul/18 22:14
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #6050: [BEAM-4658] 
Update pipeline representation in runner support libraries to handle timers.
URL: https://github.com/apache/beam/pull/6050
 
 
   Note that I modified several runner libraries related to portability to 
support multiple inputs. It was painful because of the use of too many 
abstractions. Hopefully I'll have an opportunity to cut down on how many there 
are.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126927)
Time Spent: 20m  (was: 10m)

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: 

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=126925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126925
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:11
Start Date: 24/Jul/18 22:11
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6049: [BEAM-4790] Remove 
CODEOWNERs.
URL: https://github.com/apache/beam/pull/6049
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/CODEOWNERS b/CODEOWNERS
deleted file mode 100644
index dc4b5da650e..000
--- a/CODEOWNERS
+++ /dev/null
@@ -1,131 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is used for auto-assigning reviewers to PRs.
-#
-# The last matching rule wins. For details, see:
-# https://help.github.com/articles/about-codeowners/
-
-# Per-language reviewers.
-*.go @herohde
-*.java @lukecwik @kennknowles @aaltay
-*.py @pabloem @robertwb @aaltay @charlesccychen
-
-# SDK components reviewers.
-/sdks/go/ @herohde
-
-/sdks/java/ @aaltay @kennknowles
-/sdks/java/core @lukecwik
-#/sdks/java/extensions
-/sdks/java/extensions/google-cloud-platform-core @lukecwik @chamikaramj
-#/sdks/java/extensions/jackson
-#/sdks/java/extensions/join-library
-#/sdks/java/extensions/protobuf
-#/sdks/java/extensions/sketching
-#/sdks/java/extensions/sorter
-/sdks/java/extensions/sql @akedin @apilloud @amaliujia @XuMingmin @xumingming 
@kennknowles
-/sdks/java/container @herohde @aaltay
-/sdks/java/fn-execution @lukecwik @angoenka
-/sdks/java/nexmark @echauchot @aromanenko-dev
-/sdks/java/build-tools @jasonkuster @swegner
-/sdks/java/harness @lukecwik
-/sdks/java/io @jbonofre @lukecwik @chamikaramj @timrobertson100
-/sdks/java/io/parquet @lgajowy @jbonofre @aromanenko-dev
-/sdks/java/io/common @chamikaramj @timrobertson100
-/sdks/java/io/file-based-io-tests @lgajowy @kkucharc @chamikaramj 
@timrobertson100
-/sdks/java/io/elasticsearch-tests @echauchot @timrobertson100
-#/sdks/java/io/amazon-web-services
-/sdks/java/io/google-cloud-platform @lukecwik @chamikaramj
-/sdks/java/io/hadoop-input-format @aromanenko-dev @timrobertson100
-/sdks/java/io/solr @timrobertson100
-/sdks/java/io/kinesis @aromanenko-dev
-/sdks/java/io/redis @jbonofre
-/sdks/java/io/mongodb @jbonofre
-/sdks/java/io/hcatalog @timrobertson100
-/sdks/java/io/elasticsearch @echauchot @jbonofre @timrobertson100
-/sdks/java/io/jdbc @jbonofre @timrobertson100
-/sdks/java/io/mqtt @jbonofre
-/sdks/java/io/hadoop-file-system @timrobertson100
-/sdks/java/io/amqp @jbonofre
-/sdks/java/io/cassandra @jbonofre @echauchot
-#/sdks/java/io/tika
-#/sdks/java/io/xml
-#/sdks/java/io/kafka
-/sdks/java/io/jms @jbonofre
-/sdks/java/io/hbase @timrobertson100 @aromanenko-dev
-/sdks/java/io/hadoop-common @timrobertson100
-#/sdks/java/maven-archetypes
-/sdks/java/javadoc @melap
-
-/sdks/python/ @aaltay @charlesccychen
-/sdks/python/scripts @aaltay
-/sdks/python/container @herohde @aaltay @charlesccychen
-/sdks/python/apache_beam/runners @pabloem @charlesccychen
-/sdks/python/apache_beam/utils @pabloem @charlesccychen
-/sdks/python/apache_beam/examples @aaltay @charlesccychen
-/sdks/python/apache_beam/internal @aaltay @charlesccychen
-/sdks/python/apache_beam/tools @pabloem @charlesccychen
-/sdks/python/apache_beam/coders @charlesccychen
-/sdks/python/apache_beam/portability @charlesccychen @robertwb @aaltay
-/sdks/python/apache_beam/typehints @charlesccychen
-/sdks/python/apache_beam/options @pabloem @charlesccychen
-/sdks/python/apache_beam/transforms @pabloem @aaltay @charlesccychen
-/sdks/python/apache_beam/testing @pabloem @aaltay
-/sdks/python/apache_beam/metrics @pabloem @aaltay
-/sdks/python/apache_beam/io @chamikaramj @aaltay @udim
-
-# Build and test infrastructure reviewers.
-#/.github
-/.test-infra @lgajowy
-/.test-infra/jenkins @lgajowy @kkucharc @echauchot
-/.test-infra/jupyter @udim
-/.test-infra/kubernetes @lgajowy 

[beam] 01/01: [BEAM-4790] Remove CODEOWNERs.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d2ac08da2dccce8930432fae1ec7c30953880b69
Merge: e98ff7c 0c88a6e
Author: Lukasz Cwik 
AuthorDate: Tue Jul 24 15:10:57 2018 -0700

[BEAM-4790] Remove CODEOWNERs.

 CODEOWNERS | 131 -
 1 file changed, 131 deletions(-)

[beam] branch master updated (e98ff7c -> d2ac08d)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e98ff7c  Merge pull request #6046: [BEAM-4847] Fix java OOM errors 
during compile by setting maxWorkers, Xms and Xmx to reasonable values
 add 0c88a6e  Remove CODEOWNERs.
 new d2ac08d  [BEAM-4790] Remove CODEOWNERs.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CODEOWNERS | 131 -
 1 file changed, 131 deletions(-)
 delete mode 100644 CODEOWNERS

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126924
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:09
Start Date: 24/Jul/18 22:09
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #6046: 
[BEAM-4847] Fix java OOM errors during compile by setting maxWorkers, Xms and 
Xmx to reasonable values
URL: https://github.com/apache/beam/pull/6046#discussion_r204927772
 
 

 ##
 File path: .test-infra/jenkins/CommonJobProperties.groovy
 ##
 @@ -188,16 +189,9 @@ class CommonJobProperties {
 
 // Ensure that parallel workers don't exceed total available memory.
 
-// TODO(BEAM-4230): OperatingSystemMXBeam incorrectly reports total 
memory; hard-code for now
-// Jenkins machines are GCE n1-highmem-16, with 104 GB of memory
-// def os = 
(com.sun.management.OperatingSystemMXBean)java.lang.management.ManagementFactory.getOperatingSystemMXBean()
-// def totalMemoryMb = os.getTotalPhysicalMemorySize() / (1024*1024)
-def totalMemoryMb = 104 * 1024
-// Jenkins uses 2 executors to schedule concurrent jobs, so ensure that 
each executor uses only half the
-// machine memory.
-def totalExecutorMemoryMb = totalMemoryMb / 2
-def perWorkerMemoryMb = totalExecutorMemoryMb / maxWorkers
-context.switches("-Dorg.gradle.jvmargs=-Xmx${(int)perWorkerMemoryMb}m")
+// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (3g/8g).
 
 Review comment:
   The JVM does not garbage collect eagerly and grows to the max heap size and 
then performs GC so its very likely that all workers will use max heap size if 
they run for any prolonged period of time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126924)
Time Spent: 2h 10m  (was: 2h)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke reassigned BEAM-4852:
--

Assignee: Robert Burke

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=126922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126922
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:00
Start Date: 24/Jul/18 22:00
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #6049: [BEAM-4790] Remove 
CODEOWNERs.
URL: https://github.com/apache/beam/pull/6049#issuecomment-407566099
 
 
   R: @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126922)
Time Spent: 1h 20m  (was: 1h 10m)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?focusedWorklogId=126923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126923
 ]

ASF GitHub Bot logged work on BEAM-4847:


Author: ASF GitHub Bot
Created on: 24/Jul/18 22:00
Start Date: 24/Jul/18 22:00
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6046: [BEAM-4847] Fix java OOM errors during compile by setting maxWorkers, 
Xms and Xmx to reasonable values
URL: https://github.com/apache/beam/pull/6046#discussion_r204925771
 
 

 ##
 File path: .test-infra/jenkins/CommonJobProperties.groovy
 ##
 @@ -188,16 +189,9 @@ class CommonJobProperties {
 
 // Ensure that parallel workers don't exceed total available memory.
 
-// TODO(BEAM-4230): OperatingSystemMXBeam incorrectly reports total 
memory; hard-code for now
-// Jenkins machines are GCE n1-highmem-16, with 104 GB of memory
-// def os = 
(com.sun.management.OperatingSystemMXBean)java.lang.management.ManagementFactory.getOperatingSystemMXBean()
-// def totalMemoryMb = os.getTotalPhysicalMemorySize() / (1024*1024)
-def totalMemoryMb = 104 * 1024
-// Jenkins uses 2 executors to schedule concurrent jobs, so ensure that 
each executor uses only half the
-// machine memory.
-def totalExecutorMemoryMb = totalMemoryMb / 2
-def perWorkerMemoryMb = totalExecutorMemoryMb / maxWorkers
-context.switches("-Dorg.gradle.jvmargs=-Xmx${(int)perWorkerMemoryMb}m")
+// For [BEAM-4847], hardcode Xms and Xmx to reasonable values (3g/8g).
 
 Review comment:
   Only if all 24 workers are at max heap size? I'd expect a distribution among 
them where many are not max'd out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126923)
Time Spent: 2h  (was: 1h 50m)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4854) Beam Dependency Update Request: groupA:artifactA 2.0

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4854:
---

 Summary: Beam Dependency Update Request: groupA:artifactA 2.0
 Key: BEAM-4854
 URL: https://issues.apache.org/jira/browse/BEAM-4854
 Project: Beam
  Issue Type: Bug
  Components: dependencies
Reporter: Beam JIRA Bot
Assignee: yifan zou




2018-07-24 14:49:22.764361

Please review and upgrade the groupA:artifactA to the latest version 
2.0 

cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4853) Beam Dependency Update Request: groupA

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4853:
---

 Summary: Beam Dependency Update Request: groupA
 Key: BEAM-4853
 URL: https://issues.apache.org/jira/browse/BEAM-4853
 Project: Beam
  Issue Type: Bug
  Components: dependencies
Reporter: Beam JIRA Bot




2018-07-24 14:49:21.295726

Please review and upgrade the groupA to the latest version None 

cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=126910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126910
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 24/Jul/18 21:42
Start Date: 24/Jul/18 21:42
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5955: [BEAM-4076] Enable 
schemas for more runners
URL: https://github.com/apache/beam/pull/5955#issuecomment-407561785
 
 
   retest this please.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126910)
Time Spent: 15.5h  (was: 15h 20m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=126908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-126908
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 24/Jul/18 21:39
Start Date: 24/Jul/18 21:39
Worklog Time Spent: 10m 
  Work Description: udim opened a new pull request #6049: [BEAM-4790] 
Remove CODEOWNERs.
URL: https://github.com/apache/beam/pull/6049
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 126908)
Time Spent: 1h 10m  (was: 1h)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> 

[jira] [Commented] (BEAM-4790) Make it easier to select coder reviewers for PRs

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554827#comment-16554827
 ] 

Udi Meiri commented on BEAM-4790:
-

CODEOWNERS isn't a good solution. See mailing list thread above.


> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)