[beam] 01/01: Merge pull request #5930: [BEAM-4562] [SQL] Apply JDBC rules globally

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6fd2834f0506b12b5fd394eed76618f89ae82d1e
Merge: 6230523 b45d673
Author: XuMingmin 
AuthorDate: Fri Jul 13 22:57:50 2018 -0700

Merge pull request #5930: [BEAM-4562] [SQL] Apply JDBC rules globally

 .../main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (6230523 -> 6fd2834)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6230523  Merge pull request #5950: [BEAM-4787] Ignore generated vendor 
files in python container build
 add b45d673  [SQL] Apply JDBC rules globally
 new 6fd2834  Merge pull request #5930: [BEAM-4562] [SQL] Apply JDBC rules 
globally

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4562) [SQL] Fix INSERT VALUES in JdbcDriver

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4562?focusedWorklogId=123210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123210
 ]

ASF GitHub Bot logged work on BEAM-4562:


Author: ASF GitHub Bot
Created on: 14/Jul/18 05:57
Start Date: 14/Jul/18 05:57
Worklog Time Spent: 10m 
  Work Description: XuMingmin closed pull request #5930: [BEAM-4562] [SQL] 
Apply JDBC rules globally
URL: https://github.com/apache/beam/pull/5930
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
index 4b66e93d514..8d54ed09613 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
@@ -81,7 +81,7 @@
   Thread.currentThread().setContextClassLoader(origLoader);
 }
 // inject beam rules into planner
-Hook.PLANNER.addThread(
+Hook.PLANNER.add(
 new Function() {
   @Override
   public Void apply(RelOptPlanner planner) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123210)
Time Spent: 2h  (was: 1h 50m)

> [SQL] Fix INSERT VALUES in JdbcDriver 
> --
>
> Key: BEAM-4562
> URL: https://issues.apache.org/jira/browse/BEAM-4562
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Executing INSERT VALUES against JdbcDriver fails. Executing similar 
> statements against BeamSqlEnv works fine. Example:
> {code:java}
> TestTableProvider tableProvider = new TestTableProvider();
> Connection connection = JdbcDriver.connect(tableProvider);
> connection
> .createStatement()
> .executeUpdate("CREATE TABLE person (id BIGINT, name VARCHAR) TYPE 
> 'test'");
> connection.createStatement().executeQuery("INSERT INTO person VALUES(3, 
> 'yyy')");
> {code}
>  Output:
> {code}
> java.sql.SQLException: Error while executing SQL "INSERT INTO person 
> VALUES(3, 'yyy')": Node [rel#9:Subset#1.ENUMERABLE.[]] could not be 
> implemented; planner state:
> Root: rel#9:Subset#1.ENUMERABLE.[]
> Original rel:
> BeamIOSinkRel(subset=[rel#9:Subset#1.ENUMERABLE.[]], table=[[beam, person]], 
> operation=[INSERT], flattened=[false]): rowcount = 1.0, cumulative cost = 
> {1.0 rows, 0.0 cpu, 0.0 io}, id = 6
>   LogicalValues(subset=[rel#5:Subset#0.NONE.[]], tuples=[[{ 3, 'yyy' }]]): 
> rowcount = 1.0, cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io}, id = 0
> Sets:
> Set#0, type: RecordType(BIGINT id, VARCHAR name)
>   rel#5:Subset#0.NONE.[], best=null, importance=0.81
>   rel#0:LogicalValues.NONE.[[0, 1], [1]](type=RecordType(BIGINT 
> id, VARCHAR name),tuples=[{ 3, 'yyy' }]), rowcount=1.0, cumulative cost={inf}
>   rel#14:Subset#0.BEAM_LOGICAL.[], best=null, importance=0.81
>   rel#20:Subset#0.ENUMERABLE.[], best=rel#19, importance=0.405
>   rel#19:EnumerableValues.ENUMERABLE.[[0, 1], 
> [1]](type=RecordType(BIGINT id, VARCHAR name),tuples=[{ 3, 'yyy' }]), 
> rowcount=1.0, cumulative cost={1.0 rows, 1.0 cpu, 0.0 io}
> Set#1, type: RecordType(BIGINT ROWCOUNT)
>   rel#7:Subset#1.BEAM_LOGICAL.[], best=null, importance=0.9
>   
> rel#6:BeamIOSinkRel.BEAM_LOGICAL.[](input=rel#5:Subset#0.NONE.[],table=[beam, 
> person],operation=INSERT,flattened=false), rowcount=1.0, cumulative cost={inf}
>   
> rel#15:BeamIOSinkRel.BEAM_LOGICAL.[](input=rel#14:Subset#0.BEAM_LOGICAL.[],table=[beam,
>  person],operation=INSERT,flattened=false), rowcount=1.0, cumulative 
> cost={inf}
>   rel#9:Subset#1.ENUMERABLE.[], best=null, importance=1.0
>   
> rel#10:AbstractConverter.ENUMERABLE.[](input=rel#7:Subset#1.BEAM_LOGICAL.[],convention=ENUMERABLE,sort=[]),
>  rowcount=1.0, cumulative cost={inf}
>   
> rel#11:BeamEnumerableConverter.ENUMERABLE.[](input=rel#7:Subset#1.BEAM_LOGICAL.[]),
>  rowcount=1.0, cumulative cost={inf}{code}



--

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123178
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 14/Jul/18 03:09
Start Date: 14/Jul/18 03:09
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5941: [BEAM-4076] Schema 
utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#issuecomment-404994501
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123178)
Time Spent: 11h 20m  (was: 11h 10m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=123163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123163
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:55
Start Date: 14/Jul/18 01:55
Worklog Time Spent: 10m 
  Work Description: udim opened a new pull request #5952: [BEAM-3744] 
Python PubSub API Fixes and Tests
URL: https://github.com/apache/beam/pull/5952
 
 
   Fixes attributes and adds an integration test for Dataflow.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123163)
Time Spent: 9h 10m  (was: 9h)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=123162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123162
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:51
Start Date: 14/Jul/18 01:51
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5948: [BEAM-4417] Fix the 
expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404990487
 
 
   Ah, it fails on master and I think it's a known issue. @apilloud , 
@amaliujia can you take a look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123162)
Time Spent: 5h 50m  (was: 5h 40m)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=123160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123160
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:45
Start Date: 14/Jul/18 01:45
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5948: [BEAM-4417] Fix the 
expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404990146
 
 
   Looking at it, can reproduce, but not sure how these changes could have 
affected the failing test. Probably it was already broken, checking on master


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123160)
Time Spent: 5h 40m  (was: 5.5h)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=123156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123156
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:24
Start Date: 14/Jul/18 01:24
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5951: [BEAM-4790] Replace 
Github reviewer auto-assignment with suggest_reviewers.py script
URL: https://github.com/apache/beam/pull/5951#issuecomment-404988848
 
 
   The idea is to run it like a precommit test on Jenkins, and have it report 
back the list of suggested reviewers. Still a work in progress, so for now 
users should run it manually. The current Github-based auto-assignment is 
problematic.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123156)
Time Spent: 40m  (was: 0.5h)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=123149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123149
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:02
Start Date: 14/Jul/18 01:02
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #5951: [BEAM-4790] Replace 
Github reviewer auto-assignment with suggest_reviewers.py script
URL: https://github.com/apache/beam/pull/5951#issuecomment-404987381
 
 
   So are users supposed to run `suggest_reviewers.py` or is automatically run 
somewhere?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123149)
Time Spent: 20m  (was: 10m)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=123150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123150
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 14/Jul/18 01:02
Start Date: 14/Jul/18 01:02
Worklog Time Spent: 10m 
  Work Description: holdenk edited a comment on issue #5951: [BEAM-4790] 
Replace Github reviewer auto-assignment with suggest_reviewers.py script
URL: https://github.com/apache/beam/pull/5951#issuecomment-404987381
 
 
   So are devs supposed to run `suggest_reviewers.py` or is automatically run 
somewhere?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123150)
Time Spent: 0.5h  (was: 20m)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4790) Make it easier to select coder reviewers for PRs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?focusedWorklogId=123147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123147
 ]

ASF GitHub Bot logged work on BEAM-4790:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:58
Start Date: 14/Jul/18 00:58
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5951: [BEAM-4790] Replace 
Github reviewer auto-assignment with suggest_reviewers.py script
URL: https://github.com/apache/beam/pull/5951#issuecomment-404987057
 
 
   @tweise Created a JIRA. Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123147)
Time Spent: 10m
Remaining Estimate: 0h

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Davor Bonaci
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4790) Make it easier to select coder reviewers for PRs

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-4790:
---

Assignee: Udi Meiri  (was: Davor Bonaci)

> Make it easier to select coder reviewers for PRs
> 
>
> Key: BEAM-4790
> URL: https://issues.apache.org/jira/browse/BEAM-4790
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The original idea was to use Github's CODEOWNERS feature, but that turned out 
> to not work well for Apache Beam.
> Thread: 
> https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E
> Document discussing this idea:
> https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8
> Mailing list thread:
> https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4790) Make it easier to select coder reviewers for PRs

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-4790:
---

 Summary: Make it easier to select coder reviewers for PRs
 Key: BEAM-4790
 URL: https://issues.apache.org/jira/browse/BEAM-4790
 Project: Beam
  Issue Type: Improvement
  Components: project-management
Reporter: Udi Meiri
Assignee: Davor Bonaci


The original idea was to use Github's CODEOWNERS feature, but that turned out 
to not work well for Apache Beam.
Thread: 
https://lists.apache.org/thread.html/9cc3191b21489c855ed502beade6d453b18e26eec46cb2c3dfe810e9@%3Cdev.beam.apache.org%3E

Document discussing this idea:
https://docs.google.com/document/d/184Tn0ONJ2ENHBVrhIes4seVuBSkww7xX5xb5W2VZwm8/edit#heading=h.l7mcrg761zp8

Mailing list thread:
https://lists.apache.org/thread.html/6138d08c551e254b5f13b26c6ba06579a49a4694f4d13ad6d164689a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1056

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] FileSystems should throw exception if filesystem for schema is 
not found

[ankurgoenka] Make schema pattern stricter

[ankurgoenka] Adding tet cases for matchResouces

[github] [BEAM-4752] Add dill compatibility for older versions of dill (#5949)

--
[...truncated 17.48 MB...]
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.421Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.703Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.753Z: Elided trivial flatten 
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.806Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.849Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.887Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.934Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:55.964Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:56.005Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:56.051Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:56.102Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Jul 14, 2018 12:48:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:47:56.144Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[jira] [Created] (BEAM-4789) BigQueryIO WRITE_TRUNCATE documentation is out of date

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-4789:
-

 Summary: BigQueryIO WRITE_TRUNCATE documentation is out of date
 Key: BEAM-4789
 URL: https://issues.apache.org/jira/browse/BEAM-4789
 Project: Beam
  Issue Type: Improvement
  Components: io-java-gcp, sdk-py-core
Reporter: Ahmet Altay
Assignee: Chamikara Jayalath


Apache Beam 2.5 documentation 
([https://beam.apache.org/documentation/sdks/javadoc/2.5.0/]), states that
 
"WRITE_TRUNCATE
public static final BigQueryIO.Write.WriteDisposition WRITE_TRUNCATE
Specifies that write should replace a table.
The replacement *may occur in multiple steps* - for instance by first removing 
the existing table, then creating a replacement, then filling it in. *This is 
not an atomic operation*, and external programs may see the table in any of 
these intermediate steps."
 
However WRITE_TRUNCATE is an atomic BigQuery operation. The documentation needs 
to be updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #101

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] FileSystems should throw exception if filesystem for schema is 
not found

[ankurgoenka] Make schema pattern stricter

[ankurgoenka] Adding tet cases for matchResouces

[github] [BEAM-4752] Add dill compatibility for older versions of dill (#5949)

--
[...truncated 16.83 MB...]
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 as step s23
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
 as step s24
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize 
as step s25
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key as step s26
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 as step s27
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey
 as step s28
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 as step s29
Jul 14, 2018 12:35:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
 as step s30
Jul 14, 2018 12:35:32 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests//testpipeline-jenkins-0714003528-5e4d3ffe/output/results/staging/
Jul 14, 2018 12:35:32 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <81480 bytes, hash njrc1wUmffcaaQDoBn2l4A> to 
gs://temp-storage-for-end-to-end-tests//testpipeline-jenkins-0714003528-5e4d3ffe/output/results/staging/pipeline-njrc1wUmffcaaQDoBn2l4A.pb

org.apache.beam.examples.WindowedWordCountIT > 
testWindowedWordCountInBatchDynamicSharding STANDARD_OUT
Dataflow SDK version: 2.6.0-SNAPSHOT

org.apache.beam.examples.WindowedWordCountIT > 
testWindowedWordCountInBatchDynamicSharding STANDARD_ERROR
Jul 14, 2018 12:35:33 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-07-13_17_35_32-17863579163430142422?project=apache-beam-testing

org.apache.beam.examples.WindowedWordCountIT > 
testWindowedWordCountInBatchDynamicSharding STANDARD_OUT
Submitted job: 2018-07-13_17_35_32-17863579163430142422

org.apache.beam.examples.WindowedWordCountIT > 
testWindowedWordCountInBatchDynamicSharding STANDARD_ERROR
Jul 14, 2018 12:35:33 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-07-13_17_35_32-17863579163430142422
Jul 14, 2018 12:35:33 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-07-13_17_35_32-17863579163430142422 with 0 
expected assertions.
Jul 14, 2018 12:35:45 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:35:32.816Z: Autoscaling is enabled for job 
2018-07-13_17_35_32-17863579163430142422. The number of workers will be between 
1 and 1000.
Jul 14, 2018 12:35:45 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:35:32.858Z: Autoscaling was automatically enabled for 
job 2018-07-13_17_35_32-17863579163430142422.
Jul 14, 2018 12:35:45 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:35:35.308Z: Checking required Cloud APIs are enabled.
Jul 14, 2018 12:35:45 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-14T00:35:35.492Z: Checking 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #5547

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123132
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:17
Start Date: 14/Jul/18 00:17
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5941: [BEAM-4076] Schema 
utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#issuecomment-404983816
 
 
   Thanks! LGTM!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123132)
Time Spent: 11h 10m  (was: 11h)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 11h 10m
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #597

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123131
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:16
Start Date: 14/Jul/18 00:16
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5941: [BEAM-4076] Schema 
utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#issuecomment-404983697
 
 
   @apilloud  new tests have been pushed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123131)
Time Spent: 11h  (was: 10h 50m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123130=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123130
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:15
Start Date: 14/Jul/18 00:15
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #5950: [BEAM-4787] Ignore 
generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.gitignore b/.gitignore
index 204f22fde87..8868b6e9f24 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,7 +10,8 @@
 **/.gogradle/**/*
 **/gogradle.lock
 **/build/**/*
-sdks/go/**/vendor/**/*
+sdks/**/vendor/**/*
+runners/**/vendor/**/*
 **/.gradletasknamecache
 
 # Ignore files generated by the Maven build process.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123130)
Time Spent: 1h 20m  (was: 1h 10m)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #841

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] FileSystems should throw exception if filesystem for schema is 
not found

[ankurgoenka] Make schema pattern stricter

[ankurgoenka] Adding tet cases for matchResouces

[github] [BEAM-4752] Add dill compatibility for older versions of dill (#5949)

--
[...truncated 3.37 KB...]
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6234836463457250743.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4470217277029429197.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1450130023742789318.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/0d/12/13a3117bbd2230043aa32dcfa2198c33269665eaa1a8fa26174ce49b338f/pywinrm-0.3.0-py2.py3-none-any.whl
Collecting timeout-decorator (from -r PerfKitBenchmarker/requirements.txt (line 
26))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0)
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15)) (1.0)
Collecting colorama; extra == "windows" (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
  Using cached 
https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4)
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached 
https://files.pythonhosted.org/packages/42/a9/7e99652c6bc619d19d58cdd8c47560730eb5825d43a7e25db2e1d776ceb7/xmltodict-0.11.0-py2.py3-none-any.whl
Requirement already satisfied: cryptography>=1.3 in 
/usr/local/lib/python2.7/dist-packages (from requests-ntlm>=0.3.0->pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25)) 

[jira] [Resolved] (BEAM-4787) Ignore genrated vendored files for python container

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved BEAM-4787.

   Resolution: Fixed
Fix Version/s: 2.6.0

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4784) Python SDK harness container build fails

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543890#comment-16543890
 ] 

Ahmet Altay commented on BEAM-4784:
---

Thank you for confirming. I will update the release thread to avoid confusion.

I also agree with your open question.

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (2ca320c -> 6230523)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2ca320c  [BEAM-4752] Add dill compatibility for older versions of dill 
(#5949)
 add d52d995  Ignore generated vendor files in python container build
 new 6230523  Merge pull request #5950: [BEAM-4787] Ignore generated vendor 
files in python container build

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .gitignore | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5950: [BEAM-4787] Ignore generated vendor files in python container build

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 623052353a55a7ff7488498673aaa0d934740a60
Merge: 2ca320c d52d995
Author: Thomas Weise 
AuthorDate: Fri Jul 13 17:15:00 2018 -0700

Merge pull request #5950: [BEAM-4787] Ignore generated vendor files in 
python container build

 .gitignore | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Jenkins build is back to normal : beam_PostCommit_Py_ValCont #229

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-4784) Python SDK harness container build fails

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543888#comment-16543888
 ] 

Thomas Weise commented on BEAM-4784:


Nope, but still curious to know how this stale dependency cache issue could 
occur?

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4784) Python SDK harness container build fails

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-4784:
---
Fix Version/s: (was: 2.6.0)

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=123113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123113
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:08
Start Date: 14/Jul/18 00:08
Worklog Time Spent: 10m 
  Work Description: cclauss removed a comment on issue #5916: [BEAM-4751] 
fix missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#issuecomment-404744858
 
 
   Please resolve conflict.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123113)
Time Spent: 1h 20m  (was: 1h 10m)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=123111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123111
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 14/Jul/18 00:05
Start Date: 14/Jul/18 00:05
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5948: [BEAM-4417] Fix the 
expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404982620
 
 
   The change LGTM, however it is breaking tests. 
`:beam-sdks-java-extensions-sql:integrationTest` is the failing test. 
   
   cc: @akedin will take a look it error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123111)
Time Spent: 5.5h  (was: 5h 20m)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4784) Python SDK harness container build fails

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543876#comment-16543876
 ] 

Ahmet Altay commented on BEAM-4784:
---

[~thw] is this still a blocker?

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.6.0
>
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4432) Performance tests need a way to generate Synthetic data

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4432?focusedWorklogId=123100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123100
 ]

ASF GitHub Bot logged work on BEAM-4432:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:24
Start Date: 13/Jul/18 23:24
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5519: [BEAM-4432] Adding 
Sources to produce Synthetic output for Batch pipelines
URL: https://github.com/apache/beam/pull/5519#issuecomment-404978005
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123100)
Time Spent: 8.5h  (was: 8h 20m)

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123098
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:23
Start Date: 13/Jul/18 23:23
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5941: [BEAM-4076] Schema 
utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#issuecomment-404977688
 
 
   @apilloud new tests are coming, I just replied first to the comments I could 
fix quickly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123098)
Time Spent: 10h 50m  (was: 10h 40m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4751) Finish io futurize stage 2: fix the missing pylint3 check in tox.ini

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4751?focusedWorklogId=123090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123090
 ]

ASF GitHub Bot logged work on BEAM-4751:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:18
Start Date: 13/Jul/18 23:18
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5916: [BEAM-4751] fix 
missing pylint3 check for io subpackage
URL: https://github.com/apache/beam/pull/5916#issuecomment-404977122
 
 
   R: @tvalentyn @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123090)
Time Spent: 1h 10m  (was: 1h)

> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini
> 
>
> Key: BEAM-4751
> URL: https://issues.apache.org/jira/browse/BEAM-4751
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Matthias Feys
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Finish io futurize stage 2: fix the missing pylint3 check in tox.ini



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4788) Test Auto Jira Subtask

[yifan zou (JIRA)]

yifan zou created BEAM-4788:
---

 Summary: Test Auto Jira Subtask
 Key: BEAM-4788
 URL: https://issues.apache.org/jira/browse/BEAM-4788
 Project: Beam
  Issue Type: Sub-task
  Components: dependencies
Reporter: yifan zou
Assignee: yifan zou


No actions required



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123084
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:13
Start Date: 13/Jul/18 23:13
Worklog Time Spent: 10m 
  Work Description: angoenka edited a comment on issue #5950: [BEAM-4787] 
Ignore generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404975918
 
 
   Sure, I will add the following to the list. I hope that if we accidentally 
ignore valid file then we will catch it soon in the build.
   
   ```
   sdks/**/vendor/**/*
   runners/**/vendor/**/*
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123084)
Time Spent: 1h 10m  (was: 1h)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123083
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:12
Start Date: 13/Jul/18 23:12
Worklog Time Spent: 10m 
  Work Description: angoenka edited a comment on issue #5950: [BEAM-4787] 
Ignore generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404975918
 
 
   Sure, I will add the following to the list. I hope that if we accidentally 
ignore valid file then the we will catch it soon in the build.
   
   ```
   sdks/**/vendor/**/*
   runners/**/vendor/**/*
   ```
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123083)
Time Spent: 1h  (was: 50m)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123081
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:11
Start Date: 13/Jul/18 23:11
Worklog Time Spent: 10m 
  Work Description: angoenka edited a comment on issue #5950: [BEAM-4787] 
Ignore generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404975918
 
 
   Sure, I will add the following to the list. I hope that if we accidentally 
ignore valid file then the we will catch it soon in the build.
   
   `sdks/**/vendor/**/*
   
   runners/**/vendor/**/*`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123081)
Time Spent: 50m  (was: 40m)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-4694) Test Auto JIRA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reopened BEAM-4694:
-

reopen for test purpose

> Test Auto JIRA
> --
>
> Key: BEAM-4694
> URL: https://issues.apache.org/jira/browse/BEAM-4694
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> Creating a JIRA by using python API. No actions needed. 
>  cc: [~chamikara]/n Test appending description/n Test appending description



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123080
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 23:11
Start Date: 13/Jul/18 23:11
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5950: [BEAM-4787] Ignore 
generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404975918
 
 
   Sure, I will add the following to the list. I hope that if we accidentally 
ignore valid file then the we will catch it soon in the build.
   
   `sdks/**/vendor/**/*
   runners/**/vendor/**/*`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123080)
Time Spent: 40m  (was: 0.5h)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4694) Test Auto JIRA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4694:

Description: 
Creating a JIRA by using python API. No actions needed. 
 cc: [~chamikara]/n Test appending description/n Test appending description

  was:
Creating a JIRA by using python API. No actions needed. 
 cc: [~chamikara]/n Test appending description


> Test Auto JIRA
> --
>
> Key: BEAM-4694
> URL: https://issues.apache.org/jira/browse/BEAM-4694
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> Creating a JIRA by using python API. No actions needed. 
>  cc: [~chamikara]/n Test appending description/n Test appending description



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4694) Test Auto JIRA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4694:

Description: 
Creating a JIRA by using python API. No actions needed. 
 cc: [~chamikara]/n Test appending description

  was:
Creating a JIRA by using python API. No actions needed. 
 cc: [~chamikara]


> Test Auto JIRA
> --
>
> Key: BEAM-4694
> URL: https://issues.apache.org/jira/browse/BEAM-4694
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> Creating a JIRA by using python API. No actions needed. 
>  cc: [~chamikara]/n Test appending description



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4694) Test Auto JIRA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4694:

Component/s: (was: testing)

> Test Auto JIRA
> --
>
> Key: BEAM-4694
> URL: https://issues.apache.org/jira/browse/BEAM-4694
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> Creating a JIRA by using python API. No actions needed. 
>  cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4694) Test Auto JIRA

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4694:

Component/s: dependencies

> Test Auto JIRA
> --
>
> Key: BEAM-4694
> URL: https://issues.apache.org/jira/browse/BEAM-4694
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> Creating a JIRA by using python API. No actions needed. 
>  cc: [~chamikara]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-4752] Add dill compatibility for older versions of dill (#5949)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2ca320c  [BEAM-4752] Add dill compatibility for older versions of dill 
(#5949)
2ca320c is described below

commit 2ca320cc4265d2bdc8b678579b01597cea3cd837
Author: Ahmet Altay 
AuthorDate: Fri Jul 13 15:40:39 2018 -0700

[BEAM-4752] Add dill compatibility for older versions of dill (#5949)

* Add dill compatibility for older versions of dill
---
 sdks/python/apache_beam/internal/pickler.py | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/sdks/python/apache_beam/internal/pickler.py 
b/sdks/python/apache_beam/internal/pickler.py
index a4d20b9..211430b 100644
--- a/sdks/python/apache_beam/internal/pickler.py
+++ b/sdks/python/apache_beam/internal/pickler.py
@@ -44,6 +44,12 @@ import dill
 # TODO: Remove this once Beam depends on dill >= 0.2.8
 if not getattr(dill, 'dill', None):
   dill.dill = dill._dill
+  sys.modules['dill.dill'] = dill._dill
+
+# TODO: Remove once Dataflow has containers with a preinstalled dill >= 0.2.8
+if not getattr(dill, '_dill', None):
+  dill._dill = dill.dill
+  sys.modules['dill._dill'] = dill.dill
 
 
 def _is_nested_class(cls):

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123077
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:40
Start Date: 13/Jul/18 22:40
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5949: [BEAM-4752] Add dill 
compatibility for older versions of dill
URL: https://github.com/apache/beam/pull/5949
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/internal/pickler.py 
b/sdks/python/apache_beam/internal/pickler.py
index a4d20b9d043..211430bd60f 100644
--- a/sdks/python/apache_beam/internal/pickler.py
+++ b/sdks/python/apache_beam/internal/pickler.py
@@ -44,6 +44,12 @@
 # TODO: Remove this once Beam depends on dill >= 0.2.8
 if not getattr(dill, 'dill', None):
   dill.dill = dill._dill
+  sys.modules['dill.dill'] = dill._dill
+
+# TODO: Remove once Dataflow has containers with a preinstalled dill >= 0.2.8
+if not getattr(dill, '_dill', None):
+  dill._dill = dill.dill
+  sys.modules['dill._dill'] = dill.dill
 
 
 def _is_nested_class(cls):


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123077)
Time Spent: 3h 20m  (was: 3h 10m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123075
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:34
Start Date: 13/Jul/18 22:34
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5950: [BEAM-4787] Ignore 
generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404970402
 
 
   There are a few more vendor files that should me moved out of sight:
   ```
runners/gcp/gcemd/vendor/
runners/gcp/gcsproxy/vendor/
sdks/java/container/vendor/
sdks/python/container/vendor/
   ```
   I think the existing entry `sdks/go/**/vendor/**/*` can be augmented to 
capture the other sdk container builds also? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123075)
Time Spent: 0.5h  (was: 20m)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4784) Python SDK harness container build fails

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543807#comment-16543807
 ] 

Thomas Weise commented on BEAM-4784:


Interesting. I had run clean before, but only for the container and not taking 
into account the sdist dependencies.

Now the sdist build succeeds and shows 
{code:java}
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5){code}
instead of 
{code:java}
Collecting protobuf>=3.5.0.post1 (from grpcio-tools<2,>=1.3.5){code}
Does this mean the grpcio-tools dependency changed in a way that does not cause 
an update?

The good news is that my example pipeline again works with the latest master. 
Thanks!

 

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.6.0
>
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123071
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:20
Start Date: 13/Jul/18 22:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #5941: 
[BEAM-4076] Schema utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#discussion_r202486289
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
 ##
 @@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.transforms;
+
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/** A set of utilities for converting between different objects supporting 
schemas. */
+@Experimental(Kind.SCHEMAS)
+public class Convert {
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The input {@link PCollection} must have a schema attached. The output 
collection will have
+   * the same schema as the iput.
+   */
+  public static  PTransform, PCollection> 
toRows() {
+return to(Row.class);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  Class clazz) {
+return to(clazz);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  TypeDescriptor typeDescriptor) {
+return to(typeDescriptor);
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  Class clazz) {
+return to(TypeDescriptor.of(clazz));
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  TypeDescriptor typeDescriptor) {
+return new ConvertTransform<>(typeDescriptor);
+  }
+
+  private static class ConvertTransform
+  extends PTransform, PCollection> {
+@Nullable TypeDescriptor outputTypeDescriptor = null;
+SchemaCoder outputSchemaCoder;
+
+ConvertTransform(Class outputClass) {
+  this(TypeDescriptor.of(outputClass));
+}
+
+ConvertTransform(TypeDescriptor outputTypeDescriptor) {
+  this.outputTypeDescriptor = outputTypeDescriptor;
+}
+
+@Override
+@SuppressWarnings("unchecked")
 
 Review comment:
   no, it's due to the cast in the toRow branch


This is an automated message from the Apache Git Service.
To respond to 

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123072
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:20
Start Date: 13/Jul/18 22:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #5941: 
[BEAM-4076] Schema utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#discussion_r202486290
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
 ##
 @@ -301,6 +301,18 @@ public String getName() {
 return setCoder(SchemaCoder.of(schema, toRowFunction, fromRowFunction));
   }
 
+  /** Returns whether this {@link PCollection} has an attached schema. */
+  @Experimental(Kind.SCHEMAS)
+  public boolean hasSchema() {
+return getCoder() instanceof SchemaCoder;
+  }
+
+  /** Returns the attached schema, or null if there is none. */
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123072)
Time Spent: 10h 40m  (was: 10.5h)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123068
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:20
Start Date: 13/Jul/18 22:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #5941: 
[BEAM-4076] Schema utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#discussion_r202486266
 
 

 ##
 File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
 ##
 @@ -182,6 +184,33 @@ public boolean equals(Object o) {
 && Objects.equals(getFields(), other.getFields());
   }
 
+  /** Returns true if two Schemas have the same fields, but possibly in 
different orders. */
+  public boolean equivalent(Schema other) {
+List otherFields =
+other
+.getFields()
+.stream()
+.sorted(Comparator.comparing(Field::getName))
+.collect(Collectors.toList());
+List actualFields =
+getFields()
+.stream()
+.sorted(Comparator.comparing(Field::getName))
+.collect(Collectors.toList());
+if (otherFields.size() != actualFields.size()) {
 
 Review comment:
   Good call, done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123068)
Time Spent: 10h  (was: 9h 50m)

> Schema followups
> 
>
> Key: BEAM-4076
> URL: https://issues.apache.org/jira/browse/BEAM-4076
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, dsl-sql, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> This umbrella bug contains subtasks with followups for Beam schemas, which 
> were moved from SQL to the core Java SDK and made to be type-name-based 
> rather than coder based.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123069
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:20
Start Date: 13/Jul/18 22:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #5941: 
[BEAM-4076] Schema utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#discussion_r202486278
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
 ##
 @@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.transforms;
+
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/** A set of utilities for converting between different objects supporting 
schemas. */
+@Experimental(Kind.SCHEMAS)
+public class Convert {
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The input {@link PCollection} must have a schema attached. The output 
collection will have
+   * the same schema as the iput.
+   */
+  public static  PTransform, PCollection> 
toRows() {
+return to(Row.class);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  Class clazz) {
+return to(clazz);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  TypeDescriptor typeDescriptor) {
+return to(typeDescriptor);
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  Class clazz) {
+return to(TypeDescriptor.of(clazz));
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  TypeDescriptor typeDescriptor) {
+return new ConvertTransform<>(typeDescriptor);
+  }
+
+  private static class ConvertTransform
+  extends PTransform, PCollection> {
+@Nullable TypeDescriptor outputTypeDescriptor = null;
+SchemaCoder outputSchemaCoder;
 
 Review comment:
   moved


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123069)
Time Spent: 10h 10m  (was: 10h)

> Schema followups
> 

[jira] [Work logged] (BEAM-4076) Schema followups

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4076?focusedWorklogId=123070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123070
 ]

ASF GitHub Bot logged work on BEAM-4076:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:20
Start Date: 13/Jul/18 22:20
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #5941: 
[BEAM-4076] Schema utilities for converting between types
URL: https://github.com/apache/beam/pull/5941#discussion_r202486286
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
 ##
 @@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.schemas.transforms;
+
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/** A set of utilities for converting between different objects supporting 
schemas. */
+@Experimental(Kind.SCHEMAS)
+public class Convert {
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The input {@link PCollection} must have a schema attached. The output 
collection will have
+   * the same schema as the iput.
+   */
+  public static  PTransform, PCollection> 
toRows() {
+return to(Row.class);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  Class clazz) {
+return to(clazz);
+  }
+
+  /**
+   * Convert a {@link PCollection} into a {@link PCollection}.
+   *
+   * The output schema will be inferred using the schema registry. A schema 
must be registered
+   * for this type, or the conversion will fail.
+   */
+  public static  PTransform, PCollection> 
fromRows(
+  TypeDescriptor typeDescriptor) {
+return to(typeDescriptor);
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  Class clazz) {
+return to(TypeDescriptor.of(clazz));
+  }
+
+  /**
+   * Convert a {@link PCollection} to a {@link PCollection}.
+   *
+   * This function allows converting between two types as long as the two 
types have
+   * compatible schemas. Two schemas are said to be compatible 
if they recursively
+   * have fields with the same names, but possibly different orders.
+   */
+  public static  PTransform, 
PCollection> to(
+  TypeDescriptor typeDescriptor) {
+return new ConvertTransform<>(typeDescriptor);
+  }
+
+  private static class ConvertTransform
+  extends PTransform, PCollection> {
+@Nullable TypeDescriptor outputTypeDescriptor = null;
 
 Review comment:
   Fine to change for now. In the future, might have to make this nullable 
though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123070)
Time Spent: 10h 20m 

[beam] branch master updated (c14c975 -> 21979d7)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c14c975  [BEAM-4752] Add support for newer dill dependency (#5931)
 add 728591b  FileSystems should throw exception if filesystem for schema 
is not found
 add 3fff58c  Make schema pattern stricter
 add ad4b057  Adding tet cases for matchResouces
 new 21979d7  Merge pull request #5808: FileSystems should throw exception 
if filesystem for schema is not found

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/io/FileSystems.java  |  8 
 .../java/org/apache/beam/sdk/io/FileSystemsTest.java   | 18 ++
 2 files changed, 22 insertions(+), 4 deletions(-)

[beam] 01/01: Merge pull request #5808: FileSystems should throw exception if filesystem for schema is not found

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 21979d7dd21f2840145a751af34f99ee06386c25
Merge: c14c975 ad4b057
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 13 15:20:35 2018 -0700

Merge pull request #5808: FileSystems should throw exception if filesystem 
for schema is not found

FileSystems should throw exception if filesystem for schema is not found

 .../main/java/org/apache/beam/sdk/io/FileSystems.java  |  8 
 .../java/org/apache/beam/sdk/io/FileSystemsTest.java   | 18 ++
 2 files changed, 22 insertions(+), 4 deletions(-)

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123066
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:18
Start Date: 13/Jul/18 22:18
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404967884
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123066)
Time Spent: 1h 10m  (was: 1h)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123065
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 22:18
Start Date: 13/Jul/18 22:18
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404967862
 
 
   `ElasticsearchIOTest` again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123065)
Time Spent: 1h  (was: 50m)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123060
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:58
Start Date: 13/Jul/18 21:58
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5949: [BEAM-4752] Add dill 
compatibility for older versions of dill
URL: https://github.com/apache/beam/pull/5949#issuecomment-404964520
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123060)
Time Spent: 3h 10m  (was: 3h)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123059
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:57
Start Date: 13/Jul/18 21:57
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5950: [BEAM-4787] Ignore 
generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950#issuecomment-404964205
 
 
   R: @aaltay @tweise 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123059)
Time Spent: 20m  (was: 10m)

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor files which should be 
> ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4787) Ignore genrated vendored files for python container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4787?focusedWorklogId=123058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123058
 ]

ASF GitHub Bot logged work on BEAM-4787:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:56
Start Date: 13/Jul/18 21:56
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #5950: [BEAM-4787] 
Ignore generated vendor files in python container build
URL: https://github.com/apache/beam/pull/5950
 
 
   Ignore generated files for python container build
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123058)
Time Spent: 10m
Remaining Estimate: 0h

> Ignore genrated vendored files for python container
> ---
>
> Key: BEAM-4787
> URL: https://issues.apache.org/jira/browse/BEAM-4787
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Python container build generates a bunch of vendor 

[jira] [Created] (BEAM-4787) Ignore genrated vendored files for python container

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-4787:
--

 Summary: Ignore genrated vendored files for python container
 Key: BEAM-4787
 URL: https://issues.apache.org/jira/browse/BEAM-4787
 Project: Beam
  Issue Type: Bug
  Components: dependencies
Reporter: Ankur Goenka
Assignee: Ankur Goenka


Python container build generates a bunch of vendor files which should be 
ignored in git



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4784) Python SDK harness container build fails

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543769#comment-16543769
 ] 

Ankur Goenka commented on BEAM-4784:


I am not able to reproduce it here is a working build on master 
[https://scans.gradle.com/s/2cq26ubquy7ly/timeline?task=2a5wa25ihupj4] 

 

Suggestion: Try ./gradlew clean

and then use 
./gradlew -p sdks/python/container docker --scan --no-daemon 

to build

> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.6.0
>
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=123056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123056
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:40
Start Date: 13/Jul/18 21:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-404961158
 
 
   @aaltay PTAL. Thanks~


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123056)
Time Spent: 26h 20m  (was: 26h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 26h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3446) RedisIO non-prefix read operations

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543736#comment-16543736
 ] 

Ismaël Mejía commented on BEAM-3446:


[~jbonofre] I reassigned this to Vinay since he is working in the PR from some 
days on and I am doing the review at this moment.

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-redis
>Reporter: Vinay varma
>Assignee: Vinay varma
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4786) Distinct has bad parallelism characteristics

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-4786:
---

 Summary: Distinct has bad parallelism characteristics
 Key: BEAM-4786
 URL: https://issues.apache.org/jira/browse/BEAM-4786
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Pablo Estrada
Assignee: Pablo Estrada


Distinct groups first and then drops extra elements. Should drop elements in 
the mappers, and later in reducers.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java#L100



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3446) RedisIO non-prefix read operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3446?focusedWorklogId=123051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123051
 ]

ASF GitHub Bot logged work on BEAM-3446:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:21
Start Date: 13/Jul/18 21:21
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5841: 
Fixes https://issues.apache.org/jira/browse/BEAM-3446.
URL: https://github.com/apache/beam/pull/5841#discussion_r202475941
 
 

 ##
 File path: 
sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java
 ##
 @@ -279,28 +290,31 @@ public void processElement(ProcessContext 
processContext) throws Exception {
   while (!finished) {
 ScanResult scanResult = jedis.scan(cursor, scanParams);
 List keys = scanResult.getResult();
-
-Pipeline pipeline = jedis.pipelined();
-if (keys != null) {
-  for (String key : keys) {
-pipeline.get(key);
-  }
-  List values = pipeline.syncAndReturnAll();
-  for (int i = 0; i < values.size(); i++) {
-processContext.output(KV.of(keys.get(i), (String) values.get(i)));
-  }
+for (String k : keys) {
+  processContext.output(k);
 }
-
 cursor = scanResult.getStringCursor();
 if ("0".equals(cursor)) {
   finished = true;
 }
   }
 }
+  }
+  /** A {@link DoFn} requesting Redis server to get key/value pairs. */
+  private static class ReadFn extends BaseReadFn> {
 
-@Teardown
-public void teardown() {
-  jedis.close();
+ReadFn(RedisConnectionConfiguration connectionConfiguration) {
+  super(connectionConfiguration);
+}
+
+@ProcessElement
+public void processElement(ProcessContext processContext) throws Exception 
{
+  String key = processContext.element();
+
+  String value = jedis.get(key);
 
 Review comment:
   Hi, sorry I have missed your message. The idea is that we should add the 
DoFn startBundle and finishBundle methods and create a method in the Read to 
define the size of the maximum amount of elements that we will request, then 
you will build the collection of the keys that are going to be requested in the 
processElement, but you  won't do the request in the processElement but in the 
finishBundle method by doing a MGET request with the defined number of elements 
of the batch, we should choose a default min size e.g. 1000. It is similar to 
what other IOs do in the Write (see withBatchSize in ElasticsearchIO or SolrIO, 
for ref.
   
https://github.com/apache/beam/blob/c14c975224af417dcdc74fed8b0d893be742e9d7/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/SolrIO.java#L805-L829


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123051)
Time Spent: 3h 10m  (was: 3h)

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-redis
>Reporter: Vinay varma
>Assignee: Vinay varma
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

svn commit: r28108 - in /dev/beam/2.4.0/test_wheels: ./ apache_beam-2.5.0.rc2-cp27-cp27mu-manylinux1_i686.whl

[pabloem]

Author: pabloem
Date: Fri Jul 13 21:12:55 2018
New Revision: 28108

Log:
Upload python wheels

Added:
dev/beam/2.4.0/test_wheels/

dev/beam/2.4.0/test_wheels/apache_beam-2.5.0.rc2-cp27-cp27mu-manylinux1_i686.whl
   (with props)

Added: 
dev/beam/2.4.0/test_wheels/apache_beam-2.5.0.rc2-cp27-cp27mu-manylinux1_i686.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.4.0/test_wheels/apache_beam-2.5.0.rc2-cp27-cp27mu-manylinux1_i686.whl
--
svn:mime-type = application/octet-stream

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123037
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:04
Start Date: 13/Jul/18 21:04
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404953812
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123037)
Time Spent: 50m  (was: 40m)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123036
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 21:04
Start Date: 13/Jul/18 21:04
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404953788
 
 
   Hit flaps in `ElasticsearchIOTest`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123036)
Time Spent: 40m  (was: 0.5h)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4784) Python SDK harness container build fails

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-4784:
---
Description: 
A new build failure has surfaced for ./gradlew -p sdks/python/container docker  
(fails in :beam-sdks-python:sdist)

It can be reproduced even on a commit that succeeded before. Basically it 
attempts to retrieve protobuf for several minutes before it finally fails with 
"RuntimeError: maximum recursion depth exceeded"

 

  was:
A new build failure has surfaced for ./gradlew -p sdks/python/container docker

It can be reproduced even on a commit that succeeded before. Basically it 
attempts to retrieve protobuf for several minutes before it finally fails with 
"RuntimeError: maximum recursion depth exceeded"

 


> Python SDK harness container build fails
> 
>
> Key: BEAM-4784
> URL: https://issues.apache.org/jira/browse/BEAM-4784
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
> Fix For: 2.6.0
>
>
> A new build failure has surfaced for ./gradlew -p sdks/python/container 
> docker  (fails in :beam-sdks-python:sdist)
> It can be reproduced even on a commit that succeeded before. Basically it 
> attempts to retrieve protobuf for several minutes before it finally fails 
> with "RuntimeError: maximum recursion depth exceeded"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3919) checkpoint can not work with flink 1.4.1,1.4.2

[Joe Malt (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543690#comment-16543690
 ] 

Joe Malt commented on BEAM-3919:


Do you know whether this issue still affects Flink 1.5.0 and Beam 2.5.0? If so, 
do you have an ETA on fixing it?

> checkpoint can not work with flink 1.4.1,1.4.2
> --
>
> Key: BEAM-3919
> URL: https://issues.apache.org/jira/browse/BEAM-3919
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.3.0, 2.4.0
>Reporter: eisig
>Assignee: Harshal Tripathi
>Priority: Critical
>
> When submmit application to flink cluster(1.4.1,1.4.2) with checkpoint 
> enabled. 
> Job fail whith exception:
> java.lang.NoSuchMethodError: 
> org.apache.flink.streaming.api.operators.HeapInternalTimerService.snapshotTimersForKeyGroup(Lorg/apache/flink/core/memory/DataOutputViewStreamWrapper;I)V
>  
> It seems that 
> `org.apache.flink.streaming.api.operators.HeapInternalTimerService.snapshotTimersForKeyGroup`.
>   was changed in flink1.4.1.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4432) Performance tests need a way to generate Synthetic data

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4432?focusedWorklogId=123034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123034
 ]

ASF GitHub Bot logged work on BEAM-4432:


Author: ASF GitHub Bot
Created on: 13/Jul/18 20:46
Start Date: 13/Jul/18 20:46
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5519: [BEAM-4432] Adding 
Sources to produce Synthetic output for Batch pipelines
URL: https://github.com/apache/beam/pull/5519#issuecomment-404949448
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123034)
Time Spent: 8h 20m  (was: 8h 10m)

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3446) RedisIO non-prefix read operations

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3446:
--

Assignee: Vinay varma  (was: Jean-Baptiste Onofré)

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-redis
>Reporter: Vinay varma
>Assignee: Vinay varma
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=123026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123026
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 13/Jul/18 20:24
Start Date: 13/Jul/18 20:24
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5948: [BEAM-4417] Fix the 
expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404943901
 
 
   Run Java PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123026)
Time Spent: 5h 20m  (was: 5h 10m)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=123024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123024
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 13/Jul/18 20:15
Start Date: 13/Jul/18 20:15
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5948: [BEAM-4417] Fix the 
expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404941933
 
 
   I will review this today. Thank you @ElliottBrossard 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123024)
Time Spent: 5h 10m  (was: 5h)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4706) BigQueryTornadoesIT cannot be run using integrationTest and performanceTest tasks

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4706?focusedWorklogId=123023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123023
 ]

ASF GitHub Bot logged work on BEAM-4706:


Author: ASF GitHub Bot
Created on: 13/Jul/18 20:02
Start Date: 13/Jul/18 20:02
Worklog Time Spent: 10m 
  Work Description: ElliottBrossard commented on issue #5948: [BEAM-4706] 
Fix the expected encoding of BigQuery's NUMERIC type when reading from Avro
URL: https://github.com/apache/beam/pull/5948#issuecomment-404938916
 
 
   @kennknowles I'd appreciate it if you could take a look at this PR as well, 
thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123023)
Time Spent: 2h 10m  (was: 2h)

> BigQueryTornadoesIT cannot be run using integrationTest and performanceTest 
> tasks
> -
>
> Key: BEAM-4706
> URL: https://issues.apache.org/jira/browse/BEAM-4706
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> It seems that we cannot run this test using tasks from BuildModulePlugin 
> designed to run such tests. Those are not included in build.gradle of 
> examples module. There's a possibility that some other tests are not able to 
> run due to this reason (this also has to be checked).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123022
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 20:00
Start Date: 13/Jul/18 20:00
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5946: Revert "[BEAM-4752] 
Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404938388
 
 
   @alanmyrvold 
   
   It seems like 0.2.8 produces serialized data that is not compatible with 
0.2.6 (https://issues.apache.org/jira/browse/BEAM-4785 to fix it. There are 
some blockers though.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123022)
Time Spent: 3h  (was: 2h 50m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4785) Upgrade dill dependency to a new version and fix it

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-4785:
-

 Summary: Upgrade dill dependency to a new version and fix it
 Key: BEAM-4785
 URL: https://issues.apache.org/jira/browse/BEAM-4785
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Ahmet Altay


We currently depend on 'dill>=0.2.6,<=0.2.8.2',

Dill produces incompatible data across versions, we need to pin it again and 
use a newer version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123019
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:57
Start Date: 13/Jul/18 19:57
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5946: Revert "[BEAM-4752] 
Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404937789
 
 
   Closing this, because https://github.com/apache/beam/pull/5949 has a fix for 
the issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123019)
Time Spent: 2h 40m  (was: 2.5h)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123018
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:57
Start Date: 13/Jul/18 19:57
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5949: [BEAM-4752] Add dill 
compatibility for older versions of dill
URL: https://github.com/apache/beam/pull/5949#issuecomment-404937677
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123018)
Time Spent: 2.5h  (was: 2h 20m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123020
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:57
Start Date: 13/Jul/18 19:57
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5946: Revert "[BEAM-4752] 
Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123020)
Time Spent: 2h 50m  (was: 2h 40m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123017
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:56
Start Date: 13/Jul/18 19:56
Worklog Time Spent: 10m 
  Work Description: aaltay opened a new pull request #5949: [BEAM-4752] Add 
dill compatibility for older versions of dill
URL: https://github.com/apache/beam/pull/5949
 
 
   Fixing the post commit issue related to older dill version.
   
   R: @charlesccychen 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123017)
Time Spent: 2h 20m  (was: 2h 10m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123015
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:52
Start Date: 13/Jul/18 19:52
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404936494
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123015)
Time Spent: 0.5h  (was: 20m)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4706) BigQueryTornadoesIT cannot be run using integrationTest and performanceTest tasks

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4706?focusedWorklogId=123011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123011
 ]

ASF GitHub Bot logged work on BEAM-4706:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:35
Start Date: 13/Jul/18 19:35
Worklog Time Spent: 10m 
  Work Description: ElliottBrossard opened a new pull request #5948: 
[BEAM-4706] Fix the expected encoding of BigQuery's NUMERIC type when reading 
from Avro
URL: https://github.com/apache/beam/pull/5948
 
 
   The NUMERIC data type is represented in Avro using the BYTES type with the 
DECIMAL logical type.
   
   I verified this change using BigQueryTornadoes.java, which I modified 
(locally) to write NUMERIC values to BigQuery, then read them back using 
Dataflow and write them to a new BigQuery table. The test failed as expected 
without this change, but now passes, and I can see the correct content in the 
table.
   
   Context: https://github.com/apache/beam/pull/5755#issuecomment-402241675


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123011)
Time Spent: 2h  (was: 1h 50m)

> BigQueryTornadoesIT cannot be run using integrationTest and performanceTest 
> tasks
> -
>
> Key: BEAM-4706
> URL: https://issues.apache.org/jira/browse/BEAM-4706
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> It seems that we cannot run this test using tasks from BuildModulePlugin 
> designed to run such tests. Those are not included in build.gradle of 
> examples module. There's a possibility that some other tests are not able to 
> run due to this reason (this also has to be checked).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4771) Update BeamSQL Walkthrough documentation to align with latest version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4771?focusedWorklogId=123010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123010
 ]

ASF GitHub Bot logged work on BEAM-4771:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:31
Start Date: 13/Jul/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #495: [BEAM-4771]Rename 
RowType to Schema in SQL walkthrough
URL: https://github.com/apache/beam-site/pull/495#issuecomment-404931661
 
 
   R: @apilloud @melap 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123010)
Time Spent: 20m  (was: 10m)

> Update BeamSQL Walkthrough documentation to align with latest version
> -
>
> Key: BEAM-4771
> URL: https://issues.apache.org/jira/browse/BEAM-4771
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, website
>Affects Versions: 2.5.0
>Reporter: Akanksha Sharma
>Assignee: Reuven Lax
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> As I see, in 2.5 BeamSQL had been changed to work with Schema.
> The sample code provided in 
> [https://beam.apache.org/documentation/dsls/sql/walkthrough/] does not 
> compile with Beam 2.5, and needs to be updated.
>  
> Row.withRowType(appType)
>  
> The above mentioned line needs to be adapted to use schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4771) Update BeamSQL Walkthrough documentation to align with latest version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4771?focusedWorklogId=123009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123009
 ]

ASF GitHub Bot logged work on BEAM-4771:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:28
Start Date: 13/Jul/18 19:28
Worklog Time Spent: 10m 
  Work Description: akedin opened a new pull request #495: 
[BEAM-4771]Rename RowType to Schema in SQL walkthrough
URL: https://github.com/apache/beam-site/pull/495
 
 
   *Please* add a meaningful description for your change here.
   
   Once your pull request has been opened and assigned a number, please edit the
   URL below, replacing `PULL_REQUEST_NUMBER` with the number of your pull 
request.
   
   
http://apache-beam-website-pull-requests.storage.googleapis.com/PULL_REQUEST_NUMBER/index.html
   
   Finally, it will help us expedite review of your Pull Request if you tag
   someone (e.g. @username) to look at it.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123009)
Time Spent: 10m
Remaining Estimate: 0h

> Update BeamSQL Walkthrough documentation to align with latest version
> -
>
> Key: BEAM-4771
> URL: https://issues.apache.org/jira/browse/BEAM-4771
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, website
>Affects Versions: 2.5.0
>Reporter: Akanksha Sharma
>Assignee: Reuven Lax
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As I see, in 2.5 BeamSQL had been changed to work with Schema.
> The sample code provided in 
> [https://beam.apache.org/documentation/dsls/sql/walkthrough/] does not 
> compile with Beam 2.5, and needs to be updated.
>  
> Row.withRowType(appType)
>  
> The above mentioned line needs to be adapted to use schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123008
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:26
Start Date: 13/Jul/18 19:26
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #5946: Revert 
"[BEAM-4752] Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404930508
 
 
   The dockerfile is currently pinning the version to 0.2.6
   
https://github.com/apache/beam/blob/c14c975224af417dcdc74fed8b0d893be742e9d7/sdks/python/container/Dockerfile#L47


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123008)
Time Spent: 2h 10m  (was: 2h)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123007
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:18
Start Date: 13/Jul/18 19:18
Worklog Time Spent: 10m 
  Work Description: charlesccychen edited a comment on issue #5946: Revert 
"[BEAM-4752] Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404925443
 
 
   Looked at this a bit more and my guess is that the version of dill differs 
from job submission (latest, 0.2.8) and on the SDK harness when deserializing 
(0.2.6).  So at submission time, the pickled output references `dill._dill`, 
but when run on the SDK harness, this isn't defined since it uses the old 
version.  So maybe the safe thing to do is to add a similar branch that also 
populates `dill._dill` if it isn't there.  (Hopefully we can remove that once 
we converge to >=0.2.8)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123007)
Time Spent: 2h  (was: 1h 50m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud reassigned BEAM-4761:


Assignee: Andrew Pilloud  (was: Kenneth Knowles)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123006
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:15
Start Date: 13/Jul/18 19:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5947: [BEAM-4774] Nexmark 
SQL Postcommits
URL: https://github.com/apache/beam/pull/5947#issuecomment-404927609
 
 
   R: @echauchot 
   cc: @akedin @amaliujia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123006)
Time Spent: 20m  (was: 10m)

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4774) Intergrate Nexmark SQL with Perfkit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4774?focusedWorklogId=123005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123005
 ]

ASF GitHub Bot logged work on BEAM-4774:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:14
Start Date: 13/Jul/18 19:14
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #5947: [BEAM-4774] 
Nexmark SQL Postcommits
URL: https://github.com/apache/beam/pull/5947
 
 
   This adds the Nexmark SQL jobs to the post commits.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123005)
Time Spent: 10m
Remaining Estimate: 0h

> Intergrate Nexmark SQL with Perfkit
> ---
>
> Key: BEAM-4774
> URL: https://issues.apache.org/jira/browse/BEAM-4774
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We should publish to a dashboard like pure JAVA does. See BEAM-4283.



--
This message was sent by 

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123002
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 19:06
Start Date: 13/Jul/18 19:06
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5946: Revert 
"[BEAM-4752] Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404925443
 
 
   Looked at this a bit more and my guess is that the version of dill differs 
from job submission (latest, 0.2.8) and on the SDK harness when deserializing 
(0.2.6).  So at submission time, the pickled output references `dill._dill`, 
but when run on the SDK harness, this isn't defined since it uses the old 
version.  So maybe the safe thing to do is to add a similar branch that also 
populates `dill._dill` if it isn't there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123002)
Time Spent: 1h 50m  (was: 1h 40m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=123000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-123000
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 18:59
Start Date: 13/Jul/18 18:59
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5946: Revert "[BEAM-4752] 
Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404923617
 
 
   > Thanks, LGTM. I don't quite understand how the change breaks the SDK 
harness though:
   
   I do not understand yet either. I will investigate this afternoon.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 123000)
Time Spent: 1h 40m  (was: 1.5h)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=122999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122999
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 18:58
Start Date: 13/Jul/18 18:58
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5946: Revert 
"[BEAM-4752] Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946#issuecomment-404923467
 
 
   Thanks, LGTM.  I don't quite understand how the change breaks the SDK 
harness though:
   
   ```
   DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
   java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error 
received from SDK harness for instruction -83: Traceback (most recent call 
last):
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 134, in _execute
   response = task()
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 169, in 
   self._execute(lambda: worker.do_instruction(work), work)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 215, in do_instruction
   request.instruction_id)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 234, in process_bundle
   self.data_channel_factory)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 230, in __init__
   self.ops = self.create_execution_tree(self.process_bundle_descriptor)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 272, in create_execution_tree
   descriptor.transforms, key=topological_height, reverse=True)])
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 207, in wrapper
   result = cache[args] = func(*args)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 255, in get_operation
   in descriptor.transforms[transform_id].outputs.items()
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 254, in 
   for tag, pcoll_id
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 207, in wrapper
   result = cache[args] = func(*args)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 255, in get_operation
   in descriptor.transforms[transform_id].outputs.items()
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 254, in 
   for tag, pcoll_id
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 207, in wrapper
   result = cache[args] = func(*args)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 258, in get_operation
   transform_id, transform_consumers)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 362, in create_operation
   return creator(self, transform_id, transform_proto, payload, consumers)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 499, in create
   serialized_fn, parameter.side_inputs)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 537, in _create_pardo_operation
   dofn_data = pickler.loads(serialized_fn)
 File 
"/usr/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", line 
238, in loads
   return dill.loads(s)
 File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 277, in 
loads
   return load(file)
 File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 266, in 
load
   obj = pik.load()
 File "/usr/local/lib/python2.7/pickle.py", line 864, in load
   dispatch[key](self)
 File "/usr/local/lib/python2.7/pickle.py", line 1096, in load_global
   klass = self.find_class(module, name)
 File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 423, in 
find_class
   return StockUnpickler.find_class(self, module, name)
 File "/usr/local/lib/python2.7/pickle.py", line 1130, in find_class
   __import__(module)
   ImportError: No module named _dill
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure 

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=122997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122997
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 18:54
Start Date: 13/Jul/18 18:54
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5931: [BEAM-4752] Add 
support for newer dill dependency
URL: https://github.com/apache/beam/pull/5931#issuecomment-404922377
 
 
   Thank you @alanmyrvold. Sent https://github.com/apache/beam/pull/5946 to 
rollback.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 122997)
Time Spent: 1h 20m  (was: 1h 10m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=122996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122996
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 18:54
Start Date: 13/Jul/18 18:54
Worklog Time Spent: 10m 
  Work Description: aaltay opened a new pull request #5946: Revert 
"[BEAM-4752] Add support for newer dill dependency"
URL: https://github.com/apache/beam/pull/5946
 
 
   Reverts apache/beam#5931
   
   R: @alanmyrvold 
   
   Breaks post commit tests: 
https://builds.apache.org/job/beam_PostCommit_Py_ValCont/228/console


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 122996)
Time Spent: 1h 10m  (was: 1h)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1055

[Apache Jenkins Server]

See 


Changes:

[github] [BEAM-4752] Add support for newer dill dependency (#5931)

--
[...truncated 19.28 MB...]
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:37.987Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.026Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.055Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForKeys/Read
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.086Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.131Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/ParDo(CollectWindows)
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.170Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.216Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Write mutations to Spanner into 
SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.262Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.301Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.362Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write 
mutations to Cloud Spanner/Partition input
Jul 13, 2018 6:43:43 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-13T18:43:38.406Z: Fusing consumer SpannerIO.Write/To 

[jira] [Work logged] (BEAM-4752) Import error in apache_beam.internal.pickler: "'module' object has no attribute 'dill'"

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4752?focusedWorklogId=122995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122995
 ]

ASF GitHub Bot logged work on BEAM-4752:


Author: ASF GitHub Bot
Created on: 13/Jul/18 18:42
Start Date: 13/Jul/18 18:42
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #5931: [BEAM-4752] Add 
support for newer dill dependency
URL: https://github.com/apache/beam/pull/5931#issuecomment-404919223
 
 
   I see a test error in 
https://builds.apache.org/job/beam_PostCommit_Py_ValCont/228
   
   Dataflow logs contain error "No module named _dill"
   
   Could not load main session: Traceback (most recent call last): File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 125, in main _load_main_session(semi_persistent_directory) File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 190, in _load_main_session pickler.load_session(session_file) File 
"/usr/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", line 
260, in load_session return dill.load_session(file_path) File 
"/usr/local/lib/python2.7/site-packages/dill/dill.py", line 363, in 
load_session module = unpickler.load() File 
"/usr/local/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) 
File "/usr/local/lib/python2.7/pickle.py", line 1096, in load_global klass = 
self.find_class(module, name) File 
"/usr/local/lib/python2.7/site-packages/dill/dill.py", line 423, in find_class 
return StockUnpickler.find_class(self, module, name) File 
"/usr/local/lib/python2.7/pickle.py", line 1130, in find_class 
__import__(module) ImportError: No module named _dill Traceback (most recent 
call last): File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 125, in main _load_main_session(semi_persistent_directory) File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py",
 line 190, in _load_main_session pickler.load_session(session_file) File 
"/usr/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", line 
260, in load_session return dill.load_session(file_path) File 
"/usr/local/lib/python2.7/site-packages/dill/dill.py", line 363, in 
load_session module = unpickler.load() File 
"/usr/local/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) 
File "/usr/local/lib/python2.7/pickle.py", line 1096, in load_global klass = 
self.find_class(module, name) File 
"/usr/local/lib/python2.7/site-packages/dill/dill.py", line 423, in find_class 
return StockUnpickler.find_class(self, module, name) File 
"/usr/local/lib/python2.7/pickle.py", line 1130, in find_class 
__import__(module) ImportError: No module named _dill


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 122995)
Time Spent: 1h  (was: 50m)

> Import error in apache_beam.internal.pickler: "'module' object has no 
> attribute 'dill'"
> ---
>
> Key: BEAM-4752
> URL: https://issues.apache.org/jira/browse/BEAM-4752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
> Environment: CentOS Linux release 7.4.1708
> Python 2.7.13
>Reporter: Barry Hart
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> I'm seeing the following error (stack trace below). I looked at the module 
> structure of the {{dill}} library, and it does not have a {{dill}} submodule 
> (although it *does* have a {{_dill}} submodule). I think the correct way to 
> reference {{Pickler}} is simply {{dill.Pickler.}}
> {noformat}
> Traceback (most recent call last):
>   File "script/beam_run_model.py", line 29, in 
> import apache_beam as beam
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 84, in 
> import apache_beam.internal.pickler
>   File 
> "/usr/local/pyenv/versions/2.7.13/lib/python2.7/site-packages/apache_beam/internal/pickler.py",
>  line 107, in 
> dill.dill.Pickler.dispatch[type])
> AttributeError: 'module' object has no attribute 'dill'{noformat}
> Oddly, I have successfully used Beam 2.4.0 in the past with this version of 
> Dill.  ¯_(ツ)_/¯



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5546

[Apache Jenkins Server]

See 


Changes:

[github] [BEAM-4752] Add support for newer dill dependency (#5931)

--
[...truncated 14.46 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type