[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=340641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340641 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 08/Nov/19 18:04 Start Date: 08/Nov/19 18:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9686: [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 340641) Time Spent: 16.5h (was: 16h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 16.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=340640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-340640 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 08/Nov/19 18:04 Start Date: 08/Nov/19 18:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9686: [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-551930923 Merging since the tests passed on 0834b416ab45cc3afbac35f5b9d3c5d6104528bb, and second commit does not affect any tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 340640) Time Spent: 16h 20m (was: 16h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 16h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=339456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-339456 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/Nov/19 16:21 Start Date: 06/Nov/19 16:21 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9686: [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-550385798 Sorry, I referenced a wrong Dill issue in my last comment, I meant to link https://github.com/uqfoundation/dill/issues/341. I'll add a commit to this branch to fix this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 339456) Time Spent: 16h 10m (was: 16h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 16h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=339125=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-339125 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/Nov/19 02:19 Start Date: 06/Nov/19 02:19 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#discussion_r342890040 ## File path: sdks/python/setup.py ## @@ -106,8 +106,7 @@ def get_version(): 'avro>=1.8.1,<2.0.0; python_version < "3.0"', 'avro-python3>=1.8.1,<2.0.0; python_version >= "3.0"', 'crcmod>=1.7,<2.0', -# Dill doesn't guarantee comatibility between releases within minor version. -'dill>=0.3.0,<0.3.1', +'dill>=0.3.1.1,<0.4.0', Review comment: Updated. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 339125) Time Spent: 16h (was: 15h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 16h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=338280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338280 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 04/Nov/19 18:22 Start Date: 04/Nov/19 18:22 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#discussion_r342198241 ## File path: sdks/python/setup.py ## @@ -106,8 +106,7 @@ def get_version(): 'avro>=1.8.1,<2.0.0; python_version < "3.0"', 'avro-python3>=1.8.1,<2.0.0; python_version >= "3.0"', 'crcmod>=1.7,<2.0', -# Dill doesn't guarantee comatibility between releases within minor version. -'dill>=0.3.0,<0.3.1', +'dill>=0.3.1.1,<0.4.0', Review comment: Also, please keep the comment, and link https://github.com/uqfoundation/dill/issues/347 in the comment. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338280) Time Spent: 15h 50m (was: 15h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 15h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=338269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338269 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 04/Nov/19 18:18 Start Date: 04/Nov/19 18:18 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#discussion_r342196700 ## File path: sdks/python/setup.py ## @@ -106,8 +106,7 @@ def get_version(): 'avro>=1.8.1,<2.0.0; python_version < "3.0"', 'avro-python3>=1.8.1,<2.0.0; python_version >= "3.0"', 'crcmod>=1.7,<2.0', -# Dill doesn't guarantee comatibility between releases within minor version. -'dill>=0.3.0,<0.3.1', +'dill>=0.3.1.1,<0.4.0', Review comment: Let's restrict this to < 0.3.2 due to possible backwards-incompatibility issues. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338269) Time Spent: 15h 40m (was: 15.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 15h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=337139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337139 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 01/Nov/19 02:18 Start Date: 01/Nov/19 02:18 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-548639816 @tvalentyn Sure. Wait for a while. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337139) Time Spent: 15.5h (was: 15h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 15.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=336959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336959 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/19 17:57 Start Date: 31/Oct/19 17:57 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-548496713 Hi @lazylynx , could you please give this PR another try after resolving conflicts? We should have fixed the test failures now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336959) Time Spent: 15h 20m (was: 15h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 15h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=336957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336957 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/19 17:56 Start Date: 31/Oct/19 17:56 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686 update dill minimum version to 0.3.1.1 in setup.py and re-add unittest for keyword-only arguments from #9237 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=321025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321025 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 01/Oct/19 02:04 Start Date: 01/Oct/19 02:04 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321025) Time Spent: 15h (was: 14h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 15h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=321024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321024 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 01/Oct/19 02:04 Start Date: 01/Oct/19 02:04 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9686: [WIP][BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-536826908 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321024) Time Spent: 14h 50m (was: 14h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 14h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=320428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320428 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Sep/19 10:05 Start Date: 30/Sep/19 10:05 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9686: [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686#issuecomment-536494716 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 320428) Time Spent: 14h 40m (was: 14.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 14h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=320225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320225 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 29/Sep/19 14:35 Start Date: 29/Sep/19 14:35 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9686: [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments URL: https://github.com/apache/beam/pull/9686 update dill minimum version to 0.3.1.1 in setup.py and re-add unittest for keyword-only arguments from #9237 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=318473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318473 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Sep/19 17:31 Start Date: 25/Sep/19 17:31 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-535128812 @mmckerns Thanks for the update. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 318473) Time Spent: 14h 20m (was: 14h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 14h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=318447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318447 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Sep/19 16:34 Start Date: 25/Sep/19 16:34 Worklog Time Spent: 10m Work Description: mmckerns commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-535104886 The `dill` release was pushed back a few days in dealing with some 3.8 issues, and now will happen on the 27th. I try not to slide the release dates back, but we didn't allocate time sufficiently for the extent of the changes that came with pickling in 3.8. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 318447) Time Spent: 14h 10m (was: 14h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 14h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=318394=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318394 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Sep/19 15:01 Start Date: 25/Sep/19 15:01 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-535065606 > @tvalentyn: should be before the end of the month. Trying to get in some more 3.8/3.7 compat, and currently it's slated for release on Sept 22. Hi @mmckerns do you have by chance an updated timeline for next release? I don't know how complicated dill releases are, but would be happy to know if you have any suggestions for how other dill users can help with releases. Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 318394) Time Spent: 13h 50m (was: 13h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Time Spent: 13h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=310163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310163 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Sep/19 22:47 Start Date: 10/Sep/19 22:47 Worklog Time Spent: 10m Work Description: mmckerns commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-530149542 I know (https://github.com/uqfoundation/dill/issues/223) I attach a milestone to each issue, but that doesn't give a full summary as some features are not reflected in the issues. It's on the todo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 310163) Time Spent: 13h 40m (was: 13.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 13h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309900 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Sep/19 15:21 Start Date: 10/Sep/19 15:21 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529987284 @mmckerns Thanks for the update. Looking forward to that release. By the way, it would be helpful for Dill users to have a summary of release notes associated with releases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309900) Time Spent: 13.5h (was: 13h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 13.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309426 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Sep/19 01:03 Start Date: 10/Sep/19 01:03 Worklog Time Spent: 10m Work Description: mmckerns commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529724421 @tvalentyn: should be before the end of the month. Trying to get in some more 3.8/3.7 compat, and currently it's slated for release on Sept 22. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309426) Time Spent: 13h 20m (was: 13h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 13h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309264 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 21:29 Start Date: 09/Sep/19 21:29 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529674708 FYI, I have checked that apache_beam.transforms.transforms_keyword_only_args_test_py3 that we added in this PR passes with dill==0.3.1.dev0. @mmckerns you by chance have an idea when dill 0.3.1 may be released? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309264) Time Spent: 13h 10m (was: 13h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 13h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309130=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309130 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 18:00 Start Date: 09/Sep/19 18:00 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517#issuecomment-529596278 Thanks @chamikaramj . Reasons for revert were also discussed on the original PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309130) Time Spent: 13h (was: 12h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 13h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309115 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 17:42 Start Date: 09/Sep/19 17:42 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309115) Time Spent: 12h 50m (was: 12h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309114 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 17:41 Start Date: 09/Sep/19 17:41 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517#issuecomment-529589272 LGTM. Thanks. To clarify, I believe this is being reverted since the monkey patch caused some user pipelines to fail with an infinite recursion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309114) Time Spent: 12h 40m (was: 12.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309026 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 16:21 Start Date: 09/Sep/19 16:21 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517#issuecomment-529556754 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309026) Time Spent: 12h 20m (was: 12h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309027 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 16:21 Start Date: 09/Sep/19 16:21 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529557084 thanks, @lazylynx and @mmckerns . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309027) Time Spent: 12.5h (was: 12h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=309025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309025 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 16:20 Start Date: 09/Sep/19 16:20 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517#issuecomment-529556754 R: chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309025) Time Spent: 12h 10m (was: 12h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308919=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308919 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 14:30 Start Date: 09/Sep/19 14:30 Worklog Time Spent: 10m Work Description: mmckerns commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529502880 the relevant changes are merged into `dill`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308919) Time Spent: 12h (was: 11h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308571 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 09/Sep/19 01:06 Start Date: 09/Sep/19 01:06 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529266407 @tvalentyn Understood. Sorry for bothering. @mmckerns Thanks. I'm waiting for your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308571) Time Spent: 11h 50m (was: 11h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308469 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 08/Sep/19 10:05 Start Date: 08/Sep/19 10:05 Worklog Time Spent: 10m Work Description: mmckerns commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529188225 @tvalentyn: I agree, the monkey-patch is better pushed into `dill`. I have several 3.7/3.8 specific changes in the pipeline now, and this was one I hadn't gotten to yet. Will review the PR and move forward to get it into `dill`, which is a cleaner solution. Thanks @lazylynx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308469) Time Spent: 11h 40m (was: 11.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308284 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/Sep/19 03:34 Start Date: 07/Sep/19 03:34 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517#issuecomment-529068201 R: @lazylynx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308284) Time Spent: 11.5h (was: 11h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308282 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/Sep/19 03:33 Start Date: 07/Sep/19 03:33 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-529068152 One Beam SDK users who use master branch reported that monkey-patch caused their pipeline to fail with what appears to be an infinite recursion from new_save_reduce. If line 163 is removed, the problem disappears. The problem persists with the monkey patch, even if the body of new_save_reduce is changed to ``` StockPickler.save_reduce(self, func, args, *other_args, **kwargs) ``` @lazylynx I have not investigated that report yet but I'd suggest to revert this change in the mean time until we understand a better way to patch, which may not be necessary soon given your outstanding changes to dill. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308282) Time Spent: 11h 10m (was: 11h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=308283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308283 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/Sep/19 03:33 Start Date: 07/Sep/19 03:33 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9517: Revert "[BEAM-5878] support DoFns with Keyword-only arguments" URL: https://github.com/apache/beam/pull/9517 Reverts apache/beam#9237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308283) Time Spent: 11h 20m (was: 11h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305868 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 20:38 Start Date: 03/Sep/19 20:38 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-527629636 Merged, thanks again. As discussed before, consider making the change to Dill and reverting the monkeypatching here after Dill picks up the changes. @mmckerns is Dill maintainer and can probably help with the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305868) Time Spent: 11h (was: 10h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 11h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305866 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 20:33 Start Date: 03/Sep/19 20:33 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305866) Time Spent: 10h 50m (was: 10h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305714 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 17:24 Start Date: 03/Sep/19 17:24 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-527557087 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305714) Time Spent: 10h 40m (was: 10.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305676 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 16:12 Start Date: 03/Sep/19 16:12 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-527529133 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305676) Time Spent: 10h 20m (was: 10h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305677 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 16:12 Start Date: 03/Sep/19 16:12 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-527529187 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305677) Time Spent: 10.5h (was: 10h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305675 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 16:11 Start Date: 03/Sep/19 16:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-527529083 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305675) Time Spent: 10h 10m (was: 10h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305674 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Sep/19 16:11 Start Date: 03/Sep/19 16:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r320358173 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: No worries, and thanks a lot, @lazylynx ! The change looks good to me. I'm going to re-run a few tests to make sure that they still pass with this change combined with a recent upgrade of dill version (https://github.com/apache/beam/pull/9419). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305674) Time Spent: 10h (was: 9h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 10h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=305028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305028 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 02/Sep/19 00:53 Start Date: 02/Sep/19 00:53 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r319783108 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: @tvalentyn Sorry for late. PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305028) Time Spent: 9h 50m (was: 9h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=303296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-303296 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 28/Aug/19 23:26 Start Date: 28/Aug/19 23:26 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r318833002 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: Thanks, @lazylynx . It would be nice to add this to next Beam release that is not be cut in ~2 weeks. Thanks for your help. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 303296) Time Spent: 9h 40m (was: 9.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=302151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302151 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 27/Aug/19 16:04 Start Date: 27/Aug/19 16:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r317834907 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: Can we use a more generic signature of new_save_reduce, for example `def new_save_reduce(self, func, args, **kwargs)`? The problem is that we assume a particular version of the API for pickle.save_reduce here, and we can see that it will change in Python 3.8, see https://github.com/python/cpython/blob/c75f0e5bdee3cfaba9fd5b3a8549dec0aba01ebe/Lib/pickle.py#L619. I think with a generic definition of `new_save_reduce` we can still update `args` list , and pass `**kwargs` to `pickler.save_reduce`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302151) Time Spent: 9.5h (was: 9h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=302029=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302029 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 27/Aug/19 14:30 Start Date: 27/Aug/19 14:30 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r318114869 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: Quite so. I'll work on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302029) Time Spent: 9h 20m (was: 9h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=301584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-301584 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 26/Aug/19 23:11 Start Date: 26/Aug/19 23:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r317834907 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +# TODO: Remove this once uqfoundation/dill#313 is fixed +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, Review comment: Can we use a more generic signature of new_save_reduce, for example `def new_save_reduce(self, func, args, **kwargs)`? The problem is that we assume a particular version of the API for pickle.save_reduce here, and we can see that it will change in Python 3.8, see https://github.com/python/cpython/blob/c75f0e5bdee3cfaba9fd5b3a8549dec0aba01ebe/Lib/pickle.py#L619. I think with a generic definition of `new_save_reduce` we can still update `args` list , and pass `**kwargs` to `picker.save_reduce`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 301584) Time Spent: 9h 10m (was: 9h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300956 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Aug/19 23:47 Start Date: 25/Aug/19 23:47 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-524674534 @aaltay @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300956) Time Spent: 9h (was: 8h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300955 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Aug/19 23:47 Start Date: 25/Aug/19 23:47 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r317420153 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions Review comment: Added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300955) Time Spent: 8h 50m (was: 8h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300954 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Aug/19 23:46 Start Date: 25/Aug/19 23:46 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r317420147 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +# fkwdefaults is not None when called from new_save_reduce, skip None check Review comment: Surely be redundant. Removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300954) Time Spent: 8h 40m (was: 8.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300953 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 25/Aug/19 23:46 Start Date: 25/Aug/19 23:46 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r317420119 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +# fkwdefaults is not None when called from new_save_reduce, skip None check +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, + dictitems=None, obj=None): +pickler = super(dill.dill.Pickler, self) +if func is _create_function and \ +getattr(obj, '__kwdefaults__', None) is not None: + pickler.save_reduce(func=_create_function_has_kwdefaults, + args=args + (getattr(obj, '__kwdefaults__', None),), + state=state, listitems=listitems, dictitems=dictitems, + obj=obj) +else: + pickler.save_reduce(func, args, state, listitems, dictitems, obj) Review comment: This call back calls `save_reduce` function of `pickle.Pickler`, super class of `dill.dill.Pickler`. So `new_save_reduce` would not be called. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300953) Time Spent: 8.5h (was: 8h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300631=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300631 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 24/Aug/19 00:09 Start Date: 24/Aug/19 00:09 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-524497249 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300631) Time Spent: 8h 20m (was: 8h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=300628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300628 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 24/Aug/19 00:00 Start Date: 24/Aug/19 00:00 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-524495973 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300628) Time Spent: 8h 10m (was: 8h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=298826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298826 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 21/Aug/19 16:21 Start Date: 21/Aug/19 16:21 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r316277567 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +# fkwdefaults is not None when called from new_save_reduce, skip None check +func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, + dictitems=None, obj=None): +pickler = super(dill.dill.Pickler, self) +if func is _create_function and \ +getattr(obj, '__kwdefaults__', None) is not None: + pickler.save_reduce(func=_create_function_has_kwdefaults, + args=args + (getattr(obj, '__kwdefaults__', None),), + state=state, listitems=listitems, dictitems=dictitems, + obj=obj) +else: + pickler.save_reduce(func, args, state, listitems, dictitems, obj) Review comment: Would not this call back into `new_save_reduce` since you patched it in L164? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298826) Time Spent: 7h 50m (was: 7h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=298827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298827 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 21/Aug/19 16:21 Start Date: 21/Aug/19 16:21 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r316277821 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +# fkwdefaults is not None when called from new_save_reduce, skip None check Review comment: Is this comment needed? If we check for None, what would it be for? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298827) Time Spent: 8h (was: 7h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=298828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298828 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 21/Aug/19 16:21 Start Date: 21/Aug/19 16:21 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r316278667 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions Review comment: Could we add a TODO to remove this after https://github.com/uqfoundation/dill/issues/313 is fixed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298828) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 8h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=298674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298674 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 21/Aug/19 13:24 Start Date: 21/Aug/19 13:24 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-523454912 @tvalentyn PTAL Thank you for comments, especially with combine function! And I have started working on https://github.com/uqfoundation/dill/issues/313 and will create PR before long. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298674) Time Spent: 7h 40m (was: 7.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 7h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=298311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-298311 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 20/Aug/19 23:33 Start Date: 20/Aug/19 23:33 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-523235161 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 298311) Time Spent: 7.5h (was: 7h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292469 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 07:06 Start Date: 10/Aug/19 07:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312693535 ## File path: sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py ## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for side inputs.""" + +from __future__ import absolute_import + +import logging +import unittest + +import apache_beam as beam +from apache_beam.testing.test_pipeline import TestPipeline +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to + + +class KeywordOnlyArgsTests(unittest.TestCase): + + # Enable nose tests running in parallel + _multiprocess_can_split_ = True + + def test_side_input_keyword_only_args(self): +pipeline = TestPipeline() + +def sort_with_side_inputs(x, *s, reverse=False): + for y in s: +yield sorted([x] + y, reverse=reverse) + +pcol = pipeline | 'start' >> beam.Create([1, 2]) +side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input. +result1 = pcol | 'compute1' >> beam.FlatMap( +sort_with_side_inputs, +beam.pvalue.AsList(side), reverse=True) +assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1') + +result2 = pcol | 'compute2' >> beam.FlatMap( + sort_with_side_inputs, + beam.pvalue.AsList(side)) +assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2') + +result3 = pcol | 'compute3' >> beam.FlatMap( + sort_with_side_inputs) +assert_that(result3, equal_to([]), label='assert3') + +result4 = pcol | 'compute4' >> beam.FlatMap( + sort_with_side_inputs, reverse=True) +assert_that(result4, equal_to([]), label='assert4') + +pipeline.run() + + def test_combine_keyword_only_args(self): +pipeline = TestPipeline() + +def bounded_sum(values, *s, bound=500): Review comment: The reason you had to change `test_combine_keyword_only_args` is that this scenario is actually nondeterministic. From https://beam.apache.org/documentation/programming-guide/#combine > Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. By asserting that `result2` is `[49]` imply that a runner will compute partial sum 3 times, which will add 13 three times. So we rely on an implementation detail of the Direct Runner (which executes this test). The test would be deterministic if we add zeros, e.g. `beam.CombineGlobally(bounded_sum, 0, 0)`. This would make the test more stable towards future changes in direct runner, and fit the purpose of this PR to exercise keyword-only arguments. We can also come up with a better example. And, thanks a lot for calling out changes to the tests! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292469) Time Spent: 7h 10m (was: 7h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority:
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292470 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 07:06 Start Date: 10/Aug/19 07:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312693535 ## File path: sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py ## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for side inputs.""" + +from __future__ import absolute_import + +import logging +import unittest + +import apache_beam as beam +from apache_beam.testing.test_pipeline import TestPipeline +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to + + +class KeywordOnlyArgsTests(unittest.TestCase): + + # Enable nose tests running in parallel + _multiprocess_can_split_ = True + + def test_side_input_keyword_only_args(self): +pipeline = TestPipeline() + +def sort_with_side_inputs(x, *s, reverse=False): + for y in s: +yield sorted([x] + y, reverse=reverse) + +pcol = pipeline | 'start' >> beam.Create([1, 2]) +side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input. +result1 = pcol | 'compute1' >> beam.FlatMap( +sort_with_side_inputs, +beam.pvalue.AsList(side), reverse=True) +assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1') + +result2 = pcol | 'compute2' >> beam.FlatMap( + sort_with_side_inputs, + beam.pvalue.AsList(side)) +assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2') + +result3 = pcol | 'compute3' >> beam.FlatMap( + sort_with_side_inputs) +assert_that(result3, equal_to([]), label='assert3') + +result4 = pcol | 'compute4' >> beam.FlatMap( + sort_with_side_inputs, reverse=True) +assert_that(result4, equal_to([]), label='assert4') + +pipeline.run() + + def test_combine_keyword_only_args(self): +pipeline = TestPipeline() + +def bounded_sum(values, *s, bound=500): Review comment: The reason you had to change `test_combine_keyword_only_args` is that this scenario is actually nondeterministic. From https://beam.apache.org/documentation/programming-guide/#combine > Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. By asserting that `result2` is `[49]` we imply that a runner will compute partial sum 3 times, which will add 13 three times. So we rely on an implementation detail of the Direct Runner (which executes this test). The test would be deterministic if we add zeros, e.g. `beam.CombineGlobally(bounded_sum, 0, 0)`. This would make the test more stable towards future changes in direct runner, and fit the purpose of this PR to exercise keyword-only arguments. We can also come up with a better example. And, thanks a lot for calling out changes to the tests! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292470) Time Spent: 7h 20m (was: 7h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292468 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 07:05 Start Date: 10/Aug/19 07:05 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312693535 ## File path: sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py ## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for side inputs.""" + +from __future__ import absolute_import + +import logging +import unittest + +import apache_beam as beam +from apache_beam.testing.test_pipeline import TestPipeline +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to + + +class KeywordOnlyArgsTests(unittest.TestCase): + + # Enable nose tests running in parallel + _multiprocess_can_split_ = True + + def test_side_input_keyword_only_args(self): +pipeline = TestPipeline() + +def sort_with_side_inputs(x, *s, reverse=False): + for y in s: +yield sorted([x] + y, reverse=reverse) + +pcol = pipeline | 'start' >> beam.Create([1, 2]) +side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input. +result1 = pcol | 'compute1' >> beam.FlatMap( +sort_with_side_inputs, +beam.pvalue.AsList(side), reverse=True) +assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1') + +result2 = pcol | 'compute2' >> beam.FlatMap( + sort_with_side_inputs, + beam.pvalue.AsList(side)) +assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2') + +result3 = pcol | 'compute3' >> beam.FlatMap( + sort_with_side_inputs) +assert_that(result3, equal_to([]), label='assert3') + +result4 = pcol | 'compute4' >> beam.FlatMap( + sort_with_side_inputs, reverse=True) +assert_that(result4, equal_to([]), label='assert4') + +pipeline.run() + + def test_combine_keyword_only_args(self): +pipeline = TestPipeline() + +def bounded_sum(values, *s, bound=500): Review comment: The reason you had to change `test_combine_keyword_only_args` is that this scenario is actually nondeterministic. From https://beam.apache.org/documentation/programming-guide/#combine ```Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.``` By asserting that `result2` is `[49]` imply that a runner will compute partial sum 3 times, which will add 13 three times. So we rely on an implementation detail of the Direct Runner (which executes this test). The test would be deterministic if we add zeros, e.g. `beam.CombineGlobally(bounded_sum, 0, 0)`. This would make the test more stable towards future changes in direct runner, and fit the purpose of this PR to exercise keyword-only arguments. We can also come up with a better example. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292468) Time Spent: 7h (was: 6h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 7h > Remaining
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292459 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 06:12 Start Date: 10/Aug/19 06:12 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312690779 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +if fkwdefaults is not None: + func.__kwdefaults__ = fkwdefaults +return func + + def new_save_reduce(self, func, args, state=None, listitems=None, + dictitems=None, obj=None): +pickler = super(dill.dill.Pickler, self) +if func is _create_function and \ Review comment: nit: implicit usage of brackets is preferred to backslash for line continuation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292459) Time Spent: 6h 50m (was: 6h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292458 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 06:12 Start Date: 10/Aug/19 06:12 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312690816 ## File path: sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py ## @@ -0,0 +1,106 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for side inputs.""" + +from __future__ import absolute_import + +import logging +import unittest + +import apache_beam as beam +from apache_beam.testing.test_pipeline import TestPipeline +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to + + +class KeywordOnlyArgsTests(unittest.TestCase): + + # Enable nose tests running in parallel + _multiprocess_can_split_ = True + + def test_side_input_keyword_only_args(self): +pipeline = TestPipeline() + +def sort_with_side_inputs(x, *s, reverse=False): Review comment: Could you please add a test where we have a function with keyword-only arguments, that don't have assigned default values? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292458) Time Spent: 6h 40m (was: 6.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=292457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292457 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 10/Aug/19 06:12 Start Date: 10/Aug/19 06:12 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r312691122 ## File path: sdks/python/apache_beam/internal/pickler.py ## @@ -136,6 +136,32 @@ def _reject_generators(unused_pickler, unused_obj): dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators +if sys.version_info[0] > 2: + # Monkey patch for dill._dill.Pickler to pickle functions + # with keyword-only args + _create_function = dill.dill._create_function + + def _create_function_has_kwdefaults(fcode, fglobals, fname=None, + fdefaults=None, fclosure=None, fdict=None, + fkwdefaults=None): +func = _create_function(fcode, fglobals, fname, fdefaults, fclosure, fdict) +if fkwdefaults is not None: Review comment: nit: we already check that kwdefaults is not None in another function. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292457) Time Spent: 6h 40m (was: 6.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=291349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-291349 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 08/Aug/19 15:32 Start Date: 08/Aug/19 15:32 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-519569187 @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 291349) Time Spent: 6.5h (was: 6h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=290626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290626 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/Aug/19 17:54 Start Date: 07/Aug/19 17:54 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-519203798 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 290626) Time Spent: 6h 20m (was: 6h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=289684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289684 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/Aug/19 13:59 Start Date: 06/Aug/19 13:59 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-518681646 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289684) Time Spent: 6h 10m (was: 6h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=289683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289683 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/Aug/19 13:59 Start Date: 06/Aug/19 13:59 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-518681577 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289683) Time Spent: 6h (was: 5h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 6h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=289682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289682 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/Aug/19 13:59 Start Date: 06/Aug/19 13:59 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#discussion_r311077705 ## File path: sdks/python/apache_beam/transforms/transforms_keyword_only_args_py3_only_test.py ## @@ -0,0 +1,115 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for side inputs.""" + +from __future__ import absolute_import + +import logging +import sys +import unittest + +import apache_beam as beam # pylint: disable=unused-import +from apache_beam.testing.test_pipeline import TestPipeline # pylint: disable=unused-import +from apache_beam.testing.util import assert_that # pylint: disable=unused-import +from apache_beam.testing.util import equal_to # pylint: disable=unused-import + +IS_PYTHON_2 = sys.version_info[0] == 2 + + +# pylint: disable=exec-used +@unittest.skipIf(IS_PYTHON_2, 'test only in python 3 with kwonly args') Review comment: We came up with a different way of adding Python3-only tests, see https://github.com/apache/beam/pull/9223. Let's follow that pattern here. We would need to rename this file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289682) Time Spent: 5h 50m (was: 5h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=288768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288768 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 05/Aug/19 08:20 Start Date: 05/Aug/19 08:20 Worklog Time Spent: 10m Work Description: lazylynx commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-518135438 @tvalentyn sorry for no notification, please start reviewing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 288768) Time Spent: 5h 40m (was: 5.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=288717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288717 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 05/Aug/19 07:11 Start Date: 05/Aug/19 07:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-518114671 Hi @lazylynx , is this change ready for review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 288717) Time Spent: 5.5h (was: 5h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=288340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288340 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Aug/19 00:28 Start Date: 03/Aug/19 00:28 Worklog Time Spent: 10m Work Description: aaltay commented on issue #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237#issuecomment-517878579 R: @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 288340) Time Spent: 5h 20m (was: 5h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: yoshiki obata >Priority: Minor > Fix For: 2.16.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=288338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288338 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Aug/19 00:19 Start Date: 03/Aug/19 00:19 Worklog Time Spent: 10m Work Description: lazylynx commented on pull request #9237: [BEAM-5878] support DoFns with Keyword-only arguments URL: https://github.com/apache/beam/pull/9237 support DoFns with Keyword-only arguments in Python 3 and add test reverted in #8750 with no syntax error in Python 2 tests condition are fixed as follows due to errors: # test_side_input_keyword_only_args ``` result2 = pcol | 'compute2' >> beam.FlatMap( sort_with_side_inputs, beam.pvalue.AsIter(side)) ``` to ``` result2 = pcol | 'compute2' >> beam.FlatMap( sort_with_side_inputs, beam.pvalue.AsList(side)) ``` # test_combine_keyword_only_args ``` assert_that(result2, equal_to([23]), label='assert2') ``` to ``` assert_that(result2, equal_to([49]), label='assert2') ``` Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=253425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253425 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Jun/19 22:10 Start Date: 03/Jun/19 22:10 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8750: Revert "[BEAM-5878] add tests for kwonly args in python 3" URL: https://github.com/apache/beam/pull/8750 Reverts apache/beam#8505 Reason: breaks postcommit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253425) Time Spent: 5h (was: 4h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=253413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253413 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 03/Jun/19 21:55 Start Date: 03/Jun/19 21:55 Worklog Time Spent: 10m Work Description: apilloud commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-498441270 After this change beam_PostCommit_Python_Verify :sdks:python:postCommitIT and other tests are failing due to a syntax error. I filed an issue: https://issues.apache.org/jira/browse/BEAM-7481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253413) Time Spent: 4h 50m (was: 4h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=252061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-252061 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/May/19 21:10 Start Date: 31/May/19 21:10 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 252061) Time Spent: 4h 40m (was: 4.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=248851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248851 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 27/May/19 13:42 Start Date: 27/May/19 13:42 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-496217367 @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248851) Time Spent: 4.5h (was: 4h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=246036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246036 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 21/May/19 13:22 Start Date: 21/May/19 13:22 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-494388653 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 246036) Time Spent: 4h 20m (was: 4h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245399 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 20/May/19 18:06 Start Date: 20/May/19 18:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#discussion_r285702955 ## File path: sdks/python/scripts/run_pylint.sh ## @@ -60,8 +60,15 @@ EXCLUDED_GENERATED_FILES=( apache_beam/portability/api/*pb2*.py ) +# Following files contain python 3 syntax and are excluded from pylint in python 2 +EXCLUDED_PYTHON3_FILES=( +"apache_beam/transforms/transforms_keyword_only_args_py3_test.py" Review comment: Let's add `# TODO(BEAM-7372): Remove this list once Python 2 is no longer supported and drop _py3_test suffix.` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245399) Time Spent: 4h (was: 3h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245396
]
ASF GitHub Bot logged work on BEAM-5878:
Author: ASF GitHub Bot
Created on: 20/May/19 18:06
Start Date: 20/May/19 18:06
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8505: [BEAM-5878]
add tests for kwonly args in python 3
URL: https://github.com/apache/beam/pull/8505#discussion_r285698767
##
File path:
sdks/python/apache_beam/transforms/transforms_keyword_only_args_py3_test.py
##
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@unittest.skipIf(sys.version_info[0] == 2,
+ 'Keyword-Only Arguments are not supported in python 2')
+@unittest.skip('TODO BEAM-5878: support kwonly args in python 3')
+class KeywordOnlyArgsTests(unittest.TestCase):
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+pipeline = TestPipeline()
+
+def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+yield sorted([x] + y, reverse=reverse)
+
+pcol = pipeline | 'start' >> beam.Create([1, 2])
+side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+result1 = pcol | 'compute1' >> beam.FlatMap(
+sort_with_side_inputs,
+beam.pvalue.AsList(side), reverse=True)
+assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1')
+
+result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsIter(side))
+assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2')
+
+result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+assert_that(result3, equal_to([]), label='assert3')
+
+result4 = pcol | 'compute4' >> beam.FlatMap(
+ sort_with_side_inputs, reverse=True)
+assert_that(result4, equal_to([]), label='assert4')
+
+pipeline.run()
+
+ def test_combine_keyword_only_args(self):
+pipeline = TestPipeline()
+
+# Keyword-only arguments are not available on Python 2
+# pylint: disable=syntax-error
+def bounded_sum(values, *s, bound=500):
+ return min(sum(values) + sum(s), bound)
+
+pcoll = pipeline | 'start' >> beam.Create([6, 3, 1])
+result1 = pcoll | 'sum1' >> beam.CombineGlobally(bounded_sum, 5, 8,
bound=20)
+result2 = pcoll | 'sum2' >> beam.CombineGlobally(bounded_sum, 5, 8)
+result3 = pcoll | 'sum3' >> beam.CombineGlobally(bounded_sum)
+result4 = pcoll | 'sum4' >> beam.CombineGlobally(bounded_sum, bound=12)
+
+assert_that(result1, equal_to([20]), label='assert1')
+assert_that(result2, equal_to([23]), label='assert2')
+assert_that(result3, equal_to([10]), label='assert3')
+assert_that(result4, equal_to([10]), label='assert4')
+
+pipeline.run()
+
+ def test_do_fn_keyword_only_args(self):
+pipeline = TestPipeline()
+
+class MyDoFn(beam.DoFn):
+ def process(self, element, *s, bound=500):
+return [min(sum(s) + element, bound)]
+
+pcoll = pipeline | 'start' >> beam.Create([6, 3, 1])
+result1 = pcoll | 'sum1' >> beam.ParDo(MyDoFn(), 5, 8, bound=15)
+result2 = pcoll | 'sum2' >> beam.ParDo(MyDoFn(), 5, 8)
+result3 = pcoll | 'sum3' >> beam.ParDo(MyDoFn())
+result4 = pcoll | 'sum4' >> beam.ParDo(MyDoFn(), bound=10)
Review comment:
You could select a smaller bound (5) here for result4 to be different.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245397
]
ASF GitHub Bot logged work on BEAM-5878:
Author: ASF GitHub Bot
Created on: 20/May/19 18:06
Start Date: 20/May/19 18:06
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8505: [BEAM-5878]
add tests for kwonly args in python 3
URL: https://github.com/apache/beam/pull/8505#discussion_r285699322
##
File path:
sdks/python/apache_beam/transforms/transforms_keyword_only_args_py3_test.py
##
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@unittest.skipIf(sys.version_info[0] == 2,
+ 'Keyword-Only Arguments are not supported in python 2')
+@unittest.skip('TODO BEAM-5878: support kwonly args in python 3')
+class KeywordOnlyArgsTests(unittest.TestCase):
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+pipeline = TestPipeline()
+
+def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+yield sorted([x] + y, reverse=reverse)
+
+pcol = pipeline | 'start' >> beam.Create([1, 2])
+side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+result1 = pcol | 'compute1' >> beam.FlatMap(
+sort_with_side_inputs,
+beam.pvalue.AsList(side), reverse=True)
+assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1')
+
+result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsIter(side))
+assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2')
+
+result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+assert_that(result3, equal_to([]), label='assert3')
+
+result4 = pcol | 'compute4' >> beam.FlatMap(
+ sort_with_side_inputs, reverse=True)
+assert_that(result4, equal_to([]), label='assert4')
+
+pipeline.run()
+
+ def test_combine_keyword_only_args(self):
+pipeline = TestPipeline()
+
+# Keyword-only arguments are not available on Python 2
+# pylint: disable=syntax-error
+def bounded_sum(values, *s, bound=500):
+ return min(sum(values) + sum(s), bound)
+
+pcoll = pipeline | 'start' >> beam.Create([6, 3, 1])
+result1 = pcoll | 'sum1' >> beam.CombineGlobally(bounded_sum, 5, 8,
bound=20)
+result2 = pcoll | 'sum2' >> beam.CombineGlobally(bounded_sum, 5, 8)
+result3 = pcoll | 'sum3' >> beam.CombineGlobally(bounded_sum)
+result4 = pcoll | 'sum4' >> beam.CombineGlobally(bounded_sum, bound=12)
Review comment:
You could select a smaller bound here for `result4` to be different.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245397)
Time Spent: 3h 40m (was: 3.5h)
> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Minor
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to
> define functions with keyword-only arguments.
> Currently Beam does not handle them
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245398
]
ASF GitHub Bot logged work on BEAM-5878:
Author: ASF GitHub Bot
Created on: 20/May/19 18:06
Start Date: 20/May/19 18:06
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8505: [BEAM-5878]
add tests for kwonly args in python 3
URL: https://github.com/apache/beam/pull/8505#discussion_r285695694
##
File path:
sdks/python/apache_beam/transforms/transforms_keyword_only_args_py3_test.py
##
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@unittest.skipIf(sys.version_info[0] == 2,
+ 'Keyword-Only Arguments are not supported in python 2')
+@unittest.skip('TODO BEAM-5878: support kwonly args in python 3')
+class KeywordOnlyArgsTests(unittest.TestCase):
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+pipeline = TestPipeline()
+
+def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+yield sorted([x] + y, reverse=reverse)
+
+pcol = pipeline | 'start' >> beam.Create([1, 2])
+side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+result1 = pcol | 'compute1' >> beam.FlatMap(
+sort_with_side_inputs,
+beam.pvalue.AsList(side), reverse=True)
+assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1')
+
+result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsIter(side))
+assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2')
+
+result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+assert_that(result3, equal_to([]), label='assert3')
+
+result4 = pcol | 'compute4' >> beam.FlatMap(
+ sort_with_side_inputs, reverse=True)
+assert_that(result4, equal_to([]), label='assert4')
+
+pipeline.run()
+
+ def test_combine_keyword_only_args(self):
+pipeline = TestPipeline()
+
+# Keyword-only arguments are not available on Python 2
+# pylint: disable=syntax-error
Review comment:
Leftover comment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 245398)
Time Spent: 3h 50m (was: 3h 40m)
> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Minor
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to
> define functions with keyword-only arguments.
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
> in our codebase that we should fix: in Python in 3.0 inspect.getargspec()
> will fail on functions with keyword-only arguments, but a new method
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
> supports them.
> There may be implications for our (best-effort) type-hints
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245400 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 20/May/19 18:06 Start Date: 20/May/19 18:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#discussion_r285706706 ## File path: sdks/python/scripts/run_pylint.sh ## @@ -60,8 +60,15 @@ EXCLUDED_GENERATED_FILES=( apache_beam/portability/api/*pb2*.py ) +# Following files contain python 3 syntax and are excluded from pylint in python 2 +EXCLUDED_PYTHON3_FILES=( +"apache_beam/transforms/transforms_keyword_only_args_py3_test.py" Review comment: I think `_py3_only_test` may be a more explicit suffix, how about we use that? Also could this be a regex pattern `.*_py3_only_test.py`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245400) Time Spent: 4h 10m (was: 4h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 4h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=245109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-245109 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 20/May/19 13:51 Start Date: 20/May/19 13:51 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-493996063 @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 245109) Time Spent: 3h 20m (was: 3h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 3h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=241319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-241319 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 13/May/19 21:09 Start Date: 13/May/19 21:09 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-491987776 It looks like the suite still fails on Py2 due to unsupported syntax. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 241319) Time Spent: 3h 10m (was: 3h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 3h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=238751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238751 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/May/19 19:49 Start Date: 07/May/19 19:49 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-490230027 Adding as exception sounds good to me. Is it possible to add and exception on Python 2 lint only? If that's hard, you can file a Jira and add a comment to clean this up eventually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238751) Time Spent: 3h (was: 2h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 3h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=238687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238687 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/May/19 17:47 Start Date: 07/May/19 17:47 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-490180960 @tvalentyn Apparently `# pylint disable=syntax-error` is not possible: https://github.com/PyCQA/pylint/issues/986. I can only think of fixing it by adding it as an exception in pylint2.7: https://github.com/apache/beam/blob/master/sdks/python/scripts/run_pylint.sh#L73. What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238687) Time Spent: 2h 50m (was: 2h 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=238449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238449 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/May/19 13:17 Start Date: 07/May/19 13:17 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-490075237 Let's try to disable a particular pylint warning and add a comment. ` # Keyword-only arguments are not available on Python 2` ` # pylint: disable=wrong-` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238449) Time Spent: 2h 40m (was: 2.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 2h 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=238339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238339 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 07/May/19 09:28 Start Date: 07/May/19 09:28 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-490006329 @tvalentyn The test currently fails on pylint 2.7 because the syntax of keyword only args gives a syntax-error in python 2. Should I add a pylint disable syntax-error comment or is there an other way to skip this file for python 2? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 238339) Time Spent: 2.5h (was: 2h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=238338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-238338
]
ASF GitHub Bot logged work on BEAM-5878:
Author: ASF GitHub Bot
Created on: 07/May/19 09:23
Start Date: 07/May/19 09:23
Worklog Time Spent: 10m
Work Description: Juta commented on pull request #8505: [BEAM-5878] add
tests for kwonly args in python 3
URL: https://github.com/apache/beam/pull/8505#discussion_r281545285
##
File path:
sdks/python/apache_beam/transforms/tranforms_keyword_only_args_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@unittest.skipIf(sys.version_info[0] == 2,
+ 'Keyword-Only Arguments are not supported in python 2')
+@unittest.skip('TODO BEAM-5878: support kwonly args in python 3')
+class KeywordOnlyArgsTests(unittest.TestCase):
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+pipeline = TestPipeline()
+
+def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+yield sorted([x] + y, reverse=reverse)
+
+pcol = pipeline | 'start' >> beam.Create([1, 2])
+side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+result1 = pcol | 'compute1' >> beam.FlatMap(
+sort_with_side_inputs,
+beam.pvalue.AsList(side), reverse=True)
+assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1')
+
+result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsIter(side))
+assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2')
+
+result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+assert_that(result3, equal_to([]), label='assert3')
Review comment:
yes because it joins with the side input and in this case there is no side
input
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 238338)
Time Spent: 2h 20m (was: 2h 10m)
> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to
> define functions with keyword-only arguments.
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
> in our codebase that we should fix: in Python in 3.0 inspect.getargspec()
> will fail on functions with keyword-only arguments, but a new method
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
> supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only
> arguments once Beam Python 3 tests are in a good shape.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[
https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=237883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237883
]
ASF GitHub Bot logged work on BEAM-5878:
Author: ASF GitHub Bot
Created on: 06/May/19 16:11
Start Date: 06/May/19 16:11
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8505: [BEAM-5878]
add tests for kwonly args in python 3
URL: https://github.com/apache/beam/pull/8505#discussion_r281252834
##
File path:
sdks/python/apache_beam/transforms/tranforms_keyword_only_args_test.py
##
@@ -0,0 +1,109 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+from __future__ import absolute_import
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@unittest.skipIf(sys.version_info[0] == 2,
+ 'Keyword-Only Arguments are not supported in python 2')
+@unittest.skip('TODO BEAM-5878: support kwonly args in python 3')
+class KeywordOnlyArgsTests(unittest.TestCase):
+ # Enable nose tests running in parallel
+ _multiprocess_can_split_ = True
+
+ def test_side_input_keyword_only_args(self):
+pipeline = TestPipeline()
+
+def sort_with_side_inputs(x, *s, reverse=False):
+ for y in s:
+yield sorted([x] + y, reverse=reverse)
+
+pcol = pipeline | 'start' >> beam.Create([1, 2])
+side = pipeline | 'side' >> beam.Create([3, 4]) # 2 values in side input.
+result1 = pcol | 'compute1' >> beam.FlatMap(
+sort_with_side_inputs,
+beam.pvalue.AsList(side), reverse=True)
+assert_that(result1, equal_to([[4,3,1], [4,3,2]]), label='assert1')
+
+result2 = pcol | 'compute2' >> beam.FlatMap(
+ sort_with_side_inputs,
+ beam.pvalue.AsIter(side))
+assert_that(result2, equal_to([[1,3,4], [2,3,4]]), label='assert2')
+
+result3 = pcol | 'compute3' >> beam.FlatMap(
+ sort_with_side_inputs)
+assert_that(result3, equal_to([]), label='assert3')
Review comment:
(didn't look into the logic in detail yet) is the result supposed to be
empty here and in line 64?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 237883)
Time Spent: 2h 10m (was: 2h)
> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to
> define functions with keyword-only arguments.
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
> in our codebase that we should fix: in Python in 3.0 inspect.getargspec()
> will fail on functions with keyword-only arguments, but a new method
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
> supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only
> arguments once Beam Python 3 tests are in a good shape.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=237863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237863 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/May/19 15:44 Start Date: 06/May/19 15:44 Worklog Time Spent: 10m Work Description: Juta commented on issue #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505#issuecomment-489668866 R: @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 237863) Time Spent: 2h (was: 1h 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Juta Staes >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=237831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-237831 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 06/May/19 15:15 Start Date: 06/May/19 15:15 Worklog Time Spent: 10m Work Description: Juta commented on pull request #8505: [BEAM-5878] add tests for kwonly args in python 3 URL: https://github.com/apache/beam/pull/8505 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) Portable | --- |
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=161280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161280 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/18 18:49 Start Date: 31/Oct/18 18:49 Worklog Time Spent: 10m Work Description: aaltay closed pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py index 47222a18a78..033650048fd 100644 --- a/sdks/python/apache_beam/runners/common.py +++ b/sdks/python/apache_beam/runners/common.py @@ -150,8 +150,13 @@ def __init__(self, obj_to_invoke, method_name): 'a \'RestrictionProvider\'. Received %r instead.' % obj_to_invoke) -args, _, _, defaults = core.get_function_arguments( +fullargspec = core.get_function_arguments( obj_to_invoke, method_name) + +# TODO(BEAM-5878) support kwonlyargs on Python 3. +args = fullargspec[0] +defaults = fullargspec[3] + defaults = defaults if defaults else [] method_value = getattr(obj_to_invoke, method_name) self.method_value = method_value diff --git a/sdks/python/apache_beam/transforms/core.py b/sdks/python/apache_beam/transforms/core.py index 5505de55522..9027b97ac50 100644 --- a/sdks/python/apache_beam/transforms/core.py +++ b/sdks/python/apache_beam/transforms/core.py @@ -20,7 +20,6 @@ from __future__ import absolute_import import copy -import inspect import logging import random import re @@ -64,6 +63,7 @@ from apache_beam.typehints.decorators import TypeCheckError from apache_beam.typehints.decorators import WithTypeHints from apache_beam.typehints.decorators import get_type_hints +from apache_beam.typehints.decorators import getfullargspec from apache_beam.typehints.trivial_inference import element_type from apache_beam.typehints.typehints import is_consistent_with from apache_beam.utils import urns @@ -275,14 +275,14 @@ def restriction_coder(self): def get_function_arguments(obj, func): """Return the function arguments based on the name provided. If they have a _inspect_function attached to the class then use that otherwise default - to the python inspect library. + to the modified version of python inspect library. """ func_name = '_inspect_%s' % func if hasattr(obj, func_name): f = getattr(obj, func_name) return f() f = getattr(obj, func) - return inspect.getargspec(f) + return getfullargspec(f) class _DoFnParam(object): @@ -442,12 +442,21 @@ def is_process_bounded(self): def _fn_takes_side_inputs(fn): try: -argspec = inspect.getargspec(fn) +argspec = getfullargspec(fn) except TypeError: # We can't tell; maybe it does. return True is_bound = isinstance(fn, types.MethodType) and fn.__self__ is not None - return len(argspec.args) > 1 + is_bound or argspec.varargs or argspec.keywords + + try: +varkw = argspec.varkw +kwonlyargs = argspec.kwonlyargs + except AttributeError: # Python 2 +varkw = argspec.keywords +kwonlyargs = [] + + return (len(argspec.args) + len(kwonlyargs) > 1 + is_bound or + argspec.varargs or varkw) class CallableWrapperDoFn(DoFn): @@ -515,7 +524,7 @@ def _process_argspec_fn(self): return getattr(self._fn, '_argspec_fn', self._fn) def _inspect_process(self): -return inspect.getargspec(self._process_argspec_fn()) +return getfullargspec(self._process_argspec_fn()) class CombineFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn): diff --git a/sdks/python/apache_beam/transforms/ptransform.py b/sdks/python/apache_beam/transforms/ptransform.py index 9939278d380..c512d9f3aa6 100644 --- a/sdks/python/apache_beam/transforms/ptransform.py +++ b/sdks/python/apache_beam/transforms/ptransform.py @@ -37,7 +37,6 @@ class and wrapper class that allows lambda functions to be used as from __future__ import absolute_import import copy -import inspect import itertools import operator import os @@ -61,6 +60,7 @@ class and wrapper class that allows lambda functions to be used as from apache_beam.typehints.decorators import TypeCheckError from apache_beam.typehints.decorators import WithTypeHints from apache_beam.typehints.decorators import getcallargs_forhints +from apache_beam.typehints.decorators import getfullargspec from apache_beam.typehints.trivial_inference import instance_to_type from apache_beam.typehints.typehints import
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=161262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161262 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/18 18:15 Start Date: 31/Oct/18 18:15 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#issuecomment-434793488 @aaltay Thanks for the review! PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161262) Time Spent: 1.5h (was: 1h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160886 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 22:04 Start Date: 30/Oct/18 22:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229503113 ## File path: sdks/python/apache_beam/typehints/decorators.py ## @@ -259,15 +263,30 @@ def _unpack_positional_arg_hints(arg, hint): def getcallargs_forhints(func, *typeargs, **typekwargs): """Like inspect.getcallargs, but understands that Tuple[] and an Any unpack. """ - argspec = inspect.getargspec(func) + argspec = getfullargspec(func) # Turn Tuple[x, y] into (x, y) so getcallargs can do the proper unpacking. packed_typeargs = [_unpack_positional_arg_hints(arg, hint) for (arg, hint) in zip(argspec.args, typeargs)] packed_typeargs += list(typeargs[len(packed_typeargs):]) + + # Monkeypatch inspect.getfullargspec to allow passing non-function objects. + # getfullargspec (getargspec on Python 2) are used by inspect.getcallargs. + if _use_legacy_getargspec: # Python 2 +inspect.getargspec = getfullargspec + else: +inspect.getfullargspec = getfullargspec + try: callargs = inspect.getcallargs(func, *packed_typeargs, **typekwargs) except TypeError as e: raise TypeCheckError(e) + finally: +# Revert monkey-patch. Review comment: I agree. Added a TODO to the relevant JIRA issue. We can do that in a separate change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160886) Time Spent: 1h 20m (was: 1h 10m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160884 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 22:04 Start Date: 30/Oct/18 22:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229502868 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -442,12 +442,21 @@ def is_process_bounded(self): def _fn_takes_side_inputs(fn): try: -argspec = inspect.getargspec(fn) +argspec = getfullargspec(fn) except TypeError: # We can't tell; maybe it does. return True is_bound = isinstance(fn, types.MethodType) and fn.__self__ is not None - return len(argspec.args) > 1 + is_bound or argspec.varargs or argspec.keywords + + try: +varkw = argspec.varkw +kwonlyargs = argspec.kwonlyargs + except AttributeError: # Python 2 +varkw = argspec.keywords +kwonlyargs = [] + + return (len(argspec.args) + len(kwonlyargs) > 1 + is_bound or Review comment: We preserve the original intent: to check that total number of arguments is more than 1, or 2 if the method is bound. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160884) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160885 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 22:04 Start Date: 30/Oct/18 22:04 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229503245 ## File path: sdks/python/apache_beam/typehints/decorators.py ## @@ -112,38 +112,42 @@ def foo((a, b)): _MethodDescriptorType = type(str.upper) # pylint: enable=invalid-name +try: + _original_getfullargspec = inspect.getfullargspec + _use_legacy_getargspec = False +except AttributeError: # Python 2 + _original_getfullargspec = inspect.getargspec + _use_legacy_getargspec = True -# Monkeypatch inspect.getargspec to allow passing non-function objects. -# This is needed to use higher-level functions such as getcallargs. -_original_getargspec = inspect.getargspec - -def getargspec(func): +def getfullargspec(func): try: -return _original_getargspec(func) +return _original_getfullargspec(func) except TypeError: if isinstance(func, type): - argspec = getargspec(func.__init__) + argspec = getfullargspec(func.__init__) del argspec.args[0] return argspec elif callable(func): try: -return _original_getargspec(func.__call__) +return _original_getfullargspec(func.__call__) except TypeError: # Return an ArgSpec with at least one positional argument, # and any number of other (positional or keyword) arguments -# whose name won't match any real agument. +# whose name won't match any real argument. # Arguments with the %unknown% prefix will be ignored in the type # checking code. -return inspect.ArgSpec( -['_'], '__unknown__varargs', '__unknown__keywords', ()) +try: Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160885) Time Spent: 1h 10m (was: 1h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160782 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 19:01 Start Date: 30/Oct/18 19:01 Worklog Time Spent: 10m Work Description: aaltay commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229443338 ## File path: sdks/python/apache_beam/transforms/core.py ## @@ -442,12 +442,21 @@ def is_process_bounded(self): def _fn_takes_side_inputs(fn): try: -argspec = inspect.getargspec(fn) +argspec = getfullargspec(fn) except TypeError: # We can't tell; maybe it does. return True is_bound = isinstance(fn, types.MethodType) and fn.__self__ is not None - return len(argspec.args) > 1 + is_bound or argspec.varargs or argspec.keywords + + try: +varkw = argspec.varkw +kwonlyargs = argspec.kwonlyargs + except AttributeError: # Python 2 +varkw = argspec.keywords +kwonlyargs = [] + + return (len(argspec.args) + len(kwonlyargs) > 1 + is_bound or Review comment: why `len(kwonlyargs) > 1` ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160782) Time Spent: 50m (was: 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160783 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 19:01 Start Date: 30/Oct/18 19:01 Worklog Time Spent: 10m Work Description: aaltay commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229444391 ## File path: sdks/python/apache_beam/typehints/decorators.py ## @@ -259,15 +263,30 @@ def _unpack_positional_arg_hints(arg, hint): def getcallargs_forhints(func, *typeargs, **typekwargs): """Like inspect.getcallargs, but understands that Tuple[] and an Any unpack. """ - argspec = inspect.getargspec(func) + argspec = getfullargspec(func) # Turn Tuple[x, y] into (x, y) so getcallargs can do the proper unpacking. packed_typeargs = [_unpack_positional_arg_hints(arg, hint) for (arg, hint) in zip(argspec.args, typeargs)] packed_typeargs += list(typeargs[len(packed_typeargs):]) + + # Monkeypatch inspect.getfullargspec to allow passing non-function objects. + # getfullargspec (getargspec on Python 2) are used by inspect.getcallargs. + if _use_legacy_getargspec: # Python 2 +inspect.getargspec = getfullargspec + else: +inspect.getfullargspec = getfullargspec + try: callargs = inspect.getcallargs(func, *packed_typeargs, **typekwargs) except TypeError as e: raise TypeCheckError(e) + finally: +# Revert monkey-patch. Review comment: I think we should fully stop doing the monkey patching. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160783) Time Spent: 1h (was: 50m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160781 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 30/Oct/18 19:01 Start Date: 30/Oct/18 19:01 Worklog Time Spent: 10m Work Description: aaltay commented on a change in pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#discussion_r229443871 ## File path: sdks/python/apache_beam/typehints/decorators.py ## @@ -112,38 +112,42 @@ def foo((a, b)): _MethodDescriptorType = type(str.upper) # pylint: enable=invalid-name +try: + _original_getfullargspec = inspect.getfullargspec + _use_legacy_getargspec = False +except AttributeError: # Python 2 + _original_getfullargspec = inspect.getargspec + _use_legacy_getargspec = True -# Monkeypatch inspect.getargspec to allow passing non-function objects. -# This is needed to use higher-level functions such as getcallargs. -_original_getargspec = inspect.getargspec - -def getargspec(func): +def getfullargspec(func): try: -return _original_getargspec(func) +return _original_getfullargspec(func) except TypeError: if isinstance(func, type): - argspec = getargspec(func.__init__) + argspec = getfullargspec(func.__init__) del argspec.args[0] return argspec elif callable(func): try: -return _original_getargspec(func.__call__) +return _original_getfullargspec(func.__call__) except TypeError: # Return an ArgSpec with at least one positional argument, # and any number of other (positional or keyword) arguments -# whose name won't match any real agument. +# whose name won't match any real argument. # Arguments with the %unknown% prefix will be ignored in the type # checking code. -return inspect.ArgSpec( -['_'], '__unknown__varargs', '__unknown__keywords', ()) +try: Review comment: You can use `_use_legacy_getargspec`. You do not need another `try/except`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160781) Time Spent: 50m (was: 40m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160292 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 29/Oct/18 23:04 Start Date: 29/Oct/18 23:04 Worklog Time Spent: 10m Work Description: tvalentyn edited a comment on issue #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#issuecomment-434110293 R: @aaltay CC: @robertwb @splovyt @RobbeSneyders @markflyhigh Also thanks to @ruoyu who did initial investigation and came up with a patch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160292) Time Spent: 40m (was: 0.5h) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=160285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-160285 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 29/Oct/18 22:52 Start Date: 29/Oct/18 22:52 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#issuecomment-434110293 R: @aaltay CC: @robertwb @splovyt @RobbeSneyders @markflyhigh Also thanks to Ruoyu who did initial investigation and came up with a patch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 160285) Time Spent: 0.5h (was: 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
