[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331764 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 22/Oct/19 01:44 Start Date: 22/Oct/19 01:44 Worklog Time Spent: 10m Work Description: udim commented on pull request #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331764) Time Spent: 1.5h (was: 1h 20m) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331752 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 22/Oct/19 01:10 Start Date: 22/Oct/19 01:10 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805#issuecomment-544770107 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331752) Time Spent: 1h 20m (was: 1h 10m) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331740 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 22/Oct/19 00:56 Start Date: 22/Oct/19 00:56 Worklog Time Spent: 10m Work Description: udim commented on issue #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805#issuecomment-544767624 @chamikaramj does this LGTY? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331740) Time Spent: 1h 10m (was: 1h) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331581 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 21/Oct/19 18:57 Start Date: 21/Oct/19 18:57 Worklog Time Spent: 10m Work Description: udim commented on issue #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805#issuecomment-544657577 Opened: https://issues.apache.org/jira/browse/BEAM-8447 to add a datastore_matcher.py This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331581) Time Spent: 1h (was: 50m) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331579 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 21/Oct/19 18:54 Start Date: 21/Oct/19 18:54 Worklog Time Spent: 10m Work Description: udim commented on issue #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805#issuecomment-544656056 > LGTM other than one comment. Thanks. > > Do you think it's worth adding this to the IT ? This should probably be tested but the existing IT only verifies counts of entities, not actual content. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 331579) Time Spent: 50m (was: 40m) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[
https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=331571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-331571
]
ASF GitHub Bot logged work on BEAM-8405:
Author: ASF GitHub Bot
Created on: 21/Oct/19 18:28
Start Date: 21/Oct/19 18:28
Worklog Time Spent: 10m
Work Description: udim commented on pull request #9805: [BEAM-8405]
Support embedded Datastore entities
URL: https://github.com/apache/beam/pull/9805#discussion_r337171539
##
File path: sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
##
@@ -47,30 +49,52 @@ def setUp(self):
# Don't do any network requests.
_http=mock.MagicMock())
+ def _assert_keys_equal(self, beam_type, client_type, expected_project):
+self.assertEqual(beam_type.path_elements[0], client_type.kind)
+self.assertEqual(beam_type.path_elements[1], client_type.id)
+self.assertEqual(expected_project, client_type.project)
+
def testEntityToClientEntity(self):
+# Test conversion from Beam type to client type.
k = Key(['kind', 1234], project=self._PROJECT)
kc = k.to_client_key()
-exclude_from_indexes = ('efi1', 'efi2')
+exclude_from_indexes = ('datetime', 'key')
e = Entity(k, exclude_from_indexes=exclude_from_indexes)
-ref = Key(['kind2', 1235])
-e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
+properties = {
+ 'datetime': datetime.datetime.utcnow(),
+ 'key_ref': Key(['kind2', 1235]),
+ 'bool': True,
+ 'float': 1.21,
+ 'int': 1337,
+ 'unicode': 'text',
+ 'bytes': b'bytes',
+ 'geopoint': GeoPoint(0.123, 0.456),
+ 'none': None,
+ 'list': [1, 2, 3],
+ 'entity': Entity(Key(['kind', 111])),
+ 'dict': {'property': 5},
+}
+e.set_properties(properties)
ec = e.to_client_entity()
self.assertEqual(kc, ec.key)
self.assertSetEqual(set(exclude_from_indexes), ec.exclude_from_indexes)
self.assertEqual('kind', ec.kind)
self.assertEqual(1234, ec.id)
-self.assertEqual('kind2', ec['ref'].kind)
-self.assertEqual(1235, ec['ref'].id)
-self.assertEqual(self._PROJECT, ec['ref'].project)
-
- def testEntityFromClientEntity(self):
Review comment:
I merged the two tests together (see "Test reverse conversion" below).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 331571)
Time Spent: 40m (was: 0.5h)
> Python: Datastore: add support for embedded entities
> -
>
> Key: BEAM-8405
> URL: https://issues.apache.org/jira/browse/BEAM-8405
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The conversion methods to/from the client entity type should be updated to
> support an embedded Entity.
> https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[
https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=330686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-330686
]
ASF GitHub Bot logged work on BEAM-8405:
Author: ASF GitHub Bot
Created on: 18/Oct/19 17:49
Start Date: 18/Oct/19 17:49
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #9805:
[BEAM-8405] Support embedded Datastore entities
URL: https://github.com/apache/beam/pull/9805#discussion_r336605472
##
File path: sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
##
@@ -47,30 +49,52 @@ def setUp(self):
# Don't do any network requests.
_http=mock.MagicMock())
+ def _assert_keys_equal(self, beam_type, client_type, expected_project):
+self.assertEqual(beam_type.path_elements[0], client_type.kind)
+self.assertEqual(beam_type.path_elements[1], client_type.id)
+self.assertEqual(expected_project, client_type.project)
+
def testEntityToClientEntity(self):
+# Test conversion from Beam type to client type.
k = Key(['kind', 1234], project=self._PROJECT)
kc = k.to_client_key()
-exclude_from_indexes = ('efi1', 'efi2')
+exclude_from_indexes = ('datetime', 'key')
e = Entity(k, exclude_from_indexes=exclude_from_indexes)
-ref = Key(['kind2', 1235])
-e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
+properties = {
+ 'datetime': datetime.datetime.utcnow(),
+ 'key_ref': Key(['kind2', 1235]),
+ 'bool': True,
+ 'float': 1.21,
+ 'int': 1337,
+ 'unicode': 'text',
+ 'bytes': b'bytes',
+ 'geopoint': GeoPoint(0.123, 0.456),
+ 'none': None,
+ 'list': [1, 2, 3],
+ 'entity': Entity(Key(['kind', 111])),
+ 'dict': {'property': 5},
+}
+e.set_properties(properties)
ec = e.to_client_entity()
self.assertEqual(kc, ec.key)
self.assertSetEqual(set(exclude_from_indexes), ec.exclude_from_indexes)
self.assertEqual('kind', ec.kind)
self.assertEqual(1234, ec.id)
-self.assertEqual('kind2', ec['ref'].kind)
-self.assertEqual(1235, ec['ref'].id)
-self.assertEqual(self._PROJECT, ec['ref'].project)
-
- def testEntityFromClientEntity(self):
Review comment:
We don't need this test anymore ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 330686)
Time Spent: 0.5h (was: 20m)
> Python: Datastore: add support for embedded entities
> -
>
> Key: BEAM-8405
> URL: https://issues.apache.org/jira/browse/BEAM-8405
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The conversion methods to/from the client entity type should be updated to
> support an embedded Entity.
> https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=328871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328871 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 16/Oct/19 00:13 Start Date: 16/Oct/19 00:13 Worklog Time Spent: 10m Work Description: udim commented on issue #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805#issuecomment-542455698 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328871) Time Spent: 20m (was: 10m) > Python: Datastore: add support for embedded entities > - > > Key: BEAM-8405 > URL: https://issues.apache.org/jira/browse/BEAM-8405 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The conversion methods to/from the client entity type should be updated to > support an embedded Entity. > https://github.com/apache/beam/blob/603d68aafe9bdcd124d28ad62ad36af01e7a7403/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L216-L240 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8405) Python: Datastore: add support for embedded entities
[ https://issues.apache.org/jira/browse/BEAM-8405?focusedWorklogId=328869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328869 ] ASF GitHub Bot logged work on BEAM-8405: Author: ASF GitHub Bot Created on: 16/Oct/19 00:11 Start Date: 16/Oct/19 00:11 Worklog Time Spent: 10m Work Description: udim commented on pull request #9805: [BEAM-8405] Support embedded Datastore entities URL: https://github.com/apache/beam/pull/9805 Python SDK already supports keys as property values, add entities as well. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[