[
https://issues.apache.org/jira/browse/BEAM-9245?focusedWorklogId=763928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-763928
]
ASF GitHub Bot logged work on BEAM-9245:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Apr/22 23:14
Start Date: 28/Apr/22 23:14
Worklog Time Spent: 10m
Work Description: codecov[bot] commented on PR #17499:
URL: https://github.com/apache/beam/pull/17499#issuecomment-1112737203
#
[Codecov](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
Report
> Merging
[#17499](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(df1e637) into
[master](https://codecov.io/gh/apache/beam/commit/07f30d221e4b285b23b74c3509d77b62388b7bb4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
(07f30d2) will **decrease** coverage by `0.00%`.
> The diff coverage is `66.66%`.
```diff
@@ Coverage Diff @@
## master #17499 +/- ##
==========================================
- Coverage 73.93% 73.92% -0.01%
==========================================
Files 689 689
Lines 90397 90399 +2
==========================================
- Hits 66832 66831 -1
- Misses 22381 22384 +3
Partials 1184 1184
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.65% <66.66%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click
here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment)
to find out more.
| [Impacted
Files](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
| Coverage Δ | |
|---|---|---|
|
[...python/apache\_beam/io/gcp/datastore/v1new/types.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy90eXBlcy5weQ==)
| `92.98% <66.66%> (-0.77%)` | :arrow_down: |
|
[sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5)
| `91.47% <0.00%> (-0.78%)` | :arrow_down: |
|
[sdks/python/apache\_beam/ml/inference/api.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2FwaS5weQ==)
| `90.00% <0.00%> (-0.48%)` | :arrow_down: |
|
[sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=)
| `93.03% <0.00%> (-0.39%)` | :arrow_down: |
|
[...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==)
| `90.18% <0.00%> (-0.31%)` | :arrow_down: |
|
[sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=)
| `90.27% <0.00%> (-0.15%)` | :arrow_down: |
|
[...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==)
| `93.51% <0.00%> (-0.13%)` | :arrow_down: |
|
[.../python/apache\_beam/ml/inference/sklearn\_loader.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3NrbGVhcm5fbG9hZGVyLnB5)
| `92.68% <0.00%> (ø)` | |
|
[...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==)
| `93.08% <0.00%> (+0.64%)` | :arrow_up: |
|
[...dks/python/apache\_beam/examples/cookbook/coders.py](https://codecov.io/gh/apache/beam/pull/17499/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svY29kZXJzLnB5)
| `63.15% <0.00%> (+0.99%)` | :arrow_up: |
------
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Last update
[07f30d2...df1e637](https://codecov.io/gh/apache/beam/pull/17499?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Read the [comment
docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
Issue Time Tracking
-------------------
Worklog Id: (was: 763928)
Time Spent: 50m (was: 40m)
> Unable to pull datatore Entity which contains dict properties
> -------------------------------------------------------------
>
> Key: BEAM-9245
> URL: https://issues.apache.org/jira/browse/BEAM-9245
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.18.0
> Reporter: Colin Le Nost
> Priority: P3
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hello, we are facing a small bug while reading Datastore entities using
> ReadFromDatastore transform (python SDK, 2.17 & 2.18)
> We are unable to retrieve entities that contain a dictionary. We think there
> is implicit casting from these properties into Datastore entity, but when the
> client is trying to retrieve the entity using the key, it breaks (because
> this entity has no key).
> h2. Stacktrace
> {code:python}
> File
> ".../venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/datastoreio.py",
> line 269, in process
> yield types.Entity.from_client_entity(client_entity)
> File
> ".../venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
> line 225, in from_client_entity
> value = Entity.from_client_entity(value)
> File
> ".../venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
> line 219, in from_client_entity
> Key.from_client_key(client_entity.key),
> File
> ".../venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
> line 156, in from_client_key
> return Key(client_key.flat_path, project=client_key.project,
> AttributeError: 'NoneType' object has no attribute 'flat_path' [while running
> 'Read from datastore/Read']
> {code}
>
> h2. Here is some code to reproduce:
> # Insert a datastore entity using the given function
> # Run the dataflow pipeline using DirectRunner
>
> {code:python}
> import apache_beam as beam
> from google.cloud import datastore
> from apache_beam.io.gcp.datastore.v1new.types import Query
> from apache_beam.io.gcp.datastore.v1new.datastoreio import ReadFromDatastore
> from apache_beam.options.pipeline_options import StandardOptions,
> PipelineOptions
> DATASTORE_KIND = "my_entity_kind"
> PROJECT_ID = "my_project_id"
> def create_datastore_entity():
> client = datastore.Client(PROJECT_ID)
> key = client.key(DATASTORE_KIND, "my_task")
> entity = client.get(key=key)
> if entity is not None:
> raise Exception("Existing entity")
> else:
> entity_dict = {"regular_field": "test", "nested_field": {"field1":
> "my_field1"}}
> entity = datastore.Entity(key=key)
> entity_dict = {k: v for k, v in entity_dict.items()}
> entity.update(entity_dict)
> client.put(entity)
> def my_func(element):
> print(element)
> return element
> def run():
> pipeline_options = PipelineOptions()
> pipeline_options.view_as(StandardOptions).runner = "DirectRunner"
> p = beam.Pipeline(options=pipeline_options)
> my_ds_query = Query(kind=DATASTORE_KIND, project=PROJECT_ID,)
> p | "Read from datastore" >> ReadFromDatastore(
> query=my_ds_query
> ) | "Print entity" >> beam.Map(my_func)
> p.run().wait_until_finish()
> if __name__ == "__main__":
> create_datastore_entity()
> run()
> {code}
> h2.
> Workaround
> Currently, we mocked the library using this code (modifying the Entity class,
> in `sdks/python/apache_beam/io/gcp/datastore/v1new/types.py`, aka this
> [line|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py#L231]
> ).
> {code:python}
> @staticmethod
> def from_client_entity(client_entity):
> res = Entity(
> Key.from_client_key(client_entity.key),
> exclude_from_indexes=set(client_entity.exclude_from_indexes))
> for name, value in client_entity.items():
> if isinstance(value, key.Key):
> value = Key.from_client_key(value)
> if isinstance(value, entity.Entity):
> if value.key:
> value = Entity.from_client_entity(value)
> else:
> value = {k:v for k,v in value.items()}
> res.properties[name] = value
> return res
> {code}
> If the workaround works for you, I can do the PR.
>
> Thanks, Colin
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
