[
https://issues.apache.org/jira/browse/BEAM-11979?focusedWorklogId=566527&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-566527
]
ASF GitHub Bot logged work on BEAM-11979:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Mar/21 20:33
Start Date: 15/Mar/21 20:33
Worklog Time Spent: 10m
Work Description: codecov[bot] edited a comment on pull request #14237:
URL: https://github.com/apache/beam/pull/14237#issuecomment-799709508
# [Codecov](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=h1) Report
> Merging
[#14237](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=desc) (7aea04f)
into
[master](https://codecov.io/gh/apache/beam/commit/2d9c66647ecbc9aca6a8618168d44e4c815cea19?el=desc)
(2d9c666) will **increase** coverage by `0.14%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14237 +/- ##
==========================================
+ Coverage 83.04% 83.18% +0.14%
==========================================
Files 469 469
Lines 58472 58989 +517
==========================================
+ Hits 48556 49071 +515
- Misses 9916 9918 +2
```
| [Impacted
Files](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=tree) | Coverage
Δ | |
|---|---|---|
|
[...sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS91dGlscy9pbnRlcmFjdGl2ZV91dGlscy5weQ==)
| `92.85% <0.00%> (-2.39%)` | :arrow_down: |
|
[.../internal/clients/dataflow/dataflow\_v1b3\_client.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9ydW5uZXJzL2RhdGFmbG93L2ludGVybmFsL2NsaWVudHMvZGF0YWZsb3cvZGF0YWZsb3dfdjFiM19jbGllbnQucHk=)
| `71.34% <0.00%> (-1.47%)` | :arrow_down: |
|
[...srcs/sdks/python/apache\_beam/coders/avro\_record.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9jb2RlcnMvYXZyb19yZWNvcmQucHk=)
| `88.88% <0.00%> (-1.12%)` | :arrow_down: |
|
[...cs/sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9kYXRhZnJhbWUvZnJhbWVfYmFzZS5weQ==)
| `89.65% <0.00%> (-0.85%)` | :arrow_down: |
|
[...on/apache\_beam/runners/portability/spark\_runner.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9ydW5uZXJzL3BvcnRhYmlsaXR5L3NwYXJrX3J1bm5lci5weQ==)
| `55.81% <0.00%> (-0.71%)` | :arrow_down: |
|
[...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9jb2RlcnMvY29kZXJzX3Rlc3RfY29tbW9uLnB5)
| `98.31% <0.00%> (-0.59%)` | :arrow_down: |
|
[.../srcs/sdks/python/apache\_beam/dataframe/convert.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9kYXRhZnJhbWUvY29udmVydC5weQ==)
| `90.36% <0.00%> (-0.55%)` | :arrow_down: |
|
[...on/apache\_beam/runners/portability/flink\_runner.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9ydW5uZXJzL3BvcnRhYmlsaXR5L2ZsaW5rX3J1bm5lci5weQ==)
| `61.11% <0.00%> (-0.30%)` | :arrow_down: |
|
[...on/apache\_beam/runners/dataflow/dataflow\_runner.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93X3J1bm5lci5weQ==)
| `81.63% <0.00%> (-0.26%)` | :arrow_down: |
|
[.../srcs/sdks/python/apache\_beam/coders/coder\_impl.py](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0NvbW1pdC9zcmMvc2Rrcy9weXRob24vdGVzdC1zdWl0ZXMvdG94L3B5MzgvYnVpbGQvc3Jjcy9zZGtzL3B5dGhvbi9hcGFjaGVfYmVhbS9jb2RlcnMvY29kZXJfaW1wbC5weQ==)
| `95.30% <0.00%> (-0.23%)` | :arrow_down: |
| ... and [50
more](https://codecov.io/gh/apache/beam/pull/14237/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=continue).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=footer). Last
update
[1f35e2f...e2e0f11](https://codecov.io/gh/apache/beam/pull/14237?src=pr&el=lastupdated).
Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 566527)
Time Spent: 1h 20m (was: 1h 10m)
> Can't use ReadFromMongoDB with a datetime in filter
> ---------------------------------------------------
>
> Key: BEAM-11979
> URL: https://issues.apache.org/jira/browse/BEAM-11979
> Project: Beam
> Issue Type: Bug
> Components: io-py-mongodb
> Affects Versions: 2.28.0
> Reporter: Gaël
> Assignee: Yichi Zhang
> Priority: P2
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> I'm having an issue while using a filter containing a datetime.
> This filter works directly in pymongo but not in ReadFromMongoDB.
> _BoundedMongoSource.display_data() seems to be the source of the issue.
>
> Fixing the line 267:
> {code:java}
> res['filter'] = json.dumps(self.filter){code}
> by using:
>
> {code:java}
> # from bson import json_util
> res['filter'] = json.dumps(self.filter, default=json_util.default){code}
> Here is an example of my code :
> {code:java}
> import apache_beam as beam
> from apache_beam.io import ReadFromMongoDB
> import datetime
> inputs_query = {"created_at": { "$gte": datetime.datetime.now() } }
> with beam.Pipeline() as p:
> p_inputs = (p | 'Read Mongo Inputs' >> ReadFromMongoDB(uri=mongo_db_uri,
> db=db,
> coll=input_coll,
>
> filter=inputs_query
> )
> | 'Count all Inputs' >> beam.combiners.Count.Globally()
> | 'Print Inputs' >> beam.Map(print)
> )
> {code}
> I get the following error :
> {code:java}
> mongomicrotest.py:19: FutureWarning: ReadFromMongoDB is experimental.
> | 'Print Inputs' >> beam.Map(print)
> Traceback (most recent call last):
> File "mongomicrotest.py", line 19, in <module>
> | 'Print Inputs' >> beam.Map(print)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py",
> line 1058, in __ror__
> return self.transform.__ror__(pvalueish, self.label)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py",
> line 573, in __ror__
> result = p.apply(self, pvalueish, label)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 646, in apply
> return self.apply(transform, pvalueish)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 689, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 188, in apply
> return m(transform, input, options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 218, in apply_PTransform
> return transform.expand(input)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py",
> line 163, in expand
> return pcoll | iobase.Read(self._mongo_source)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pvalue.py",
> line 141, in __or__
> return self.pipeline.apply(ptransform, self)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 689, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 188, in apply
> return m(transform, input, options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 218, in apply_PTransform
> return transform.expand(input)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/iobase.py",
> line 894, in expand
> display_data = self.source.display_data() or {}
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py",
> line 267, in display_data
> res['filter'] = json.dumps(self.filter)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/__init__.py",
> line 231, in dumps
> return _default_encoder.encode(obj)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 199, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 257, in iterencode
> return _iterencode(o, 0)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 180, in default
> o.__class__.__name__)
> TypeError: Object of type 'datetime' is not JSON serializable
> {code}
> Maybe there is a way to correctly pass a datetime in the filter ?
> This is a blocker in our company project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
