[
https://issues.apache.org/jira/browse/BEAM-11979?focusedWorklogId=566508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-566508
]
ASF GitHub Bot logged work on BEAM-11979:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Mar/21 19:40
Start Date: 15/Mar/21 19:40
Worklog Time Spent: 10m
Work Description: y1chi opened a new pull request #14237:
URL: https://github.com/apache/beam/pull/14237
…O display data
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
| ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
| ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/)
| --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
| ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/)
<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/)
| [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | --- | --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 566508)
Remaining Estimate: 0h
Time Spent: 10m
> Can't use ReadFromMongoDB with a datetime in filter
> ---------------------------------------------------
>
> Key: BEAM-11979
> URL: https://issues.apache.org/jira/browse/BEAM-11979
> Project: Beam
> Issue Type: Bug
> Components: io-py-mongodb
> Affects Versions: 2.28.0
> Reporter: Gaël
> Assignee: Yichi Zhang
> Priority: P2
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm having an issue while using a filter containing a datetime.
> This filter works directly in pymongo but not in ReadFromMongoDB.
> _BoundedMongoSource.display_data() seems to be the source of the issue.
>
> Fixing the line 267:
> {code:java}
> res['filter'] = json.dumps(self.filter){code}
> by using:
>
> {code:java}
> # from bson import json_util
> res['filter'] = json.dumps(self.filter, default=json_util.default){code}
> Here is an example of my code :
> {code:java}
> import apache_beam as beam
> from apache_beam.io import ReadFromMongoDB
> import datetime
> inputs_query = {"created_at": { "$gte": datetime.datetime.now() } }
> with beam.Pipeline() as p:
> p_inputs = (p | 'Read Mongo Inputs' >> ReadFromMongoDB(uri=mongo_db_uri,
> db=db,
> coll=input_coll,
>
> filter=inputs_query
> )
> | 'Count all Inputs' >> beam.combiners.Count.Globally()
> | 'Print Inputs' >> beam.Map(print)
> )
> {code}
> I get the following error :
> {code:java}
> mongomicrotest.py:19: FutureWarning: ReadFromMongoDB is experimental.
> | 'Print Inputs' >> beam.Map(print)
> Traceback (most recent call last):
> File "mongomicrotest.py", line 19, in <module>
> | 'Print Inputs' >> beam.Map(print)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py",
> line 1058, in __ror__
> return self.transform.__ror__(pvalueish, self.label)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py",
> line 573, in __ror__
> result = p.apply(self, pvalueish, label)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 646, in apply
> return self.apply(transform, pvalueish)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 689, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 188, in apply
> return m(transform, input, options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 218, in apply_PTransform
> return transform.expand(input)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py",
> line 163, in expand
> return pcoll | iobase.Read(self._mongo_source)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pvalue.py",
> line 141, in __or__
> return self.pipeline.apply(ptransform, self)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py",
> line 689, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 188, in apply
> return m(transform, input, options)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py",
> line 218, in apply_PTransform
> return transform.expand(input)
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/iobase.py",
> line 894, in expand
> display_data = self.source.display_data() or {}
> File
> "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py",
> line 267, in display_data
> res['filter'] = json.dumps(self.filter)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/__init__.py",
> line 231, in dumps
> return _default_encoder.encode(obj)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 199, in encode
> chunks = self.iterencode(o, _one_shot=True)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 257, in iterencode
> return _iterencode(o, 0)
> File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py",
> line 180, in default
> o.__class__.__name__)
> TypeError: Object of type 'datetime' is not JSON serializable
> {code}
> Maybe there is a way to correctly pass a datetime in the filter ?
> This is a blocker in our company project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
