nevi_me created BEAM-2639:
-----------------------------
Summary: Unbounded Source for MongoDB
Key: BEAM-2639
URL: https://issues.apache.org/jira/browse/BEAM-2639
Project: Beam
Issue Type: Bug
Components: sdk-java-extensions
Affects Versions: 2.0.0
Reporter: nevi_me
Assignee: Davor Bonaci
Priority: Minor
The current MongoDB source is bounded, which means that we can't build
streaming pipelines directly from MongoDB.
MongoDB publishes changes in each collection through the oplog. Would it be
possible to create a connector that reads the oplog to create an unbounded
source?
As an oplog is only available through replication, this creates that
dependency. We would need to also consider whether a polling method (using the
ObjectId) could be an appropriate fallback.
Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
