shink opened a new pull request, #16003: URL: https://github.com/apache/pulsar/pull/16003
### Motivation Now, the MongoDB source connector only supports the incremental message synchronization. This PR adds support for full message synchronization. Since MongDB 4.0, we can set the starting point for the change stream by the `startAtOperationTime` field. So, we can set it to `0` to make start point the earliest. See https://www.mongodb.com/docs/v4.2/reference/method/db.collection.watch/ for more information. ### Modifications 1. Improve config object. There are some commonalities and differences between sink configuration and source configuration. So, I created an abstract class called `MongoAbstractConfig` which contains the commonalities between them. MongoSourceConfig` and `MongoSinkConfig` contain the unique content. 2. Add support for full message synchronization in the source connector. ```java if (SyncType.FULL_SYNC.equals(mongoSourceConfig.getSyncType())) { // sync currently existing messages // startAtOperationTime is the starting point for the change stream // setting startAtOperationTime to 0 means the start point is the earliest // see https://www.mongodb.com/docs/v4.2/reference/method/db.collection.watch/ for more information stream.startAtOperationTime(new BsonTimestamp(0L)); } ``` ### Verifying this change - [ ] Make sure that the change passes the CI checks. ### Does this pull request potentially affect one of the following parts: No. ### Documentation Check the box below or label this PR directly. Need to update docs? - [x] `doc-required` (Your PR needs to update docs and you will update later) - [ ] `doc-not-needed` (Please explain why) - [ ] `doc` (Your PR contains doc changes) - [ ] `doc-complete` (Docs have been already added) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
