tuteng commented on a change in pull request #5715: [Issues 5692]Add document
for mongodb connector
URL: https://github.com/apache/pulsar/pull/5715#discussion_r349390966
##########
File path: site2/docs/io-cdc-debezium.md
##########
@@ -305,3 +312,114 @@ This example shows how to change the data of a
PostgreSQL table using the Pulsar
----- got message -----
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":107}}�{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"db"},{"type":"int64","optional":true,"field":"ts_usec"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"string","optional":true,"field":"schema"},{"type":"string","optional":true,"field":"table"},{"type":"boolean","optional":true,"default":false,"field":"snapshot"},{"type":"boolean","optional":true,"field":"last_snapshot_record"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"before":{"id":107,"name":"rocks","description":"box
of assorted
rocks","weight":5.3},"after":{"id":107,"name":"1111111111","description":"box
of assorted
rocks","weight":5.3},"source":{"version":"0.9.2.Final","connector":"postgresql","name":"dbserver1","db":"postgres","ts_usec":1559208957661080,"txId":577,"lsn":23862872,"schema":"inventory","table":"products","snapshot":false,"last_snapshot_record":null},"op":"u","ts_ms":1559208957692}}
```
+## Example of MongoDB
+
+You need to create a configuration file before using the Pulsar Debezium
connector.
+
+* JSON
+
+ ```json
+ {
+ "mongodb.hosts": "rs0/mongodb:27017",
+ "mongodb.name": "dbserver1",
+ "mongodb.user": "debezium",
+ "mongodb.password": "dbz",
+ "mongodb.task.id": "1",
+ "schema.whitelist": "inventory",
+ "pulsar.service.url": "pulsar://127.0.0.1:6650"
+ }
+ ```
+
+* YAML
+
+ You can create a `debezium-postgres-source-config.yaml` file and copy the
[contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/resources/debezium-postgres-source-config.yaml)
below to the `debezium-postgres-source-config.yaml` file.
+
+ ```yaml
+ tenant: "public"
+ namespace: "default"
+ name: "debezium-mongodb-source"
+ topicName: "debezium-mongodb-topic"
+ archive: "connectors/pulsar-io-debezium-mongodb-{{pulsar:version}}.nar"
+ parallelism: 1
+
+ configs:
+
+ ## config for pg, docker image: debezium/example-postgress:0.8
+ "mongodb.hosts": "rs0/mongodb:27017",
+ "mongodb.name": "dbserver1",
+ "mongodb.user": "debezium",
+ "mongodb.password": "dbz",
+ "mongodb.task.id": "1",
+ "schema.whitelist": "inventory",
+
+ ## PULSAR_SERVICE_URL_CONFIG
+ pulsar.service.url: "pulsar://127.0.0.1:6650"
+ ```
+
+### Usage
+
+This example shows how to change the data of a MongoDB table using the Pulsar
Debezium connector.
+
+
+1. Start a PostgreSQL server with a database from which Debezium can capture
changes.
+
+ ```bash
+ $ docker pull debezium/example-mongodb:0.10
+ $ docker run -d -it --rm --name pulsar-mongodb -e MONGODB_USER=mongodb -e
MONGODB_PASSWORD=mongodb -p 27017:27017 debezium/example-mongodb:0.10
+ ```
+
+2. Start a Pulsar service locally in standalone mode.
+
+ ```bash
+ $ bin/pulsar standalone
+ ```
+
+3. Start the Pulsar Debezium connector in local run mode using one of the
following methods.
+
+ * Use the **JSON** configuration file as shown previously.
+
+ Make sure the nar file is available at
`connectors/pulsar-io-mongodb-postgres-{{pulsar:version}}.nar`.
+
+ ```bash
+ $ bin/pulsar-admin source localrun \
+ --archive
connectors/pulsar-io-debezium-postgres-{{pulsar:version}}.nar \
+ --name debezium-mongodb-source \
+ --destination-topic-name debezium-mongodb-topic \
+ --tenant public \
+ --namespace default \
+ --source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name":
"dbserver1","mongodb.user": "debezium","mongodb.password":
"dbz","mongodb.task.id": "1","schema.whitelist":
"inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}'
+ ```
+
+ * Use the **YAML** configuration file as shown previously.
+
+ ```bash
+ $ bin/pulsar-admin source localrun \
+ --source-config-file debezium-mongodb-source-config.yaml
+ ```
+
+4. Subscribe the topic _sub-products_ for the _inventory.products_ table.
+
+ ```
+ $ bin/pulsar-client consume -s "sub-products"
public/default/dbserver1.inventory.products -n 0
+ ```
+
+5. Start a MongoDB client in docker.
+
+ ```bash
+ $ docker exec -it pulsar-mongodb /bin/bash
+ ```
+
+6. A MongoDB client pops out.
+
+ Use the following commands to init the data.
+
+ ``` bash
+ ./usr/local/bin/init-inventory.sh
Review comment:
I think this should be done after the database initialization is completed
because the replication set should be initialized first. After initializing the
replication set, you need to add a DNS resolution of the container name under
your /etc/hosts file, otherwise, the connector cannot find this address. After
the script is initialized, new permissions need to be added. In order to
configure the replication set, root permission can be temporarily used for
testing. You can view the status and configuration of replication sets through
`rs.status()` and `rs.conf()`.
Update permissions:
```
docker exec -it pulsar-mongodb /bin/bash
mongo -u admin -p admin localhost:27017/admin
db.updateUser('admin', { pwd: 'admin', roles: [ {role: "root", db: "admin" }
] });
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
