PeterZh6 opened a new pull request, #11356: URL: https://github.com/apache/inlong/pull/11356
Fixes [[Feature][Sort] Add new source metrics for sort-connector-mongodb-cdc-v1.15 #11355](https://github.com/apache/inlong/issues/11355) **Coauthored with [yangyang-12-wq](https://github.com/yangyang-12-wq)** ### Motivation The purpose of this PR is to enhance observability for the sort-connector-mongodb-cdc-v1.15 by introducing new source metrics. These metrics will provide detailed insights into the deserialization process and checkpoint management, facilitating better monitoring, troubleshooting, and optimization for users. ### Modifications Basically the same way of modifying as that of #11130 **Deserialization Metrics:** Added counters to track successful and failed deserialization attempts (`numDeserializeSuccess`, `numDeserializeError`). Added latency gauge to measure time taken for deserialization (`deserializeTimeLag`). **SnapshotState Metrics:** Added counters for the number of snapshots created (`numSnapshotCreate`) and errors encountered during snapshot operations (`numSnapshotError`). **NotifyComplete Metrics:** Added a counter to track completed snapshots (`numCompletedSnapshots`). Added latency gauge for the time between snapshot creation and checkpoint completion (`snapshotToCheckpointTimeLag`). ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. - [ ] This change is already covered by existing tests, such as: *(please describe tests)* - [x] This change added tests and can be verified as follows: Can use the same way of verification as that in #11130, an simpler way, however, can be used in the following way. - Use an End-to-End test called `inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Mongodb2StarRocksTest.java`. - Add a while loop after all the checks in `testMongodbUpdateAndDelete` to stop the testing container from being torn down. - Add ``` 'inlong.metric.labels' = 'groupId=pulsarGroup&streamId=pulsarStream&nodeId=pulsarNode' ``` in the source connection option of `inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/resources/flinkSql/pulsar_test.sql`. - Run the maven test. - Wait until all the tests down(should take a while), visit `localhost:8081`, which is the url for `Flink Web Dashboard`. - Click the operator, and check the `metrics` column. The result should be like this: <img width="1280" alt="MongoDBMetrics1" src="https://github.com/user-attachments/assets/ffafe7d7-2b0a-487e-b01c-b969d0adc689";> ### Documentation - Does this pull request introduce a new feature? Yes - If yes, how is the feature documented? Docs in `inlong-website` repo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
