rajeshkumarchandolu opened a new pull request, #11993: URL: https://github.com/apache/ozone/pull/11993
## What changes are proposed in this pull request? SCM shows up in a trace only as one server span for a block allocation, so the time spent choosing a container, creating a pipeline or committing through SCM Ratis is invisible. This PR adds OpenTelemetry coverage for SCM on top of the tracing helpers merged in HDDS-17581 (#11852). It is part of the tracing epic HDDS-13679. The OM, datanode, client and Freon changes are in separate PRs. **Spans added (service `StorageContainerManager`)** | Area | Span | How | |---|---|---| | Block protocol | `SCMBlockProtocolServer.allocateBlock` | `createProxy` | | Container protocol | `SCMClientProtocolServer.getContainerWithPipeline` / `getContainerWithPipelineBatch` / `listContainer` | `createProxy` | | Block manager | `BlockManagerImpl.allocateBlock` (one per allocated block) | `createProxy` | | Writable containers | `WritableRatisContainerProvider.getContainer` | `createProxy` | | SCM Ratis | `SCMRatisServerImpl.submitRequest` | `createProxy` | **Attributes and events added** | Where | Attributes | |---|---| | SCM server span (`AllocateScmBlock`, `GetContainerWithPipeline`, `GetContainerWithPipelineBatch`, `ListContainer`, other request types) | `scm.cmd_type`, `ozone.status` (set when the request fails) | | `SCMBlockProtocolServer.allocateBlock` | `size`, `num`, `replication`, `allocated` | | `BlockManagerImpl.allocateBlock` | `hdds.container.id`, `hdds.block.local_id`, `pipeline.id` | | `WritableRatisContainerProvider.getContainer` | `scm.pipeline.created` when SCM creates a new pipeline | | Failures | exception recorded as a span event and span status ERROR, event `scm.allocate.failed` with the replication when no container is found | ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17593 ## How was this patch tested? * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD SUCCESS * Unit tests with `TestSpanCollector` assertions, 168 tests in 6 classes pass: `TestBlockManager`, `TestWritableRatisContainerProvider`, `TestScmBlockLocationProtocolServerSideTranslatorPB`, `TestStorageContainerLocationProtocolServerSideTranslatorPB`, `TestSCMBlockProtocolServer`, `TestSCMClientProtocolServer` * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files * Build in the fork: [`build-branch` workflow run on `tracing-scm-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42641847069) (in progress) * Manual check in Jaeger, master versus this branch: The Jaeger runs used the docker compose cluster (`ozone` with `monitoring.yaml`, 3 datanodes) and the same commands on `origin/master` and on this branch. Screenshots are on the [tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs) branch. Click an image to open it at full size. | Command | Spans on master | Spans with this PR | Jaeger screenshot | |---|---|---|---| | `ozone sh key put` (4 MB) | 64 | 77 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-put-4m-side-by-side.png) | | `ozone sh key get` (4 MB) | 40 | 41 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-get-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-get-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-get-4m-side-by-side.png) | | `ozone fs -put` (4 MB) | 1129 | 1142 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/fs-put-4m-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/fs-put-4m-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/fs-put-4m-side-by-side.png) | | `ozone freon ockg -n 3 -s 4MB` | 191 | 218 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/freon-ockg-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/freon-ockg-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/freon-ockg-side-by-side.png) | | `ozone admin container list` | 4 | 5 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-list-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-list-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-list-side-by-side.png) | | `ozone admin container info 999999` | 4 | 5 | [before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-info-missing-before.png) / [after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-info-missing-after.png) / [side by side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-info-missing-side-by-side.png) | OM and datanode spans are unchanged by this PR. Commands that do not call SCM, for example `ozone sh volume create`, have the same spans as on master. **ozone sh key put: SCM block allocation under AllocateScmBlock** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/sh-key-put-4m-side-by-side.png) **ozone admin container list** [](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/scm/admin-container-list-side-by-side.png) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
