rajeshkumarchandolu opened a new pull request, #11995:
URL: https://github.com/apache/ozone/pull/11995

   ## What changes are proposed in this pull request?
   
   A datanode write or read currently shows one span per command, with nothing 
about the chunk or block work behind it or
   which container and block it touched. This PR adds OpenTelemetry coverage 
for the datanode container write and read path
   on top of the tracing helpers merged in HDDS-17581 (#11852). It is part of 
the tracing epic HDDS-13679. The OM, SCM,
   client and Freon changes are in separate PRs.
   
   **Spans added (service `HddsDatanodeService.<datanode uuid>`)**
   
   | Area | Span | How |
   |---|---|---|
   | Chunk manager | `ChunkManagerDispatcher.writeChunk` / `readChunk` / 
`finishWriteChunks`, `FilePerBlockStrategy.writeChunk` / `readChunk` / 
`finishWriteChunks` | `createProxy` |
   | Block manager | `BlockManagerImpl.putBlock` / `getBlock` | `createProxy` |
   | Token verifier | `NoopTokenVerifier.verify` (or the real verifier when 
security is enabled) | `createProxy` |
   | Volume choosing | `CapacityVolumeChoosingPolicy.chooseVolume` | 
`createProxy` |
   | Container create | `HddsDispatcher.createContainer` (auto-create on the 
first write) | local span |
   
   The existing dispatcher spans (`WriteChunk`, `PutBlock`, `ReadChunk`, 
`GetBlock`, and others) keep their names.
   
   **Attributes added**
   
   | Where | Attributes |
   |---|---|
   | Every dispatcher span | `dn.cmd_type`, `hdds.container.id`, 
`hdds.datanode.uuid`, `hdds.datanode.hostname`, `ozone.status` (the container 
`Result`) |
   | `WriteChunk` | `hdds.block.local_id`, `hdds.chunk.offset`, 
`hdds.chunk.length`, `dn.stage` (`WRITE_DATA`, `COMMIT_DATA` or `COMBINED`) |
   | `PutBlock` | `hdds.block.local_id`, `hdds.block.bcs_id` |
   | Container state | `dn.container.state` when a failed write checks the 
container state before marking it unhealthy |
   | Failures | exception recorded as a span event and span status ERROR |
   
   `hdds.block.bcs_id` is recorded in `KeyValueHandler#handlePutBlock` after 
the datanode assigns the block commit sequence ID,
   because the client always sends 0. It therefore shows the real value (2, 6, 
10 and 14 for four blocks in the manual run).
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17594
   
   ## How was this patch tested?
   
   * `mvn clean install -DskipTests -DskipShade -DskipRecon -DskipDocs`: BUILD 
SUCCESS
   * Unit tests with `TestSpanCollector` assertions, 147 run with 0 failures 
(20 skipped by existing assumptions):
     `TestHddsDispatcher` (status, exception, stage, container state, container 
auto-create, BCSID assigned by the
     datanode), `TestKeyValueHandler`, `TestBlockManagerImpl`, 
`TestChunkManagerDummyImpl`
   * `checkstyle.sh`: clean. `rat.sh`: no findings in the changed files
   * Build in the fork: [`build-branch` workflow run on 
`tracing-datanode-spans`](https://github.com/rajeshkumarchandolu/ozone/actions/runs/42643332452)
 (in progress)
   * Manual check in Jaeger, master versus this branch:
   
   The Jaeger runs used the docker compose cluster (`ozone` with 
`monitoring.yaml`, 3 datanodes) and the same
   commands on `origin/master` and on this branch. Screenshots are on the
   
[tracing-proofs](https://github.com/rajeshkumarchandolu/ozone/tree/tracing-proofs)
 branch. Click an image to open it
   at full size.
   
   | Command | Spans on master | Spans with this PR | Jaeger screenshot |
   |---|---|---|---|
   | `ozone sh key put` (4 MB) | 64 | 154 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-side-by-side.png)
 |
   | `ozone sh key get` (4 MB) | 40 | 60 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-side-by-side.png)
 |
   | `ozone fs -put` (4 MB) | 1129 | 1213 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-put-4m-side-by-side.png)
 |
   | `ozone fs -cat` (4 MB) | 2073 | 2849 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/fs-cat-4m-side-by-side.png)
 |
   | `ozone freon ockg -n 3 -s 4MB` | 191 | 443 | 
[before](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-before.png)
 / 
[after](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-after.png)
 / [side by 
side](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/freon-ockg-side-by-side.png)
 |
   
   OM and SCM spans are unchanged by this PR. Commands without datanode I/O 
(volume, bucket, key info, list) are identical.
   
   Each chunk read or write on each replica adds up to three internal spans. 
`ozone fs -cat` reads in small chunks, so the
   datanode part adds about 780 spans for a 4 MB key.
   
   **ozone sh key put: datanode WriteChunk and PutBlock subtree on each 
replica**
   
   [![ozone sh key put: datanode WriteChunk and PutBlock subtree on each 
replica](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-side-by-side.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-put-4m-side-by-side.png)
   
   **ozone sh key get: datanode GetBlock and ReadChunk**
   
   [![ozone sh key get: datanode GetBlock and 
ReadChunk](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-side-by-side.png)](https://raw.githubusercontent.com/rajeshkumarchandolu/ozone/tracing-proofs/datanode/sh-key-get-4m-side-by-side.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to