diegomrsantos opened a new issue, #4176: URL: https://github.com/apache/iggy/issues/4176
Part of #4174. Affected areas: Iggy server, Performance. `walk_disk_chunk` in [poll_plan.rs](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/partitions/src/poll_plan.rs) decodes a complete batch and verifies checksums when integrity validation is enabled. [select_batch_slice](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/partitions/src/journal.rs) then walks records to select the requested slice. Adjacent small polls can revisit batch data, and locating a late record can require scanning preceding records. Earlier diagnostic profiles placed substantial CPU time in the inclusive checksum and hash category: | Workload | Baseline sampled CPU, µs per poll | Candidate sampled CPU, µs per poll | | --- | ---: | ---: | | Explicit offset, without concurrent writes | 100.474 | 101.810 | | Explicit offset, with concurrent writes | 109.583 | 110.524 | Profiled baseline: `dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e`. Profiled candidate: `a699ff91c6b9126502ea112ac676d96d8d107975`. Values are descriptive arithmetic means of two captures per revision and workload, with 33,000 completed polls per capture and opposite version orders. The workload used one shard, one sequential TCP consumer, explicit offsets, automatic commits disabled, and one message with a 256 byte payload per poll. The optional producer targeted 50,000,000 payload bytes per second in batches of 100. Linux perf sampled software `cpu-clock` at 499 Hz in Docker Desktop's ARM64 Linux VM on an M1 Pro. This category includes called functions and overlaps its callers. It cannot be added to their CPU totals or interpreted as elapsed latency. CPU with writes includes serving the producer. The substantial shared cost does not establish repeated verification, avoidable work, or the cause of the latest performance difference. The parent reproduces shared historical evidence; the detailed diagnostic reports and raw captures remain archived locally. [verify_and_recompute_batch_checksum](https://github.com/apache/iggy/blob/8f1c0e260076f9fbe49539e904e43895bc24c3fe/core/binary_protocol/src/batch.rs) already validates message bodies and recomputes the batch checksum during one frame walk. Removing a supposed duplicate pass over all message bodies is therefore not a supported starting point. The investigation should: 1. Count bytes hashed, records visited, distinct batches and repeated visits per returned message. Attribute current checksum samples to their callers and distinguish hashing from record selection. 2. Vary encoded batch size, requested message count and the requested position within a batch. Determine whether cost comes from verifying a large batch, revisiting the same batch, locating a late record, or the hash implementation. 3. Share read and batch counters with #4175. Keep implementation experiments separate so that changes to read sizing and changes to verification are not confounded. 4. Evaluate one equivalent verification or parsing improvement when the measurements identify material avoidable work. Confirm a component saving through the real poll operation with fresh paired timings, CPU and memory accounting, including larger polls and concurrent writes. Preserve the configured integrity contract, wire checksums, message body validation, malformed frame rejection, offsets, timestamps and contiguous results. Disabling verification changes the contract. Remembering that a file range was once valid is not equivalent to checking newly read bytes: later corruption and path reuse matter. Any retained verified buffer needs bounded memory use, correct identity and explicit invalidation. Completion means a reproducible account of the cost and a decision. Adopt a change only when it preserves these contracts and provides a worthwhile saving through polling. A finding that repeated work is negligible or that a proposed cache is not justified also completes the investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
