spetz opened a new pull request, #4146: URL: https://github.com/apache/iggy/pull/4146
A persisted topic drove far more disk traffic than the bytes it ingested, and the excess barely moved when the ingest rate rose. The cost was per acknowledgment, not per byte. Most of it was not the prepare bytes. Publishing the WAL frontier created a temporary inode, wrote one block into it, fdatasynced it, renamed it over the frontier and fsynced the directory, once per batch. On a journaling filesystem every acknowledgment paid metadata transactions for that lifecycle, and it was the fourth serialized barrier in a path that needed two. The frontier is now two slots in one file, published by overwriting the older slot in place: no create, no rename, no directory barrier. A torn slot fails its checksum and its partner holds the previous publication, which is what the rename guaranteed. An unreadable newest slot is the case that loses, so recovery then walks past the frontier it could read and adopts the records already on disk, refusing rather than truncating when any of them fails to verify. The body and WAL barriers now overlap, covering different inodes. Group commit is bounded by arrival instead of a byte budget charged against body bytes the WAL does not write. The index fdatasync moves off the flush path onto the seal and checkpoint barriers that WAL reclamation already requires. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
