HesandaLiyanage commented on PR #3193:
URL: https://github.com/apache/james-project/pull/3193#issuecomment-5774373981
Hi @chibenwa and @Arsnael,
All requested changes and feedbacks have been addressed and pushed as
individual atomic commits:
### 1. Structural & Algorithmic Improvements
- **BlobStoreDAO API**: `readRange` now returns `Publisher<Blob>` /
`Mono<Blob>`, allowing `ChunkedBlobStoreDAO` to propagate blob metadata and
slot CRC checks through the blob store hierarchy.
- **Zstd Delegation**: Removed decompression from `BlobSlot` /
`ChunkedBlobStoreDAO`; chunk slots preserve `content-encoding` and
`content-original-size` so `ZstdBlobStoreDAO` handles decompression
transparently.
- **Metadata Preservation**: Chunk format and slot serialization now
preserve and restore all blob metadata in chunk slots.
- **Prefix Pushdown**: Implemented generation prefix pushdown when listing
candidate blobs with `BlobStoreDAO.listBlobs(bucket, prefix)`.
- **Pre-Windowing Size Filtering**: Candidate blobs exceeding
`maxPackableSize` are filtered out prior to windowing and kept as standalone
objects.
- **Size-Based Windowing**: Candidate windowing is now driven by cumulative
byte size up to `chunkTargetSize` rather than pure item count.
- **Single Candidate Handling**: Batches with a single candidate are skipped
without packing into chunks.
- **Dedicated Deletion Method**: Extracted reference replacement and
original blob deletion into `updateReferencesAndDeleteOriginalBlobs`.
- **Streaming References**: Replaced in-memory reference loading with
on-demand windowed queries against `BlobReferenceMappingSource` to avoid heap
exhaustion.
- **Two Distinct Tasks**: Split compaction into `InitialBlobCompactionTask`
(for standalone blobs of a completed generation) and `GCBlobCompactionTask`
(for reclaiming space from dead slots and merging small chunks), each with its
own DTO and WebAdmin scope.
### 2. Guice Dependencies in Non-Cassandra Environments
- Fixed Guice wiring in `BlobCompactionModule` to make `BlobIdUpdater` and
`BlobReferenceMappingSource` optional when non-Cassandra backends (e.g.
Postgres, Memory) are used, resolving the CI failure noted by @Arsnael.
### 3. Cassandra Mailbox Mappers Fallback
- Added `onErrorResume(ObjectNotFoundException.class, ...)` in both
`CassandraMessageMapper` and `CassandraMessageIdMapper` on the
`FetchType.HEADERS` fast path, falling back to `messageDAOV3` which retains
canonical compacted chunk slot references.
### 4. Coherence Under Failure Integration Tests
- Added integration tests in `BlobCompactionAlgorithmTest` verifying store
coherence when reference updates fail during initial compaction and GC
recompaction (original standalone blobs and chunks are preserved, readable, and
consistent).
### 5. Layout Matrix Integration Tests (200 Mails)
- Added `BlobCompactionLayoutIntegrationTest` on `MemoryBlobStoreDAO`
covering 5 distinct layout scenarios (200 mails each):
1. *Uniform small population (all active)*: yields single chunk,
standalone deleted, integrity constraints verified.
2. *Partially unreferenced population (50% unreferenced)*: compacts only
active candidates, preserves uncompacted.
3. *Mixed sizes with oversized blobs*: preserves oversized standalone
blobs, compacts remaining.
4. *Multi-generation interleaved population*: compacts only target
generation blobs.
5. *Two-stage compaction with subsequent deletion and GC recompaction*:
purges dead slots, yields new compact chunk, and ensures non-existent metadata
returns 404.
### 6. Documentation
- Added documentation in `docs/` covering:
- Architecture: `ChunkedBlobStoreDAO` and S3 Object Compaction concepts
(`docs/modules/servers/partials/architecture/blobstore.adoc`).
- WebAdmin: endpoints for initial compaction, GC recompaction, and full
compaction (`docs/modules/servers/partials/operate/webadmin.adoc`).
All changes are completed and tested. Thank you both for the detailed review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
