Hi all,
As discussed with Benoit on PR #3193
<https://github.com/apache/james-project/pull/3193> (JAMES-4231 S3 Object
Compaction), the full implementation (~8,000 LOC) is too large for an
effective single review. We are keeping PR #3193 as an open draft
reference and proposing to break the implementation into smaller,
incremental sub-PRs targeted against a dedicated feature branch
(`JAMES-4231`).
Here is the proposed breakdown and order of submission:
1. [DOC] ADR-0076 & Documentation (~270 LOC)
- Documents the chunk file layout, suffix index, virtual addressing
scheme, and operational
semantics. Zero runtime code risk.
2. [API] BlobStoreDAO Range Reads & BlobIdUpdater SPI (~300 LOC)
- Adds `BlobStoreDAO.readRange`, `ChunkMarker` regex validation,
`BlobIdUpdater` interface, and abstract range read contract tests.
3. [BLOB-S3] S3BlobStoreDAO HTTP Byte-Range Reads (~140 LOC)
- Implements single-request ranged read (`Range: bytes=X-Y`) in
S3BlobStoreDAO with Testcontainers verification.
4. [STORAGE-STRATEGY] BloomFilter GC Compaction Exclusion (~50 LOC)
- Configures BloomFilterGCAlgorithm to skip virtual chunk addresses
so valid slots are not purged during unreferenced blob cleanup.
5. [BLOB-COMPACTION] Core Compaction Module (~5,500 LOC, self-contained)
- New `server/blob/blob-compaction` module: ChunkFormat, ChunkId,
ChunkedBlobStoreDAO, BlobCompactionAlgorithm, tasks, and 68 unit/layout
tests. Completely isolated from the running server until wired.
6. [CASSANDRA] CassandraBlobIdUpdater & Fallback Reconciler (~560 LOC)
- Reference updates across Cassandra tables, and self-healing
metadata reconciler upon missing blob 404s.
7. [POSTGRES] PostgresBlobIdUpdater (~330 LOC)
- Keyset-paginated message/attachment table scanning and batched
reference updating for PostgreSQL deployments (PR #3216).
8. [DISTRIBUTED] Guice Wiring, WebAdmin REST Endpoints & MinIO E2E
(~1,100 LOC)
- Exposes `DELETE /blobs?scope=compaction` in WebAdmin, fail-closed
Guice decorator selection, and end-to-end MinIO compaction integration
tests.
Feedback on the ordering or scope of these sub-PRs is welcome!
Best regards,
Hesanda Liyanage
