Hello Jean. I think we mostly agree.
> features should declare their own logical bucket Remains a question: where ? Ideally we can maintain a 1 <-> mapping between logical and s3 bucket: alligning the concepts would make things easier I presume. I would be tempted, given this pre-requisite, to answer the above question by "in blob.properties" to keeps things grouped in the configuration. Also, thinking about migration of the configuration, do we attempt to compute the "logical s3 bucket names" from the previous conf format? In which case we mostly transfered today complexity? Or shall we just refuse to stop if those properties are missing, requiring exxplicit user configuration - safer but it puts more weights onto the user... (sorry for the formatting results below - I presume that's a drawback of eating its own dogfood)-- Best regards, Benoit TELLIER General manager of Linagora VIETNAM. Product owner for Twake-Mail product. Chairman of the Apache James project. Mail: [email protected] Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) Le janv. 25, 2026 11:21 AM, de Jean Helou <[email protected]>Hi benoit, The inlined discussion is becoming hard to read so I'll attempt a summary below :D There are 2 different bucket concepts : A logical blobstore bucket and a physical s3 bucket. Both concepts currently overlap in the S3 implementation making the code confusing without extensive analysis. I see this a a barrier to entry for external contributors. Each of the BlobStore usecase can require distinct settings both for physical storage and for deduplication, this is not currently possible for most use cases because many use cases are stored in a single logical (and physical) bucket. It would be beneficial to have separate logical buckets for each use case and allow custom mapping of each use case to physical bucket storage. I think we agree that : - any refactoring should allow to preserve the existing physical storage structure - features should declare their own logical bucket, assemble a BlobStore which uses this bucket, the logical bucket name must be configurable with a default value. This can be done without refactoring the BlobStoreApi - changing the deleted message vault structure goes in the right direction. I'm not as clear as you seem to be about the breadth of the refactoring required, I'll investigate a POC for the mailrepository-s3 to see if I can get a better grasp because I don't envision it as a superhero scale refactoring at the moment so I may be missing something. Jean Le ven. 23 janv. 2026 à 20:48, Benoit TELLIER a écrit : > Hello Jean, > Answer inlined. > -- > > Best regards, > > Benoit TELLIER > > General manager of Linagora VIETNAM. > Product owner for Twake-Mail product. > Chairman of the Apache James project. > > Mail: [email protected] > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > Le janv. 20, 2026 10:48 PM, de Jean Helou Hello, > > An interesting discussion which helps me understand better the state of S3 > storage in james > > This is the case for > > > > - mailbox - RabbitMQ mail queue - Cassandra + Postgres mail > repositories > > - Attachments > > They share the default bucket in order to be deduplicated. > > This cross-feature dedulication is explicitly desired CF > > > github.com/apache/james-project/blob/aef61d6fc9655698a8bff1521398ff110388e2d1/server/apps/distributed-app/src/test/java/org/apache/james/WithCassandraDeduplicationBlobStoreTest.java#L113 > > > > > While it is currently implemented this way, I don't think deduplicating > accross use cases with different storage expectations is such a good idea > mailqueues are transient storage ( a few days ) while mailboxes are long > term storage ( until the heat death of the universe if my old gmail is any > indication ) > mailrepositories I consider transient too but I may be missing use cases. ( > all I can think of is spooling and dead letters/error neither of which I > would expect to store for more than a few weeksOverall +1 > I would go even further: each of those use case may have distinct settings > for instance regarding reduplication. > Deduplication makes sense for mailbox in an enterprise context but not for > mail queue. > > Also there is 6th use case for the DEFAULT BlobStore bucket : the > mailrepository-blob which we introduced with matthieu starting may 2022 > and july 2024 > > > The current choice is either a full s3 bucket per feature or a single s3> > > bucket for all blobstore buckets.> - no blob.properties prefix, james > uses > > one top level buckets per feature> - blob.properties prefix : james > stores > > EVERY blob in a single bucket > > To be faily honest there's nothing harder to migrate than a S3 naming > > layout, I'm reluctant to changes affecting it. > > > > The ADR should aim to guide future S3 naming layouts, we can document > legacy exceptions but ensure our future is brighter. > > > > Furthermore I believe we would be better served being more declarative > and > > have each feature explicitly declare its buckets (and object name > prefix?) > > > > Its S3 bucket or its Blobstore bucket ? ( sorry I'm really asking, this is > not me being pedantic)You are not, I only have a bad answer: both... > At the feature level I would like to speak of james blob store bucket. > And as an ops, if a james bucket is encoded as a S3 bucket I want to > control and be very explicit about the S3 bucket name. > Not sure what the object name prefix you mention is : if there is an object > name prefix wouldn't that defeat deduplication strategies entirely ? > Do you mean introducing a `objectstorage.objectPrefix` that would somehow > interact with `objectstorage.namespace` to allow for nesting our BlobStore > buckets under a single S3 bucket ? > > Changing the meaning of blob.properties objectstorage.bucketPrefix ( > > > github.com/apache/james-project/blob/aef61d6fc9655698a8bff1521398ff110388e2d1/server/apps/distributed-app/sample-configuration/blob.properties#L66 > ) > > would create even more confusion on the topic and so adding yet another > > "prefix" concept in this file. > > > > I understand operating off 3 buckets might not be ideal but it is likely > > to still be very manageable no? > > > Indeed I was, once again, misled by the naming of the configuration > parameters and the unexpected hardcoded naming scheme. And it seems > reviewers were confused enough to let the mailrepository-blob use the > default bucket too even though it doesn't make much sense either. > > There is the "DEFAULT" BlobStore bucket which is named "default" in the > code thus trying to use a top level s3 bucket name "default" out of the > box. This mapping can be controlled by using the `objectstorage.namespace` > property which allows to bind the "DEFAULT" Blobstore bucket to an > arbitrary S3 bucket name. > > There is also an `objectstorage.bucketPrefix` property which has no effect > whatsoever (if I understand correctly) on the mapping of the "DEFAULT" > BlobStore bucket but partially controls the names of the remaining > BlobStore buckets. The `objectstorage.namespace` property has no effect at > all on these buckets ( which still confuses the hell out of me even as I > write this message ). > > Note that the `objectstorage.bucketPrefix` cannot contain a path separator > which is not checked until the application blows in your face because the > S3 connector refuses an invalid bucketname leaving a very confused user > (yes this happened to me as I couldn't give bucket creation rigths to my > james user and had to use pre-existing buckets so I tried a lot of things) > > We have > - `jmap-uploads` which is necessarily a top level S3 bucket ( it can only > be influenced by `objectstorage.bucketPrefix`) I'm not exacly sure what the > upload-{ww} UploadBucketName is but it seems specific to the cassandra > implementation and the name of which cannot be configured apart from the > prefix. > - deleted-messages-{YYYY-MM} which is necessarily a set of dynamic toplevel > buckets possibly with a unique prefix to avoid collision but requiring > elevated application permissions in order to allow creating the buckets > when the time marker overflows. Which better explains the current PR > > I'm not familiar with JMAP uploads so I can only make a guess as to what a > proper storage policy would be: the appended week number suggests that the > objects are rather long lived. In the context of email, the name and long > lived object interrogates as to why we wouldn't want them deduplicated > with, let's say, Attachments as a guess. Something which is effectively > impossible today since we can't configure physical bucket storage per > BlobStoreBucket use case. > > So indeed, today, james has 3 S3 buckets mapped to 3 BlobStore buckets > including the DEFAULT BlobStore bucket which is used by 6 different > usecases. > I haven't checked if there is additional namespacing in the form of virtual > folders to try and avoid collisions between the use cases. I know > mailrepository-blob uses the mailrepository paths as virtual folders but > they are created at the toplevel of the bucket so listing my server's > bucket toplevel I see /var/mail/error which is not so great in hindsight ( > I should probably have prepended a `/mailrepositories` prefix to get > `/mailrepositories/var/mail/error` in order to properly namespace the > feature within the bucket ) > > I have spent too long with security engineers lately and I can't help > wonder how this could be subverted to gain unauthorized access or to let an > attacked destroy data it shouldn't be able to access. > > > A path forward could be to : > - Stop binding to BlobStore.DEFAULT_BUCKET_NAME_QUALIFIER all over the > place and use configuration from each feature tell us what the desired > bucket name should be ( using "default" as default value to maintain > retrocompatibility ) so the guice bindings would target a > MAILREPOSITORY_BUCKET_NAME_QUALIFIER and a > DELETED_MESSAGE_VAULT_BUCKET_NAME_QUALIFIER and put the actual name there, > ready to be overriden by configuration instead of being hardcoded in the > features. > - Implement activable feature based namespacing beyond the S3 bucket name > so each feature can be mixed or isolated through configuration properties > allowing to retain the existing "everything in a single basket" mashup but > opening the door to better isolationOverall +1 > Let me sum it up to see if I understand well: - the common building block > is to be the blob store DAO - blob store (eventually) no longer needs > bucket as an API parameter - each feature needs to assemble it's blob store > (bucket, deduplication, eventually caching, etc...) based on its own > configuration. > > I'm not saying this should all be done in the initial deleted message vaul > PRIf it's deemed a useful improvement for the blob store meta-problem, > let's move this forward. > > > but it could become the conclusion of the ADR and provide a > refactoringguide. As long as the configuration allows to keep using the > existing S3 > naming scheme ( both bucket names and paths ) the nightmare of migrating an > S3 naming layout would remain nicely mitigated in the closet.+1 refactor > and reorganize code but keep on-disk structure. > > What do you think ?Like a boiling frog. This seems like a lot of work for > unwanted complexity we (linagora at least) succeed to live with. I feel > more concerned with the operational pain (bucket count time bomb over my > head) than the logical pain I did get used to.On a personal note I can live > with this code complexity on the very long term. Maybe the massive code > changes are unneeded, and we could simplify this for our users by:1. > Killing the deleted message vault bucket as proposed here 2. Having the few > feature declare bucket they use. And don't attempt a likely to fail > superhero refactoring. > Have a good weekend, > Benoit > > regards, > Jean > > > > Le janv. 19, 2026 12:16 PM, de Jean Helou To sum up > > what my take of the discussions so far : > > > > On the specific case of the deleted message vault, the proposed > > implementation : > > > > Introduce a logical bucket (Blobstore) prefix to the time based > structure. > > this allows storing all deleted messages in a single S3 bucket or > combined > > with the underlying S3 prefix capability having a single toplevel > > "directory" for all these messages. All new messages will be written to > the > > new structure, both the new and old addressing schemes will be readable. > > The old addressing scheme will be announced as deprecated but maintained > > for 5 years ( ~ about 3 or 4 major releases ?) ensuring every user who > > updates on a reasonable timescale will have a proper upgrade. > > The name of the logical blobstore bucket will be configurable, enabling > > users who don't use an S3 prefix to use the bucket of their choice for > this > > use case. > > > > I agree this is better than the current implementation and vote in favor( > > +1 ). > > I think we should add comments to the fallkback specific code to flag the > > expected date of removal in the code, I for one guarantee that I will > > forget about it within such a long time span. > > > > With regards to the ADR : as it is currently worded I stil vote against. > > (-1) since the vote was initiated on both topics my compound vote must be > > -1 at least as it is worded today, > > My position is that : > > - blobstore bucket and s3 bucket should not be conflated > > - each feature should use a unique blobstore bucket > > - how blobstore buckets are stored in s3 buckets should be configurable > > > > I argue that some features share similar storage properties/requirement > and > > could live in a single s3 bucket with a storage class policy while other > > features require different storage properties/requirements and could live > > in different s3 bucket (s) with their own different storage class policy, > > if we are going to have an ADR about bucket usage we should not ignore > > future features. while the fixed set of 3 buckets may be acceptable > today, > > do we really want to force users to configure 10 different buckets in the > > future. > > > > The current choice is either a full s3 bucket per feature or a single s3 > > bucket for all blobstore buckets. > > - no blob.properties prefix, james uses one top level buckets per feature > > - blob.properties prefix : james stores EVERY blob in a single bucket > > > > My proposal is to move forward with the implemetation of the deleted > > message vault but reboot the ADR discussion. > > > > Jean > > > > > > Le ven. 16 janv. 2026 à 09:14, Rene Cordier a > écrit > > : > > > > > Hello, > > > > > > Overall +1 on my side. > > > > > > Regarding the second topic, to add a few more details: technically the > > > blobstore deleted message vault does not change much in the end. There > > > was no new V2 implementation in the end (maybe likely should update the > > > ADR regarding this point). We identified that we could go forward in > the > > > current vault code to add those modifications: > > > > > > - append is new, but we just write into the new single bucket so it > does > > > not collide with the current version. We keep the old code around just > > > for testing (put the tag for it) the fallback, but it's not used > anymore > > > outside of tests. > > > > > > - read and delete: the code does not change! as we do not change the > > > underlying design with metadatas, it actually works for old buckets and > > > the new single one without code mifications there. > > > > > > - purge task: that's where there is an old code / new code mixed > > > together. I did put a deprecated tag on the concerned old part. If we > > > remove it in +2 releases for example that should be alright? Or even > > > more. Honestly the fallback ain't too bad here. If you don't have old > > > buckets the old purge code part will just do nothing! > > > > > > Hope this helps clarify some points :) > > > > > > Regards, > > > > > > Rene. > > > > > > On 1/15/26 22:59, Benoit TELLIER wrote: > > > > Hello Jean > > > > > > > > Topic 1: > > > > > > > > I think an ADR structure rework discussion was started on the mailing > > > list already. > > > > Regarding this very example: > > > > - The principal of operating off a smal number of fixed buckets is > > > likely an architecture decision. > > > > Worth recording in an ADR. It could decide to define an > > architecture > > > principle. > > > > Then in the *context* we could list offending features and > plugins > > > and *consequences* list needed refactorings. > > > > - Having an ADR presenting why we have a plugin to fix a generic > > > problem in the email world seems like a reasonnable choice to me. I > often > > > end up, in architecture discussions (and no later than yesterday!) to > get > > > discussion on this topic. Hence it feels justified to get a track > record. > > > > > > > > Also this seems dangerous to me to say "not relevant, that's a > plugin" > > > as everything (the mailbox, the event-bus, etc) is somehow a plugin > with > > > guice binding and concern specific implementations and not others. > > > > > > > > Topic 2: Answers inlined. > > > > > > > > -- > > > > > > > > Best regards, > > > > > > > > Benoit TELLIER > > > > > > > > General manager of Linagora VIETNAM. > > > > Product owner for Twake-Mail product. > > > > Chairman of the Apache James project. > > > > > > > > Mail: [email protected] > > > > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > > > > > > > > > > Le janv. 15, 2026 4:13 PM, de Jean Helou Since > you > > > called a formal vote I must vote -1 I may change my vote after > > > > the discussion has taken place. > > > > > > > > 2 topics on which I would like to see some discussion before voting : > > > > > > > > Topic 1 Why an ADR for an implementation detail in a plugin ? > > > > Maybe I'm just too new to james and/or need a refresher in what james > > > > considers and architectural decision. > > > > It feels to me that this should be part of the README for the > plugin. I > > > see > > > > that introducing the feature (the deleted message store) was also > done > > > > through an ADR which seems a bit suprising to me. > > > > I dont disagree on the content of the files themselves (thought the > ADR > > > > formalism is a bit weird for a README), it just doesn't match with my > > > > understanding of what ADRs are. > > > > I personnally would prefer to merge the content of both files in a > > README > > > > file on the plugin itself (to document the why and the implementation > > > > details of the plugin) and list the plugin out somewhere in the > > > > documentation ( it probably already is) to let people know the > feature > > > > exist and where to find details about it. > > > > > > > > On the plugin evolution itself > > > > The currenty behaviour is to create a full s3 bucket following the > > > pattern > > > > `deleted-messages-[year]-[month]-01` (it is unclear to me what the 01 > > > > represents, is it the day ?) > > > > > > > > Means first day of the month and was adopted to get something that > > looks > > > like a date if I recall well/ > > > > > > > > > > > > > > > > > > > > According to the change proposed in the "ADR", the plugin would now > > store > > > > the same contents under a virtual path within a single bucker ` > > > > [year]/[month]/[blob_id]` > > > > The proposed migration strategy is : > > > > - - write only on the single bucket, fall back if necessary on old > > > buckets > > > > for read and delete > > > > - - add the single bucket usage case to the purge task, that would do > > > > cleaning on both new and old buckets. > > > > I think we should work out right now for how long we intend to > maintain > > > the > > > > fallback and old behaviour in the clean task > > > > > > > > > > > > +1 typical retention is 1 year so going for 2 years seems reasonable. > > > Likely a bit more to further ease the upgrade path. > > > > > > > > Probably also clearly communicate (changelog comes to mind) on the > > > > deprecation of the old behaviour so we can eventually remove both the > > > > fallback and the cleaning task. > > > > > > > > +1 > > > > > > > > However this has consequences on users ugrade path. As proposed they > > will > > > > have to install a version which has both the new behaviour and the > > > fallback > > > > for at least the retention period of all the deleted messages in > their > > > > system. > > > > > > > > Would you consider providing a migration tool that allows them to > move > > > > their deleted messages to the new scheme and fast forward on the > > > versions ? > > > > (maybe even skip as each versions embarks all the migrations from the > > > > previous versions IIRC) > > > > > > > > > > > > To be fair given it's simplicity I'd rather support the fallback 5 > > years > > > rather than moving blobs around. Personal taste. > > > > > > > > > > > > I was away from home and had a small car crash so I didn't have time > to > > > > look into 2902 yet. I had a quick look while writing this message > and I > > > was > > > > suprised to see an API change that affects the cassandra > implementation > > > and > > > > introduces something similar to a blobid factory (and used as such) > but > > > > with a different type. I left a comment to that effect and will > > continue > > > > the review (probably tomorrow or during the weekend) > > > > > > > > I saw it and I will craft something around it, it seems like a > relevant > > > remark. > > > > > > > > > > > > jean > > > > > > > > Le jeu. 15 janv. 2026 à 14:23, Benoit TELLIER > > > a > > > > écrit : > > > > > > > >> Hi all, > > > >> > > > >> > > > >> > > > >> > > > >> I would like to call a vote on the following change: > > > >> > > > >> - github.com/apache/james-project/pull/2894 ADR: Deleted message > > > >> vault single bucket usage > > > >> > > > >> - github.com/apache/james-project/pull/2902 which is the > > > >> implementation of the aforementioned ADR > > > >> > > > >> > > > >> The use of monthly buckets is problematic with certain S3 suppliers > > that > > > >> limit their count, and require extensive rights onto the object > store > > > >> endpoints. > > > >> The proposal would address solely this problematic feature and not > > > attempt > > > >> to refactor in depth the way James' bucket are encoded onto the S3 > > > endpoint. > > > >> > > > >> (More context is provided onto the relevant ADR) > > > >> > > > >> > > > >> > > > >> This vote is open for at least 72 hours and requires a simple > > majority. > > > >> > > > >> > > > >> Please vote: > > > >> > > > >> > > > >> - +1 approve > > > >> > > > >> > > > >> - 0 no opinion > > > >> > > > >> > > > >> - -1 disapprove (please explain) > > > >> > > > >> > > > >> > > > >> Thanks, > > > >> > > > >> -- > > > >> > > > >> Best regards, > > > >> > > > >> Benoit TELLIER > > > >> > > > >> General manager of Linagora VIETNAM. > > > >> Product owner for Twake-Mail product. > > > >> Chairman of the Apache James project. > > > >> > > > >> Mail: [email protected] > > > >> Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > >> > > > >> > > > >> > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > >
