Re: [ext] Re: The future of SIS
* Pedro Ribeiro via dovecot : > Hello everyone! > I'm reviving the topic just to add that after reconstructing our storage with > SIS disabled the occupied space increased from 5.3TB to 9.6TB, almost > doubling! > It's a feature promoting storage efficiency, I think it demands some > ponderation the advantages of keeping or improving the module. Amen to that. I think the deduplication is a useful, storage efficient way -- especially in large environments with lots of users (who get the same mails :) ) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | https://www.charite.de smime.p7s Description: S/MIME cryptographic signature ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Hello everyone! I'm reviving the topic just to add that after reconstructing our storage with SIS disabled the occupied space increased from 5.3TB to 9.6TB, almost doubling! It's a feature promoting storage efficiency, I think it demands some ponderation the advantages of keeping or improving the module. I just have to thank all the people involved in this software for the great work and contribution to the community, with or without SIS in the future! regards. On 16/10/2023 14:00, Chris Candreva wrote: On Mon, 16 Oct 2023, Marc wrote: Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication. In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful. -- Com os melhores cumprimentos, [https://www.net.ipl.pt/ipl_img/logo_ipl_email.png] Pedro Ribeiro Departamento_de_Sistemas_de_Informação_e_Comunicações_-_IPLNet Serviços da Presidência Telf. +351 210 464 700 / Ext. 80101 smime.p7s Description: S/MIME Cryptographic Signature ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> On 17/10/2023 03:26 EEST Jan Bramkamp wrote: > > > On 16.10.23 13:17, Pedro Ribeiro via dovecot wrote: > > Hello to everyone! > > Ooops, we are using SIS, guess the solution for a similar optimization will > > be > > a native deduplicated filesystem. > > A block level de-duplicating filesystem can only deduplicate data that > is aligned to block boundaries. E-mail attachments tend to move around > in to a different alignment in each copy stored into a different > mailbox. Unless the storage format is designed to split off the > attachments into files there is not much to be gained by block level > dedup. So for the foreseeable future I'll have to stay off Dovecot 3.x > or add four to five times more storage to both my IMAP servers since my > users love to send big documents to multiple recipients. > > Is this an attempt to figuring out the pain tolerance of existing users > before they fork the project or pay up the Danegeld? > SIS won't be available even if you paid up the Danegeld. You can use mail_attachment_fs with posix driver. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
On 16.10.23 13:17, Pedro Ribeiro via dovecot wrote: Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem. A block level de-duplicating filesystem can only deduplicate data that is aligned to block boundaries. E-mail attachments tend to move around in to a different alignment in each copy stored into a different mailbox. Unless the storage format is designed to split off the attachments into files there is not much to be gained by block level dedup. So for the foreseeable future I'll have to stay off Dovecot 3.x or add four to five times more storage to both my IMAP servers since my users love to send big documents to multiple recipients. Is this an attempt to figuring out the pain tolerance of existing users before they fork the project or pay up the Danegeld? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
On 2023-10-18 3:35 a.m., Marc wrote: Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem. Interesting. How do you tell dovecot to do that ? I thought I read about something like this, mail_location = ATTACHMENTS=/attachment but now you have made me read the docs[1] I can't really find it. @Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative. https://doc.dovecot.org/settings/core/#core_setting-mail_attachment_dir I hope this option will not be obsoleted. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> Le 18 oct. 2023 à 09:35, Marc a écrit : > >> Dovecot has this option to store attachments separately not? So I am >> not sure this is then still a problem. >> >> >> >> Interesting. How do you tell dovecot to do that ? >> > > I thought I read about something like this, > > mail_location = ATTACHMENTS=/attachment > > but now you have made me read the docs[1] I can't really find it. > > @Aki maybe if this SIS is phased out, it is good to offer a solution that > stores the attachments separately? I think that would allow current SIS users > to implement something alternative. > Thanks for the pointer. Thanks to it, I found it in the documentation. It was supposed to be defined like this in v2.0.0, but it is now a core setting (and is only available for sd/mdbox storage): mail_attachment_dir • Default: • Values: String The directory in which to store mail attachments. With sdbox and mdbox, mail attachments can be saved to external files, which also allows single-instance storage of them. If no value is specified, attachment saving to external files is disabled. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> Dovecot has this option to store attachments separately not? So I am > not sure this is then still a problem. > > > > Interesting. How do you tell dovecot to do that ? > I thought I read about something like this, mail_location = ATTACHMENTS=/attachment but now you have made me read the docs[1] I can't really find it. @Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative. [1] https://doc.dovecot.org/configuration_manual/mail_location/# ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Le 17 oct. 2023 à 16:34, Marc a écrit : The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication. [1] https://docs.ceph.com/en/reef/dev/deduplication/ Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective. Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem. Interesting. How do you tell dovecot to do that ? Is it really worse bothering deploying a whole Ceph cluster for that ? No you should not get ceph just for this. But ceph brings you nice redundancy, distributed storage. I am totally fan of it. Me too. I’m using it extensively to store multi terabytes of data, but it may be overkill if you don’t need all of this. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
--- Original Message --- On Tuesday, October 17th, 2023 at 15:27, Filip Hanes via dovecot wrote: > Other S3 implementation is Minio on top of any posix filesystem - you can > choose which fills your needs. Minio is great in general, the only thing I would say it its a little bit weird to setup if you're in a VM environment. It was really based around physical hosts, so you need to replicate that on VMs (i.e. 3 x virtual disks per VM so that the error encoding stuff works just like it would on physical hosts). But certainly compared to Ceph its a lot easier on the sysadmin side ! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > The problem is a bit what everyone understands as s3. I associate > this indeed also with an http endpoint on object storage. But the ceph > plugin skips this http and talks directly to object store. I don't think > you would like to operate on this http level. If I look at this page of > ceph[1], it also looks like you do not want to get yourself involved in > deduplication. > > [1] > https://docs.ceph.com/en/reef/dev/deduplication/ > > > > > Moreover, following Filip remark about block deduplication, having any kind > of deduplication that is not optimized for the email case (where > attachments are always embed in slightly different documents) would make it > ineffective. Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem. > Is it really worse bothering deploying a whole Ceph cluster for that ? > No you should not get ceph just for this. But ceph brings you nice redundancy, distributed storage. I am totally fan of it. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Le 17 oct. 2023 à 13:12, Marc a écrit : Is s3 not to slow for this? I think the clue is in the name "s3- compatible". Clearly calling out to "real" (AWS) S3 would be a non-starter. But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably? I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency. S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage. The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication. [1] https://docs.ceph.com/en/reef/dev/deduplication/ Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective. Is it really worse bothering deploying a whole Ceph cluster for that ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > 17.10.2023 12:22, Filip Hanes via dovecot пишет: > > S3-compatible storage is very good for multi-server installations where > you need redundancy, availability. S3 is basically HTTP server so you can > code your own logic on stored emails, balancers, caches, deduplication, > compression, encryption it does't need to be off-the-shelf storage. > > > is S3 better then cephfs? > The drawbacks of cephfs is you need to have the mds. If you scale the mds you could have some issues. I think even in newer ceph releases you need to start pin them on pools / directories. I still have issues with cephfs mounts locking up on a hyperconverged setup so I am not using it in production, but I am still on a older version. The flip side to using the mds, is that it is caching a lot of meta data so in theory you could have a better performance with cephfs than writing directly to rados. Writing to rados directly seems to me the most stable. What I thought was super strange about the s3/radowsgw layer is that if you rename a file, the file is actually copied to a new name. It is not renamed. I am not sure if this is a standard and still like this, but s3 is just developed for a different use. So it depends on how you use s3/radosgw, object storage directly or cephfs. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
17.10.2023 12:22, Filip Hanes via dovecot пишет: S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the- shelf storage. is S3 better then cephfs? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > > > If you are using Ubuntu, OpenZFS is readily available, and support > deduplication natively. > > > I thought nobody sane actually used ZFS dedup because it eats RAM for > breakfast, lunch and dinner ? > What an interesting and informing reading lately!! Thanks everyone!! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> >>> > Is s3 not to slow for this? > > >>> I think the clue is in the name "s3-compatible". > >>> > >>> Clearly calling out to "real" (AWS) S3 would be a non-starter. > >>> > >>> But a local installation of something like CEPH, MinIO or whatever on > the > >>> same LAN ? I'd think that should be workable, no ? > >> Do you know of anything that does this reliably? > >> > >> I tested a few years ago with ceph[1] but at that point there was some > issues where it had a 2x write applification (on top of the 3x) if I > remember correctly. > > All of this is if not dead end will be a lots of complexity and > inefficiency and a lot of waste of money. Only the application know how to > things efficiently and with consistency. > > S3-compatible storage is very good for multi-server installations where you > need redundancy, availability. S3 is basically HTTP server so you can code > your own logic on stored emails, balancers, caches, deduplication, > compression, encryption it does't need to be off-the-shelf storage. The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication. [1] https://docs.ceph.com/en/reef/dev/deduplication/ ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
--- Original Message --- On Tuesday, October 17th, 2023 at 06:46, Jean-Daniel Dupas wrote: > > If you are using Ubuntu, OpenZFS is readily available, and support > deduplication natively. I thought nobody sane actually used ZFS dedup because it eats RAM for breakfast, lunch and dinner ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> Day 16. 10. 2023 21:30, Emmanuel Fusté wrote: > > Le 16/10/2023 à 19:44, Marc a écrit : >>> Is s3 not to slow for this? >>> I think the clue is in the name "s3-compatible". >>> >>> Clearly calling out to "real" (AWS) S3 would be a non-starter. >>> >>> But a local installation of something like CEPH, MinIO or whatever on the >>> same LAN ? I'd think that should be workable, no ? >> Do you know of anything that does this reliably? >> >> I tested a few years ago with ceph[1] but at that point there was some >> issues where it had a 2x write applification (on top of the 3x) if I >> remember correctly. > All of this is if not dead end will be a lots of complexity and inefficiency > and a lot of waste of money. Only the application know how to things > efficiently and with consistency. S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> Day 17. 10. 2023 7:46, Jean-Daniel Dupas wrote: > > If you are using Ubuntu, OpenZFS is readily available, and support > deduplication natively. > Else it is also available on other platforms, but may require more setup. Filesystems does not have deduplication effective for emails. They mostly use block based deduplication, which does not work when the same attachment in second body is shifted even one byte, like when any header value is longer/shorter. Deduplication using rolling hash (rsync) is better but still different clients can encode same part in different ways. Even dovecot SIS is not very sophisticated in this, but it is better than block based deduplication. Good deduplication can lower storage to almost 1/3 of full email bodies. -- Filip Hanes ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> Le 16 oct. 2023 à 15:51, Marc a écrit : > >>> Hello to everyone! >>> Ooops, we are using SIS, guess the solution for a similar optimization >> will be >>> a native deduplicated filesystem. >> >> did you really mean deduplicated or distributed? >> > > I think this duduplicating. Storage systems are offering such solutions. I > think ceph has something like this, although I am not sure for rbd disk > images. I think it makes more sense to have something like this done by a fs > or storage solution. If you are using Ubuntu, OpenZFS is readily available, and support deduplication natively. Else it is also available on other platforms, but may require more setup. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Le 16/10/2023 à 19:44, Marc a écrit : Is s3 not to slow for this? I think the clue is in the name "s3-compatible". Clearly calling out to "real" (AWS) S3 would be a non-starter. But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably? I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency. The actual system is not perfect, but simple and robust and there is room for improvements. Dovecot is a very solid base. The replicator removing is sad, but aligned with the business pressure open-xchange business: Their customers are only big players all-ready wasting lots of money on "enterprise grade" architecture and big and expensive cloud services. It is perfectly aligned with their customer view of efficiency... At the same time, this customers have a very distorted view of the cost of pure service. They want a price at the lowest common denominator of human cost over the planet... I think that open-xchange only try to off cut all possible maintenance cost from the "product". It it too much ? Will it pay off ? Who live will see... From over more than thirty years of experience of observing the open source world, projects that achieve "world domination" are NEVER the ones that sacrifices the technical goal for business reasons/pressure even if it take decades to be recognized for it true value. Now that Exchange has a robust multinode replication and distribution system dovecot will loose it own. Ironic isn't it :-) Emmanuel. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > Is s3 not to slow for this? > > > > I think the clue is in the name "s3-compatible". > > Clearly calling out to "real" (AWS) S3 would be a non-starter. > > But a local installation of something like CEPH, MinIO or whatever on the > same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably? I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. [1] https://github.com/ceph-dovecot/dovecot-ceph-plugin ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> Is s3 not to slow for this? > I think the clue is in the name "s3-compatible". Clearly calling out to "real" (AWS) S3 would be a non-starter. But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > Interesting, nice they use this rust, I am curious how they define this > scaling. What I don't get is why are they messing with smtp. I always get a > bad feeling when a company is trying to do everything. Good they are using rust and even better they've had an independent security audit (https://www.stalw.art/blog/security-audit). On the scaling side, maybe see the storage page ? (https://www.stalw.art/docs/storage/overview). The metadata is stored in a database which can be replicated. And the mails themselves can be stored in filesystem or "S3-compatible" storage, and so there are scaling options there too ? But clearly some experimentation is required to see how it works in practice. Are they messing with SMTP ? As I understand it its an IMAP/JMAP server. And (like Dovecot) it has LMTP for getting mail into it from e.g. Postfix ? From my reading of the docs it looks like SMTP is only there if you don't want to use LMTP to get mail into it ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> > Well, so Laura is absolutely right ... > > > "Things like dsync will be GONE in the community version." > > That's not right, dsync is still there. Replicator is not, so dsync can't be > triggered automatically by dovecot after changes to the mailbox Well, to be fair : 1. I said what I said based on the video. And the video seemed pretty clear cut to me ? 2. Its not there in the form that many (most ?) people would use it for (i.e. with Replicator). 3. Then Aki came along and said "there is no hidden cache of code going into 3.0 that will not be open source". When the video kind of makes it clear 3.0 Pro with all its new features (e.g. multi-server) is very much going to be a closed-source job. And that the present open-source version is, just like they say in the video, is going to be "supported for single-server use only". Therefore the waters are still very much muddy overall. The dsync question might well have now been clarified somewhat. But the rest of "how much 3.0 Pro will we see in open source" ? If we're being generous we would say muddy waters, but my gut feeling is the video made clear their direction of travel in that the present Open Source version will continue as-is with updates and support, bu won't be getting any of the fancy new features and functionality that 3.0 Pro is. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> >> > >> What is being removed is the replicator plugin (that used dsync). > That's what is being referred to in the video. Replicator hasn't been > actively maintained for years now so this was dead code anyway. > > > > > > Well, so Laura is absolutely right ... > > "Things like dsync will be GONE in the community version." > > That's not right, dsync is still there. Replicator is not, so dsync can't > be > triggered automatically by dovecot after changes to the mailbox (which is > very handy for a simple "nearly live" Dovecot-aware backup mechanism > when used on what is mostly treated as a single server setup, but with a > replica on another machine). > > There are other ways to run dsync, though not as convenient. > > I'm still trying to decide how best to do this after replicator is gone. > If the community version had obox (s3-compatible) storage I think I'd > probably use that and e.g. replicated minio as backend, this seems like > it would be more robust. As it is, I'll probably scan mailboxes for > timestamp changes every few minutes and fire off dsync as needed. > Is s3 not to slow for this? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > If that is the case, well then I have to find another way to keep mails > in sync between 2 mailservers. Hope the community will find a new solution! > > > > I have been keeping one eye on Stalwart (https://stalw.art/) for a while > now. > > I haven't tested it as yet, but I'm very much tempted to get a test > instance up and running. > Interesting, nice they use this rust, I am curious how they define this scaling. What I don't get is why are they messing with smtp. I always get a bad feeling when a company is trying to do everything. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
On 2023-10-16 2:30 a.m., Michael Slusarz via dovecot wrote: To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. So if we are using "mail_attachment_fs = sis posix" in 2.4 it will become "mail_attachment_fs = posix" ? i.e. separating attachments still supported but without SIS? Thanks for any clarification. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> > If that is the case, well then I have to find another way to keep mails in > sync between 2 mailservers. Hope the community will find a new solution! > I have been keeping one eye on Stalwart (https://stalw.art/) for a while now. I haven't tested it as yet, but I'm very much tempted to get a test instance up and running. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > Hello to everyone! > > Ooops, we are using SIS, guess the solution for a similar optimization > will be > > a native deduplicated filesystem. > > did you really mean deduplicated or distributed? > I think this duduplicating. Storage systems are offering such solutions. I think ceph has something like this, although I am not sure for rbd disk images. I think it makes more sense to have something like this done by a fs or storage solution. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Hi, Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem. did you really mean deduplicated or distributed? Regards Bjoern ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
On Mon, 16 Oct 2023, Marc wrote: > Is this feature really useful? I can imagine if you are twitter or ig and > everyone is posting the same video this could be usefull. Are there any stats > on this available, so you know what to expect implementing deduplication. In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful. -- --- Chris Candreva -- ch...@westnet.com -- http://www.westnet.com/~chris ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> Ooops, we are using SIS, guess the solution for a similar optimization > will be a native deduplicated filesystem. > Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Hello to everyone!
Ooops, we are using SIS, guess the solution for a similar optimization will be
a native deduplicated filesystem.
For server synchronization (non "realtime") we are using "imapsync" ( https://
imapsync.lamiral.info/ )
regards!
On 16/10/23 08:11, Taavi Ansper via dovecot wrote:
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
plugin {
mail_replica = tcp:example.com:12345
}
If that is the case, well then I have to find another way to keep
mails in sync between 2 mailservers. Hope the community will find a
new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the
video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that
does mail synchronization between a source account to a
destination account. As Aki said, this is not going
anywhere. This is a necessary tool for any kind of
migrations, for example. dsync is under active
maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used
dsync). That's what is being referred to in the video.
Replicator hasn't been actively maintained for years now so
this was dead code anyway.
To answer the OP: sis is also being removed and should
not be used by any new installation. Code remains to read
data written by the old plug-in so that these installations
don't require a migration between 2.3 and 2.4. This is
another plugin that hasn't be actively maintained in years,
and has all kinds of limitations that prevent it from
running at scale.
Neither replicator nor sis is code that is moving from
open to closed source. These plugins aren't used in Pro.
They are unmaintained so they are being removed, as happens
with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot
wrote:
FUD ?
I knew someone would accuse me of that which is
why I linked to the video from the horse's mouth,
I transcribe what the speaker said:
"there will be an open source version, but that
open source version will be maintained for single
server use only. we are actually taking out
anything any actually kinda' involves multiple
servers, dsync replication and err some other
stuff. so dovecot will be a fully-featured single
node server"
--- Original Message ---
On Friday, October 13th, 2023 at 19:37, Aki Tuomi
wrote:
Dear Laura, please don't spread FUD
that you made up.
Dsync is not going anywhere, and we are
not close-sourcing Dovecot Core. There
is not a trove of code going into
Dovecot 3.0 that "never sees the
daylight".
Thank you,
Aki
On 13/10/2023 21:10 EEST
Laura Smith via dovecot
dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot
Community user, don't waste
your time reading the Dovecot
Pro release notes.
To expand:
I think you have to
understand that lots of
things that are going into
Dovecot 3 (Pro) will never
see the light of day in the
community edition.
In addition, Dovecot have
publicly quite plainly
announced in public that they
are actively removing all
multi-server related
functionality from Dovecot
Community.
I don't think the community
has quite yet grasped it.
Things like dsync will be
GONE in the community
version.
If you don't believe me, look
at this video, about 15
minutes in:
https://youtu.be/s-
JYrjCKshA?feature=shared=912
Re: The future of SIS
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
plugin {
mail_replica = tcp:example.com:12345
}
If that is the case, well then I have to find another way to keep mails in sync
between 2 mailservers. Hope the community will find a new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the video, although I
could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from
the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be
maintained for single server use only. we are actually taking out anything any actually
kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot
will be a fully-featured single node server"
--- Original Message ---
On Friday, October 13th, 2023 at 19:37, Aki Tuomi
wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a
trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you,
Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the
Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot
3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they
are actively removing all multi-server related functionality from Dovecot
Community.
I don't think the community has quite yet grasped it. Things like dsync will be
GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in:
https://youtu.be/s-JYrjCKshA?feature=shared=912
--- Original Message ---
On Friday, October 13th, 2023 at 17:15, Sebastian Marsching
sebast...@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was
planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I
stumbled across the following notice in the documentation for Dovecot 3.0
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
16.10.2023 10:30, Michael Slusarz via dovecot пишет: Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better. "dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally. What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway. Well, so Laura is absolutely right ... To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code. michael On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote: FUD ? I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said: "there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server" --- Original Message --- On Friday, October 13th, 2023 at 19:37, Aki Tuomi wrote: Dear Laura, please don't spread FUD that you made up. Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight". Thank you, Aki On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote: TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes. To expand: I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition. In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community. I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version. If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared=912 --- Original Message --- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebast...@marsching.com wrote: Hi, I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0 ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better. "dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally. What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway. To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code. michael > On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote: > > > FUD ? > > I knew someone would accuse me of that which is why I linked to the video > from the horse's mouth, I transcribe what the speaker said: > > "there will be an open source version, but that open source version will be > maintained for single server use only. we are actually taking out anything > any actually kinda' involves multiple servers, dsync replication and err some > other stuff. so dovecot will be a fully-featured single node server" > > > > > --- Original Message --- > On Friday, October 13th, 2023 at 19:37, Aki Tuomi > wrote: > > > > Dear Laura, please don't spread FUD that you made up. > > > > Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. > > There is not a trove of code going into Dovecot 3.0 that "never sees the > > daylight". > > > > Thank you, > > Aki > > > > > On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org > > > wrote: > > > > > > TL;DR If you are a Dovecot Community user, don't waste your time reading > > > the Dovecot Pro release notes. > > > > > > To expand: > > > > > > I think you have to understand that lots of things that are going into > > > Dovecot 3 (Pro) will never see the light of day in the community edition. > > > > > > In addition, Dovecot have publicly quite plainly announced in public that > > > they are actively removing all multi-server related functionality from > > > Dovecot Community. > > > > > > I don't think the community has quite yet grasped it. Things like dsync > > > will be GONE in the community version. > > > > > > If you don't believe me, look at this video, about 15 minutes in: > > > https://youtu.be/s-JYrjCKshA?feature=shared=912 > > > > > > --- Original Message --- > > > On Friday, October 13th, 2023 at 17:15, Sebastian Marsching > > > sebast...@marsching.com wrote: > > > > > > > Hi, > > > > > > > > I am currently in the process of planning a new deployment of Dovecot. > > > > I was planning to use mdbox or sdbox with “mail_attachment_fs = sis > > > > posix”, but I stumbled across the following notice in the documentation > > > > for Dovecot 3.0 > > > > ___ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
On 10/14/23 03:26, Laura Smith via dovecot wrote: > FUD ? > > I knew someone would accuse me of that which is why I linked to the video > from the horse's mouth, I transcribe what the speaker said: > > "there will be an open source version, but that open source version will be > maintained for single server use only. we are actually taking out anything > any actually kinda' involves multiple servers, dsync replication and err some > other stuff. so dovecot will be a fully-featured single node server" > Yes. Aki, it would be much appreciated if you can deliver your point in the form of a clarification of what the product manager actually said in that video, in very clear language. Thanks! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
FUD ? I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said: "there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server" --- Original Message --- On Friday, October 13th, 2023 at 19:37, Aki Tuomi wrote: > Dear Laura, please don't spread FUD that you made up. > > Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. > There is not a trove of code going into Dovecot 3.0 that "never sees the > daylight". > > Thank you, > Aki > > > On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote: > > > > TL;DR If you are a Dovecot Community user, don't waste your time reading > > the Dovecot Pro release notes. > > > > To expand: > > > > I think you have to understand that lots of things that are going into > > Dovecot 3 (Pro) will never see the light of day in the community edition. > > > > In addition, Dovecot have publicly quite plainly announced in public that > > they are actively removing all multi-server related functionality from > > Dovecot Community. > > > > I don't think the community has quite yet grasped it. Things like dsync > > will be GONE in the community version. > > > > If you don't believe me, look at this video, about 15 minutes in: > > https://youtu.be/s-JYrjCKshA?feature=shared=912 > > > > --- Original Message --- > > On Friday, October 13th, 2023 at 17:15, Sebastian Marsching > > sebast...@marsching.com wrote: > > > > > Hi, > > > > > > I am currently in the process of planning a new deployment of Dovecot. I > > > was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, > > > but I stumbled across the following notice in the documentation for > > > Dovecot 3.0 > > > ___ > > > dovecot mailing list -- dovecot@dovecot.org > > > To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Dear Laura, please don't spread FUD that you made up. Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight". Thank you, Aki > On 13/10/2023 21:10 EEST Laura Smith via dovecot wrote: > > > TL;DR If you are a Dovecot Community user, don't waste your time reading the > Dovecot Pro release notes. > > To expand: > > I think you have to understand that lots of things that are going into > Dovecot 3 (Pro) will never see the light of day in the community edition. > > In addition, Dovecot have publicly quite plainly announced in public that > they are actively removing all multi-server related functionality from > Dovecot Community. > > I don't think the community has quite yet grasped it. Things like dsync will > be GONE in the community version. > > If you don't believe me, look at this video, about 15 minutes in: > https://youtu.be/s-JYrjCKshA?feature=shared=912 > > --- Original Message --- > On Friday, October 13th, 2023 at 17:15, Sebastian Marsching > wrote: > > > > Hi, > > > > I am currently in the process of planning a new deployment of Dovecot. I > > was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, > > but I stumbled across the following notice in the documentation for Dovecot > > 3.0 > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes. To expand: I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition. In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community. I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version. If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared=912 --- Original Message --- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching wrote: > Hi, > > I am currently in the process of planning a new deployment of Dovecot. I was > planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I > stumbled across the following notice in the documentation for Dovecot 3.0 ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
The future of SIS
Hi, I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0 (https://doc.dovecot.org/3.0/settings/core/#core_setting-mail_attachment_fs): > Changed in version 2.4.0 (CE): SIS is deprecated and writing of SIS files is > disabled. Reading is supported for now, any missing SIS attachments are > replaced with files filled with spaces. > > Changed in version 3.0.0 (Pro): SIS is deprecated and writing of SIS files is > disabled. Reading is supported for now, any missing SIS attachments are > replaced with files filled with spaces. And https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/ says: > Saving new mails’ attachments via fs-sis is disabled, but reading SIS > attachments is still supported. Missing SIS attachments are replaced with > files filled with spaces. What is not entirely clear from this comment is whether this only applied to “sis posix” or also to “sis-queue posix”. Does anybody whether sis-queue is also affected by this? I saw that the notice was added to the dovecot/documentation Git repository in April 2023, but I couldn’t find any further information there or the main branch of the dovecot/core repository. Maybe, the code for 2.4 / 3.0 has not made it into that branch or repository yet. SIS sounds like a neat concept and I would like to use it, but obviously I do not want to use a soon-to-be-deprecated feature in a completely new deployment. So, does anybody know an alternative solution for attachment deduplication (except file-system level deduplication)? Thanks, Sebastian smime.p7s Description: S/MIME cryptographic signature ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
