Re: The future of SIS
> Le 16 oct. 2023 à 15:51, Marc a écrit : > >>> Hello to everyone! >>> Ooops, we are using SIS, guess the solution for a similar optimization >> will be >>> a native deduplicated filesystem. >> >> did you really mean deduplicated or distributed? >> > > I think this duduplicating. Storage systems are offering such solutions. I > think ceph has something like this, although I am not sure for rbd disk > images. I think it makes more sense to have something like this done by a fs > or storage solution. If you are using Ubuntu, OpenZFS is readily available, and support deduplication natively. Else it is also available on other platforms, but may require more setup. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Le 16/10/2023 à 19:44, Marc a écrit : Is s3 not to slow for this? I think the clue is in the name "s3-compatible". Clearly calling out to "real" (AWS) S3 would be a non-starter. But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably? I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency. The actual system is not perfect, but simple and robust and there is room for improvements. Dovecot is a very solid base. The replicator removing is sad, but aligned with the business pressure open-xchange business: Their customers are only big players all-ready wasting lots of money on "enterprise grade" architecture and big and expensive cloud services. It is perfectly aligned with their customer view of efficiency... At the same time, this customers have a very distorted view of the cost of pure service. They want a price at the lowest common denominator of human cost over the planet... I think that open-xchange only try to off cut all possible maintenance cost from the "product". It it too much ? Will it pay off ? Who live will see... From over more than thirty years of experience of observing the open source world, projects that achieve "world domination" are NEVER the ones that sacrifices the technical goal for business reasons/pressure even if it take decades to be recognized for it true value. Now that Exchange has a robust multinode replication and distribution system dovecot will loose it own. Ironic isn't it :-) Emmanuel. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > Is s3 not to slow for this? > > > > I think the clue is in the name "s3-compatible". > > Clearly calling out to "real" (AWS) S3 would be a non-starter. > > But a local installation of something like CEPH, MinIO or whatever on the > same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably? I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. [1] https://github.com/ceph-dovecot/dovecot-ceph-plugin ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> Is s3 not to slow for this? > I think the clue is in the name "s3-compatible". Clearly calling out to "real" (AWS) S3 would be a non-starter. But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > Interesting, nice they use this rust, I am curious how they define this > scaling. What I don't get is why are they messing with smtp. I always get a > bad feeling when a company is trying to do everything. Good they are using rust and even better they've had an independent security audit (https://www.stalw.art/blog/security-audit). On the scaling side, maybe see the storage page ? (https://www.stalw.art/docs/storage/overview). The metadata is stored in a database which can be replicated. And the mails themselves can be stored in filesystem or "S3-compatible" storage, and so there are scaling options there too ? But clearly some experimentation is required to see how it works in practice. Are they messing with SMTP ? As I understand it its an IMAP/JMAP server. And (like Dovecot) it has LMTP for getting mail into it from e.g. Postfix ? From my reading of the docs it looks like SMTP is only there if you don't want to use LMTP to get mail into it ? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> > Well, so Laura is absolutely right ... > > > "Things like dsync will be GONE in the community version." > > That's not right, dsync is still there. Replicator is not, so dsync can't be > triggered automatically by dovecot after changes to the mailbox Well, to be fair : 1. I said what I said based on the video. And the video seemed pretty clear cut to me ? 2. Its not there in the form that many (most ?) people would use it for (i.e. with Replicator). 3. Then Aki came along and said "there is no hidden cache of code going into 3.0 that will not be open source". When the video kind of makes it clear 3.0 Pro with all its new features (e.g. multi-server) is very much going to be a closed-source job. And that the present open-source version is, just like they say in the video, is going to be "supported for single-server use only". Therefore the waters are still very much muddy overall. The dsync question might well have now been clarified somewhat. But the rest of "how much 3.0 Pro will we see in open source" ? If we're being generous we would say muddy waters, but my gut feeling is the video made clear their direction of travel in that the present Open Source version will continue as-is with updates and support, bu won't be getting any of the fancy new features and functionality that 3.0 Pro is. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> >> > >> What is being removed is the replicator plugin (that used dsync). > That's what is being referred to in the video. Replicator hasn't been > actively maintained for years now so this was dead code anyway. > > > > > > Well, so Laura is absolutely right ... > > "Things like dsync will be GONE in the community version." > > That's not right, dsync is still there. Replicator is not, so dsync can't > be > triggered automatically by dovecot after changes to the mailbox (which is > very handy for a simple "nearly live" Dovecot-aware backup mechanism > when used on what is mostly treated as a single server setup, but with a > replica on another machine). > > There are other ways to run dsync, though not as convenient. > > I'm still trying to decide how best to do this after replicator is gone. > If the community version had obox (s3-compatible) storage I think I'd > probably use that and e.g. replicated minio as backend, this seems like > it would be more robust. As it is, I'll probably scan mailboxes for > timestamp changes every few minutes and fire off dsync as needed. > Is s3 not to slow for this? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > > > If that is the case, well then I have to find another way to keep mails > in sync between 2 mailservers. Hope the community will find a new solution! > > > > I have been keeping one eye on Stalwart (https://stalw.art/) for a while > now. > > I haven't tested it as yet, but I'm very much tempted to get a test > instance up and running. > Interesting, nice they use this rust, I am curious how they define this scaling. What I don't get is why are they messing with smtp. I always get a bad feeling when a company is trying to do everything. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
On 2023-10-16 2:30 a.m., Michael Slusarz via dovecot wrote: To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. So if we are using "mail_attachment_fs = sis posix" in 2.4 it will become "mail_attachment_fs = posix" ? i.e. separating attachments still supported but without SIS? Thanks for any clarification. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Extract body preview in lua notify script
Hi, I am trying to extract short text preview from mail body in function dovecot_lua_notify_end_txn. I want to send notification about new email with perex included. How can it be done? Is it possible? thanks -- Filip Hanes ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
> > If that is the case, well then I have to find another way to keep mails in > sync between 2 mailservers. Hope the community will find a new solution! > I have been keeping one eye on Stalwart (https://stalw.art/) for a while now. I haven't tested it as yet, but I'm very much tempted to get a test instance up and running. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> > Hello to everyone! > > Ooops, we are using SIS, guess the solution for a similar optimization > will be > > a native deduplicated filesystem. > > did you really mean deduplicated or distributed? > I think this duduplicating. Storage systems are offering such solutions. I think ceph has something like this, although I am not sure for rbd disk images. I think it makes more sense to have something like this done by a fs or storage solution. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Hi, Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem. did you really mean deduplicated or distributed? Regards Bjoern ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
On Mon, 16 Oct 2023, Marc wrote: > Is this feature really useful? I can imagine if you are twitter or ig and > everyone is posting the same video this could be usefull. Are there any stats > on this available, so you know what to expect implementing deduplication. In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful. -- --- Chris Candreva -- ch...@westnet.com -- http://www.westnet.com/~chris ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: The future of SIS
> Ooops, we are using SIS, guess the solution for a similar optimization > will be a native deduplicated filesystem. > Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Hello to everyone!
Ooops, we are using SIS, guess the solution for a similar optimization will be
a native deduplicated filesystem.
For server synchronization (non "realtime") we are using "imapsync" ( https://
imapsync.lamiral.info/ )
regards!
On 16/10/23 08:11, Taavi Ansper via dovecot wrote:
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
plugin {
mail_replica = tcp:example.com:12345
}
If that is the case, well then I have to find another way to keep
mails in sync between 2 mailservers. Hope the community will find a
new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the
video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that
does mail synchronization between a source account to a
destination account. As Aki said, this is not going
anywhere. This is a necessary tool for any kind of
migrations, for example. dsync is under active
maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used
dsync). That's what is being referred to in the video.
Replicator hasn't been actively maintained for years now so
this was dead code anyway.
To answer the OP: sis is also being removed and should
not be used by any new installation. Code remains to read
data written by the old plug-in so that these installations
don't require a migration between 2.3 and 2.4. This is
another plugin that hasn't be actively maintained in years,
and has all kinds of limitations that prevent it from
running at scale.
Neither replicator nor sis is code that is moving from
open to closed source. These plugins aren't used in Pro.
They are unmaintained so they are being removed, as happens
with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot
wrote:
FUD ?
I knew someone would accuse me of that which is
why I linked to the video from the horse's mouth,
I transcribe what the speaker said:
"there will be an open source version, but that
open source version will be maintained for single
server use only. we are actually taking out
anything any actually kinda' involves multiple
servers, dsync replication and err some other
stuff. so dovecot will be a fully-featured single
node server"
--- Original Message ---
On Friday, October 13th, 2023 at 19:37, Aki Tuomi
wrote:
Dear Laura, please don't spread FUD
that you made up.
Dsync is not going anywhere, and we are
not close-sourcing Dovecot Core. There
is not a trove of code going into
Dovecot 3.0 that "never sees the
daylight".
Thank you,
Aki
On 13/10/2023 21:10 EEST
Laura Smith via dovecot
dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot
Community user, don't waste
your time reading the Dovecot
Pro release notes.
To expand:
I think you have to
understand that lots of
things that are going into
Dovecot 3 (Pro) will never
see the light of day in the
community edition.
In addition, Dovecot have
publicly quite plainly
announced in public that they
are actively removing all
multi-server related
functionality from Dovecot
Community.
I don't think the community
has quite yet grasped it.
Things like dsync will be
GONE in the community
version.
If you don't believe me, look
at this video, about 15
minutes in:
https://youtu.be/s-
JYrjCKshA?feature=shared=912
Re: The future of SIS
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
plugin {
mail_replica = tcp:example.com:12345
}
If that is the case, well then I have to find another way to keep mails in sync
between 2 mailservers. Hope the community will find a new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the video, although I
could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from
the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be
maintained for single server use only. we are actually taking out anything any actually
kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot
will be a fully-featured single node server"
--- Original Message ---
On Friday, October 13th, 2023 at 19:37, Aki Tuomi
wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a
trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you,
Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the
Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot
3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they
are actively removing all multi-server related functionality from Dovecot
Community.
I don't think the community has quite yet grasped it. Things like dsync will be
GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in:
https://youtu.be/s-JYrjCKshA?feature=shared=912
--- Original Message ---
On Friday, October 13th, 2023 at 17:15, Sebastian Marsching
sebast...@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was
planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I
stumbled across the following notice in the documentation for Dovecot 3.0
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
16.10.2023 10:30, Michael Slusarz via dovecot пишет: Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better. "dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally. What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway. Well, so Laura is absolutely right ... To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code. michael On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote: FUD ? I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said: "there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server" --- Original Message --- On Friday, October 13th, 2023 at 19:37, Aki Tuomi wrote: Dear Laura, please don't spread FUD that you made up. Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight". Thank you, Aki On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote: TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes. To expand: I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition. In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community. I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version. If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared=912 --- Original Message --- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebast...@marsching.com wrote: Hi, I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0 ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: The future of SIS
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better. "dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally. What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway. To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code. michael > On 10/13/2023 1:26 PM MDT Laura Smith via dovecot wrote: > > > FUD ? > > I knew someone would accuse me of that which is why I linked to the video > from the horse's mouth, I transcribe what the speaker said: > > "there will be an open source version, but that open source version will be > maintained for single server use only. we are actually taking out anything > any actually kinda' involves multiple servers, dsync replication and err some > other stuff. so dovecot will be a fully-featured single node server" > > > > > --- Original Message --- > On Friday, October 13th, 2023 at 19:37, Aki Tuomi > wrote: > > > > Dear Laura, please don't spread FUD that you made up. > > > > Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. > > There is not a trove of code going into Dovecot 3.0 that "never sees the > > daylight". > > > > Thank you, > > Aki > > > > > On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org > > > wrote: > > > > > > TL;DR If you are a Dovecot Community user, don't waste your time reading > > > the Dovecot Pro release notes. > > > > > > To expand: > > > > > > I think you have to understand that lots of things that are going into > > > Dovecot 3 (Pro) will never see the light of day in the community edition. > > > > > > In addition, Dovecot have publicly quite plainly announced in public that > > > they are actively removing all multi-server related functionality from > > > Dovecot Community. > > > > > > I don't think the community has quite yet grasped it. Things like dsync > > > will be GONE in the community version. > > > > > > If you don't believe me, look at this video, about 15 minutes in: > > > https://youtu.be/s-JYrjCKshA?feature=shared=912 > > > > > > --- Original Message --- > > > On Friday, October 13th, 2023 at 17:15, Sebastian Marsching > > > sebast...@marsching.com wrote: > > > > > > > Hi, > > > > > > > > I am currently in the process of planning a new deployment of Dovecot. > > > > I was planning to use mdbox or sdbox with “mail_attachment_fs = sis > > > > posix”, but I stumbled across the following notice in the documentation > > > > for Dovecot 3.0 > > > > ___ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
