This is an automated email from the ASF dual-hosted git repository. rcordier pushed a commit to branch postgresql in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 8682b6ca33e420d4cc101074f6c69147f4a50915 Author: vttran <[email protected]> AuthorDate: Mon Jul 1 09:01:37 2024 +0700 [Antora] Make partial for server Architecture section --- ...s.png => specialized-instances-distributed.png} | Bin .../{storage.png => storage_james_distributed.png} | Bin .../architecture/consistency-model.adoc | 119 +------- .../consistency_model_data_replication_extend.adoc | 41 +++ .../architecture/implemented-standards.adoc | 118 +------- .../pages/distributed/architecture/index.adoc | 315 +-------------------- .../architecture/mailqueue_combined_extend.adoc | 4 + .../architecture/specialized-instances.adoc | 38 +-- .../partials/architecture/consistency-model.adoc | 65 +++++ .../architecture/implemented-standards.adoc | 5 +- .../architecture/index.adoc | 102 ++++--- .../architecture/specialized-instances.adoc | 21 +- 12 files changed, 194 insertions(+), 634 deletions(-) diff --git a/docs/modules/servers/assets/images/specialized-instances.png b/docs/modules/servers/assets/images/specialized-instances-distributed.png similarity index 100% rename from docs/modules/servers/assets/images/specialized-instances.png rename to docs/modules/servers/assets/images/specialized-instances-distributed.png diff --git a/docs/modules/servers/assets/images/storage.png b/docs/modules/servers/assets/images/storage_james_distributed.png similarity index 100% rename from docs/modules/servers/assets/images/storage.png rename to docs/modules/servers/assets/images/storage_james_distributed.png diff --git a/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc b/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc index bc4babba01..af72b7d810 100644 --- a/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc +++ b/docs/modules/servers/pages/distributed/architecture/consistency-model.adoc @@ -1,79 +1,14 @@ = Distributed James Server — Consistency Model :navtitle: Consistency Model -This page presents the consistency model used by the Distributed Server and -points to the tools built around it. +:backend-name: cassandra +:backend-name-cap: Cassandra +:server-name: Distributed James Server +:mailet-repository-path-prefix: cassandra +:xref-base: distributed +:data_replication_extend: servers:distributed/architecture/consistency_model_data_replication_extend.adoc -== Data Replication - -The Distributed Server relies on different storage technologies, all having their own -consistency models. - -These data stores replicate data in order to enforce some level of availability. We call -this process replication. By consistency, we mean the ability for all replica to hold the -same data. By availability, we mean the ability for a replica to answer a request. - -In distributed systems, link:https://en.wikipedia.org/wiki/CAP_theorem[according to the CAP theorem], -as we will necessarily encounter network partitions, then trade-offs need to be made between -consistency and availability. - -This section details this trade-off for data stores used by the Distributed Server. - -=== Cassandra consistency model - -link:https://cassandra.apache.org/[Cassandra] is an -link:https://en.wikipedia.org/wiki/Eventual_consistency[eventually consistent] data store. -This means that replica can hold diverging data, but are guaranteed to converge over time. - -Several mechanisms are built in Cassandra to enforce this convergence, and need to be -leveraged by *Distributed Server Administrator*. Namely -link:https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html[nodetool repair], -link:https://cassandra.apache.org/doc/latest/operating/hints.html[Hinted hand-off] and -link:https://cassandra.apache.org/doc/latest/operating/read_repair.html[Read repair]. - -The Distributed Server tries to mitigate inconsistencies by relying on -link:https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/dml/dmlConfigConsistency.html[QUORUM] read and write levels. -This means that a majority of replica are needed for read and write operations to be performed. - -Critical business operations, like UID allocation, rely on strong consistency mechanisms brought by -link:https://www.datastax.com/blog/2013/07/lightweight-transactions-cassandra-20[lightweight transaction]. - -==== About multi data-center setups - -As strong consistency is required for some operation, especially regarding IMAP monotic UID and MODSEQ generation, -and as lightweight transactions are slow across data centers, running James with a -link:https://docs.datastax.com/en/ddac/doc/datastax_enterprise/production/DDACmultiDCperWorkloadType.html[multi data-center] -Cassandra setup is discouraged. - -However, xref:distributed/configure/cassandra.adoc[this page] enables setting alternative read level, -which could be acceptable regarding limited requirements. `LOCAL_QUORUM` coupled with `LOCAL_SERIAL` -is likely the only scalable setup. Some options were added to turn off SERIAL consistency usage for message -and mailbox management. However, the use of Lightweight Transaction cannot be disabled for UIDs and ModSeqs. - -Running the Distributed Server IMAP server in a multi datacenter setup will likely result either in data loss, -or very slow operations - as we rely on monotic UID generation, without strong consistency, UIDs could be allocated -several times. - -We did wire a multi-DC friendly distributed, POP3 only server that leverages acceptable performance while staying -consistent. This is achieved by having a reduced feature set - supporting only the POP3 server and using messageIds as -identifiers (generated without synchronisation using TimeUUIDs). You can find this application -link:https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app[on GitHub]. In the future, -JMAP support could be added, but requires followup developments as some components critically depends on UIDs -(for instance the search). - -=== OpenSearch consistency model - -OpenSearch relies on link:https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch[strong consistency] -with home grown algorithm. - -The 6.x release line, that the distributed server is using is known to be slow to recover from failures. - -Be aware that data is asynchronously indexed in OpenSearch, changes will be eventually visible. - -=== RabbitMQ consistency model - -The Distributed Server relies out of the box on a single RabbitMQ server, thus consistency concerns -are not (yet) applicable. Availability concerns are applicable. +include::partial$architecture/consistency-model.adoc[] == Denormalization @@ -86,43 +21,11 @@ level across denormalization tables. We write to a "table of truth" first, then duplicate the data to denormalization tables. -The Distributed server offers several mechanisms to mitigate these inconsistencies: +The {server-name} offers several mechanisms to mitigate these inconsistencies: - - Writes to denormalization tables are retried. - - Some xref:distributed/operate/guide.adoc#_solving_cassandra_inconsistencies[SolveInconsistencies tasks] are exposed and are able to heal a given denormalization table. +- Writes to denormalization tables are retried. +- Some xref:{xref-base}/operate/guide.adoc#_solving_cassandra_inconsistencies[SolveInconsistencies tasks] are exposed and are able to heal a given denormalization table. They reset the "deduplication tables" content to the "table of truth" content. - - link:https://github.com/apache/james-project/blob/master/src/adr/0042-applicative-read-repairs.md[Read repairs], +- link:https://github.com/apache/james-project/blob/master/src/adr/0042-applicative-read-repairs.md[Read repairs], when implemented for a given denormalization, enables auto-healing. When an inconsistency is detected, They reset the "deduplication tables" entry to the "table of truth" entry. - -== Consistency across data stores - -The Distributed Server leverages several data stores: - - - Cassandra is used for metadata storage - - OpenSearch for search - - Object Storage for large object storage - -Thus the Distributed Server also offers mechanisms to enforce consistency across data stores. - -=== Write path organisation - -The primary data stores are composed of Cassandra for metadata and Object storage for binary data. - -To ensure the data referenced in Cassandra is pointing to a valid object in the object store, we write -the object store payload first, then write the corresponding metadata in Cassandra. - -Such a procedure avoids metadata pointing to unexisting blobs, however might lead to some unreferenced -blobs. - -=== Cassandra <=> OpenSearch - -After being written to the primary stores (namely Cassandra & Object Storage), email content is -asynchronously indexed into OpenSearch. - -This process, called the EventBus, which retries temporary errors, and stores transient errors for -later admin-triggered retries is described further xref:distributed/operate/guide.adoc#_mailbox_event_bus[here]. -His role is to spread load and limit inconsistencies. - -Furthermore, some xref:distributed/operate/guide.adoc#_usual_troubleshooting_procedures[re-indexing tasks] -enables to re-synchronise OpenSearch content with the primary data stores diff --git a/docs/modules/servers/pages/distributed/architecture/consistency_model_data_replication_extend.adoc b/docs/modules/servers/pages/distributed/architecture/consistency_model_data_replication_extend.adoc new file mode 100644 index 0000000000..d6e3cee915 --- /dev/null +++ b/docs/modules/servers/pages/distributed/architecture/consistency_model_data_replication_extend.adoc @@ -0,0 +1,41 @@ +=== Cassandra consistency model + +link:https://cassandra.apache.org/[Cassandra] is an +link:https://en.wikipedia.org/wiki/Eventual_consistency[eventually consistent] data store. +This means that replica can hold diverging data, but are guaranteed to converge over time. + +Several mechanisms are built in Cassandra to enforce this convergence, and need to be +leveraged by *Distributed Server Administrator*. Namely +link:https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html[nodetool repair], +link:https://cassandra.apache.org/doc/latest/operating/hints.html[Hinted hand-off] and +link:https://cassandra.apache.org/doc/latest/operating/read_repair.html[Read repair]. + +The {server-name} tries to mitigate inconsistencies by relying on +link:https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/dml/dmlConfigConsistency.html[QUORUM] read and write levels. +This means that a majority of replica are needed for read and write operations to be performed. + +Critical business operations, like UID allocation, rely on strong consistency mechanisms brought by +link:https://www.datastax.com/blog/2013/07/lightweight-transactions-cassandra-20[lightweight transaction]. + +==== About multi data-center setups + +As strong consistency is required for some operation, especially regarding IMAP monotic UID and MODSEQ generation, +and as lightweight transactions are slow across data centers, running James with a +link:https://docs.datastax.com/en/ddac/doc/datastax_enterprise/production/DDACmultiDCperWorkloadType.html[multi data-center] +Cassandra setup is discouraged. + +However, xref:{xref-base}/configure/cassandra.adoc[this page] enables setting alternative read level, +which could be acceptable regarding limited requirements. `LOCAL_QUORUM` coupled with `LOCAL_SERIAL` +is likely the only scalable setup. Some options were added to turn off SERIAL consistency usage for message +and mailbox management. However, the use of Lightweight Transaction cannot be disabled for UIDs and ModSeqs. + +Running the {server-name} IMAP server in a multi datacenter setup will likely result either in data loss, +or very slow operations - as we rely on monotic UID generation, without strong consistency, UIDs could be allocated +several times. + +We did wire a multi-DC friendly distributed, POP3 only server that leverages acceptable performance while staying +consistent. This is achieved by having a reduced feature set - supporting only the POP3 server and using messageIds as +identifiers (generated without synchronisation using TimeUUIDs). You can find this application +link:https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app[on GitHub]. In the future, +JMAP support could be added, but requires followup developments as some components critically depends on UIDs +(for instance the search). diff --git a/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc b/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc index 566e40a9b3..82f085c438 100644 --- a/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc +++ b/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc @@ -1,120 +1,6 @@ = Distributed James Server — Implemented standards :navtitle: Implemented standards -This page details standards implemented by the distributed server. - -== Message formats - - - link:https://datatracker.ietf.org/doc/html/rfc5322[RFC-5322] Internet Message Format (MIME) - - link:https://datatracker.ietf.org/doc/html/rfc2045[RFC-2045] Format of Internet Message Bodies - - link:https://datatracker.ietf.org/doc/html/rfc3464[RFC-3464] An Extensible Message Format for Delivery Status Notifications - - James allow emmit DSNs from the mailet container. - - link:https://datatracker.ietf.org/doc/html/rfc8098[RFC-8098] Message Disposition Notification - -== TLS & authentication - -- link:https://datatracker.ietf.org/doc/html/rfc2595.html[RFC-2595] TLS for IMAP, POP3, SMTP (StartTLS) -- link:https://datatracker.ietf.org/doc/html/rfc8314.html[RFC-8314] Implicit TLS -- link:https://www.rfc-editor.org/rfc/rfc4959.html[RFC-4959] SASL IR: Initial client response -- link:https://datatracker.ietf.org/doc/html/rfc4616[RFC-4616] SASL plain authentication -- link:https://datatracker.ietf.org/doc/html/rfc8314.html[RFC-7628] SASL for OAUTH -- Implemented for IMAP and SMTP -- Support for OIDC standard only. - -== SMTP - -- link:https://datatracker.ietf.org/doc/html/rfc5321[RFC-5321] SMTP Protocol -- link:https://datatracker.ietf.org/doc/html/rfc974[RFC-974] MAIL ROUTING AND THE DOMAIN SYSTEM -- link:https://www.rfc-editor.org/rfc/rfc3461[RFC-3461] Simple Mail Transfer Protocol (SMTP) Service Extension for Delivery Status Notifications (DSNs) - - Requires extra configuration. -- link:https://datatracker.ietf.org/doc/html/rfc1652[RFC-1652] SMTP Service Extension for 8bit-MIME transport -- link:https://datatracker.ietf.org/doc/html/rfc1830[RFC-1830] SMTP Service Extensions for Transmission of Large and Binary MIME Messages -- link:https://datatracker.ietf.org/doc/html/rfc1869[RFC-1869] SMTP Service Extensions -- link:https://datatracker.ietf.org/doc/html/rfc1870[RFC-1870] SMTP Service Extension for Message Size Declaration -- link:https://datatracker.ietf.org/doc/html/rfc1891[RFC-1891] SMTP Service Extension for Delivery Status Notifications -- link:https://datatracker.ietf.org/doc/html/rfc1893[RFC-1893] Enhanced Mail System Status Codes -- link:https://datatracker.ietf.org/doc/html/rfc2034[RFC-2034] SMTP Service Extension for Returning Enhanced Error Codes -- link:https://datatracker.ietf.org/doc/html/rfc2142[RFC-2142] Mailbox Names For Common Services, Roles And Functions -- link:https://datatracker.ietf.org/doc/html/rfc2197[RFC-2197] SMTP Service Extension for Command Pipelining -- link:https://datatracker.ietf.org/doc/html/rfc2554[RFC-2554] ESMTP Service Extension for Authentication -- link:https://datatracker.ietf.org/doc/html/rfc1893[RFC-1893] Enhanced Mail System Status Codes - -== LMTP - - - link:https://james.apache.org/server/rfclist/lmtp/rfc2033.txt[RFC-2033] LMTP Local Mail Transfer Protocol - -== IMAP - -The following IMAP specifications are implemented: - - - link:https://datatracker.ietf.org/doc/html/rfc3501.html[RFC-3501] INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1 - - link:https://datatracker.ietf.org/doc/html/rfc2177.html[RFC-2177] IMAP IDLE (mailbox scoped push notifications) - - link:https://www.rfc-editor.org/rfc/rfc9208.html[RFC-9208] IMAP QUOTA Extension - - link:https://datatracker.ietf.org/doc/html/rfc2342.html[RFC-2342] IMAP namespace - - link:https://datatracker.ietf.org/doc/html/rfc2088.html[RFC-2088] IMAP non synchronized literals - - link:https://datatracker.ietf.org/doc/html/rfc4315.html[RFC-4315] IMAP UIDPLUS - - link:https://datatracker.ietf.org/doc/html/rfc5464.html[RFC-5464] IMAP Metadata (annotations on mailboxes) - - link:https://datatracker.ietf.org/doc/html/rfc4551.html[RFC-4551] IMAP Condstore - - link:https://datatracker.ietf.org/doc/html/rfc5162.html[RFC-5162] IMAP QRESYNC (synchronisation semantic for deleted messages) - - We don't store a log of deleted modseq thus clients should rely on known sequences mechanism to optimize exchanges. - - link:https://datatracker.ietf.org/doc/html/rfc4978.html[RFC-4978] IMAP Compress (optional) - - link:https://datatracker.ietf.org/doc/html/rfc5161.html[RFC-5161] IMAP ENABLE - - link:https://datatracker.ietf.org/doc/html/rfc6851.html[RFC-6851] IMAP MOVE command - - link:https://datatracker.ietf.org/doc/html/rfc5182.html[RFC-5182] IMAP Extension for Referencing the Last SEARCH Result - - link:https://datatracker.ietf.org/doc/html/rfc5032.html[RFC-5032] IMAP WITHIN (for relative date search semantic) - - link:https://datatracker.ietf.org/doc/html/rfc4731.html[RFC-4731] IMAP ESEARCH: extentions for IMAP search: new options like min, max, count. - - link:https://datatracker.ietf.org/doc/html/rfc3348.html[RFC-3348] IMAP Child Mailbox Extension - - link:https://www.rfc-editor.org/rfc/rfc8508.html[RFC-8508] IMAP Replace Extension - - link:https://www.rfc-editor.org/rfc/rfc7889.html[RFC-7889] IMAP Extension for APPENDLIMIT - - link:https://www.rfc-editor.org/rfc/rfc8474.html[RFC-8474] IMAP Extension for Object Identifiers - - link:https://datatracker.ietf.org/doc/html/rfc2971.html[RFC-2971] IMAP ID Extension - - link:https://datatracker.ietf.org/doc/html/rfc8438.html[RFC-8438] IMAP Extension for STATUS=SIZE - - link:https://www.rfc-editor.org/rfc/rfc5258.html[RFC-5258] IMAP LIST Command Extensions - - link:https://www.rfc-editor.org/rfc/rfc5819.html[RFC-5819] IMAP4 Extension for Returning STATUS Information in Extended LIST - - link:https://www.rfc-editor.org/rfc/rfc8440.html[RFC-8440] IMAP4 Extension for Returning MYRIGHTS Information in Extended LIST - - link:https://www.rfc-editor.org/rfc/rfc8440.html[RFC-6154] IMAP LIST Extension for Special-Use Mailboxes - - link:https://www.rfc-editor.org/rfc/rfc8514.html[RFC-8514] IMAP SAVEDATE Extension - - link:https://www.rfc-editor.org/rfc/rfc8514.html[RFC-9394] IMAP PARTIAL Extension for Paged SEARCH and FETCH - -Partially implemented specifications: - - - link:https://datatracker.ietf.org/doc/html/rfc4314.html[RFC-4314] IMAP ACL - - ACLs can be created and managed but mailbox not belonging to one account cannot, as of today, be accessed in IMAP. - -== JMAP - - - link:https://datatracker.ietf.org/doc/html/rfc8620[RFC-8620] Json Metadata Application Protocol (JMAP) - - link:https://datatracker.ietf.org/doc/html/rfc8621[RFC-8621] JMAP for emails - - link:https://datatracker.ietf.org/doc/html/rfc8887[RFC-8887] JMAP over websockets - - link:https://datatracker.ietf.org/doc/html/rfc9007.html[RFC-9007] Message Delivery Notifications with JMAP. - - link:https://datatracker.ietf.org/doc/html/rfc8030.html[RFC-8030] Web PUSH: JMAP enable sending push notifications through a push gateway. - -https://jmap.io/[JMAP] is intended to be a new standard for email clients to connect to mail -stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more -generic. It does not replace MTA-to-MTA SMTP transmission. - -The link:https://github.com/apache/james-project/tree/master/server/protocols/jmap-rfc-8621/doc/specs/spec[annotated documentation] -presents the limits of the JMAP RFC-8621 implementation part of the Apache James project. - -Some methods / types are not yet implemented, some implementations are naive, and the PUSH is not supported yet. - -Users are invited to read these limitations before using actively the JMAP RFC-8621 implementation, and should ensure their -client applications only uses supported operations. - -== POP3 - - - link:https://www.ietf.org/rfc/rfc1939.txt[RFC-1939] Post Office Protocol - Version 3 - -== ManageSieve - -Support for manageSieve is experimental. - - - link:https://datatracker.ietf.org/doc/html/rfc5804[RFC-5804] A Protocol for Remotely Managing Sieve Scripts - -== Sieve - - - link:https://datatracker.ietf.org/doc/html/rfc5228[RFC-5228] Sieve: An Email Filtering Language - - link:https://datatracker.ietf.org/doc/html/rfc5173[RFC-5173] Sieve Email Filtering: Body Extension - - link:https://datatracker.ietf.org/doc/html/rfc5230[RFC-5230] Sieve Email Filtering: Vacation Extension - +:server-name: Distributed James Server +include::partial$architecture/implemented-standards.adoc[] \ No newline at end of file diff --git a/docs/modules/servers/pages/distributed/architecture/index.adoc b/docs/modules/servers/pages/distributed/architecture/index.adoc index ba5c25541a..a36103cc8c 100644 --- a/docs/modules/servers/pages/distributed/architecture/index.adoc +++ b/docs/modules/servers/pages/distributed/architecture/index.adoc @@ -1,308 +1,13 @@ = Distributed James Server — Architecture :navtitle: Architecture -This sections presents the Distributed Server architecture. - -== Storage - -In order to deliver its promises, the Distributed Server leverages the following storage strategies: - -image::storage.png[Storage responsibilities for the Distributed Server] - - * *Cassandra* is used for metadata storage. Cassandra is efficient for a very high workload of small queries following -a known pattern. - * The *blob store* storage interface is responsible of storing potentially large binary data. For instance - email bodies, headers or attachments. Different technologies can be used: *Cassandra*, or S3 compatible *Object Storage* -(S3 or Swift). - * *OpenSearch* component empowers full text search on emails. It also enables querying data with unplanned access -patterns. OpenSearch throughput do not however match the one of Cassandra thus its use is avoided upon regular workloads. - * *RabbitMQ* enables James nodes of a same cluster to collaborate together. It is used to implement connected protocols, -notification patterns as well as distributed resilient work queues and mail queue. - * *Tika* (optional) enables text extraction from attachments, thus improving full text search results. - * *link:https://spamassassin.apache.org/[SpamAssassin] or link:https://rspamd.com/[Rspamd]* (optional) can be used for Spam detection and user feedback is supported. - -xref:distributed/architecture/consistency-model.adoc[This page] further details Distributed James consistency model. - -== Protocols - -The following protocols are supported and can be used to interact with the Distributed Server: - -* *SMTP* -* *IMAP* -* xref:distributed/operate/webadmin.adoc[WebAdmin] REST Administration API -* *LMTP* -* *POP3* - -The following protocols should be considered experimental: - -* *JMAP* (RFC-8620 &RFC-8621 specifications and known limitations of the James implementation are defined link:https://github.com/apache/james-project/tree/master/server/protocols/jmap-rfc-8621/doc[here]) -* *ManagedSieve* - -Read more on xref:distributed/architecture/implemented-standards.adoc[implemented standards]. - -== Topology - -While it is perfectly possible to deploy homogeneous James instances, with the same configuration and thus the same -protocols and the same responsibilities one might want to investigate in -xref:distributed/architecture/specialized-instances.adoc['Specialized instances']. - -== Components - -This section presents the various components of the Distributed server, providing context about -their interactions, and about their implementations. - -=== High level view - -Here is a high level view of the various server components and their interactions: - -image::server-components.png[Server components mobilized for SMTP & IMAP] - - - The SMTP protocol receives a mail, and enqueue it on the MailQueue - - The MailetContainer will start processing the mail Asynchronously and will take business decisions like storing the - email locally in a user mailbox. The behaviour of the MailetContainer is highly customizable thanks to the Mailets and - the Matcher composibility. - - The Mailbox component is responsible of storing a user's mails. - - The user can use the IMAP or the JMAP protocol to retrieve and read his mails. - -These components will be presented more in depth below. - -=== Mail processing - -Mail processing allows to take asynchronously business decisions on -received emails. - -Here are its components: - -* The `spooler` takes mail out of the mailQueue and executes mail -processing within the `mailet container`. -* The `mailet container` synchronously executes the user defined logic. -This `logic' is written through the use of `mailet`, `matcher` and -`processor`. -* A `mailet` represents an action: mail modification, envelop -modification, a side effect, or stop processing. -* A `matcher` represents a condition to execute a mailet. -* A `processor` is a flow of pair of `matcher` and `mailet` executed -sequentially. The `ToProcessor` mailet is a `goto` instruction to start -executing another `processor` -* A `mail repository` allows storage of a mail as part of its -processing. Standard configuration relies on the following mail -repository: -** `cassandra://var/mail/error/` : unexpected errors that occurred -during mail processing. Emails impacted by performance related -exceptions, or logical bug within James code are typically stored here. -These mails could be reprocessed once the cause of the error is fixed. -The `Mail.error` field can help diagnose the issue. Correlation with -logs can be achieved via the use of the `Mail.name` field. -** `cassandra://var/mail/address-error/` : mail addressed to a -non-existing recipient of a handled local domain. These mails could be -reprocessed once the user is created, for instance. -** `cassandra://var/mail/relay-denied/` : mail for whom relay was -denied: missing authentication can, for instance, be a cause. In -addition to prevent disasters upon miss configuration, an email review -of this mail repository can help refine a host spammer blacklist. -** `cassandra://var/mail/rrt-error/` : runtime error upon Recipient -Rewriting occurred. This is typically due to a loop. - -=== Mail Queue - -An email queue is a mandatory component of SMTP servers. It is a system -that creates a queue of emails that are waiting to be processed for -delivery. Email queuing is a form of Message Queuing – an asynchronous -service-to-service communication. A message queue is meant to decouple a -producing process from a consuming one. An email queue decouples email -reception from email processing. It allows them to communicate without -being connected. As such, the queued emails wait for processing until -the recipient is available to receive them. As James is an Email Server, -it also supports mail queue as well. - -==== Why Mail Queue is necessary - -You might often need to check mail queue to make sure all emails are -delivered properly. At first, you need to know why email queues get -clogged. Here are the two core reasons for that: - -* Exceeded volume of emails - -Some mailbox providers enforce email rate limits on IP addresses. The -limits are based on the sender reputation. If you exceeded this rate and -queued too many emails, the delivery speed will decrease. - -* Spam-related issues - -Another common reason is that your email has been busted by spam -filters. The filters will let the emails gradually pass to analyze how -the rest of the recipients react to the message. If there is slow -progress, it’s okay. Your email campaign is being observed and assessed. -If it’s stuck, there could be different reasons including the blockage -of your IP address. - -==== Why combining Cassandra, RabbitMQ and Object storage for MailQueue - -* RabbitMQ ensures the messaging function, and avoids polling. -* Cassandra enables administrative operations such as browsing, deleting -using a time series which might require fine performance tuning (see -http://cassandra.apache.org/doc/latest/operating/index.html[Operating -Casandra documentation]). -* Object Storage stores potentially large binary payload. - -However the current design do not implement delays. Delays allow to -define the time a mail have to be living in the mailqueue before being -dequeued and is used for example for exponential wait delays upon remote -delivery retries, or - -=== Mailbox - -Storage for emails belonging for users. - -Metadata are stored in Cassandra while headers, bodies and attachments are stored -within the xref:#_blobstore[BlobStore]. - -==== Search index - -Emails are indexed asynchronously in OpenSearch via the xref:#_event_bus[EventBus] -in order to empower advanced and fast email full text search. - -Text extraction can be set up using link:https://tika.apache.org/[Tika], allowing -to extract the text from attachment, allowing to search your emails based on the attachment -textual content. In such case, the OpenSearch indexer will call a Tika server prior -indexing. - -==== Quotas - -Current Quotas of users are hold in a Cassandra projection. Limitations can be defined via -user, domain or globally. - -==== Event Bus - -Distributed James relies on an event bus system to enrich mailbox capabilities. Each -operation performed on the mailbox will trigger related events, that can -be processed asynchronously by potentially any James node on a -distributed system. - -Many different kind of events can be triggered during a mailbox -operation, such as: - -* `MailboxEvent`: event related to an operation regarding a mailbox: -** `MailboxDeletion`: a mailbox has been deleted -** `MailboxAdded`: a mailbox has been added -** `MailboxRenamed`: a mailbox has been renamed -** `MailboxACLUpdated`: a mailbox got its rights and permissions updated -* `MessageEvent`: event related to an operation regarding a message: -** `Added`: messages have been added to a mailbox -** `Expunged`: messages have been expunged from a mailbox -** `FlagsUpdated`: messages had their flags updated -** `MessageMoveEvent`: messages have been moved from a mailbox to an -other -* `QuotaUsageUpdatedEvent`: event related to quota update - -Mailbox listeners can register themselves on this event bus system to be -called when an event is fired, allowing to do different kind of extra -operations on the system, like: - -* Current quota calculation -* Message indexation with OpenSearch -* Mailbox annotations cleanup -* Ham/spam reporting to Spam filtering system -* … - -==== Deleted Messages Vault - -Deleted Messages Vault is an interesting feature that will help James -users have a chance to: - -* retain users deleted messages for some time. -* restore & export deleted messages by various criteria. -* permanently delete some retained messages. - -If the Deleted Messages Vault is enabled when users delete their mails, -and by that we mean when they try to definitely delete them by emptying -the trash, James will retain these mails into the Deleted Messages -Vault, before an email or a mailbox is going to be deleted. And only -administrators can interact with this component via -wref:webadmin.adoc#_deleted-messages-vault[WebAdmin] REST APIs]. - -However, mails are not retained forever as you have to configure a -retention period before using it (with one-year retention by default if -not defined). It’s also possible to permanently delete a mail if needed. - -=== Data - -Storage for domains and users. - -Domains are persisted in Cassandra. - -Users can be managed in Cassandra, or via a LDAP (read only). - -=== Recipient rewrite tables - -Storage of Recipients Rewriting rules, in Cassandra. - -==== Mapping types - -James allows using various mapping types for better expressing the intent of your address rewriting logic: - -* *Domain mapping*: Rewrites the domain of mail addresses. Use it for technical purposes, user will not -be allowed to use the source in their FROM address headers. Domain mappings can be managed via the CLI and -added via xref:distributed/operate/webadmin.adoc#_domain_mappings[WebAdmin] -* *Domain aliases*: Rewrites the domain of mail addresses. Express the idea that both domains can be used -inter-changeably. User will be allowed to use the source in their FROM address headers. Domain aliases can -be managed via xref:distributed/operate/webadmin.adoc#_get_the_list_of_aliases_for_a_domain[WebAdmin] -* *Forwards*: Replaces the source address by another one. Vehicles the intent of forwarding incoming mails -to other users. Listing the forward source in the forward destinations keeps a local copy. User will not be -allowed to use the source in their FROM address headers. Forward can -be managed via xref:distributed/operate/webadmin.adoc#_address_forwards[WebAdmin] -* *Groups*: Replaces the source address by another one. Vehicles the intent of a group registration: group -address will be swapped by group member addresses (Feature poor mailing list). User will not be -allowed to use the source in their FROM address headers. Groups can -be managed via xref:distributed/operate/webadmin.adoc#_address_group[WebAdmin] -* *Aliases*: Replaces the source address by another one. Represents user owned mail address, with which -he can interact as if it was his main mail address. User will be allowed to use the source in their FROM -address headers. Aliases can be managed via xref:distributed/operate/webadmin.adoc#_address_aliases[WebAdmin] -* *Address mappings*: Replaces the source address by another one. Use for technical purposes, this mapping type do -not hold specific intent. Prefer using one of the above mapping types... User will not be allowed to use the source -in their FROM address headers. Address mappings can be managed via the CLI or via -xref:distributed/operate/webadmin.adoc#_address_mappings[WebAdmin] -* *Regex mappings*: Applies the regex on the supplied address. User will not be allowed to use the source -in their FROM address headers. Regex mappings can be managed via the CLI or via -xref:distributed/operate/webadmin.adoc#_regex_mapping[WebAdmin] -* *Error*: Throws an error upon processing. User will not be allowed to use the source -in their FROM address headers. Errors can be managed via the CLI - -=== BlobStore - -Stores potentially large binary data. - -Mailbox component, Mail Queue component, Deleted Message Vault -component relies on it. - -Supported backends include S3 compatible ObjectStorage (link:https://wiki.openstack.org/wiki/Swift[Swift], S3 API). - -Encryption can be configured on top of ObjectStorage. - -Blobs can currently be deduplicated in order to reduce storage space. This means that two blobs with -the same content will be stored one once. - -The downside is that deletion is more complicated, and a garbage collection needs to be run. A first implementation -based on bloom filters can be used and triggered using the WebAdmin REST API. - -=== Task Manager - -Allows to control and schedule long running tasks run by other -components. Among other it enables scheduling, progress monitoring, -cancellation of long running tasks. - -Distributed James leverage a task manager using Event Sourcing and RabbitMQ for messaging. - -=== Event sourcing - -link:https://martinfowler.com/eaaDev/EventSourcing.html[Event sourcing] implementation -for the Distributed server stores events in Cassandra. It enables components -to rely on event sourcing technics for taking decisions. - -A short list of usage are: - -* Data leak prevention storage -* JMAP filtering rules storage -* Validation of the MailQueue configuration -* Sending email warnings to user close to their quota -* Implementation of the TaskManager +:backend-name: cassandra +:server-name: Distributed James Server +:backend-storage-introduce: Cassandra is used for metadata storage. Cassandra is efficient for a very high workload of small queries following a known pattern. +:storage-picture-file-name: storage_james_distributed.png +:mailet-repository-path-prefix: cassandra +:xref-base: distributed +:mailqueue-combined-extend-backend: , Cassandra +:mailqueue-combined-extend: servers:distributed/architecture/mailqueue_combined_extend.adoc + +include::partial$architecture/index.adoc[] \ No newline at end of file diff --git a/docs/modules/servers/pages/distributed/architecture/mailqueue_combined_extend.adoc b/docs/modules/servers/pages/distributed/architecture/mailqueue_combined_extend.adoc new file mode 100644 index 0000000000..2e381417e5 --- /dev/null +++ b/docs/modules/servers/pages/distributed/architecture/mailqueue_combined_extend.adoc @@ -0,0 +1,4 @@ +* Cassandra enables administrative operations such as browsing, deleting +using a time series which might require fine performance tuning (see +http://cassandra.apache.org/doc/latest/operating/index.html[Operating +Cassandra documentation]). \ No newline at end of file diff --git a/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc b/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc index 5c7365da4b..03a412f0e2 100644 --- a/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc +++ b/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc @@ -1,39 +1,7 @@ = Distributed James Server — Specialized instances :navtitle: Specialized instances -While it is perfectly possible to deploy homogeneous James instances, with the same configuration and thus the same -protocols and the same responsibilities one might want to investigate in 'Specialized instances'. +:server-name: Distributed James Server +:specialized-instances-file-name: specialized-instances-distributed.png -This deployment topology consists of Distributed James servers with heterogeneous configurations on top of shared -data-bases. Groups of James servers will thus handle various protocols and have different responsibilities. - -This approach limits cascading failures across protocols and services. Think of *OutOfMemoryErrors*, Cassandra driver -queue overuse, CPUs starvation, etc. - -However, we can't speak of micro-services here: all James instances runs the same code, James is still a monolith, and -databases need to be shared across instances. - -image::specialized-instances.png[Example of Specialized instances topology] - -We speak of: - - - **Front-line servers** serves protocols. James enables to easily turn protocols on and off. Typically, each protocol would - be isolated in its own group of James instances: james-imap, james-jmap, james-smtp, james-webadmin, etc... Refer to - protocols configuration files to learn more. - - - **Back-office servers** handles other services like: - - - Mail processing. - - Remote delivery. - - Event processing. - - Task execution. - -Front-line servers will likely not handle back office responsibilities (but be sure to have back-office servers that do!). - - - xref:distributed/configure/mailetcontainer.adoc[Mail processing can be switched off]. - - xref:distributed/configure/listeners.adoc[Mailbox event processing can be switched off]. - - xref:distributed/configure/rabbitmq.adoc[Task execution can be switched off]. - - Remote Delivery service is not started if the RemoteDelivery mailet is not positioned in mailetcontainer.xml. - -Of course, the above instances can be collocated at will, to reach some intermediate deployments with fewer -instances to mitigate costs. \ No newline at end of file +include::partial$architecture/specialized-instances.adoc[] \ No newline at end of file diff --git a/docs/modules/servers/partials/architecture/consistency-model.adoc b/docs/modules/servers/partials/architecture/consistency-model.adoc new file mode 100644 index 0000000000..c104535b0a --- /dev/null +++ b/docs/modules/servers/partials/architecture/consistency-model.adoc @@ -0,0 +1,65 @@ +This page presents the consistency model used by the {server-name} and +points to the tools built around it. + +== Data Replication + +The {server-name} relies on different storage technologies, all having their own +consistency models. + +These data stores replicate data in order to enforce some level of availability. We call +this process replication. By consistency, we mean the ability for all replica to hold the +same data. By availability, we mean the ability for a replica to answer a request. + +In distributed systems, link:https://en.wikipedia.org/wiki/CAP_theorem[according to the CAP theorem], +as we will necessarily encounter network partitions, then trade-offs need to be made between +consistency and availability. + +This section details this trade-off for data stores used by the {server-name}. + +=== OpenSearch consistency model + +OpenSearch relies on link:https://opensearch.org/docs/latest/tuning-your-cluster/[strong consistency] +with home-grown algorithm. + +The 2.x release line, that the distributed server is using, is known to provide faster recovery. + +Be aware that data is asynchronously indexed in OpenSearch, changes will be eventually visible. + +=== RabbitMQ consistency model + +The {server-name} relies out of the box on a single RabbitMQ server, thus consistency concerns +are not (yet) applicable. Availability concerns are applicable. + +include::{data_replication_extend}[] + +== Consistency across data stores + +The {server-name} leverages several data stores: + + - {backend-name} is used for metadata storage + - OpenSearch for search + - Object Storage for large object storage + +Thus the {server-name} also offers mechanisms to enforce consistency across data stores. + +=== Write path organisation + +The primary data stores are composed of {backend-name} for metadata and Object storage for binary data. + +To ensure the data referenced in {backend-name} is pointing to a valid object in the object store, we write +the object store payload first, then write the corresponding metadata in {backend-name}. + +Such a procedure avoids metadata pointing to un existing blobs, however might lead to some unreferenced +blobs. + +=== {backend-name-cap} ↔ OpenSearch + +After being written to the primary stores (namely {backend-name} & Object Storage), email content is +asynchronously indexed into OpenSearch. + +This process, called the EventBus, which retries temporary errors, and stores transient errors for +later admin-triggered retries is described further xref:{xref-base}/operate/guide.adoc#_mailbox_event_bus[here]. +His role is to spread load and limit inconsistencies. + +Furthermore, some xref:{xref-base}/operate/guide.adoc#_usual_troubleshooting_procedures[re-indexing tasks] +enables to re-synchronise OpenSearch content with the primary data stores diff --git a/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc b/docs/modules/servers/partials/architecture/implemented-standards.adoc similarity index 98% copy from docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc copy to docs/modules/servers/partials/architecture/implemented-standards.adoc index 566e40a9b3..707c1bcc7a 100644 --- a/docs/modules/servers/pages/distributed/architecture/implemented-standards.adoc +++ b/docs/modules/servers/partials/architecture/implemented-standards.adoc @@ -1,7 +1,4 @@ -= Distributed James Server — Implemented standards -:navtitle: Implemented standards - -This page details standards implemented by the distributed server. +This page details standards implemented by the {server-name}. == Message formats diff --git a/docs/modules/servers/pages/distributed/architecture/index.adoc b/docs/modules/servers/partials/architecture/index.adoc similarity index 75% copy from docs/modules/servers/pages/distributed/architecture/index.adoc copy to docs/modules/servers/partials/architecture/index.adoc index ba5c25541a..449a31c99e 100644 --- a/docs/modules/servers/pages/distributed/architecture/index.adoc +++ b/docs/modules/servers/partials/architecture/index.adoc @@ -1,35 +1,31 @@ -= Distributed James Server — Architecture -:navtitle: Architecture - -This sections presents the Distributed Server architecture. +This sections presents the {server-name} architecture. == Storage -In order to deliver its promises, the Distributed Server leverages the following storage strategies: +In order to deliver its promises, the {server-name} leverages the following storage strategies: -image::storage.png[Storage responsibilities for the Distributed Server] +image::{storage-picture-file-name}[Storage responsibilities for the {server-name}] - * *Cassandra* is used for metadata storage. Cassandra is efficient for a very high workload of small queries following -a known pattern. - * The *blob store* storage interface is responsible of storing potentially large binary data. For instance - email bodies, headers or attachments. Different technologies can be used: *Cassandra*, or S3 compatible *Object Storage* + * {backend-storage-introduce} + * The *blob store* storage interface is responsible for storing potentially large binary data. For instance + email bodies, headers or attachments. Different technologies can be used: *{backend-name}*, or S3 compatible *Object Storage* (S3 or Swift). * *OpenSearch* component empowers full text search on emails. It also enables querying data with unplanned access -patterns. OpenSearch throughput do not however match the one of Cassandra thus its use is avoided upon regular workloads. +patterns. OpenSearch throughput do not however match the one of {backend-name} thus its use is avoided upon regular workloads. * *RabbitMQ* enables James nodes of a same cluster to collaborate together. It is used to implement connected protocols, notification patterns as well as distributed resilient work queues and mail queue. * *Tika* (optional) enables text extraction from attachments, thus improving full text search results. * *link:https://spamassassin.apache.org/[SpamAssassin] or link:https://rspamd.com/[Rspamd]* (optional) can be used for Spam detection and user feedback is supported. -xref:distributed/architecture/consistency-model.adoc[This page] further details Distributed James consistency model. +xref:{xref-base}/architecture/consistency-model.adoc[This page] further details {server-name} consistency model. == Protocols -The following protocols are supported and can be used to interact with the Distributed Server: +The following protocols are supported and can be used to interact with the {server-name}: * *SMTP* * *IMAP* -* xref:distributed/operate/webadmin.adoc[WebAdmin] REST Administration API +* xref:{xref-base}/operate/webadmin.adoc[WebAdmin] REST Administration API * *LMTP* * *POP3* @@ -38,17 +34,17 @@ The following protocols should be considered experimental: * *JMAP* (RFC-8620 &RFC-8621 specifications and known limitations of the James implementation are defined link:https://github.com/apache/james-project/tree/master/server/protocols/jmap-rfc-8621/doc[here]) * *ManagedSieve* -Read more on xref:distributed/architecture/implemented-standards.adoc[implemented standards]. +Read more on xref:{xref-base}/architecture/implemented-standards.adoc[implemented standards]. == Topology While it is perfectly possible to deploy homogeneous James instances, with the same configuration and thus the same protocols and the same responsibilities one might want to investigate in -xref:distributed/architecture/specialized-instances.adoc['Specialized instances']. +xref:{xref-base}/architecture/specialized-instances.adoc['Specialized instances']. == Components -This section presents the various components of the Distributed server, providing context about +This section presents the various components of the {server-name}, providing context about their interactions, and about their implementations. === High level view @@ -87,20 +83,20 @@ executing another `processor` * A `mail repository` allows storage of a mail as part of its processing. Standard configuration relies on the following mail repository: -** `cassandra://var/mail/error/` : unexpected errors that occurred +** `{mailet-repository-path-prefix}://var/mail/error/` : unexpected errors that occurred during mail processing. Emails impacted by performance related exceptions, or logical bug within James code are typically stored here. These mails could be reprocessed once the cause of the error is fixed. The `Mail.error` field can help diagnose the issue. Correlation with logs can be achieved via the use of the `Mail.name` field. -** `cassandra://var/mail/address-error/` : mail addressed to a +** `{mailet-repository-path-prefix}://var/mail/address-error/` : mail addressed to a non-existing recipient of a handled local domain. These mails could be reprocessed once the user is created, for instance. -** `cassandra://var/mail/relay-denied/` : mail for whom relay was +** `{mailet-repository-path-prefix}://var/mail/relay-denied/` : mail for whom relay was denied: missing authentication can, for instance, be a cause. In addition to prevent disasters upon miss configuration, an email review of this mail repository can help refine a host spammer blacklist. -** `cassandra://var/mail/rrt-error/` : runtime error upon Recipient +** `{mailet-repository-path-prefix}://var/mail/rrt-error/` : runtime error upon Recipient Rewriting occurred. This is typically due to a loop. === Mail Queue @@ -136,17 +132,15 @@ progress, it’s okay. Your email campaign is being observed and assessed. If it’s stuck, there could be different reasons including the blockage of your IP address. -==== Why combining Cassandra, RabbitMQ and Object storage for MailQueue +==== Why combining RabbitMQ, Object storage {mailqueue-combined-extend-backend} for MailQueue * RabbitMQ ensures the messaging function, and avoids polling. -* Cassandra enables administrative operations such as browsing, deleting -using a time series which might require fine performance tuning (see -http://cassandra.apache.org/doc/latest/operating/index.html[Operating -Casandra documentation]). * Object Storage stores potentially large binary payload. -However the current design do not implement delays. Delays allow to -define the time a mail have to be living in the mailqueue before being +include::{mailqueue-combined-extend}[] + +However, the current design do not implement delays. Delays allow to +define the time a mail have to be living in the mail queue before being dequeued and is used for example for exponential wait delays upon remote delivery retries, or @@ -154,7 +148,7 @@ delivery retries, or Storage for emails belonging for users. -Metadata are stored in Cassandra while headers, bodies and attachments are stored +Metadata are stored in {backend-name} while headers, bodies and attachments are stored within the xref:#_blobstore[BlobStore]. ==== Search index @@ -169,12 +163,12 @@ indexing. ==== Quotas -Current Quotas of users are hold in a Cassandra projection. Limitations can be defined via +Current Quotas of users are hold in a {backend-name} projection. Limitations can be defined via user, domain or globally. ==== Event Bus -Distributed James relies on an event bus system to enrich mailbox capabilities. Each +{server-name} relies on an event bus system to enrich mailbox capabilities. Each operation performed on the mailbox will trigger related events, that can be processed asynchronously by potentially any James node on a distributed system. @@ -191,8 +185,7 @@ operation, such as: ** `Added`: messages have been added to a mailbox ** `Expunged`: messages have been expunged from a mailbox ** `FlagsUpdated`: messages had their flags updated -** `MessageMoveEvent`: messages have been moved from a mailbox to an -other +** `MessageMoveEvent`: messages have been moved from a mailbox to another * `QuotaUsageUpdatedEvent`: event related to quota update Mailbox listeners can register themselves on this event bus system to be @@ -219,7 +212,7 @@ and by that we mean when they try to definitely delete them by emptying the trash, James will retain these mails into the Deleted Messages Vault, before an email or a mailbox is going to be deleted. And only administrators can interact with this component via -wref:webadmin.adoc#_deleted-messages-vault[WebAdmin] REST APIs]. +xref:{xref-base}/operate/webadmin.adoc#_deleted_messages_vault[WebAdmin] REST APIs. However, mails are not retained forever as you have to configure a retention period before using it (with one-year retention by default if @@ -229,13 +222,13 @@ not defined). It’s also possible to permanently delete a mail if needed. Storage for domains and users. -Domains are persisted in Cassandra. +Domains are persisted in {backend-name}. -Users can be managed in Cassandra, or via a LDAP (read only). +Users can be managed in {backend-name}, or via a LDAP (read only). === Recipient rewrite tables -Storage of Recipients Rewriting rules, in Cassandra. +Storage of Recipients Rewriting rules, in {backend-name}. ==== Mapping types @@ -243,31 +236,32 @@ James allows using various mapping types for better expressing the intent of you * *Domain mapping*: Rewrites the domain of mail addresses. Use it for technical purposes, user will not be allowed to use the source in their FROM address headers. Domain mappings can be managed via the CLI and -added via xref:distributed/operate/webadmin.adoc#_domain_mappings[WebAdmin] +added via xref:{xref-base}/operate/webadmin.adoc#_domain_mappings[WebAdmin] * *Domain aliases*: Rewrites the domain of mail addresses. Express the idea that both domains can be used inter-changeably. User will be allowed to use the source in their FROM address headers. Domain aliases can -be managed via xref:distributed/operate/webadmin.adoc#_get_the_list_of_aliases_for_a_domain[WebAdmin] +be managed via xref:{xref-base}/operate/webadmin.adoc#_get_the_list_of_aliases_for_a_domain[WebAdmin] * *Forwards*: Replaces the source address by another one. Vehicles the intent of forwarding incoming mails to other users. Listing the forward source in the forward destinations keeps a local copy. User will not be allowed to use the source in their FROM address headers. Forward can -be managed via xref:distributed/operate/webadmin.adoc#_address_forwards[WebAdmin] +be managed via xref:{xref-base}/operate/webadmin.adoc#_address_forwards[WebAdmin] * *Groups*: Replaces the source address by another one. Vehicles the intent of a group registration: group address will be swapped by group member addresses (Feature poor mailing list). User will not be allowed to use the source in their FROM address headers. Groups can -be managed via xref:distributed/operate/webadmin.adoc#_address_group[WebAdmin] +be managed via xref:{xref-base}/operate/webadmin.adoc#_address_group[WebAdmin] * *Aliases*: Replaces the source address by another one. Represents user owned mail address, with which he can interact as if it was his main mail address. User will be allowed to use the source in their FROM -address headers. Aliases can be managed via xref:distributed/operate/webadmin.adoc#_address_aliases[WebAdmin] -* *Address mappings*: Replaces the source address by another one. Use for technical purposes, this mapping type do -not hold specific intent. Prefer using one of the above mapping types... User will not be allowed to use the source -in their FROM address headers. Address mappings can be managed via the CLI or via -xref:distributed/operate/webadmin.adoc#_address_mappings[WebAdmin] -* *Regex mappings*: Applies the regex on the supplied address. User will not be allowed to use the source -in their FROM address headers. Regex mappings can be managed via the CLI or via -xref:distributed/operate/webadmin.adoc#_regex_mapping[WebAdmin] -* *Error*: Throws an error upon processing. User will not be allowed to use the source -in their FROM address headers. Errors can be managed via the CLI - +address headers. Aliases can be managed via xref:{xref-base}/operate/webadmin.adoc#_address_aliases[WebAdmin] +* *Address mappings*: Replaces the source address by another one.Use for technical purposes, this mapping type do +not hold specific intent.Prefer using one of the above mapping types... User will not be allowed to use the source +in their FROM address headers.Address mappings can be managed via the CLI or via +xref:{xref-base}/operate/webadmin.adoc#_address_mappings[WebAdmin] +* *Regex mappings*: Applies the regex on the supplied address.User will not be allowed to use the source +in their FROM address headers.Regex mappings can be managed via the CLI or via +xref:{xref-base}/operate/webadmin.adoc#_regex_mapping[WebAdmin] +* *Error*: Throws an error upon processing.User will not be allowed to use the source +in their FROM address headers.Errors can be managed via the CLI + +[#_blobstore] === BlobStore Stores potentially large binary data. @@ -291,12 +285,12 @@ Allows to control and schedule long running tasks run by other components. Among other it enables scheduling, progress monitoring, cancellation of long running tasks. -Distributed James leverage a task manager using Event Sourcing and RabbitMQ for messaging. +{server-name} leverage a task manager using Event Sourcing and RabbitMQ for messaging. === Event sourcing link:https://martinfowler.com/eaaDev/EventSourcing.html[Event sourcing] implementation -for the Distributed server stores events in Cassandra. It enables components +for the {server-name} stores events in {backend-name}. It enables components to rely on event sourcing technics for taking decisions. A short list of usage are: diff --git a/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc b/docs/modules/servers/partials/architecture/specialized-instances.adoc similarity index 59% copy from docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc copy to docs/modules/servers/partials/architecture/specialized-instances.adoc index 5c7365da4b..d8e02b1dc7 100644 --- a/docs/modules/servers/pages/distributed/architecture/specialized-instances.adoc +++ b/docs/modules/servers/partials/architecture/specialized-instances.adoc @@ -1,19 +1,16 @@ -= Distributed James Server — Specialized instances -:navtitle: Specialized instances - While it is perfectly possible to deploy homogeneous James instances, with the same configuration and thus the same protocols and the same responsibilities one might want to investigate in 'Specialized instances'. -This deployment topology consists of Distributed James servers with heterogeneous configurations on top of shared -data-bases. Groups of James servers will thus handle various protocols and have different responsibilities. +This deployment topology consists of {server-name} with heterogeneous configurations on top of shared +databases. Groups of James servers will thus handle various protocols and have different responsibilities. -This approach limits cascading failures across protocols and services. Think of *OutOfMemoryErrors*, Cassandra driver -queue overuse, CPUs starvation, etc. +This approach limits cascading failures across protocols and services. Think of *OutOfMemoryErrors*, CPUs starvation, +{backend-name} driver issue, etc. -However, we can't speak of micro-services here: all James instances runs the same code, James is still a monolith, and +However, we can't speak of microservices here: all James instances runs the same code, James is still a monolith, and databases need to be shared across instances. -image::specialized-instances.png[Example of Specialized instances topology] +image::{specialized-instances-file-name}[Example of Specialized instances topology] We speak of: @@ -30,9 +27,9 @@ We speak of: Front-line servers will likely not handle back office responsibilities (but be sure to have back-office servers that do!). - - xref:distributed/configure/mailetcontainer.adoc[Mail processing can be switched off]. - - xref:distributed/configure/listeners.adoc[Mailbox event processing can be switched off]. - - xref:distributed/configure/rabbitmq.adoc[Task execution can be switched off]. + - xref:{xref-base}/configure/mailetcontainer.adoc[Mail processing can be switched off]. + - xref:{xref-base}/configure/listeners.adoc[Mailbox event processing can be switched off]. + - xref:{xref-base}/configure/rabbitmq.adoc[Task execution can be switched off]. - Remote Delivery service is not started if the RemoteDelivery mailet is not positioned in mailetcontainer.xml. Of course, the above instances can be collocated at will, to reach some intermediate deployments with fewer --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
