This is an automated email from the ASF dual-hosted git repository. chibenwa pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f44c40f825d8f73e8e190d47b7137bb6d53c25ac Author: Benoit TELLIER <[email protected]> AuthorDate: Fri Aug 28 18:27:09 2026 +0700 [DOC] Monitor folder + database schema from src/site --- docs/modules/servers/nav.adoc | 4 + .../servers/pages/distributed/pop3-only.adoc | 165 +++++++++++++++++++++ .../pages/jpa/architecture/database-schema.adoc | 12 ++ .../modules/servers/pages/jpa/operate/folders.adoc | 13 ++ docs/modules/servers/pages/jpa/operate/index.adoc | 2 + .../pages/spring/architecture/database-schema.adoc | 16 ++ docs/modules/servers/pages/spring/index.adoc | 8 +- .../servers/pages/spring/operate/folders.adoc | 16 ++ .../servers/pages/spring/operate/index.adoc | 3 + docs/modules/servers/pages/spring/run.adoc | 6 +- .../partials/architecture/database-schema.adoc | 139 +++++++++++++++++ docs/modules/servers/partials/operate/folders.adoc | 139 +++++++++++++++++ docs/modules/servers/partials/operate/index.adoc | 5 + 13 files changed, 524 insertions(+), 4 deletions(-) diff --git a/docs/modules/servers/nav.adoc b/docs/modules/servers/nav.adoc index e920693144..1722b55b6f 100644 --- a/docs/modules/servers/nav.adoc +++ b/docs/modules/servers/nav.adoc @@ -143,6 +143,7 @@ ** xref:jpa/index.adoc[] *** xref:jpa/architecture/index.adoc[] **** xref:jpa/architecture/implemented-standards.adoc[] +**** xref:jpa/architecture/database-schema.adoc[] *** xref:jpa/run/index.adoc[] **** xref:jpa/run/run-java.adoc[Run with Java] **** xref:jpa/run/run-docker.adoc[Run with Docker] @@ -186,6 +187,7 @@ **** xref:jpa/operate/guide.adoc[] **** xref:jpa/operate/performanceChecklist.adoc[] **** xref:jpa/operate/logging.adoc[] +**** xref:jpa/operate/folders.adoc[] **** xref:jpa/operate/webadmin.adoc[] **** xref:jpa/operate/metrics.adoc[] **** xref:jpa/operate/cli.adoc[] @@ -200,6 +202,7 @@ ** xref:spring/index.adoc[] *** xref:spring/architecture/index.adoc[] **** xref:spring/architecture/implemented-standards.adoc[] +**** xref:spring/architecture/database-schema.adoc[] *** xref:spring/run.adoc[Run] *** xref:spring/configure/index.adoc[] **** Protocols @@ -232,6 +235,7 @@ ***** xref:spring/configure/mta-sts.adoc[MTA-STS] *** xref:spring/operate/index.adoc[Operate] **** xref:spring/operate/logging.adoc[] +**** xref:spring/operate/folders.adoc[] **** xref:spring/operate/jmx.adoc[] **** xref:spring/operate/cli.adoc[] **** xref:spring/operate/security.adoc[] diff --git a/docs/modules/servers/pages/distributed/pop3-only.adoc b/docs/modules/servers/pages/distributed/pop3-only.adoc new file mode 100644 index 0000000000..5e590c81e5 --- /dev/null +++ b/docs/modules/servers/pages/distributed/pop3-only.adoc @@ -0,0 +1,165 @@ += Distributed James Server — POP3 only variant +:navtitle: POP3 only variant + +:server-name: Distributed POP3 James server +:xref-base: distributed +:pages-path: distributed +:sample-configuration-prefix-url: https://github.com/apache/james-project/blob/master/server/apps/distributed-pop3-app/sample-configuration + +Alongside the xref:distributed/index.adoc[Distributed James server], the project ships a variant of it, packaged as +link:https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app[distributed-pop3-app] and +published as the `apache/james:distributed-pop3-latest` docker image. + +It is the very same application, minus IMAP: it exposes *SMTP*, *LMTP* and *POP3*. Dropping IMAP is what allows it +to be deployed *across several availability zones, or several data centers*, which the Distributed James server +cannot do. + +== Motivation: running across availability zones + +IMAP mandates that every message of a mailbox receives a *monotonically increasing UID*, and every change a +monotonically increasing MODSEQ. Cassandra, being an eventually consistent store, cannot deliver such a counter +with its regular write path: the Distributed James server therefore allocates UIDs and MODSEQs with +link:https://www.datastax.com/blog/2013/07/lightweight-transactions-cassandra-20[lightweight transactions], whose +default execution profile reads at `SERIAL` consistency. + +A `SERIAL` read is a Paxos round involving a majority of the replicas *of the whole cluster*. Stretch a cluster +over two availability zones, and every single message appended to a mailbox pays a cross-zone round trip - several +of them, actually, as a lightweight transaction that loses its race is retried. Downgrading the profile to +`LOCAL_SERIAL` gives back the latency but breaks the very guarantee that was being bought: two zones can then +allocate the same UID concurrently, which means losing emails. This is why +xref:distributed/architecture/consistency-model.adoc#_about_multi_data_center_setups[multi data-center setups are +discouraged] on the Distributed James server. + +POP3 has no such requirement. It identifies messages by their `MessageId`, which James generates as a *TimeUUID*, +locally, without any coordination. Removing IMAP thus removes the need for a monotonic counter, and with it the +last lightweight transaction of the message write path. Cassandra can then be operated with `LOCAL_QUORUM` reads +and writes, each James instance talking to the replicas of its own zone, and the cluster spanning as many zones or +data centers as your availability target requires. + +== What changes compared to the Distributed server + +Everything of the xref:distributed/architecture/index.adoc[Distributed James server] is retained - Cassandra, +OpenSearch, RabbitMQ, the BlobStore and its object storage backends, the mailet container, the distributed task +manager, the WebAdmin API - with the following substitutions: + +* The Cassandra *UID and MODSEQ providers are replaced by random generators* (`RandomUidProvider` and +`RandomModSeqProvider`, drawing from a `SecureRandom`). No lightweight transaction is involved in appending a +message any more. +* *Mailbox ACLs are dropped*: the ACL mapper is bound to a no-op implementation. Mailbox sharing is unsupported, +and the lightweight transactions the ACL mapper performs are gone as well. +* Thread resolution falls back to a naive algorithm, as the regular one relies on the mailbox metadata that is no +longer maintained. +* A dedicated `pop3metadata` Cassandra table holds, per mailbox, the identifier and the size of every message. It +is populated by a mailbox listener, so that the `STAT`, `LIST` and `UIDL` commands - which POP3 clients issue at +every single connection - are answered by *one partition read* rather than by a scan of the mailbox. + +== Supported features + +The following protocols are supported: + +* *SMTP* and *LMTP* +* *POP3* +* The xref:distributed/operate/webadmin.adoc[WebAdmin] REST administration API, and the +xref:distributed/operate/cli.adoc[command line interface] over JMX + +*IMAP is not available*: its Guice module is simply not installed. *JMAP is not supported* either: although its +modules are wired in, no *jmap.properties* ships with the application, which leaves the JMAP server disabled. +Enabling it would require follow-up developments, as several components - the search among them - critically +depend on UIDs being meaningful. + +[WARNING] +==== +The UIDs and MODSEQs this application writes are random, hence meaningless. Anything ordering or ranging over +them - IMAP first and foremost - will misbehave on that data. Do not point an IMAP server at a store written by +this variant, and do not run this variant against a store written by the regular Distributed server. +==== + +Mail processing, recipient rewriting, Sieve filtering upon local delivery, quotas, the Deleted Messages Vault and +the mail repositories all behave as they do on the Distributed James server, and are configured with the very same +files. A +link:https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app/sample-configuration[sample +configuration directory] is provided. + +== Operate + +The `pop3metadata` projection is maintained asynchronously by a mailbox listener, and can therefore diverge from +the mailbox itself - a listener failure, a message written while a node was down. A WebAdmin task recomputes it: + +.... +curl -XPOST http://ip:port/mailboxes?task=fixPop3Inconsistencies +.... + +It is exposed alongside the other xref:distributed/operate/webadmin.adoc[mailbox tasks], and its progress is +followed through the task manager, like any other James task. + +Everything else - xref:distributed/operate/guide.adoc[the operation guide], +xref:distributed/operate/logging.adoc[logging], xref:distributed/operate/metrics.adoc[metrics], +xref:distributed/operate/security.adoc[the security checklist] - applies unchanged. + +== Run with Docker + +The application is published as `apache/james:distributed-pop3-latest`, and needs the same third party +dependencies as the Distributed James server: Cassandra {cassandra-version}, RabbitMQ {rabbitmq-version} and an S3 +compatible object storage. + +[subs=attributes+] +---- +$ docker network create --driver bridge james +$ docker run -d --network james -p 9042:9042 --name=cassandra cassandra:{cassandra-version} +$ docker run -d --network james -p 5672:5672 -p 15672:15672 --name=rabbitmq rabbitmq:{rabbitmq-version}-management +$ docker run -d --network james --env 'REMOTE_MANAGEMENT_DISABLE=1' \ + --env 'SCALITY_ACCESS_KEY_ID=accessKey1' --env 'SCALITY_SECRET_ACCESS_KEY=secretKey1' \ + --name=s3 ghcr.io/scality/cloudserver:{cloudserver-version} +---- + +For security reasons you are required to generate your own keystore, and to mount it into the container: + +[source,bash] +---- +$ keytool -genkey -alias james -keyalg RSA -keystore keystore +$ docker run --network james -v $PWD/keystore:/root/conf/keystore apache/james:distributed-pop3-latest +---- + +For a quick start without creating a keystore (e.g. for development), pass the `--generate-keystore` command +argument and James will generate one with its default settings: + +[source,bash] +---- +$ docker run --network james apache/james:distributed-pop3-latest --generate-keystore +---- + +A +link:https://github.com/apache/james-project/blob/master/server/apps/distributed-pop3-app/docker-compose.yml[docker-compose] +file starting the application together with all of its dependencies is provided. + +The image can also be built locally with a regular `mvn clean install -DskipTests`, then imported: + +[source,bash] +---- +$ docker image load -i server/apps/distributed-pop3-app/target/jib-image.tar +---- + +Extra JVM flags are passed with +link:https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#jvm-flags[JAVA_TOOL_OPTIONS], and a domain +can be created upon start via an environment variable: + +[source,bash] +---- +$ docker run --network james --environment DOMAIN=domain.tld \ + -e "JAVA_TOOL_OPTIONS=-Xmx500m -Xms500m" apache/james:distributed-pop3-latest +---- + +link:https://glowroot.org/[Glowroot APM] is packaged within the image. Disabled by default, its java agent can be +enabled: + +[source,bash] +---- +$ docker run --network james -e "JAVA_TOOL_OPTIONS=-javaagent:/root/glowroot.jar" apache/james:distributed-pop3-latest +---- + +The xref:distributed/operate/cli.adoc[CLI] is available within the container: + +[source,bash] +---- +$ docker exec CONTAINER-ID james-cli ListDomains +---- diff --git a/docs/modules/servers/pages/jpa/architecture/database-schema.adoc b/docs/modules/servers/pages/jpa/architecture/database-schema.adoc new file mode 100644 index 0000000000..b954076329 --- /dev/null +++ b/docs/modules/servers/pages/jpa/architecture/database-schema.adoc @@ -0,0 +1,12 @@ += JPA James Server — Database schema +:navtitle: Database schema + +:xref-base: jpa +:pages-path: jpa +:server-name: JPA James server +:persistence-unit-name: Global +:persistence-xml-url: https://github.com/apache/james-project/blob/master/server/apps/jpa-app/src/main/resources/META-INF/persistence.xml + +include::partial$jpa/deprecated.adoc[] + +include::partial$architecture/database-schema.adoc[] diff --git a/docs/modules/servers/pages/jpa/operate/folders.adoc b/docs/modules/servers/pages/jpa/operate/folders.adoc new file mode 100644 index 0000000000..2c83806ead --- /dev/null +++ b/docs/modules/servers/pages/jpa/operate/folders.adoc @@ -0,0 +1,13 @@ += JPA James Server — Folder layout +:navtitle: Folder layout + +:xref-base: jpa +:pages-path: jpa +:server-name: JPA James server +:backend-name: the relational database +:rrt-error-repository: +:lucene-at-root: + +include::partial$jpa/deprecated.adoc[] + +include::partial$operate/folders.adoc[] diff --git a/docs/modules/servers/pages/jpa/operate/index.adoc b/docs/modules/servers/pages/jpa/operate/index.adoc index 41bb440d91..48d5bfc5ce 100644 --- a/docs/modules/servers/pages/jpa/operate/index.adoc +++ b/docs/modules/servers/pages/jpa/operate/index.adoc @@ -6,6 +6,8 @@ :server-tag: JPA :no-opensearch: +:folder-layout: + include::partial$jpa/deprecated.adoc[] include::partial$operate/index.adoc[] diff --git a/docs/modules/servers/pages/spring/architecture/database-schema.adoc b/docs/modules/servers/pages/spring/architecture/database-schema.adoc new file mode 100644 index 0000000000..55f3a3bf47 --- /dev/null +++ b/docs/modules/servers/pages/spring/architecture/database-schema.adoc @@ -0,0 +1,16 @@ += Spring James Server — Database schema +:navtitle: Database schema + +:xref-base: spring +:pages-path: spring +:server-name: Spring James server +:persistence-unit-name: James +:persistence-xml-url: https://github.com/apache/james-project/blob/master/server/apps/spring-app/src/main/resources/META-INF/persistence.xml +:no-droplist: +:no-webadmin: +:sieve-file-repository: +:file-mail-repository: + +include::partial$spring/deprecated.adoc[] + +include::partial$architecture/database-schema.adoc[] diff --git a/docs/modules/servers/pages/spring/index.adoc b/docs/modules/servers/pages/spring/index.adoc index dfe9e1c4f7..16228e2da2 100644 --- a/docs/modules/servers/pages/spring/index.adoc +++ b/docs/modules/servers/pages/spring/index.adoc @@ -75,9 +75,11 @@ link:https://github.com/apache/james-project/tree/master/server/apps/jpa-app/sam A few points deserve attention: -* *Sieve scripts are not migrated*. The Spring application stores them as files under `var/sieve`, while the JPA -application stores them in the database. Users need to re-upload their scripts. -* *The Lucene index is not migrated*: it does not live at the same place. Rebuild it with the +* *Sieve scripts are not migrated*. The Spring application stores them as files, in a `sieve/` folder of its +installation directory - see xref:spring/operate/folders.adoc[the folder layout] - while the JPA application +stores them in the database. Users need to re-upload their scripts. +* *The Lucene index is not migrated*: it does not live at the same place - `var/store/lucene` here, the working +directory itself on the JPA application. Rebuild it with the xref:jpa/operate/webadmin.adoc#_reindexing_all_mails[reIndexing WebAdmin task] once the JPA server is up. * *The mail queue is not migrated*. Let the Spring instance finish processing its spool before stopping it. * The `db` and `dbfile` mail repository protocols, backed by the legacy `JDBCMailRepository` and `sqlResources.xml`, diff --git a/docs/modules/servers/pages/spring/operate/folders.adoc b/docs/modules/servers/pages/spring/operate/folders.adoc new file mode 100644 index 0000000000..28b4112a35 --- /dev/null +++ b/docs/modules/servers/pages/spring/operate/folders.adoc @@ -0,0 +1,16 @@ += Spring James Server — Folder layout +:navtitle: Folder layout + +:xref-base: spring +:pages-path: spring +:server-name: Spring James server +:backend-name: the relational database +:spring-layout: +:spam-repositories: +:lucene-in-var: +:sieve-file-repository: +:no-webadmin: + +include::partial$spring/deprecated.adoc[] + +include::partial$operate/folders.adoc[] diff --git a/docs/modules/servers/pages/spring/operate/index.adoc b/docs/modules/servers/pages/spring/operate/index.adoc index 40a7755956..e4878b1996 100644 --- a/docs/modules/servers/pages/spring/operate/index.adoc +++ b/docs/modules/servers/pages/spring/operate/index.adoc @@ -15,6 +15,9 @@ these topics. Read more about xref:{xref-base}/operate/logging.adoc[Logging]. +The xref:{xref-base}/operate/folders.adoc[folder layout] page details what the {server-name} writes on the file +system, and which of those folders need watching. + *Administration of this application happens entirely over JMX*, as it does not ship the WebAdmin REST API. The xref:{xref-base}/operate/jmx.adoc[JMX page] lists the exposed MBeans, and the xref:{xref-base}/operate/cli.adoc[Command line interface] wraps the most common operations. Both are documented as diff --git a/docs/modules/servers/pages/spring/run.adoc b/docs/modules/servers/pages/spring/run.adoc index eb40047e7e..fdcea2947a 100644 --- a/docs/modules/servers/pages/spring/run.adoc +++ b/docs/modules/servers/pages/spring/run.adoc @@ -37,9 +37,13 @@ bin/ start scripts, and the james-cli client conf/ the configuration files, and conf/lib for the extra JARs lib/ the James JARs log/ the log files -var/ the mail repositories, the Sieve scripts, the Lucene index, the ActiveMQ store and the H2 database +sieve/ the Sieve scripts of your users, created on first use +var/ the mail repositories, the Lucene index, the ActiveMQ store and the H2 database .... +The xref:spring/operate/folders.adoc[folder layout page] details what lives where, and which of those folders need +watching. + The `conf` folder ships every configuration file with sane defaults, which you need to review before running in production. Read xref:spring/configure/index.adoc[the configuration section]. At the very least: diff --git a/docs/modules/servers/partials/architecture/database-schema.adoc b/docs/modules/servers/partials/architecture/database-schema.adoc new file mode 100644 index 0000000000..fa046d67e6 --- /dev/null +++ b/docs/modules/servers/partials/architecture/database-schema.adoc @@ -0,0 +1,139 @@ +The {server-name} keeps everything in a single relational database, mapped with the +link:https://openjpa.apache.org/[Apache OpenJPA] implementation of the Java Persistence Architecture. The schema +is *generated by OpenJPA from the entity classes* rather than maintained as SQL: there is no DDL script to run, +and James creates the missing tables on start up. + +This page describes that schema. You should not need it to operate the {server-name} - the +xref:{xref-base}/configure/index.adoc[configuration], the +xref:{xref-base}/operate/index.adoc[administration interfaces] and the +xref:{xref-base}/architecture/index.adoc[architecture page] are the supported entry points - but it is +occasionally the only way to diagnose an odd behaviour, to size a deployment, or to convince yourself that a +migration carried everything over. + +WARNING: This schema is an *implementation detail*, and is not a public interface: it changes between releases +without notice. Read it if you must, but never write to it behind James' back. Doing so bypasses every cache and +every consistency check, and the mailbox in particular maintains counters that a manual `UPDATE` will silently +desynchronise. + +The mapping is declared in +link:{persistence-xml-url}[META-INF/persistence.xml], under the `{persistence-unit-name}` persistence unit. Every +table is prefixed with `JAMES_`. + +== Users, domains and rewriting + +.Identity tables +|=== +| Table | Content + +| `JAMES_USER` +| One row per user. `USER_NAME` is the login - the full email address when +xref:{xref-base}/configure/usersrepository.adoc[virtual hosting] is enabled, the local part otherwise -, +`PASSWORD` the hash of the password and `PASSWORD_HASH_ALGORITHM` the algorithm it was computed with. Passwords +are never stored in clear. This table is empty when users are read from an LDAP. + +| `JAMES_DOMAIN` +| One row per handled domain, `DOMAIN_NAME` holding it. + +| `JAMES_RECIPIENT_REWRITE` +| The xref:{xref-base}/configure/recipientrewritetable.adoc[rewriting rules]. `USER_NAME` and `DOMAIN_NAME` form +the matched address, `TARGET_ADDRESS` the result. A single source can carry several targets, hence several rows. + +ifndef::no-droplist[] +| `JAMES_DROP_LIST` +| The xref:{xref-base}/configure/droplists.adoc[drop lists]: `OWNER_SCOPE` and `OWNER` say whose list it is +(global, per domain or per user), `DENIED_ENTITY_TYPE` and `DENIED_ENTITY` what is blocked. +endif::[] +|=== + +== Mailboxes and messages + +.Mailbox tables +|=== +| Table | Content + +| `JAMES_MAILBOX` +| One row per mailbox. `USER_NAME`, `MAILBOX_NAMESPACE` and `MAILBOX_NAME` identify it, +`MAILBOX_UID_VALIDITY` carries the IMAP `UIDVALIDITY`, and `MAILBOX_LAST_UID` and `MAILBOX_HIGHEST_MODSEQ` are the +*counters* from which the UID and the MODSEQ of the next message are allocated. Those two columns are the +serialisation point of every append to a mailbox. + +| `JAMES_MAIL` +| One row per message *in a mailbox*. Its primary key is the `(MAILBOX_ID, MAIL_UID)` pair, which is why moving a +message means deleting and re-inserting a row. `MAIL_MODSEQ` carries the modification sequence, +`MAIL_IS_ANSWERED`, `MAIL_IS_DELETED`, `MAIL_IS_DRAFT`, `MAIL_IS_FLAGGED`, `MAIL_IS_RECENT` and `MAIL_IS_SEEN` the +IMAP system flags, and `MAIL_DATE` the internal date. The message itself is in the same row: +`HEADER_BYTES` holds the headers and `MAIL_BYTES` the body, with `MAIL_BODY_START_OCTET` marking the boundary and +`MAIL_CONTENT_OCTETS_COUNT` the total size. *This table holds the emails of your users, and is by far the largest +one.* + +| `JAMES_MAIL_USERFLAG` +| The user defined IMAP flags - keywords - set on a message, one row per flag per message. + +| `JAMES_MAIL_PROPERTY` +| The parsed MIME properties of a message: content type, charset, encoding... one row per property, ordered by +`PROPERTY_LINE_NUMBER`. + +| `JAMES_SUBSCRIPTION` +| The IMAP subscriptions: a `(USER_NAME, MAILBOX_NAME)` pair per subscribed mailbox. + +| `JAMES_MAILBOX_ANNOTATION` +| The link:https://datatracker.ietf.org/doc/html/rfc5464[RFC-5464] mailbox annotations, keyed by mailbox and +annotation key. +|=== + +NOTE: Storing the message content in the same row as its metadata is what makes this schema simple, and what makes +it scale poorly: every IMAP `FETCH` of a flag reads a row whose blob columns hold the whole email. The +xref:postgres/index.adoc[Postgres James server] separates the two, keeping the metadata in the database and the +content in a xref:postgres/architecture/blobstore.adoc[BlobStore]. + +== Quotas + +Quotas are split across one table per limit, all of them holding a `VALUE` column carrying the limit itself: +`JAMES_MAX_GLOBAL_MESSAGE_COUNT` and `JAMES_MAX_Global_STORAGE` for the server wide defaults, +`JAMES_MAX_DOMAIN_MESSAGE_COUNT` and `JAMES_MAX_DOMAIN_STORAGE` per domain, `JAMES_MAX_USER_MESSAGE_COUNT` and +`JAMES_MAX_USER_STORAGE` per user. + +`JAMES_QUOTA_CURRENTQUOTA` holds the *current* consumption of each quota root - +`CURRENTQUOTA_MESSAGECOUNT` and `CURRENTQUOTA_SIZE`. It is a counter maintained incrementally as messages are +added and removed, hence it can drift from the reality of `JAMES_MAIL`. +ifndef::no-webadmin[] +Recomputing it is a xref:{xref-base}/operate/webadmin.adoc#_recomputing_current_quotas_for_users[WebAdmin task]. +endif::[] + +== Sieve + +`JAMES_SIEVE_SCRIPT` holds one row per script, with `USER_NAME`, `SCRIPT_NAME`, `SCRIPT_CONTENT`, `SCRIPT_SIZE`, +`IS_ACTIVE` and `ACTIVATION_DATE_TIME` - at most one script is active per user. `JAMES_SIEVE_QUOTA` carries the +per user limit on the total size of the scripts. + +ifdef::sieve-file-repository[] +NOTE: The {server-name} is configured with the *file based* Sieve repository, and therefore leaves these two +tables empty: the scripts live in the `sieve/` folder, as described on the +xref:{xref-base}/operate/folders.adoc[folder layout page]. The entities are nonetheless mapped, so the tables are +created on start up. +endif::[] + +== Mail repositories + +`JAMES_MAIL_REPOS` lists the declared repository urls, and `JAMES_MAIL_STORE` holds their content: one row per +parked mail, carrying its envelope (`SENDER`, `RECIPIENTS`, `REMOTE_HOST`, `REMOTE_ADDR`), its processing state +(`MESSAGE_STATE`, `ERROR_MESSAGE`, `LAST_UPDATED`), its attributes and per recipient headers, and the message +itself in `MESSAGE_BODY`. + +ifdef::file-mail-repository[] +NOTE: These two tables stay empty on the {server-name}: its +xref:{xref-base}/configure/mailrepositorystore.adoc[mailrepositorystore.xml] declares the *file* mail repository, +which writes to `var/mail` rather than to the database. +endif::[] + +== Reading the schema yourself + +The authoritative description is the set of annotated entity classes, which is worth consulting when this page +falls behind: + +* the mailbox entities under +link:https://github.com/apache/james-project/tree/master/mailbox/jpa/src/main/java/org/apache/james/mailbox/jpa[mailbox/jpa], +* everything else - users, domains, rewriting, Sieve, mail repositories, drop lists - under +link:https://github.com/apache/james-project/tree/master/server/data/data-jpa/src/main/java/org/apache/james[server/data/data-jpa]. + +The `@Table` and `@Column` annotations they carry are exactly what OpenJPA turns into the schema above. diff --git a/docs/modules/servers/partials/operate/folders.adoc b/docs/modules/servers/partials/operate/folders.adoc new file mode 100644 index 0000000000..9c545d125e --- /dev/null +++ b/docs/modules/servers/partials/operate/folders.adoc @@ -0,0 +1,139 @@ +Knowing what the {server-name} writes, and where, is needed to size a volume, to back the server up, and to tell +apart the data that can be thrown away from the data that cannot. + +ifndef::spring-layout[] +Every folder below is relative to the *working directory* - the folder James is started from, or the one given by +the `working.directory` system property. + +.... +<working directory> +├── conf/ configuration files +└── var/ the data James reads and writes + ├── mail/ mail repositories + └── store/ the embedded storage dependencies +.... +endif::[] +ifdef::spring-layout[] +Unpacking the distribution archive gives the following layout, which the rest of this page calls the *installation +directory*: + +.... +<installation directory> +├── bin/ start scripts, and the james-cli client +├── conf/ configuration files, and conf/lib for the extra JARs +├── lib/ the James JARs +├── log/ the log files +├── sieve/ the Sieve scripts of your users +└── var/ the data James reads and writes + ├── mail/ mail repositories + └── store/ the embedded storage dependencies +.... + +[IMPORTANT] +==== +The start scripts run James *from the `bin` folder*, and the application resolves its own root one level above - +`JamesServerApplicationContext` is built on a resource loader rooted at `../`. This is why the paths of the +{server-name} configuration are written the way they are, and it bites in two different ways: + +* paths going through the James file system, written `file://var/mail/error/`, are resolved against the +installation directory. Copy such a path as is; +* paths handed over to a third party library - the `database.url` of +xref:{xref-base}/configure/database.adoc[james-database.properties], the Lucene directory of +xref:{xref-base}/configure/indexer.adoc[indexer.xml] - are resolved by that library against the *process* working +directory, which is `bin`. They are therefore written `../var/store/...`, and *lose their leading `../` when +carried over to any other James application*, which run from the installation directory itself. +==== +endif::[] + +WARNING: `var/mail` does *not* contain the mails of your users. Those live in the mailbox, that is to say in +{backend-name}. `var/mail` only holds the mails that were set aside during processing. + +== var/mail: the mail repositories + +Every folder under `var/mail` is a xref:{xref-base}/configure/mailrepositorystore.adoc[mail repository], that is +to say a place where the xref:{xref-base}/configure/mailetcontainer.adoc[mailet container] parks a mail it could +not deliver. Which ones exist depends entirely on your *mailetcontainer.xml*; the shipped configuration declares: + +* `error/` - a mail that threw an exception during processing; +* `address-error/` - a mail addressed to a local user that does not exist; +* `relay-denied/` - a mail an unauthenticated sender tried to relay; +* `relay-limit-exceeded/` - a mail that looped, as detected by the `RelayLimit` matcher; +ifdef::rrt-error-repository[] +* `rrt-error/` - a mail that hit a loop in the xref:{xref-base}/configure/recipientrewritetable.adoc[recipient +rewriting rules]. +endif::[] +ifdef::spam-repositories[] +* `spam/` - a mail flagged by the anti-spam handlers; +* `over-quota-error/` - a mail that could not be delivered because the recipient was over quota. +endif::[] + +Inside such a folder, each mail is stored as *two* files sharing the same name: a `.FileObjectStore` file holding +the envelope and the mail attributes, and a `.FileStreamStore` file holding the MIME message itself. Never delete +one without the other. + +*These folders grow without bound.* Nothing prunes them: a mail parked there stays there until an operator deals +with it. They are the first place to look when the disk fills up. + +ifndef::no-webadmin[] +Rather than reading those files by hand, list and reprocess their content over +xref:{xref-base}/operate/webadmin.adoc#_reprocessing_mails_from_a_mail_repository[WebAdmin]. +endif::[] +ifdef::no-webadmin[] +The {server-name} ships no WebAdmin API, and its CLI exposes no mail repository command: these folders can only be +audited by hand. This is one of the strongest arguments for moving to the xref:jpa/index.adoc[JPA James server]. +endif::[] + +== var/store: the embedded dependencies + +`var/store` holds the state of the storage dependencies the {server-name} embeds and runs in its own process: + +* `var/store/h2db.mv.db` and its companion files - the embedded +link:https://h2database.com[H2] database, when you kept the default +xref:{xref-base}/configure/database.adoc[james-database.properties]. *This is where the mailboxes, the mails, the +users, the domains and the rewriting rules actually are.* Point James at another database and this file disappears, +your data being wherever that database keeps it. +* `var/store/activemq/brokers/KahaDB/` - the persistent store of the embedded ActiveMQ broker backing the +xref:{xref-base}/architecture/index.adoc[mail queue]. It holds the mails that are spooled but not yet processed, +and is expected to stay small on a healthy server: a KahaDB that keeps growing means the mailet container is not +keeping up with the incoming traffic. +* `var/store/activemq/blob-transfer/` - the bodies of the queued mails that were too large to be carried in the +JMS message itself, with one folder per queue (`spool`, `outgoing`). +ifdef::lucene-in-var[] +* `var/store/lucene/` - the xref:{xref-base}/configure/indexer.adoc[Lucene] full text index. It is a *derived* +projection: losing it costs a re-indexing, not your emails. +endif::[] + +ifdef::lucene-at-root[] +== The Lucene index + +The xref:{xref-base}/architecture/index.adoc[Lucene] full text index is *not* written under `var`: its segment +files - `segments_N`, `_0.cfs`, `write.lock`... - are created directly in the working directory. This is +surprising, and worth knowing before you mistake them for junk and delete them. + +The index is a derived projection: losing it costs a +xref:{xref-base}/operate/cli.adoc#_re_indexing[re-indexing], not your emails. +endif::[] + +ifdef::sieve-file-repository[] +== sieve/ + +The {server-name} stores the Sieve scripts of its users on the file system rather than in the database +(`SieveFileRepository`), in a `sieve/` folder of the installation directory - not under `var`. The folder is +created on first use. It holds one sub-folder per user, with their `.sieve` scripts, plus `.quota` files carrying +the xref:{xref-base}/configure/sieve.adoc[Sieve quotas]. + +*Back it up along with your database*: it is the one piece of user data that does not live in it, and the reason +Sieve scripts are the only thing a xref:spring/index.adoc#_moving_to_the_jpa_james_server[move to the JPA +application] does not carry over on its own. +endif::[] + +== The system temporary directory + +James also writes to the system temporary directory - `/tmp` on Linux, overridable with the `java.io.tmpdir` +system property. Incoming messages larger than the `inMemorySizeLimit` of the +xref:{xref-base}/configure/imap.adoc[IMAP] and xref:{xref-base}/configure/smtp.adoc[SMTP] servers are streamed to +a temporary file rather than held in the heap, and the mailbox does the same when appending a large message. + +These files are removed shortly after use, so the folder should stay small. Size it nonetheless for the largest +message you accept, times the number of connections that may be uploading at the same time: a temporary directory +that cannot be written to fails the delivery. diff --git a/docs/modules/servers/partials/operate/index.adoc b/docs/modules/servers/partials/operate/index.adoc index aab1f0c804..5149dc4790 100644 --- a/docs/modules/servers/partials/operate/index.adoc +++ b/docs/modules/servers/partials/operate/index.adoc @@ -6,6 +6,11 @@ these topics. Read more about xref:{xref-base}/operate/logging.adoc[Logging]. +ifdef::folder-layout[] +The xref:{xref-base}/operate/folders.adoc[folder layout] page details what the {server-name} writes on the file +system, and which of those folders need watching. +endif::[] + The xref:{xref-base}/operate/webadmin.adoc[WebAdmin Restfull administration API] is the recommended way to operate the {server-name}. It allows managing and interacting with most server components. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
