Re: 3.9.0 and S3 BlobStore

Jerry Malcolm Thu, 01 Jan 2026 16:49:22 -0800

Thanks again. You convinced me. PostGreSQL it is. I've spent the daydigging deep into postgres and getting it installed, with thepostgres/guice package 'almost' up and running.

But (as usual) I hit one snag that I can't seem to get around. I'vecreated postgres.properties and blob.properties in /opt/james/var/confand set var/conf as the working directory. I know it's processing theblob.properties (i had an omission, and it complained until i fixedit). It screamed loudly when it initially couldn't findpostgres.properties, and those errors went away when I created thepostgres.properties file. So it is definitely now finding that file. However, no matter what I put in that file (including one time with justtrash characters), I always get the same error:

00:24:45.757 [main] INFO org.apache.james.CONFIGURATION - Loadconfiguration file /opt/james/var/conf/postgres.propertiesException in thread "main" java.lang.IllegalArgumentException: You needto specify username

ChatGPT has told me two different formats for that file. But neithermakes any difference. Always the same error above. What am I doing wrong?


version 1:

host=db_pg_jwmbos.******.com
port=5432
database=jamesdb
username=********
password=***********
maxConnections=10

version 2:

postgres.host=db_pg_jwmbos.******.com
postgres.port=5432
postgres.database=jamesdb
postgres.username=*********
postgres.password=*********
postgres.maxConnections=10




On 1/1/2026 11:23 AM, Jean Helou wrote:

Hi Jerry,

I'm doing a completely clean install, and I'm already set up to use

imapsync to populate the new instance.  So no  worries about migration.

good to know

don't believe I care about spring vs guice.  I've never used a guice

version.  But spring-->guice should be transparent unless I've done  custom
spring stuff, correct (which I haven't)?

Exactly, unless you have custom spring extensions for james using either is
transparent for you

You said there were only the 4 pre-packaged assemblies with S3.  Does

that mean that (with a little work on my part), there 'could' be a
custom-packaged assembly created that would support mysql/s3?

In theory: yes. However, I looked into how big a work it would be this
morning while writing my answer to see if I should suggest it and
unfortunately I expect it to be quite involved.

there are 2 aspects :
- "short" term storage of mailqueue messages (mails that are being received
and processed or mails that are enqueued for sending, plus the various
error queues which may not be that short :D )
- long term storage of mailbox messages

The first aspect can be managed through (a bit of tortuous) configuration
but the second requires code modifications
The only mysql support is based on openJPA [1] which is a generic java
persistence system. The code that writes the message to the database when
using openJPA is org.apache.james.mailbox.jpa.mail.JPAMessageMapper if you
look at it you will see that the implementation is quite straightforward in
the protected `org.apache.james.mailbox.jpa.mail.JPAMessageMapper#save`
method
The org.apache.james.mailbox.postgres.mail.PostgresMessageMapper is
structured a bit differently and if you look at the save method, it first
stores the message body ( private method saveBodyContent) then asks
the org.apache.james.mailbox.postgres.mail.dao.PostgresMessageDAO#insert to
store the message providing it with the message from which it extracts the
headers and the blobid of the saved body content.

To enable such behaviour you would need to create a custom JPAMessageMapper
and probably a custom variant of JPAMailboxMessage that has a field to
store the blobId, create the correspond SQL migration to add the missing
column and do the dance around saving the body, then saving the message
etc. Since the MessageMapper is not created directly by the IOC container
you will need to implement a custom JPAMailboxSessionMapperFactory
Once you have done this, you will need to make the IOC container pick your
factory implementation over the standard one (whether using spring or
guice)

An alternative but probably requiring even more work would be to create a
Mysql connector using Jooq (same as the postgres connector) thats probably
a lot more code.

best regards,
Jean




On Thu, Jan 1, 2026 at 4:50 PM Jerry Malcolm <[email protected]> wrote:

Jean,

Thank you so much for the detailed explanation.  One clarification that
might make decisions easier... I'm doing a completely clean install, and
I'm already set up to use imapsync to populate the new instance.  So no
worries about migration.

I don't believe I care about spring vs guice.  I've never used a guice
version.  But spring-->guice should be transparent unless I've done
custom spring stuff, correct (which I haven't)?

So it really comes down to mysql vs. postgres, correct?  I've never used
postgres.  But installing and learning it is doable. But before I head
down that path, I would like to understand any other possibilities.  You
said there were only the 4 pre-packaged assemblies with S3.  Does that
mean that (with a little work on my part), there 'could' be a
custom-packaged assembly created that would support mysql/s3?    I'm a
programmer, but I don't purport to know the inner workings of James
wiring.  But just for exploration purposes, how big a project would it
be to make james use mysql with s3 instead of postgres with s3
(obviously requiring some code changes and wiring).  But I'm just
curious if this is something that is doable on my part?  If it's massive
changes... then on to postgres.  But I'm not adverse to a little coding,
especially if it reduces the learning curve for a new db environment.
   Or if there are plans anytime in the immediate future for james to
release a mysql/s3 option, then I might just pay a bit more for db
storage and wait it out.

Again, I appreciate you taking time on New Years Day to answer my
questions  Thanks again.

Jerry


On 1/1/2026 3:38 AM, Jean Helou wrote:

Hello jerry,

I'm afraid you are a bit out of luck: james has a modular architecture

and

there are multiple implementations for most if not all modules, this

means

that the number of module combination is very high.
the packaged assemblies cover some of the combinations, some assemblies

are

configurable to make it possible to cover even more through

configuration (

at the cost of increased package size )

Unfortunately there is no pre-packaged assembly which supports both MySQL
AND S3 blobstore today.

Only 4 assemblies have blobstore capabilities :
- james distributed
- james distributed pop3
- james postgresql
- scaling pulsar smtp ( this is a demo assembly for an SMTP only server
which uses the pulsar message broker/postgres and s3 )
so that really only leaves 2 options:
- james distributed
- james postgresql

The former is designed for large enterprise deployments and relies on
complex systems
The latter is designed for small to medium deployments

Given you are using the spring/mysql james server  today and only want s3
to optimize storage costs, I think your best bet is to go with  james
postgresql. The distributed version is designed for enterprise with tens

or

hundreds of thousands of users, or for high volume servers. Cassandra is
not easy to operate.

Unless you customized it quite heavily, the spring assembly should be

using

the jpa implementation to access the database, this means you can use the
provided migration tool[1]  to move you data from JPA to Postgres.
You will have to build the migration tool from source as it is not

included

in the officially published artifacts for the project.
This is a one shot operation after which you will be able to use the
james-server-postgres-app assembly which supports both postgres and s3
blobstore.

The postgres server is an officially published artifact[2]  (there is a

bug

in the website as the title for that link is
also  james-server-distributed-guice.zip but its only the title which is
incorrect, the link goes to the right place)
You can run both servers side by side for a while, repeatedly running the
migration tool (it is designed for this) until you are satisfied the
migration is complete with the limitation that the migration only goes

one

way so mails you send/receive  with the postgres server will never be

known

by the mysql server.
A basic configuration will
- store queued messages (either incoming or outgoing) in postgres
- store mailbox messages (long term storage for received or sent

messages)

in s3
further configuration can store queued messages in s3 too if you want

With regards to your compilation problems it's hard to tell without

seeing

the actual errors, a full compilation running all tests takes about 5

hours

(there are a lot of rather slow integration tests when you build
everything), so you probably want to either skip the tests or target a
specific project

mvn clean package -DskipTests will build everything without running the
tests
mvn clean package -am -pl :$project-name will build a project and all its
dependencies running all the tests for all the built projects, you can

also

add the -DskipTests to avoid running the tests

if you encounter further errors don't hesitate to upload the output to a
pastebin or filebin of some kind and include the link, the last build for
master[3] and the 3.9.x[4] branch on the james CI are green so the build
should be ok

Cheers,
Jean

1

https://github.com/apache/james-project/tree/master/server/apps/migration/core-data-jpa-to-pg

https://www.apache.org/dyn/closer.lua/james/server/3.9.0/james-server-postgres-app.zip

3 https://ci-builds.apache.org/job/james/job/ApacheJames/job/master/
4 https://ci-builds.apache.org/job/james/job/ApacheJames/job/3.9.x/

On Wed, Dec 31, 2025 at 7:31 PM Jerry Malcolm <[email protected]>
wrote:

I am so confused.  I have been trying for 5 days to upgrade to James
3.9.0 and begin using AWS S3 blobstore.   I've been using ChatGPT for
guidance.  But I'm beginning to realize that ChatGPT is just making up
answers on the fly, giving me completely different and wrong answers
depending on the time of day.

- first answer from ChatGPT - I should install 3.9.0 spring with
JPA/mySQL and configure AWS S3 for blob storage (never could get james
to even recognize that I had blob storage configured)

    - revised answer from ChatGPT - Oh, you can't use blobstore with
JPA/mySQL... you have to get Cassandra version to use blobstore.... took
over a day to make cassandra happy with it's Java 17 requirement while
keeping James 3.9.0 happy with its required Java 21.   Now I can't find
the 'required' James 3.9.0 Spring-Cassandra version zipfile to download

- revised answer from ChatGPT - Oh, there is no Spring/Cassandra.... you
need to use 3.9.0 Guice/Cassandra, which is not recommended since Spring
is strategic even though with last release spring was going away.....
<head exploding icon>

I'm sick of 'recommendation du-jour' from ChatGPT...  I've been on James
/ James-spring for over 20 years with mySQL.  I simply want to cut my
horrendous RDS/mySQL expenses for James mail data by using S3 for blob
storage.  I'll use any configuration available that I can get to work as
long as I can have S3 blob storage.

Oh, yeah... and I cloned the 3.9.0 release source from GitHub and tried
to run maven.  maven ran for about 20 minutes, then crashed and burned
on a bunch of errors.  So I had to go with the pre-built 3.9.0 zip.

I have just one simple question.... What are my options for
configuration in order to use AWS S3 for blob storage.

Thx

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: 3.9.0 and S3 BlobStore

Reply via email to