Re: Building a PostgreSQL mailbox for James?
Hi Quan! Thanks for the initiative of the follow up! > To have a look at the planned tickets, please visit https://github.com/linagora/james-project/issues. We cannot expect people from the community to read a Linagora specific bugtracker IMO we should sum up the planned tickets for JMAP onto the dedicated JIRA ticket on Apache James bugtracker: - Detail the implementation strategy (spawn a JMAP server on top of PG as fast as possible, using memory storage where needed, then removing the mempry code in favor of newly written PG code) - List the tasks and interfaces to implement - not sure we need to give all details all of the time though... - Detail specific mechanism (I am thinking to EventStore transactionnality, data expiracy ie TTLs, etc...) This would make it easier to follow the PG work and have people eventually provide feedback. Best regards, Benoit TELLIER On 04/01/2024 08:08, Quan tran hong wrote: Hi everyone, This is another follow-up to our recent work on the Postgres topic. In the last sprint: - We migrated fully JPA code to Postgres. No more JPA code in the postgres-app now. So at this moment, we had a fully functional IMAP server on top of Postgres. - We solved the flags concurrently update issue and proposed an ADR explaining the solution. - We implemented a blob store backed by Postgres. However, it is not mature yet as we are having concurrent issues with PostgresBlobStore when Postgres persists blobs slowly using just a single connection. On that, we are working on a Postgres pool connection to improve the performance. - We succeeded in plugging RabbitMQ as the event bus, S3 as the blob store, and OpenSearch into the postgres-app. We provide configurable module choosers so users are free to choose the implementation. For example, one could select S3/File/Postgres as his blob store implementation. In the coming time, we plan to implement the JMAP storage backed by Postgres. To have a look at the planned tickets, please visit https://github.com/linagora/james-project/issues. Your feedback would be appreciated. Regards, Quan Vào Th 6, 15 thg 12, 2023 vào lúc 18:45 Quan tran hong < quan.tranhong1...@gmail.com> đã viết: Hi everyone, Yet another following up on our recent Postgres work. In the last sprint, our team successfully migrated the mailbox core module `apache-james-mailbox-postgres` fully from JPA to Postgres. We did some performance tests with the IMAP server backed by Postgres-app and the result was very impressive: postgres-app outperformed JPA about 20 times (at least look at the 99 percentile) with even less resource usage (CPU and memory) compared to the jpa-app. For more details: https://github.com/linagora/james-project/issues/4936 We are working on an ADR to explain our schema decision on the core tables like mailbox and message tables, and also solving the flags concurrently update issue. In the coming time, we plan to: - Migrate other remaining minor components from JPA code to Postgres. - Plug the RabbitMQ, S3, OpenSearch into the Postgres-app. On that, we would try to implement alternative implementations with Postgres e.g. BlobStore backed by Postgres. We would provide configurable module choosers so users are free to choose the implementation. Best regards, Quan Vào Th 2, 27 thg 11, 2023 vào lúc 19:55 Benoit TELLIER < btell...@apache.org> đã viết: Thanks for the write up Quan! I would add that we shall rights ADRs about important decisions on the PostgreSQL implementation. So far I identified... - table structure for the postgresql mailbox - how deletes are handled - concurrency control for flag updates Those ADR would be opened on the postgresql branch and eventually make it to the James master branch. Best regards, Benoit TELLIER On 27/11/2023 12:47, Quan tran hong wrote: Hi everyone, This is a follow-up on the Postgres implementation topic. We have been working on this topic for a while and your feedback and expertise about Postgres would be helpful :-) For short: - On the progress: We have implemented the SubscriptionMapper, UsersRepository, DomainList, and MailboxMapper. We are working on replacing MessageMapper (a big one) and other components. We planned to have a fully functional Postgres mailbox before moving to other blocks like object storage, search, JMAP support... Of course, we are hungry to have a performance test for the Postgres mailbox, but it is too early ATM while the core components are not ready. - We use jOOQ library + r2dbc-postgresql so far, and the development experience is overall good. The jOOQ documentation and examples are not always ready TBH, but by doing a bit of research and ChatGPT, we managed to get jOOQ work so far. - Sometimes Guice binding for Postgres code is not easy because we are mixing up the JPA and Postgres code for now, which leads to some dependent issues. So far developing James, at least for me, I experienced mostly NoSQL and not
Re: Building a PostgreSQL mailbox for James?
Hi everyone, This is another follow-up to our recent work on the Postgres topic. In the last sprint: - We migrated fully JPA code to Postgres. No more JPA code in the postgres-app now. So at this moment, we had a fully functional IMAP server on top of Postgres. - We solved the flags concurrently update issue and proposed an ADR explaining the solution. - We implemented a blob store backed by Postgres. However, it is not mature yet as we are having concurrent issues with PostgresBlobStore when Postgres persists blobs slowly using just a single connection. On that, we are working on a Postgres pool connection to improve the performance. - We succeeded in plugging RabbitMQ as the event bus, S3 as the blob store, and OpenSearch into the postgres-app. We provide configurable module choosers so users are free to choose the implementation. For example, one could select S3/File/Postgres as his blob store implementation. In the coming time, we plan to implement the JMAP storage backed by Postgres. To have a look at the planned tickets, please visit https://github.com/linagora/james-project/issues. Your feedback would be appreciated. Regards, Quan Vào Th 6, 15 thg 12, 2023 vào lúc 18:45 Quan tran hong < quan.tranhong1...@gmail.com> đã viết: > Hi everyone, > > Yet another following up on our recent Postgres work. > > In the last sprint, our team successfully migrated the mailbox core module > `apache-james-mailbox-postgres` fully from JPA to Postgres. We did some > performance tests with the IMAP server backed by Postgres-app and the > result was very impressive: postgres-app outperformed JPA about 20 times > (at least look at the 99 percentile) with even less resource usage (CPU and > memory) compared to the jpa-app. For more details: > https://github.com/linagora/james-project/issues/4936 > > We are working on an ADR to explain our schema decision on the core tables > like mailbox and message tables, and also solving the flags concurrently > update issue. > > In the coming time, we plan to: > - Migrate other remaining minor components from JPA code to Postgres. > - Plug the RabbitMQ, S3, OpenSearch into the Postgres-app. > On that, we would try to implement alternative implementations with > Postgres e.g. BlobStore backed by Postgres. We would provide configurable > module choosers so users are free to choose the implementation. > > Best regards, > > Quan > > Vào Th 2, 27 thg 11, 2023 vào lúc 19:55 Benoit TELLIER < > btell...@apache.org> đã viết: > >> Thanks for the write up Quan! >> >> I would add that we shall rights ADRs about important decisions on the >> PostgreSQL implementation. >> >> So far I identified... >> >> - table structure for the postgresql mailbox >> - how deletes are handled >> - concurrency control for flag updates >> >> Those ADR would be opened on the postgresql branch and eventually make >> it to the James master branch. >> >> Best regards, >> >> Benoit TELLIER >> >> On 27/11/2023 12:47, Quan tran hong wrote: >> > Hi everyone, >> > >> > This is a follow-up on the Postgres implementation topic. We have been >> > working on this topic for a while and your feedback and expertise about >> > Postgres would be helpful :-) >> > >> > For short: >> > - On the progress: We have implemented the SubscriptionMapper, >> > UsersRepository, DomainList, and MailboxMapper. We are working on >> replacing >> > MessageMapper (a big one) and other components. We planned to have a >> > fully functional Postgres mailbox before moving to other blocks like >> object >> > storage, search, JMAP support... Of course, we are hungry to have a >> > performance test for the Postgres mailbox, but it is too early ATM while >> > the core components are not ready. >> > - We use jOOQ library + r2dbc-postgresql so far, and the development >> > experience is overall good. The jOOQ documentation and examples are not >> > always ready TBH, but by doing a bit of research and ChatGPT, we >> managed to >> > get jOOQ work so far. >> > - Sometimes Guice binding for Postgres code is not easy because we are >> > mixing up the JPA and Postgres code for now, which leads to some >> dependent >> > issues. >> > >> > So far developing James, at least for me, I experienced mostly NoSQL and >> > not SQL. So your SQL expertise would definitely help us with its best >> > practices. >> > For more details on what we did and plan to do on the Postgres topic, >> you >> > can kindly review them at: >> > >> https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql >> > >> > Best regards, >> > >> > Quan >> > >> > >> > Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler < >> > matth...@baechler-craftsmanship.fr> đã viết: >> > >> >> Hi Benoit, >> >> >> >> This topic has been discussed for years, I'm happy you finally draw a >> plan >> >> for it. >> >> >> >> To me, the aim for Postgres is small to middle size deployment with >> >> minimal dependencies. >> >> >> >> In that regard, having an
Re: Building a PostgreSQL mailbox for James?
Hi everyone, Yet another following up on our recent Postgres work. In the last sprint, our team successfully migrated the mailbox core module `apache-james-mailbox-postgres` fully from JPA to Postgres. We did some performance tests with the IMAP server backed by Postgres-app and the result was very impressive: postgres-app outperformed JPA about 20 times (at least look at the 99 percentile) with even less resource usage (CPU and memory) compared to the jpa-app. For more details: https://github.com/linagora/james-project/issues/4936 We are working on an ADR to explain our schema decision on the core tables like mailbox and message tables, and also solving the flags concurrently update issue. In the coming time, we plan to: - Migrate other remaining minor components from JPA code to Postgres. - Plug the RabbitMQ, S3, OpenSearch into the Postgres-app. On that, we would try to implement alternative implementations with Postgres e.g. BlobStore backed by Postgres. We would provide configurable module choosers so users are free to choose the implementation. Best regards, Quan Vào Th 2, 27 thg 11, 2023 vào lúc 19:55 Benoit TELLIER < btell...@apache.org> đã viết: > Thanks for the write up Quan! > > I would add that we shall rights ADRs about important decisions on the > PostgreSQL implementation. > > So far I identified... > > - table structure for the postgresql mailbox > - how deletes are handled > - concurrency control for flag updates > > Those ADR would be opened on the postgresql branch and eventually make > it to the James master branch. > > Best regards, > > Benoit TELLIER > > On 27/11/2023 12:47, Quan tran hong wrote: > > Hi everyone, > > > > This is a follow-up on the Postgres implementation topic. We have been > > working on this topic for a while and your feedback and expertise about > > Postgres would be helpful :-) > > > > For short: > > - On the progress: We have implemented the SubscriptionMapper, > > UsersRepository, DomainList, and MailboxMapper. We are working on > replacing > > MessageMapper (a big one) and other components. We planned to have a > > fully functional Postgres mailbox before moving to other blocks like > object > > storage, search, JMAP support... Of course, we are hungry to have a > > performance test for the Postgres mailbox, but it is too early ATM while > > the core components are not ready. > > - We use jOOQ library + r2dbc-postgresql so far, and the development > > experience is overall good. The jOOQ documentation and examples are not > > always ready TBH, but by doing a bit of research and ChatGPT, we managed > to > > get jOOQ work so far. > > - Sometimes Guice binding for Postgres code is not easy because we are > > mixing up the JPA and Postgres code for now, which leads to some > dependent > > issues. > > > > So far developing James, at least for me, I experienced mostly NoSQL and > > not SQL. So your SQL expertise would definitely help us with its best > > practices. > > For more details on what we did and plan to do on the Postgres topic, you > > can kindly review them at: > > > https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql > > > > Best regards, > > > > Quan > > > > > > Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler < > > matth...@baechler-craftsmanship.fr> đã viết: > > > >> Hi Benoit, > >> > >> This topic has been discussed for years, I'm happy you finally draw a > plan > >> for it. > >> > >> To me, the aim for Postgres is small to middle size deployment with > >> minimal dependencies. > >> > >> In that regard, having an implementation that spans across all > >> infrastructure needs is a must have. > >> > >> So my take would be let's implement everything with PG: blob storage, > >> search, messaging and various data storage like Event Sourcing and plain > >> data. > >> > >> For a user, it will always be possible to plug another piece of > >> infrastructure if need be (like having better search or store more > blobs, > >> etc). > >> > >> The only nice-to-have to me would be the multi-tenant goal as you can > >> always spawn another James instance by domain (and you can use the same > PG > >> if you want by using several databases). > >> > >> To answer the last questions: I would definitely be interested in using > >> this implementation (I use JPA for now). I could marginally contribute > to > >> it as I have experience with PG but my time is very limited (unless > someone > >> wants to sponsor my work, of course). I can donate some code related to > >> Event Sourcing has I have an implementation of an Event Store on top of > PG > >> and some code around messaging. Let me know if you are interested in > that > >> contributation. > >> > >> In term of strategy, I think that would help James gain popularity among > >> hobbyist and small businesses, so I think it worth trying. > >> > >> Cheers, > >> > >> -- Matthieu Baechler > >> > >> --- Original Message --- > >> On Friday, October 6th,
Re: Building a PostgreSQL mailbox for James?
Thanks for the write up Quan! I would add that we shall rights ADRs about important decisions on the PostgreSQL implementation. So far I identified... - table structure for the postgresql mailbox - how deletes are handled - concurrency control for flag updates Those ADR would be opened on the postgresql branch and eventually make it to the James master branch. Best regards, Benoit TELLIER On 27/11/2023 12:47, Quan tran hong wrote: Hi everyone, This is a follow-up on the Postgres implementation topic. We have been working on this topic for a while and your feedback and expertise about Postgres would be helpful :-) For short: - On the progress: We have implemented the SubscriptionMapper, UsersRepository, DomainList, and MailboxMapper. We are working on replacing MessageMapper (a big one) and other components. We planned to have a fully functional Postgres mailbox before moving to other blocks like object storage, search, JMAP support... Of course, we are hungry to have a performance test for the Postgres mailbox, but it is too early ATM while the core components are not ready. - We use jOOQ library + r2dbc-postgresql so far, and the development experience is overall good. The jOOQ documentation and examples are not always ready TBH, but by doing a bit of research and ChatGPT, we managed to get jOOQ work so far. - Sometimes Guice binding for Postgres code is not easy because we are mixing up the JPA and Postgres code for now, which leads to some dependent issues. So far developing James, at least for me, I experienced mostly NoSQL and not SQL. So your SQL expertise would definitely help us with its best practices. For more details on what we did and plan to do on the Postgres topic, you can kindly review them at: https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql Best regards, Quan Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler < matth...@baechler-craftsmanship.fr> đã viết: Hi Benoit, This topic has been discussed for years, I'm happy you finally draw a plan for it. To me, the aim for Postgres is small to middle size deployment with minimal dependencies. In that regard, having an implementation that spans across all infrastructure needs is a must have. So my take would be let's implement everything with PG: blob storage, search, messaging and various data storage like Event Sourcing and plain data. For a user, it will always be possible to plug another piece of infrastructure if need be (like having better search or store more blobs, etc). The only nice-to-have to me would be the multi-tenant goal as you can always spawn another James instance by domain (and you can use the same PG if you want by using several databases). To answer the last questions: I would definitely be interested in using this implementation (I use JPA for now). I could marginally contribute to it as I have experience with PG but my time is very limited (unless someone wants to sponsor my work, of course). I can donate some code related to Event Sourcing has I have an implementation of an Event Store on top of PG and some code around messaging. Let me know if you are interested in that contributation. In term of strategy, I think that would help James gain popularity among hobbyist and small businesses, so I think it worth trying. Cheers, -- Matthieu Baechler --- Original Message --- On Friday, October 6th, 2023 at 23:48, Benoit TELLIER < btell...@linagora.com> wrote: Hey there! The goal: deliver James "stateless email server" concept to smaller deployments than those addressable with the Distributed server. Why Postgres? Rock solid. And more options than other SQL stores (see below) The requirements would be: - Leverage the blobStore for binary storage (email bodies + attachements). Those big binaries are not meant to be stored into SQL rows - blaming you, JPA! - Bring choice on blob store : PGSQL native solution ( https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3 - Bring choice on search: PGSQL native solution ( https://www.postgresql.org/docs/current/textsearch.html ) for small deployments OR OpenSearch - Bring choice on PubSub: PGSQL native solution ( https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQ - Enforce strict tenant isolation: domain A won't access domain B data even if we screw up James access control layer. This can be done with Row security https://www.postgresql.org/docs/current/ddl-rowsecurity.html . - Be reactive. This can be achieved by using a reactive firendly driver like r2dbc... - Ensure that we can easily run on some largely scaling postgres... CitusData ? An other outcome might be to drop JPA implementation, ideally... (we provide something similar but wy better) Ideally I would like to deliver this before september 2024... Thoughts? Would this be something interesting people in here? Would some people be interested
Re: Building a PostgreSQL mailbox for James?
Hi everyone, This is a follow-up on the Postgres implementation topic. We have been working on this topic for a while and your feedback and expertise about Postgres would be helpful :-) For short: - On the progress: We have implemented the SubscriptionMapper, UsersRepository, DomainList, and MailboxMapper. We are working on replacing MessageMapper (a big one) and other components. We planned to have a fully functional Postgres mailbox before moving to other blocks like object storage, search, JMAP support... Of course, we are hungry to have a performance test for the Postgres mailbox, but it is too early ATM while the core components are not ready. - We use jOOQ library + r2dbc-postgresql so far, and the development experience is overall good. The jOOQ documentation and examples are not always ready TBH, but by doing a bit of research and ChatGPT, we managed to get jOOQ work so far. - Sometimes Guice binding for Postgres code is not easy because we are mixing up the JPA and Postgres code for now, which leads to some dependent issues. So far developing James, at least for me, I experienced mostly NoSQL and not SQL. So your SQL expertise would definitely help us with its best practices. For more details on what we did and plan to do on the Postgres topic, you can kindly review them at: https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql Best regards, Quan Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler < matth...@baechler-craftsmanship.fr> đã viết: > Hi Benoit, > > This topic has been discussed for years, I'm happy you finally draw a plan > for it. > > To me, the aim for Postgres is small to middle size deployment with > minimal dependencies. > > In that regard, having an implementation that spans across all > infrastructure needs is a must have. > > So my take would be let's implement everything with PG: blob storage, > search, messaging and various data storage like Event Sourcing and plain > data. > > For a user, it will always be possible to plug another piece of > infrastructure if need be (like having better search or store more blobs, > etc). > > The only nice-to-have to me would be the multi-tenant goal as you can > always spawn another James instance by domain (and you can use the same PG > if you want by using several databases). > > To answer the last questions: I would definitely be interested in using > this implementation (I use JPA for now). I could marginally contribute to > it as I have experience with PG but my time is very limited (unless someone > wants to sponsor my work, of course). I can donate some code related to > Event Sourcing has I have an implementation of an Event Store on top of PG > and some code around messaging. Let me know if you are interested in that > contributation. > > In term of strategy, I think that would help James gain popularity among > hobbyist and small businesses, so I think it worth trying. > > Cheers, > > -- Matthieu Baechler > > --- Original Message --- > On Friday, October 6th, 2023 at 23:48, Benoit TELLIER < > btell...@linagora.com> wrote: > > > > > > > > Hey there! > > > > The goal: deliver James "stateless email server" concept to smaller > deployments than those addressable with the Distributed server. > > > > Why Postgres? Rock solid. And more options than other SQL stores (see > below) > > > > The requirements would be: > > - Leverage the blobStore for binary storage (email bodies + > attachements). Those big binaries are not meant to be stored into SQL rows > - blaming you, JPA! > > - Bring choice on blob store : PGSQL native solution ( > https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small > deployments OR S3 > > - Bring choice on search: PGSQL native solution ( > https://www.postgresql.org/docs/current/textsearch.html ) for small > deployments OR OpenSearch > > - Bring choice on PubSub: PGSQL native solution ( > https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQ > > - Enforce strict tenant isolation: domain A won't access domain B data > even if we screw up James access control layer. This can be done with Row > security https://www.postgresql.org/docs/current/ddl-rowsecurity.html . > > - Be reactive. This can be achieved by using a reactive firendly driver > like r2dbc... > > - Ensure that we can easily run on some largely scaling postgres... > CitusData ? > > > > An other outcome might be to drop JPA implementation, ideally... (we > provide something similar but wy better) > > > > Ideally I would like to deliver this before september 2024... > > > > Thoughts? > > Would this be something interesting people in here? > > Would some people be interested contributing to this effort? > > Would some people desire sponsoring this effort? > > > > If this is non consensual, I can also contribute this into > https://github.com/linagora/tmail-backend/ without annoying people in > here... > > > > > > > > -- > > > > Best regards, > > > > Benoit
Re: Building a PostgreSQL mailbox for James?
Hi Benoit, This topic has been discussed for years, I'm happy you finally draw a plan for it. To me, the aim for Postgres is small to middle size deployment with minimal dependencies. In that regard, having an implementation that spans across all infrastructure needs is a must have. So my take would be let's implement everything with PG: blob storage, search, messaging and various data storage like Event Sourcing and plain data. For a user, it will always be possible to plug another piece of infrastructure if need be (like having better search or store more blobs, etc). The only nice-to-have to me would be the multi-tenant goal as you can always spawn another James instance by domain (and you can use the same PG if you want by using several databases). To answer the last questions: I would definitely be interested in using this implementation (I use JPA for now). I could marginally contribute to it as I have experience with PG but my time is very limited (unless someone wants to sponsor my work, of course). I can donate some code related to Event Sourcing has I have an implementation of an Event Store on top of PG and some code around messaging. Let me know if you are interested in that contributation. In term of strategy, I think that would help James gain popularity among hobbyist and small businesses, so I think it worth trying. Cheers, -- Matthieu Baechler --- Original Message --- On Friday, October 6th, 2023 at 23:48, Benoit TELLIER wrote: > > > Hey there! > > The goal: deliver James "stateless email server" concept to smaller > deployments than those addressable with the Distributed server. > > Why Postgres? Rock solid. And more options than other SQL stores (see below) > > The requirements would be: > - Leverage the blobStore for binary storage (email bodies + attachements). > Those big binaries are not meant to be stored into SQL rows - blaming you, > JPA! > - Bring choice on blob store : PGSQL native solution ( > https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small > deployments OR S3 > - Bring choice on search: PGSQL native solution ( > https://www.postgresql.org/docs/current/textsearch.html ) for small > deployments OR OpenSearch > - Bring choice on PubSub: PGSQL native solution ( > https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQ > - Enforce strict tenant isolation: domain A won't access domain B data even > if we screw up James access control layer. This can be done with Row security > https://www.postgresql.org/docs/current/ddl-rowsecurity.html . > - Be reactive. This can be achieved by using a reactive firendly driver like > r2dbc... > - Ensure that we can easily run on some largely scaling postgres... CitusData > ? > > An other outcome might be to drop JPA implementation, ideally... (we provide > something similar but wy better) > > Ideally I would like to deliver this before september 2024... > > Thoughts? > Would this be something interesting people in here? > Would some people be interested contributing to this effort? > Would some people desire sponsoring this effort? > > If this is non consensual, I can also contribute this into > https://github.com/linagora/tmail-backend/ without annoying people in here... > > > > -- > > Best regards, > > Benoit TELLIER > > General manager of Linagora VIETNAM. > Product owner for Team-Mail product. > Chairman of the Apache James project. > > Mail: btell...@linagora.com > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Building a PostgreSQL mailbox for James?
I somehow messed up my email config and the following message never reached the list (but never got a nack either so it took me a week and 2 emails without response to detect the problem) Hello Benoit I think having a postgresql backed mailbox server sounds great It should not come as a surprise since Matthieu and I already play with deploying the pulsar+postgresql+s3 server for SMTP Mailbox is a next step once we get SMTP up and running with a reasonable level of stability. But at our place it will take some time :) So I (we) would definitely contribute, not very fast as our free time dictates but still contribute. Le ven. 6 oct. 2023 à 23:49, Benoit TELLIER a écrit : > Hey there! > > The goal: deliver James "stateless email server" concept to smaller > deployments than those addressable with the Distributed server. > > Why Postgres? Rock solid. And more options than other SQL stores (see > below) > > The requirements would be: > - Leverage the blobStore for binary storage (email bodies + > attachements). Those big binaries are not meant to be stored into SQL rows > - blaming you, JPA! > - Bring choice on blob store : PGSQL native solution ( > https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small > deployments OR S3 > - Bring choice on search: PGSQL native solution ( > https://www.postgresql.org/docs/current/textsearch.html ) for small > deployments OR OpenSearch > - Bring choice on PubSub: PGSQL native solution ( > https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR > RabbitMQ > - Enforce strict tenant isolation: domain A won't access domain B data > even if we screw up James access control layer. This can be done with Row > security https://www.postgresql.org/docs/current/ddl-rowsecurity.html . > - Be reactive. This can be achieved by using a reactive firendly driver > like r2dbc... > - Ensure that we can easily run on some largely scaling postgres... > CitusData ? > > An other outcome might be to drop JPA implementation, ideally... (we > provide something similar but wy better) > > Ideally I would like to deliver this before september 2024... > > Thoughts? > Would this be something interesting people in here? > Would some people be interested contributing to this effort? > Would some people desire sponsoring this effort? > > If this is non consensual, I can also contribute this into > https://github.com/linagora/tmail-backend/ without annoying people in > here... > > > > -- > > Best regards, > > Benoit TELLIER > > General manager of Linagora VIETNAM. > Product owner for Team-Mail product. > Chairman of the Apache James project. > > Mail: btell...@linagora.com > Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) > > > > >
Re: Building a PostgreSQL mailbox for James?
I did used Samsung email client on my personnal phone, that likely did badly HTML input that messed up ASF mailing lists. Apparently TMail to not mess up with formatting... -- Best regards, Benoit TELLIER General manager of Linagora VIETNAM. Product owner for Team-Mail product. Chairman of the Apache James project. Mail: btell...@linagora.com Tel: (0033) 6 77 26 04 58 (WhatsApp, Signal) On Oct 11, 2023 5:31 PM, from Wojtek Thanks for your comments. Yes, Loom is very new and there were recently even blocker when upgrading James beyond Java11 (serialization) though, with that out of the way I don't see bigger issues with adoption (_maybe_ Scala compatibility, but that's huge _maybe_ :-) ). Out of curiosity - what mailer have you used that garbled the messages? Wojtek On 10/10/2023 11:33, btellier wrote: > I'll answer: first thanks for the head up andenthousiasm!Regarding your > remarks, row level security is none essential requirement on james side but > an essential requirement to me ;-)Regarding JPA, sure if we have contributors > making it leave and fixing its flows there is no reason to deprecate and drop > it. Let see what happens. At least we will have an alternative.Regarding > reactive/loom topic, though a strong advocate internally to use newly > available jdk/jre, we always had issues in the community to adopt newer > jdk/jre. Second, I havent seen any detailed return of experience on reactive > / loom topic. I'm personnally expecting perfs to be somewhat similar bit loom > might still require extra scheduling compared to pure reactive > implementations... for sure loom will make it seem less handy...yes i am also > expecting citusData to work OOTB but our dockerised test suite would make > great job fact-checking this assumption, at least with smoke tests...Envoyé > depuis mon appareil Galaxy > Message d'origine De : Wojtek Date : > 10/10/2023 16:05 (GMT+07:00) À : James Developers List > Objet : Re: Building a PostgreSQL mailbox for > James? Hi Benoit,Please see comments below.On 06/10/2023 23:48, Benoit > TELLIER wrote:> Hey there!> > The goal: deliver James "stateless email > server" concept to smaller deployments than those addressable with the > Distributed server.> > Why Postgres? Rock solid. And more options than other > SQL stores (see below)> > The requirements would be:> - Leverage the > blobStore for binary storage (email bodies + attachements). Those big > binaries are not meant to be stored into SQL rows - blaming you, JPA!> - > Bring choice on blob store : PGSQL native solution ( > postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3> > - Bring choice on search: PGSQL native solution ( > postgresql.org/docs/current/textsearch.html ) for small deployments OR > OpenSearch> - Bring choice on PubSub: PGSQL native solution ( > postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQThis sounds > very good.> - Enforce strict tenant isolation: domain A won't access > domain B data even if we screw up James access control layer. This can be > done with Row security postgresql.org/docs/current/ddl-rowsecurity.html > .Handy, but not essential (perspective: small, personal mail server running > on home machine)> - Be reactive. This can be achieved by using a reactive > firendly driver like r2dbc...While reactive has it's benefits (backpressure), > I'm not sure it will be as handy with Loom.> - Ensure that we can easily > run on some largely scaling postgres... CitusData ?This should work OOTB if > we stick with Posgress?> An other outcome might be to drop JPA > implementation, ideally... (we provide something similar but wy > better)Hmm... I'm split on this. There was previous implementation using pure > JDBC, but it was deprecated/dropped in favour of using universal JPA. And JPA > "just works", but has it's limitations (I'd say storing blobs being the most > annoying).> Ideally I would like to deliver this before september 2024...> > > Thoughts?> Would this be something interesting people in here?Very much so!> > Would some people be interested contributing to this effort?> Would some > people desire sponsoring this effort?> > If this is non consensual, I can > also contribute this into github.com/linagora/tmail-backend/ without annoying > people in here...> Benoit TELLIERWould be nice to have it directly in James > main repository > :-)w.-To > unsubscribe, e-mail: server-dev-unsubscribe@james.apache.orgFor additional > commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Building a PostgreSQL mailbox for James?
Thanks for your comments. Yes, Loom is very new and there were recently even blocker when upgrading James beyond Java11 (serialization) though, with that out of the way I don't see bigger issues with adoption (_maybe_ Scala compatibility, but that's huge _maybe_ :-) ). Out of curiosity - what mailer have you used that garbled the messages? Wojtek On 10/10/2023 11:33, btellier wrote: I'll answer: first thanks for the head up andenthousiasm!Regarding your remarks, row level security is none essential requirement on james side but an essential requirement to me ;-)Regarding JPA, sure if we have contributors making it leave and fixing its flows there is no reason to deprecate and drop it. Let see what happens. At least we will have an alternative.Regarding reactive/loom topic, though a strong advocate internally to use newly available jdk/jre, we always had issues in the community to adopt newer jdk/jre. Second, I havent seen any detailed return of experience on reactive / loom topic. I'm personnally expecting perfs to be somewhat similar bit loom might still require extra scheduling compared to pure reactive implementations... for sure loom will make it seem less handy...yes i am also expecting citusData to work OOTB but our dockerised test suite would make great job fact-checking this assumption, at least with smoke tests...Envoyé depuis mon appareil Galaxy Message d'origine De : Wojtek Date : 10/10/2023 16:05 (GMT+07:00) À : James Developers List Objet : Re: Building a PostgreSQL mailbox for James? Hi Benoit,Please see comments below.On 06/10/2023 23:48, Benoit TELLIER wrote:> Hey there!> > The goal: deliver James "stateless email server" concept to smaller deployments than those addressable with the Distributed server.> > Why Postgres? Rock solid. And more options than other SQL stores (see below)> > The requirements would be:> - Leverage the blobStore for binary storage (email bodies + attachements). Those big binaries are not meant to be stored into SQL rows - blaming you, JPA!> - Bring choice on blob store : PGSQL native solution ( https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3> - Bring choice on search: PGSQL native solution ( https://www.postgresql.org/docs/current/textsearch.html ) for small deployments OR OpenSearch> - Bring choice on PubSub: PGSQL native solution ( https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQThis sounds very good.> - Enforce strict tenant isolation: domain A won't access domain B data even if we screw up James access control layer. This can be done with Row security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .Handy, but not essential (perspective: small, personal mail server running on home machine)> - Be reactive. This can be achieved by using a reactive firendly driver like r2dbc...While reactive has it's benefits (backpressure), I'm not sure it will be as handy with Loom.> - Ensure that we can easily run on some largely scaling postgres... CitusData ?This should work OOTB if we stick with Posgress?> An other outcome might be to drop JPA implementation, ideally... (we provide something similar but wy better)Hmm... I'm split on this. There was previous implementation using pure JDBC, but it was deprecated/dropped in favour of using universal JPA. And JPA "just works", but has it's limitations (I'd say storing blobs being the most annoying).> Ideally I would like to deliver this before september 2024...> > Thoughts?> Would this be something interesting people in here?Very much so!> Would some people be interested contributing to this effort?> Would some people desire sponsoring this effort?> > If this is non consensual, I can also contribute this into https://github.com/linagora/tmail-backend/ without annoying people in here...> Benoit TELLIERWould be nice to have it directly in James main repository :-)w.-To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.orgFor additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Building a PostgreSQL mailbox for James?
I'll answer: first thanks for the head up andenthousiasm!Regarding your remarks, row level security is none essential requirement on james side but an essential requirement to me ;-)Regarding JPA, sure if we have contributors making it leave and fixing its flows there is no reason to deprecate and drop it. Let see what happens. At least we will have an alternative.Regarding reactive/loom topic, though a strong advocate internally to use newly available jdk/jre, we always had issues in the community to adopt newer jdk/jre. Second, I havent seen any detailed return of experience on reactive / loom topic. I'm personnally expecting perfs to be somewhat similar bit loom might still require extra scheduling compared to pure reactive implementations... for sure loom will make it seem less handy...yes i am also expecting citusData to work OOTB but our dockerised test suite would make great job fact-checking this assumption, at least with smoke tests...Envoyé depuis mon appareil Galaxy Message d'origine De : Wojtek Date : 10/10/2023 16:05 (GMT+07:00) À : James Developers List Objet : Re: Building a PostgreSQL mailbox for James? Hi Benoit,Please see comments below.On 06/10/2023 23:48, Benoit TELLIER wrote:> Hey there!> > The goal: deliver James "stateless email server" concept to smaller deployments than those addressable with the Distributed server.> > Why Postgres? Rock solid. And more options than other SQL stores (see below)> > The requirements would be:> - Leverage the blobStore for binary storage (email bodies + attachements). Those big binaries are not meant to be stored into SQL rows - blaming you, JPA!> - Bring choice on blob store : PGSQL native solution ( https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3> - Bring choice on search: PGSQL native solution ( https://www.postgresql.org/docs/current/textsearch.html ) for small deployments OR OpenSearch> - Bring choice on PubSub: PGSQL native solution ( https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQThis sounds very good.> - Enforce strict tenant isolation: domain A won't access domain B data even if we screw up James access control layer. This can be done with Row security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .Handy, but not essential (perspective: small, personal mail server running on home machine)> - Be reactive. This can be achieved by using a reactive firendly driver like r2dbc...While reactive has it's benefits (backpressure), I'm not sure it will be as handy with Loom.> - Ensure that we can easily run on some largely scaling postgres... CitusData ?This should work OOTB if we stick with Posgress?> An other outcome might be to drop JPA implementation, ideally... (we provide something similar but wy better)Hmm... I'm split on this. There was previous implementation using pure JDBC, but it was deprecated/dropped in favour of using universal JPA. And JPA "just works", but has it's limitations (I'd say storing blobs being the most annoying).> Ideally I would like to deliver this before september 2024...> > Thoughts?> Would this be something interesting people in here?Very much so!> Would some people be interested contributing to this effort?> Would some people desire sponsoring this effort?> > If this is non consensual, I can also contribute this into https://github.com/linagora/tmail-backend/ without annoying people in here...> Benoit TELLIERWould be nice to have it directly in James main repository :-)w.-To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.orgFor additional commands, e-mail: server-dev-h...@james.apache.org
Re: Building a PostgreSQL mailbox for James?
I'll answer: first thanks for the head up andenthousiasm!Regarding your remarks, row level security is none essential requirement on james side but an essential requirement to me ;-)Regarding JPA, sure if we have contributors making it leave and fixing its flows there is no reason to deprecate and drop it. Let see what happens. At least we will have an alternative.Regarding reactive/loom topic, though a strong advocate internally to use newly available jdk/jre, we always had issues in the community to adopt newer jdk/jre. Second, I havent seen any detailed return of experience on reactive / loom topic. I'm personnally expecting perfs to be somewhat similar bit loom might still require extra scheduling compared to pure reactive implementations... for sure loom will make it seem less handy...yes i am also expecting citusData to work OOTB but our dockerised test suite would make great job fact-checking this assumption, at least with smoke tests...Envoyé depuis mon appareil Galaxy Message d'origine De : Wojtek Date : 10/10/2023 16:05 (GMT+07:00) À : James Developers List Objet : Re: Building a PostgreSQL mailbox for James? Hi Benoit,Please see comments below.On 06/10/2023 23:48, Benoit TELLIER wrote:> Hey there!> > The goal: deliver James "stateless email server" concept to smaller deployments than those addressable with the Distributed server.> > Why Postgres? Rock solid. And more options than other SQL stores (see below)> > The requirements would be:> - Leverage the blobStore for binary storage (email bodies + attachements). Those big binaries are not meant to be stored into SQL rows - blaming you, JPA!> - Bring choice on blob store : PGSQL native solution ( https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3> - Bring choice on search: PGSQL native solution ( https://www.postgresql.org/docs/current/textsearch.html ) for small deployments OR OpenSearch> - Bring choice on PubSub: PGSQL native solution ( https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQThis sounds very good.> - Enforce strict tenant isolation: domain A won't access domain B data even if we screw up James access control layer. This can be done with Row security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .Handy, but not essential (perspective: small, personal mail server running on home machine)> - Be reactive. This can be achieved by using a reactive firendly driver like r2dbc...While reactive has it's benefits (backpressure), I'm not sure it will be as handy with Loom.> - Ensure that we can easily run on some largely scaling postgres... CitusData ?This should work OOTB if we stick with Posgress?> An other outcome might be to drop JPA implementation, ideally... (we provide something similar but wy better)Hmm... I'm split on this. There was previous implementation using pure JDBC, but it was deprecated/dropped in favour of using universal JPA. And JPA "just works", but has it's limitations (I'd say storing blobs being the most annoying).> Ideally I would like to deliver this before september 2024...> > Thoughts?> Would this be something interesting people in here?Very much so!> Would some people be interested contributing to this effort?> Would some people desire sponsoring this effort?> > If this is non consensual, I can also contribute this into https://github.com/linagora/tmail-backend/ without annoying people in here...> Benoit TELLIERWould be nice to have it directly in James main repository :-)w.-To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.orgFor additional commands, e-mail: server-dev-h...@james.apache.org
Re: Building a PostgreSQL mailbox for James?
Hi Benoit, Please see comments below. On 06/10/2023 23:48, Benoit TELLIER wrote: Hey there! The goal: deliver James "stateless email server" concept to smaller deployments than those addressable with the Distributed server. Why Postgres? Rock solid. And more options than other SQL stores (see below) The requirements would be: - Leverage the blobStore for binary storage (email bodies + attachements). Those big binaries are not meant to be stored into SQL rows - blaming you, JPA! - Bring choice on blob store : PGSQL native solution ( https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small deployments OR S3 - Bring choice on search: PGSQL native solution ( https://www.postgresql.org/docs/current/textsearch.html ) for small deployments OR OpenSearch - Bring choice on PubSub: PGSQL native solution ( https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQ This sounds very good. - Enforce strict tenant isolation: domain A won't access domain B data even if we screw up James access control layer. This can be done with Row security https://www.postgresql.org/docs/current/ddl-rowsecurity.html . Handy, but not essential (perspective: small, personal mail server running on home machine) - Be reactive. This can be achieved by using a reactive firendly driver like r2dbc... While reactive has it's benefits (backpressure), I'm not sure it will be as handy with Loom. - Ensure that we can easily run on some largely scaling postgres... CitusData ? This should work OOTB if we stick with Posgress? An other outcome might be to drop JPA implementation, ideally... (we provide something similar but wy better) Hmm... I'm split on this. There was previous implementation using pure JDBC, but it was deprecated/dropped in favour of using universal JPA. And JPA "just works", but has it's limitations (I'd say storing blobs being the most annoying). Ideally I would like to deliver this before september 2024... Thoughts? Would this be something interesting people in here? Very much so! Would some people be interested contributing to this effort? Would some people desire sponsoring this effort? If this is non consensual, I can also contribute this into https://github.com/linagora/tmail-backend/ without annoying people in here... Benoit TELLIER Would be nice to have it directly in James main repository :-) w. - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
