Re: Call for vote: Apache James JSPF 1.0.4

[Quan tran hong]

+1

Quan

Vào Th 2, 10 thg 6, 2024 vào lúc 01:48 Jean Helou  đã
viết:

> +1
>
> Le ven. 7 juin 2024 à 16:32, btell...@linagora.com 
> a écrit :
>
> > +1
> >
> > On 07/06/2024 15:39, Benoit TELLIER wrote:
> > > Hi,
> > >
> > > I would like to propose a new vote for 1.0.4 release of the Apache
> > > James JSPF library.
> > >
> > > You can find:
> > >
> > >  - The maven release staged in repository.apache.org as the artifact
> > > #1083:
> > >
> https://repository.apache.org/content/repositories/orgapachejames-1083/
> > >  - The blog post for this release:
> > > https://github.com/apache/james-project/pull/2286
> > >
> > > This release fixes asynchronous JSPF executors.
> > >
> > > Voting rules:
> > >  - This is a majority approval: this release may not be vetoed.
> > >  - A quorum of 3 binding votes is required
> > >  - The vote starts at Friday 7th of June 2024, 3pm UTC+7
> > >  - The vote ends at Friday 14th of June 2024, 3pm UTC+7
> > >
> > > You can answer to it just with +1 and -1. Down-votes may be motivated.
> > >
> > > 3 binding votes are expected move forward on this release.
> > >
> > > Cheers,
> > >
> > > Benoit TELLIER
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > > For additional commands, e-mail: server-dev-h...@james.apache.org
> > >
> > >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>

Re: Making all queues on Rabbitmq quorum queue when option enabled

[Quan tran hong]

> Do you share this point of view?

+1

Quan

Vào Th 2, 15 thg 4, 2024 vào lúc 16:43 Benoit TELLIER 
đã viết:

> Hi Quan,
>
> First thanks for the job done on this topic.
>
> I know some members of the community (Karsten ?) already did significant
> work on the topic but more oriented toward the POP3 server.
>
> This work is of course welcome as it would result in a higher
> reliability for the IMAP / JMAP components.
>
>  > What do you think about making all queues on Rabbitmq quorum queue
> when option enabled? On the principle, +1 In practice that is slightly
> harder for the event bus notification queue... - We can likely afford
> losing some of those pub sub message? - The queue is tied to a
> connection, thus if the node/connection goes done it can be recreated
> elsewhere? - We would need to come up with a cleanup strategy in order
> to eventually deletes queues haging around. - Also, how relevant is this
> RabbitMQ backend pub sub implementation when compared with the work done
> with Redis? IMO the eventbus notification was the main blocker in order
> to achieve decent HA with RabbitMQ. Do you share this point of view?
> Best regards, Benoit TELLIER
>
> On 15/04/2024 09:53, Quan tran hong wrote:
> > Hi folks,
> >
> > Recently we encountered a deployment issue that used a RabbitMQ Cluster
> > where a RabbitMQ node outage (for about 1 hour) forced James service more
> > or less to be down too.
> >
> > I created a Jira ticket to report the issue:
> > https://issues.apache.org/jira/projects/JAMES/issues/JAMES-4027
> >
> > More details below for one did not read the Jira ticket yet:
> >
> > Today, when the quorum option is enabled, only some queues are quorum
> > queues, not all (e.g. event bus notification queues and Task Manager's
> > termination queues).
> >
> > I tried to reproduce the issue and here is my theory:
> >
> > The RabbitMQ node that stores the notification queues is down
> > -> James can not publish messages to RabbitMQ and causes e.g. IMAP
> SELECT,
> > STORE, APPEND, UNSELECT ... commands to fail
> > -> James keeps retrying the publish failures (retry for Group
> registration
> > which seems to rely on the classic queue too) and queues other IMAP
> > requests in the meantime.
> > -> The IMAP server queue becomes full and the exception `The IMAP server
> > has reached its maximum capacity` is thrown.
> > -> James IMAP becomes a zombie and cascading failures.
> >
> > James needs to be more fault-tolerant in this case.
> >
> > We think making all queues on Rabbitmq quorum queue when
> > `quorum.queues.enable=true` would help James be more fault tolerant on
> that
> > scenario.
> >
> > We investigated a POC athttps://
> github.com/apache/james-project/pull/2191  and
> > the full quorum queues helped James be more fault tolerant as expected.
> >
> > After full quorum queues are used, the James performance is a bit slower
> > but is still fine, and that cost is likely needed to make James more
> > reliable.
> >
> > If we use Redis backed event bus notifications, the performance is better
> > than the RabbitMQ notification quorum queues.
> >
> > What do you think about making all queues on Rabbitmq quorum queue when
> > option enabled? Feedback and review are very welcome.
> >
> > Thanks for reading.
> >
> > Quan
> >

Making all queues on Rabbitmq quorum queue when option enabled

[Quan tran hong]

Hi folks,

Recently we encountered a deployment issue that used a RabbitMQ Cluster
where a RabbitMQ node outage (for about 1 hour) forced James service more
or less to be down too.

I created a Jira ticket to report the issue:
https://issues.apache.org/jira/projects/JAMES/issues/JAMES-4027

More details below for one did not read the Jira ticket yet:

Today, when the quorum option is enabled, only some queues are quorum
queues, not all (e.g. event bus notification queues and Task Manager's
termination queues).

I tried to reproduce the issue and here is my theory:

The RabbitMQ node that stores the notification queues is down
-> James can not publish messages to RabbitMQ and causes e.g. IMAP SELECT,
STORE, APPEND, UNSELECT ... commands to fail
-> James keeps retrying the publish failures (retry for Group registration
which seems to rely on the classic queue too) and queues other IMAP
requests in the meantime.
-> The IMAP server queue becomes full and the exception `The IMAP server
has reached its maximum capacity` is thrown.
-> James IMAP becomes a zombie and cascading failures.

James needs to be more fault-tolerant in this case.

We think making all queues on Rabbitmq quorum queue when
`quorum.queues.enable=true` would help James be more fault tolerant on that
scenario.

We investigated a POC at https://github.com/apache/james-project/pull/2191 and
the full quorum queues helped James be more fault tolerant as expected.

After full quorum queues are used, the James performance is a bit slower
but is still fine, and that cost is likely needed to make James more
reliable.

If we use Redis backed event bus notifications, the performance is better
than the RabbitMQ notification quorum queues.

What do you think about making all queues on Rabbitmq quorum queue when
option enabled? Feedback and review are very welcome.

Thanks for reading.

Quan

Result of Proof Of Concept for Redis Event bus user notifications

[Quan tran hong]

Hi folks,

Following the context of the Jira ticket JAMES-3996 POC: Move RabbitMQ
Event bus user notifications to Redis

(TLDR:
we observed some annoying issues with RabbitMQ in a deployment and we think
it could be better to move at least the user notifications part of Event
Bus to Redis), I did a Proof Of Concept about that in the PR JAMES 3996
Redis event bus POC .

The POC is considered done to me, and I want to share the POC result to
James devs:
- It is *possible *(the POC worked!) to replace the Event bus user
notifications (key registration) using Redis Pub/Sub. And likely the whole
EventBus (the group registration part as well) can rely on Redis (I did not
do that part).
- I did several performance tests for the POC, and the results were *good*.
Regarding the metrics of Event Bus user notifications, Redis seems to
outperform and show more stability in response time than RabbitMQ. (For
more details, I shared on the PR)

Despite the POC has been worked and shown some prospects, I think we should
monitor it more carefully before applying it to James. Therefore, we
(Linagora) would adopt the Redis Event bus user notifications first in our
TMail project (based on James). We would keep an eye on its stability
before contributing the fine-tuned work toward James.

By the way, it seems someone from the community is interested in building
the whole EventBus using Redis cf
https://issues.apache.org/jira/browse/JAMES-3956. This POC could be a start
for that Redis EventBus.

What do you think about using Redis for EventBus? Would that sound
interesting to you?

Regards,

Quan

Re: vote : Remote build cache experiment

[Quan tran hong]

+1,

Thanks for your enthusiastic effort on the topic.

Cheers,

Quan

Vào Th 3, 20 thg 2, 2024 vào lúc 16:07 Rene Cordier 
đã viết:

> +1,
>
> If there is a chance to make our builds faster, I say we take it! Thanks
> for this work :)
>
> Rene.
>
> On 2/20/24 15:18, Jean Helou wrote:
> > Hello devs !
> >
> > I have not yet moved forward with enabling remote caching for the build
> and
> > tests in general. It is unclear to me if you are ok with trying it from
> the
> > previous discussion. So I'll call a more formal vote:
> >
> > Are you ok with experimenting enabling remote build cache for james (ci
> > write+local read) ?
> >
> > Deployment of the remote build cache would follow the documented plan
> >
> https://docs.gradle.com/enterprise/maven-build-cache/#rolling_out_the_cache_in_your_organization
> > TLDR;
> > 1. move forward with the INFRA request to get our build cache node
> > 2. test (read+write) with a few (2-3) volunteers locally (not enabled on
> CI)
> > 3. enable write on CI
> > 4, enable read for everyone
> >
> > - voting starts 2024-02-20
> > - voting ends 2024-02-27
> >
> > This is lazy consensus, this vote can be vetoed (please motivate your
> > veto), there will be a second formal vote to definitely enable the cache
> > once the experiment has been conducted and there are some results to
> share.
> >
> > jean
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James MIME4J 0.8.10

[Quan tran hong]

+1

Quan

Vào Th 5, 15 thg 2, 2024 vào lúc 09:32 Rene Cordier 
đã viết:

> +1,
>
> Rene.
>
> On 2/12/24 17:38, Benoit TELLIER wrote:
> > Typo
> >
> > The maven release staged in repository.apache.org as the artifact #1081:
> > https://repository.apache.org/content/repositories/orgapachejames-1081/
> >
> > On 12/02/2024 11:35, Benoit TELLIER wrote:
> >> Hi,
> >>
> >> I would like to propose a new vote for 0.8.10 release of the Apache
> >> MIME4J library.
> >>
> >> You can find:
> >>
> >>  - The maven release staged in repository.apache.org as the artifact
> >> #1076:
> >> https://repository.apache.org/content/repositories/orgapachejames-1076/
> >>  - The changelog:
> >> https://github.com/chibenwa/james-mime4j/blob/master/CHANGELOG.md
> >>
> >> This release ships mostly small enhancements.
> >>
> >> Voting rules:
> >>  - This is a majority approval: this release may not be vetoed.
> >>  - A quorum of 3 binding votes is required
> >>  - The vote starts at Monday 12th of February 2024, 11am UTC+2
> >>  - The vote ends at Monday 19th of February 2024, 11am UTC+2
> >>
> >> You can answer to it just with +1 and -1. Down-votes may be motivated.
> >>
> >> 3 binding votes are expected move forward on this release.
> >>
> >> Cheers,
> >>
> >> Benoit TELLIER
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>
> >>
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James 3.7.5

[Quan tran hong]

+1

Quan

Vào Th 5, 15 thg 2, 2024 vào lúc 09:30 Rene Cordier 
đã viết:

> +1,
>
> Rene.
>
> On 2/12/24 17:37, Benoit TELLIER wrote:
> > Hi,
> >
> > Taking into account the feedback of this very list,
> >
> > I would like to propose a new vote for 3.7.5 release of the Apache
> > James server.
> >
> > You can find:
> >
> >  - The maven release staged in repository.apache.org as the artifact
> > #1079:
> > https://repository.apache.org/content/repositories/orgapachejames-1079/
> >  - The changelog for 3.7.5:
> > https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >
> > This release mostly ships bug fixes.
> >
> > Voting rules:
> >  - This is a majority approval: this release may not be vetoed.
> >  - A quorum of 3 binding votes is required
> >  - The vote starts at Tuesday 12th of February 2024, 11am UTC+2
> >  - The vote ends at Tuesday 19th of February 2024, 11am UTC+2
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James 3.8.1

[Quan tran hong]

+1

Quan

Vào Th 5, 15 thg 2, 2024 vào lúc 09:28 Rene Cordier 
đã viết:

> +1,
>
> Rene.
>
> On 2/12/24 18:43, Karsten Otto wrote:
> > +1
> >
> > Looks good to me.
> >
> > Cheers,
> > Karsten OTTO
> >
> >
> > On 12.02.24 11:46 AM, Benoit TELLIER wrote:
> >> Hi,
> >>
> >> Following the feedback of this very list,
> >>
> >> I would like to propose a new vote for 3.8.1 release of the Apache
> >> James server.
> >>
> >> You can find:
> >>
> >>   - The maven release staged in repository.apache.org as the artifact
> >> #1080:
> >> https://repository.apache.org/content/repositories/orgapachejames-1080/
> >>   - The changelog for 3.8.1:
> >> https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >>
> >> This release mostly ships bug fixes.
> >>
> >> Voting rules:
> >>   - This is a majority approval: this release may not be vetoed.
> >>   - A quorum of 3 binding votes is required
> >>   - The vote starts at Monday 12th of February 2024, 11am UTC+2
> >>   - The vote ends at Monday 19th of February 2024, 11am UTC+2
> >>
> >> You can answer to it just with +1 and -1. Down-votes may be motivated.
> >>
> >> 3 binding votes are expected move forward on this release.
> >>
> >> Cheers,
> >>
> >> Benoit TELLIER
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >> For additional commands, e-mail: server-dev-h...@james.apache.org
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Develocity

[Quan tran hong]

Hi,

> The risk is caching a negative result coming from a flaky test. And I feel
like James test suite is quite unstable.

On that, maybe we can find if the Gradle extension / surefire supports
caching only if the build (test phase for example) succeeds?

Quan

Vào Th 4, 7 thg 2, 2024 vào lúc 10:22 Rene Cordier 
đã viết:

> I agree caching the test suite is very tempting, it would help gain
> so much time on the build.
>
> But yes the test suite is flaky unfortunately. Maybe we need extra work
> making it more stable first, but that's not easy task ^^'
>
> Or maybe for remote caching on test execution, maybe we can try to
> execute that in an other branch in parallel to master, giving us time to
> monitor the behavior of the build, before using it for good? Requires
> extra work, efforts and attention yes, but I hardly see how else we can
> go safely forward on this. But somebody else has a better idea maybe? :)
>
> Regards,
>
> Rene.
>
> On 2/7/24 02:58, Jean Helou wrote:
> > Thanks for trying
> >
> > As you can see in the output only about 10% of the tasks you ran were
> > cached.
> >
> > Which explains why the gains are not so great.
> >
> > There are 2 good news
> > - it is possible to configure more goals to be cached. I have an attempt
> at
> > caching scala compile (which doesn't work for now as that goal has rather
> > complex input and outputs).
> > I intend to look at build scans to cache more high CPU consuming goals
> and
> > to avoid running some goals entirely when possible (as per my comments on
> > the PR)
> >
> > Where it becomes really interesting (the second good news) is that
> surefire
> > is supported out of the box, so if one pays the cost of running the
> tests,
> > every subsequent build would only run the tests for the changed projects
> > and their dependencies
> >
> > And testing is by far the most time consuming goal in James build.
> >
> > Remote caching of test execution would have the most impact, accelerating
> > both local and CI builds because it would allow sharing the cache between
> > different machines/nodes. (Only the ci would write but everyone could
> read
> > if I understand correctly)
> >
> > The risk is caching a negative result coming from a flaky test. And I
> feel
> > like James test suite is quite unstable.
> >
> > So I'm wondering if I should go ahead with the remote build cache node
> > request to apache INFRA or revisit that request in a few months.
> >
> > Jean
> >
> >
> >
> > Le mar. 6 févr. 2024 à 11:18, Quan tran hong <
> quan.tranhong1...@gmail.com>
> > a écrit :
> >
> >> Hi Jean,
> >>
> >> I tried your work locally and the build time is truly better.
> >> `mvn clean install -DskipTests -Dmaven.skip.doc=true
> >> -Dgradle.cache.local.enabled=true`
> >>
> >> 1st build:
> >> [INFO] Total time:  22:18 min
> >> [INFO] 8053 goals, 7996 executed, 57 from cache, saving at least 13s
> >>
> >> 2nd build:
> >> [INFO] Total time:  15:07 min
> >> [INFO] 8053 goals, 7296 executed, 757 from cache, saving at least 3m 11s
> >>
> >> Thank you for the initiative :-)
> >>
> >> Quan
> >>
> >> Vào Th 3, 6 thg 2, 2024 vào lúc 14:48 Jean Helou 
> đã
> >> viết:
> >>
> >>> Hi devs,
> >>>
> >>> I realized this morning that I initiated the conversation on the wrong
> >> list
> >>> (server-user instead of server-dev) ...
> >>> see the archives
> >>> https://lists.apache.org/list.html?server-u...@james.apache.org for
> >> Rene's
> >>> positive answer
> >>>
> >>> sorry about that
> >>>
> >>> jean
> >>>
> >>> -- Forwarded message -
> >>> From: Jean Helou 
> >>> Date: Mon, Feb 5, 2024 at 3:52 PM
> >>> Subject: Develocity
> >>> To: James Users List 
> >>>
> >>>
> >>> Hello fellow james devs,
> >>>
> >>> I'm not sure how much context to provide in the email you can read more
> >> in
> >>>
> >>
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3978?filter=allopenissues
> >>> TLDR; A few months ago the apache foundation announced a sponsorship of
> >> the
> >>> "gradle" company for a tool to diagnose, monitor and optimize builds  (
> >>> called gradle enterprise at the time and 

Re: Develocity

[Quan tran hong]

Hi Jean,

I tried your work locally and the build time is truly better.
`mvn clean install -DskipTests -Dmaven.skip.doc=true
-Dgradle.cache.local.enabled=true`

1st build:
[INFO] Total time:  22:18 min
[INFO] 8053 goals, 7996 executed, 57 from cache, saving at least 13s

2nd build:
[INFO] Total time:  15:07 min
[INFO] 8053 goals, 7296 executed, 757 from cache, saving at least 3m 11s

Thank you for the initiative :-)

Quan

Vào Th 3, 6 thg 2, 2024 vào lúc 14:48 Jean Helou  đã
viết:

> Hi devs,
>
> I realized this morning that I initiated the conversation on the wrong list
> (server-user instead of server-dev) ...
> see the archives
> https://lists.apache.org/list.html?server-u...@james.apache.org for Rene's
> positive answer
>
> sorry about that
>
> jean
>
> -- Forwarded message -
> From: Jean Helou 
> Date: Mon, Feb 5, 2024 at 3:52 PM
> Subject: Develocity
> To: James Users List 
>
>
> Hello fellow james devs,
>
> I'm not sure how much context to provide in the email you can read more in
>
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3978?filter=allopenissues
>
> TLDR; A few months ago the apache foundation announced a sponsorship of the
> "gradle" company for a tool to diagnose, monitor and optimize builds  (
> called gradle enterprise at the time and renamed develocity since)
>
> I posted a first PR to plug james build in that tool
> https://github.com/apache/james-project/pull/1972
>
> This first integration brings only build scans and local caching
> - build scans can help us understand what slows our build
> - local caching allows local build perf improvements (I posted a small
> benchmark which shows a divide by 3 the time spent building
> james-server-core in a repeat build with no changes)
>
> The build scans also show that many tasks in the build are not actually
> cached so there is room for more improvement.
>
> The local cache does apply to surefire tasks,unfortunately the benefits for
> CI are very small :
>
> In the first stage we cache the compilation result itself but that
> represents a very small part of the test stages (even if we had to compile
> everything from scratch that's barely 20 minutes out of the 3h30 of the
> build so caching only marginally improves the build time at the moment
>
> the next stage would be to :
> - make more tasks cacheable
> - enable the remote build cache
>
> The remote build cache allows to share caching accross builds which means
> that if a change affects only a small portion of the build, only that
> change would effectively be recompiled and retested, so the closer to the
> "leafs" of the reactor a change would be the shorter its CI pipelines would
> be.
> I pinged INFRA in https://issues.apache.org/jira/browse/INFRA-25461 to
> know
> what the process is for enabling the remote build cache and it seems it
> starts with a mailing list thread :D
>
> note that
> - only the CI would be allowed to write to the remote build cache
> - build caches both local and remote can be disabled with properties (IIRC
> there are pretty strict constraints on release processes in the apache
> foundation)
> - I think it is possible to have the remote build cache be activated in
> readonly mode for everyone  so that anyone pulling the repo would
> experience a very fast first build (but that remains to be confirmed as I
> have never experienced it)
>
> what do you think ?
>
> jean
>

Vote to merge the Java 21 adoption for Apache James PR

[Quan tran hong]

Hi everyone,

Following the Jira ticket

and the previous mailing list discussion

regarding Java 21 adoption for James, I and Benoit put together an effort
to successfully get a green build on the Java 21 adoption PR
.

The idea is to have James migrated to Java 21 with minimal changes first,
and after that, we can leverage new Java features gradually.

Because this is a big change for James, James's community review, remarks,
and opinions on merging the Java 21 PR should be needed.

It would be good if the community could spend some time reviewing the PR,
or just upvote/downvote directly on this mail thread.

Thanks for reading.

Best regards,

Quan

Re: Call for vote: Apache James 3.8.1

[Quan tran hong]

+1

Cheers,

Quan

Vào Th 3, 9 thg 1, 2024 vào lúc 15:21 btell...@linagora.com <
btell...@linagora.com> đã viết:

> +1
>
> On 09/01/2024 09:17, Benoit TELLIER wrote:
> > Hi,
> >
> > I would like to propose a new vote for 3.8.1 release of the Apache
> > James server.
> >
> > You can find:
> >
> >  - The maven release staged in repository.apache.org as the artifact
> > #1077:
> > https://repository.apache.org/content/repositories/orgapachejames-1077/
> >  - The changelog for 3.8.1:
> > https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >
> > This release mostly ships bug fixes.
> >
> > Voting rules:
> >  - This is a majority approval: this release may not be vetoed.
> >  - A quorum of 3 binding votes is required
> >  - The vote starts at Tuesday 9th of January 2024, 9am UTC+2
> >  - The vote ends at Tuesday 16th of January 2024, 9am UTC+2
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James 3.7.5

[Quan tran hong]

+1

Cheers,

Quan


Vào Th 3, 9 thg 1, 2024 vào lúc 15:20 btell...@linagora.com <
btell...@linagora.com> đã viết:

> +1
>
> On 09/01/2024 09:15, Benoit TELLIER wrote:
> > Hi,
> >
> > I would like to propose a new vote for 3.7.5 release of the Apache
> > James server.
> >
> > You can find:
> >
> >  - The maven release staged in repository.apache.org as the artifact
> > #1079:
> > https://repository.apache.org/content/repositories/orgapachejames-1079/
> >  - The changelog for 3.7.5:
> > https://github.com/apache/james-project/blob/master/CHANGELOG.md
> >
> > This release mostly ships bug fixes.
> >
> > Voting rules:
> >  - This is a majority approval: this release may not be vetoed.
> >  - A quorum of 3 binding votes is required
> >  - The vote starts at Tuesday 9th of January 2024, 9am UTC+2
> >  - The vote ends at Tuesday 16th of January 2024, 9am UTC+2
> >
> > You can answer to it just with +1 and -1. Down-votes may be motivated.
> >
> > 3 binding votes are expected move forward on this release.
> >
> > Cheers,
> >
> > Benoit TELLIER
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Call for vote: Apache James MIME4J 0.8.10

[Quan tran hong]

+1

Cheers,

Quan

Vào Th 2, 8 thg 1, 2024 vào lúc 21:36 Benoit TELLIER 
đã viết:

> Hi,
>
> I would like to propose a new vote for 0.8.10 release of the Apache
> MIME4J library.
>
> You can find:
>
>   - The maven release staged in repository.apache.org as the artifact
> #1076:
> https://repository.apache.org/content/repositories/orgapachejames-1076/
>   - The changelog:
> https://github.com/chibenwa/james-mime4j/blob/master/CHANGELOG.md
>
> This release ships mostly small enhancements.
>
> Voting rules:
>   - This is a majority approval: this release may not be vetoed.
>   - A quorum of 3 binding votes is required
>   - The vote starts at Monday 8th of January 2024, 3pm UTC+2
>   - The vote ends at Monday 15th of January 2024, 3pm UTC+2
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> 3 binding votes are expected move forward on this release.
>
> Cheers,
>
> Benoit TELLIER
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Building a PostgreSQL mailbox for James?

[Quan tran hong]

Hi everyone,

This is another follow-up to our recent work on the Postgres topic.

In the last sprint:
- We migrated fully JPA code to Postgres. No more JPA code in the
postgres-app now. So at this moment, we had a fully functional IMAP server
on top of Postgres.
- We solved the flags concurrently update issue and proposed an ADR
explaining the solution.
- We implemented a blob store backed by Postgres.
  However, it is not mature yet as we are having concurrent issues with
PostgresBlobStore when Postgres persists blobs slowly using just a single
connection. On that, we are working on a Postgres pool connection to
improve the performance.
- We succeeded in plugging RabbitMQ as the event bus, S3 as the blob store,
and OpenSearch into the postgres-app.
  We provide configurable module choosers so users are free to choose the
implementation. For example, one could select S3/File/Postgres as his blob
store implementation.

In the coming time, we plan to implement the JMAP storage backed by
Postgres. To have a look at the planned tickets, please visit
https://github.com/linagora/james-project/issues.

Your feedback would be appreciated.

Regards,

Quan

Vào Th 6, 15 thg 12, 2023 vào lúc 18:45 Quan tran hong <
quan.tranhong1...@gmail.com> đã viết:

> Hi everyone,
>
> Yet another following up on our recent Postgres work.
>
> In the last sprint, our team successfully migrated the mailbox core module
> `apache-james-mailbox-postgres` fully from JPA to Postgres. We did some
> performance tests with the IMAP server backed by Postgres-app and the
> result was very impressive: postgres-app outperformed JPA about 20 times
> (at least look at the 99 percentile) with even less resource usage (CPU and
> memory) compared to the jpa-app. For more details:
> https://github.com/linagora/james-project/issues/4936
>
> We are working on an ADR to explain our schema decision on the core tables
> like mailbox and message tables, and also solving the flags concurrently
> update issue.
>
> In the coming time, we plan to:
> - Migrate other remaining minor components from JPA code to Postgres.
> - Plug the RabbitMQ, S3, OpenSearch into the Postgres-app.
>   On that, we would try to implement alternative implementations with
> Postgres e.g. BlobStore backed by Postgres. We would provide configurable
> module choosers so users are free to choose the implementation.
>
> Best regards,
>
> Quan
>
> Vào Th 2, 27 thg 11, 2023 vào lúc 19:55 Benoit TELLIER <
> btell...@apache.org> đã viết:
>
>> Thanks for the write up Quan!
>>
>> I would add that we shall rights ADRs about important decisions on the
>> PostgreSQL implementation.
>>
>> So far I identified...
>>
>>   - table structure for the postgresql mailbox
>>   - how deletes are handled
>>   - concurrency control for flag updates
>>
>> Those ADR would be opened on the postgresql branch and eventually make
>> it to the James master branch.
>>
>> Best regards,
>>
>> Benoit TELLIER
>>
>> On 27/11/2023 12:47, Quan tran hong wrote:
>> > Hi everyone,
>> >
>> > This is a follow-up on the Postgres implementation topic. We have been
>> > working on this topic for a while and your feedback and expertise about
>> > Postgres would be helpful :-)
>> >
>> > For short:
>> > - On the progress: We have implemented the SubscriptionMapper,
>> > UsersRepository, DomainList, and MailboxMapper. We are working on
>> replacing
>> > MessageMapper (a big one) and other components. We planned to have a
>> > fully functional Postgres mailbox before moving to other blocks like
>> object
>> > storage, search, JMAP support... Of course, we are hungry to have a
>> > performance test for the Postgres mailbox, but it is too early ATM while
>> > the core components are not ready.
>> > - We use jOOQ library + r2dbc-postgresql so far, and the development
>> > experience is overall good. The jOOQ documentation and examples are not
>> > always ready TBH, but by doing a bit of research and ChatGPT, we
>> managed to
>> > get jOOQ work so far.
>> > - Sometimes Guice binding for Postgres code is not easy because we are
>> > mixing up the JPA and Postgres code for now, which leads to some
>> dependent
>> > issues.
>> >
>> > So far developing James, at least for me, I experienced mostly NoSQL and
>> > not SQL. So your SQL expertise would definitely help us with its best
>> > practices.
>> > For more details on what we did and plan to do on the Postgres topic,
>> you
>> > can kindly review them at:
>> >
>> https://github.com/li

Re: Building a PostgreSQL mailbox for James?

[Quan tran hong]

Hi everyone,

Yet another following up on our recent Postgres work.

In the last sprint, our team successfully migrated the mailbox core module
`apache-james-mailbox-postgres` fully from JPA to Postgres. We did some
performance tests with the IMAP server backed by Postgres-app and the
result was very impressive: postgres-app outperformed JPA about 20 times
(at least look at the 99 percentile) with even less resource usage (CPU and
memory) compared to the jpa-app. For more details:
https://github.com/linagora/james-project/issues/4936

We are working on an ADR to explain our schema decision on the core tables
like mailbox and message tables, and also solving the flags concurrently
update issue.

In the coming time, we plan to:
- Migrate other remaining minor components from JPA code to Postgres.
- Plug the RabbitMQ, S3, OpenSearch into the Postgres-app.
  On that, we would try to implement alternative implementations with
Postgres e.g. BlobStore backed by Postgres. We would provide configurable
module choosers so users are free to choose the implementation.

Best regards,

Quan

Vào Th 2, 27 thg 11, 2023 vào lúc 19:55 Benoit TELLIER <
btell...@apache.org> đã viết:

> Thanks for the write up Quan!
>
> I would add that we shall rights ADRs about important decisions on the
> PostgreSQL implementation.
>
> So far I identified...
>
>   - table structure for the postgresql mailbox
>   - how deletes are handled
>   - concurrency control for flag updates
>
> Those ADR would be opened on the postgresql branch and eventually make
> it to the James master branch.
>
> Best regards,
>
> Benoit TELLIER
>
> On 27/11/2023 12:47, Quan tran hong wrote:
> > Hi everyone,
> >
> > This is a follow-up on the Postgres implementation topic. We have been
> > working on this topic for a while and your feedback and expertise about
> > Postgres would be helpful :-)
> >
> > For short:
> > - On the progress: We have implemented the SubscriptionMapper,
> > UsersRepository, DomainList, and MailboxMapper. We are working on
> replacing
> > MessageMapper (a big one) and other components. We planned to have a
> > fully functional Postgres mailbox before moving to other blocks like
> object
> > storage, search, JMAP support... Of course, we are hungry to have a
> > performance test for the Postgres mailbox, but it is too early ATM while
> > the core components are not ready.
> > - We use jOOQ library + r2dbc-postgresql so far, and the development
> > experience is overall good. The jOOQ documentation and examples are not
> > always ready TBH, but by doing a bit of research and ChatGPT, we managed
> to
> > get jOOQ work so far.
> > - Sometimes Guice binding for Postgres code is not easy because we are
> > mixing up the JPA and Postgres code for now, which leads to some
> dependent
> > issues.
> >
> > So far developing James, at least for me, I experienced mostly NoSQL and
> > not SQL. So your SQL expertise would definitely help us with its best
> > practices.
> > For more details on what we did and plan to do on the Postgres topic, you
> > can kindly review them at:
> >
> https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql
> >
> > Best regards,
> >
> > Quan
> >
> >
> > Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler <
> > matth...@baechler-craftsmanship.fr> đã viết:
> >
> >> Hi Benoit,
> >>
> >> This topic has been discussed for years, I'm happy you finally draw a
> plan
> >> for it.
> >>
> >> To me, the aim for Postgres is small to middle size deployment with
> >> minimal dependencies.
> >>
> >> In that regard, having an implementation that spans across all
> >> infrastructure needs is a must have.
> >>
> >> So my take would be let's implement everything with PG: blob storage,
> >> search, messaging and various data storage like Event Sourcing and plain
> >> data.
> >>
> >> For a user, it will always be possible to plug another piece of
> >> infrastructure if need be (like having better search or store more
> blobs,
> >> etc).
> >>
> >> The only nice-to-have to me would be the multi-tenant goal as you can
> >> always spawn another James instance by domain (and you can use the same
> PG
> >> if you want by using several databases).
> >>
> >> To answer the last questions: I would definitely be interested in using
> >> this implementation (I use JPA for now). I could marginally contribute
> to
> >> it as I have experience with PG but my time is very l

Re: Adding support for Apple proprietary IMAP push notifications

[Quan tran hong]

Hello Benoit,

> JIRA please!

Yes, it should be JIRA: https://issues.apache.org/jira/projects/JAMES/issues
.

> That's a classic trick to have optional mavven dependencies and address
several architectures at once ;-)

This is truly a trick that would help write the extension more simple while
using the extension modular also. Thank you for adding up for me!

Quan


Vào Th 6, 8 thg 12, 2023 vào lúc 17:18 Benoit TELLIER 
đã viết:

> Hello,
>
>  > That is basically my idea on the topic. You can create a ticket at
> https://github.com/linagora/james-project/issues where we can discuss
> further details on implementing this if you want.
>
> JIRA please!
>
> https://issues.apache.org/jira/projects/JAMES/issues
>
>  > Also please do not hesitate to open a draft pull request so we can
> give you soon feedback.
>
> +1
>
> On a technical standpoint I mostly agree with pointers for Quan.
>
> Thinking more about it, we likely can make only a single extension with
> all storage implementations. And rely on advanced maven tricks for
> having just the right dependencies, while exposing a single maven module.
>
> The idea would be to, within a single APSN extension:
>
>   - Define a storage interface
>   - Define a standard junit 5 contract (with an interface
> APSNStoreContract)
>   - Implement a memory implementation of it (class MemoryAPSNStore
> implementd APSNStore) which passes the test (class MemoryAPSNStoreTest
> implements APSNStoreContract)
>   - Add provided dependency for cassandra, jpa storage backend. This
> mean we can compile with them but runtime deps will be provided by James
> server.
>   - We can then implement one JPAAPSNStore, CassandraAPSNStore and
> related tests
>   - We can write a guice module for each APSNSotre:
> MemoryAPSNStoreModule, JPAAPSNStoreModule, CassandraAPSNStoreModule
>
> Configuration of the storage backend can then be done by choosing the
> corresponding guice module within extensions.properties.
>
> That's a classic trick to have optional mavven dependencies and address
> several architectures at once ;-)
>
> Thoughts Quan?
>
> Best regards,
>
> On 08/12/2023 11:09, Quan tran hong wrote:
> > Hello Andre Borie,
> >
> > Your PoC work is very cool. I am happy that someone out there succeeded
> in
> > using our IMAP command extension mechanism.
> >
> >> however I have a question as to which pattern the storage & APNS
> > long-lived client should conform to. The IMAP extension & mailbox
> listeners
> > (which James has established patterns for) are only *part* of the entire
> > solution.
> >
> > FYI, we (Linagora) developed the JMAP Firebase Cloud Messaging extension.
> > The idea is basically as same as yours:
> > - Exposing application API so users can register their notifications.
> >In our case, it was JMAP FirebaseSubscription extension (
> >
> https://github.com/linagora/tmail-backend/blob/master/docs/modules/ROOT/pages/tmail-backend/jmap-extensions/pushWithFirebase.adoc
> ).
> > In your case it was the IMAP command extension.
> >For storing and managing the subscriptions, we wrote an interface for
> the
> > storage API called `FirebaseSubscriptionRepository` (
> >
> https://github.com/linagora/tmail-backend/blob/master/tmail-backend/jmap/extensions/src/main/java/com/linagora/tmail/james/jmap/firebase/FirebaseSubscriptionRepository.java
> ).
> > The idea is that we can have multiple implementations for the storage API
> > e.g. memory one (mostly for test), Cassandra one (using Cassandra
> > database), Postgres one (using Postgres database)...
> > - Pushing the users's data change to the Push gateway using a listener.
> >In our case it was `FirebasePushListener` (
> >
> https://github.com/linagora/tmail-backend/blob/master/tmail-backend/jmap/extensions/src/main/scala/com/linagora/tmail/james/jmap/firebase/FirebasePushListener.scala#L30
> ),
> > like your listener.
> >
> > I think your PoC is already good enough, the story is just about how to
> > contribute it to James in a proper way.
> >
> > As Benoit suggested, you can contribute your extension into a new module
> > let's say `apsn-imap`:
> >
> https://github.com/apache/james-project/tree/master/third-party/apsn-imap.
> > For the storage API, you can write a simple Java interface e.g.
> > `ApnsTokenRepository` in a child module:
> >
> https://github.com/apache/james-project/tree/master/third-party/apsn-imap/token-storage-api
> > Then you can implement a simple implementation for the storage using
> memory
> > in another child module:
> >
> https://github.com/apache/james-

Re: Adding support for Apple proprietary IMAP push notifications

[Quan tran hong]

Hello Andre Borie,

Your PoC work is very cool. I am happy that someone out there succeeded in
using our IMAP command extension mechanism.

> however I have a question as to which pattern the storage & APNS
long-lived client should conform to. The IMAP extension & mailbox listeners
(which James has established patterns for) are only *part* of the entire
solution.

FYI, we (Linagora) developed the JMAP Firebase Cloud Messaging extension.
The idea is basically as same as yours:
- Exposing application API so users can register their notifications.
  In our case, it was JMAP FirebaseSubscription extension (
https://github.com/linagora/tmail-backend/blob/master/docs/modules/ROOT/pages/tmail-backend/jmap-extensions/pushWithFirebase.adoc).
In your case it was the IMAP command extension.
  For storing and managing the subscriptions, we wrote an interface for the
storage API called `FirebaseSubscriptionRepository` (
https://github.com/linagora/tmail-backend/blob/master/tmail-backend/jmap/extensions/src/main/java/com/linagora/tmail/james/jmap/firebase/FirebaseSubscriptionRepository.java).
The idea is that we can have multiple implementations for the storage API
e.g. memory one (mostly for test), Cassandra one (using Cassandra
database), Postgres one (using Postgres database)...
- Pushing the users's data change to the Push gateway using a listener.
  In our case it was `FirebasePushListener` (
https://github.com/linagora/tmail-backend/blob/master/tmail-backend/jmap/extensions/src/main/scala/com/linagora/tmail/james/jmap/firebase/FirebasePushListener.scala#L30),
like your listener.

I think your PoC is already good enough, the story is just about how to
contribute it to James in a proper way.

As Benoit suggested, you can contribute your extension into a new module
let's say `apsn-imap`:
https://github.com/apache/james-project/tree/master/third-party/apsn-imap.
For the storage API, you can write a simple Java interface e.g.
`ApnsTokenRepository` in a child module:
https://github.com/apache/james-project/tree/master/third-party/apsn-imap/token-storage-api
Then you can implement a simple implementation for the storage using memory
in another child module:
https://github.com/apache/james-project/tree/master/third-party/apsn-imap/token-storage-memory.
Or whatever implementation you like IMO, other people who want to have a
Cassandra or Postgres implementation can just base on your
`ApnsTokenRepository` API.
After having the storage API, you can integrate your PushClient and the
listener.

That is basically my idea on the topic. You can create a ticket at
https://github.com/linagora/james-project/issues
where we can discuss further details on implementing this if you want. Also
please do not hesitate to open a draft pull request so we can give you soon
feedback.

Thanks for your enthusiasm.

Regards,

Quan






Vào Th 6, 8 thg 12, 2023 vào lúc 08:47 Andre Borie 
đã viết:

> Hello,
>
> > is there a public document specifying this
>
> No explicit public specification; the closest we have are patches
> published against Dovecot (which was part of macOS Server, their
> discontinued server product).
>
> Other open-source mail server projects have used that as a specification
> to develop their own solutions:
>
> * https://github.com/freswa/dovecot-xaps-plugin
> *
> https://github.com/cyrusimap/cyrus-imapd/commit/a4470bf0f79de49030b3d66d2b385a1345c1a040
>
> > we added exactly one year ago support for IMAP extension
>
> As you saw previously I am already doing this in my proof-of-concept,
> however I have a question as to which pattern the storage & APNS long-lived
> client should conform to. The IMAP extension & mailbox listeners (which
> James has established patterns for) are only *part* of the entire solution.
>
> I am very new to this project (and Java in general) so I would appreciate
> if someone with more experience could stub out a general layout of the
> classes I should implement and which base classes to inherit from for the
> storage & long-lived APNS client. I will check out that Firebase extension
> you linked but I'd appreciate any other help/opinions.
>
>
> > all code landing in ASF james needs t be ASFV2
>
> I'm open to licensing this under whatever license - the PoC I linked to
> previously is licensed under whatever James is, I've copied the LICENSE
> file verbatim.
>
> > we can not rely on a proprietary custom external APSN bridge (external
> bridge is OK to me though, if FOSS and easy to run...).
>
> T bridge I used in my PoC is FOSS (MIT-licensed), however I am looking to
> remove the dependency on it as I find its data storage model inadequate and
> unable to support mailbox delegation, therefore this mailing list thread on
> how to reimplement the full functionality within James itself.
>
> Regards.
>
>
> > On 7 Dec 2023, at 20:13, Benoit TELLIER  wrote:
> >
> > Hello Andre.
> >
> > Thanks for letting me discover about this Apple feature. Out of
> curiosity, is there a public document specifying 

Re: Building a PostgreSQL mailbox for James?

[Quan tran hong]

Hi everyone,

This is a follow-up on the Postgres implementation topic. We have been
working on this topic for a while and your feedback and expertise about
Postgres would be helpful :-)

For short:
- On the progress: We have implemented the SubscriptionMapper,
UsersRepository, DomainList, and MailboxMapper. We are working on replacing
MessageMapper (a big one) and other components. We planned to have a
fully functional Postgres mailbox before moving to other blocks like object
storage, search, JMAP support... Of course, we are hungry to have a
performance test for the Postgres mailbox, but it is too early ATM while
the core components are not ready.
- We use jOOQ library + r2dbc-postgresql so far, and the development
experience is overall good. The jOOQ documentation and examples are not
always ready TBH, but by doing a bit of research and ChatGPT, we managed to
get jOOQ work so far.
- Sometimes Guice binding for Postgres code is not easy because we are
mixing up the JPA and Postgres code for now, which leads to some dependent
issues.

So far developing James, at least for me, I experienced mostly NoSQL and
not SQL. So your SQL expertise would definitely help us with its best
practices.
For more details on what we did and plan to do on the Postgres topic, you
can kindly review them at:
https://github.com/linagora/james-project/issues?q=is%3Aopen+is%3Aissue+label%3APostgresql

Best regards,

Quan


Vào Th 6, 13 thg 10, 2023 vào lúc 16:32 Matthieu Baechler <
matth...@baechler-craftsmanship.fr> đã viết:

> Hi Benoit,
>
> This topic has been discussed for years, I'm happy you finally draw a plan
> for it.
>
> To me, the aim for Postgres is small to middle size deployment with
> minimal dependencies.
>
> In that regard, having an implementation that spans across all
> infrastructure needs is a must have.
>
> So my take would be let's implement everything with PG: blob storage,
> search, messaging and various data storage like Event Sourcing and plain
> data.
>
> For a user, it will always be possible to plug another piece of
> infrastructure if need be (like having better search or store more blobs,
> etc).
>
> The only nice-to-have to me would be the multi-tenant goal as you can
> always spawn another James instance by domain (and you can use the same PG
> if you want by using several databases).
>
> To answer the last questions: I would definitely be interested in using
> this implementation (I use JPA for now). I could marginally contribute to
> it as I have experience with PG but my time is very limited (unless someone
> wants to sponsor my work, of course). I can donate some code related to
> Event Sourcing has I have an implementation of an Event Store on top of PG
> and some code around messaging. Let me know if you are interested in that
> contributation.
>
> In term of strategy, I think that would help James gain popularity among
> hobbyist and small businesses, so I think it worth trying.
>
> Cheers,
>
> -- Matthieu Baechler
>
> --- Original Message ---
> On Friday, October 6th, 2023 at 23:48, Benoit TELLIER <
> btell...@linagora.com> wrote:
>
>
> >
> >
> > Hey there!
> >
> > The goal: deliver James "stateless email server" concept to smaller
> deployments than those addressable with the Distributed server.
> >
> > Why Postgres? Rock solid. And more options than other SQL stores (see
> below)
> >
> > The requirements would be:
> > - Leverage the blobStore for binary storage (email bodies +
> attachements). Those big binaries are not meant to be stored into SQL rows
> - blaming you, JPA!
> > - Bring choice on blob store : PGSQL native solution (
> https://www.postgresql.org/docs/7.4/jdbc-binary-data.html ) for small
> deployments OR S3
> > - Bring choice on search: PGSQL native solution (
> https://www.postgresql.org/docs/current/textsearch.html ) for small
> deployments OR OpenSearch
> > - Bring choice on PubSub: PGSQL native solution (
> https://www.postgresql.org/docs/current/plpgsql-trigger.html ) OR RabbitMQ
> > - Enforce strict tenant isolation: domain A won't access domain B data
> even if we screw up James access control layer. This can be done with Row
> security https://www.postgresql.org/docs/current/ddl-rowsecurity.html .
> > - Be reactive. This can be achieved by using a reactive firendly driver
> like r2dbc...
> > - Ensure that we can easily run on some largely scaling postgres...
> CitusData ?
> >
> > An other outcome might be to drop JPA implementation, ideally... (we
> provide something similar but wy better)
> >
> > Ideally I would like to deliver this before september 2024...
> >
> > Thoughts?
> > Would this be something interesting people in here?
> > Would some people be interested contributing to this effort?
> > Would some people desire sponsoring this effort?
> >
> > If this is non consensual, I can also contribute this into
> https://github.com/linagora/tmail-backend/ without annoying people in
> here...
> >
> >
> >
> > --
> >
> > Best regards,
> >
> > Benoit 

Re: Proposal about deprecation and removal

[Quan tran hong]

Hi,

+1 on the Spring deprecation with caution.

Regards,

Quan

Vào Th 2, 27 thg 11, 2023 vào lúc 14:30 Benoit TELLIER <
btell...@apache.org> đã viết:

> +1 on my side.
>
> I agree deprecating Spring implementation in James as of 3.9.0 and drop
> it afterward.
>
> On 24/11/2023 17:30, Matthieu Baechler wrote:
> > Hi,
> >
> > I had a hack session with Benoit today and the sentence "this would
> break Spring" came many times along the day.
> >
> > As I'm less active on James than I used to be, I must admin I have no
> idea how popular the Spring version of James is nowadays.
> >
> > However, what strikes me when I hack on James is how the size of the
> project and its legacy makes it so slow to make progress.
> >
> > We did some deprecation and removal in the past but we have been
> conservative about that.
> >
> > I would like to argue that being conservative to preserve existing users
> may actually prevents from attracting new ones. Moreover, it probably also
> prevents new developers to involve as they are quickly overwhelmed.
> >
> > So, what would you think about removing more aggressively features and
> modules, starting with the Spring support?
> >
> > Cheers,
> >
> > -- Matthieu Baechler
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Fwd: Introduction from a new member of Apache James Community

[Quan tran hong]

Hi folks,
For some unknown reason, Thanh (vt...@lingora.com) can not send his
self-introduction mail to our mailing list, so I would like to forward the
mail for him.

-- Forwarded message -
Hi,

Let me introduce myself first. My name is Thanh. I'm from Vietnam and I'm a
third-year student at Hanoi University of Science and Technology. I major
in Information and Communication of Technology and now I'm working for
Linagora Vietnam company . My company has many open-source projects such
as: Tmail, Twake, Linshare, ...About Java team, we manage and develop a
branch of Apache James (https://github.com/apache/james-project
)

As a trainee at Java team, my work is implementing delayed send emails for
SMTP and JMAP using the FutureRelease extension.

I would love to participate in Apache James development, share my work and
contribute to the community.

Thanh.

Re: Code coverage for Apache James

[Quan tran hong]

Hi Benoit,

Thanks for your feedback.

I made some more effort regarding your feedback. My answers are inlined.

> What is the impact on the build time?

There is only 1 successful build on the PR. Its time is 4h50m which equals
the longest recently build on the master branch. However, 1 sample can not
say much about the impact IMO.

> A quick review of your work on github led me to think that code coverage
(especially the scala one) is under-evaluated. CF
http://timezra.blogspot.com/2013/10/jacoco-and-scala.html

It does not work sadly. The plugin is too old (10 years old) and based on
the legacy Jacoco version 0.6.3 (the latest version is 0.8.9). The most
important issue is that its repository artifact no longer exists.
I see a PR trying to upgrade it to a newer version, but it has been stale
for a few years.
Actually, this plugin is a custom version of Jacoco that exposes some Scala
filters (for better Scala coverage results) that are available on Jacoco.
However, I do not see how to explicitly specify the filters:
https://github.com/jacoco/jacoco/wiki/FilteringOptions

> Would there be a way to connect it to github (code coverage badge, havea
comment regarding code coverage)

Jacoco has integration with some code coverage registries
e.g Coveralls, Codecov... However, the biggest issue is managing the secret
token to update the coverage result. If we manage our own CI for James, it
would be easy to set a secret env on the CI. But it is not the case with
Apache CI which we do not have much control.

Quan

Vào Th 6, 21 thg 4, 2023 vào lúc 15:22 Benoit TELLIER 
đã viết:

> Hello Quan,
>
> Thanks for putting this effort together.
>
> This also gives us a glance at one quality metric. Your report shows a
> code coverage of 72%.
>
>   -> What is the impact on the build time?
>
>   -> A quick review of your work on github led me to think that code
> coverage (especially the scala one) is under-evaluated.
>
> CF http://timezra.blogspot.com/2013/10/jacoco-and-scala.html
>
> By taking into account that scala code coverage is under-evaluated it is
> reasonable to think that Apache James code coverage is more in the
> 80-85% range. Not bad!
>
>   -> How accessible is the report?
>
> I see that it is accessible as an archive on Jenkins, as a build artifact.
>
> Would there be a way to connect it to github (code coverage badge, have
> a comment regarding code coverage)
>
> Best regards,
>
> Benoit
>
> On 21/04/2023 09:56, Quan tran hong wrote:
> > Hi folks,
> >
> > I think the code coverage report for James would be a helpful factor to
> > acknowledge how good enough of the tests have been written and will be
> > written.
> >
> > Therefore I am experimenting and working on a POC for this adoption. The
> > first goal is to generate an aggregate report for James modules and then
> > based on the result we can observe and discuss more about actions to take
> > e.g: write more tests where are needed and helpful, or even set a code
> > coverage level for some specific modules.
> >
> > Jira ticket:
> >
> https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3903?filter=allissues
> > PR: https://github.com/apache/james-project/pull/1522
> >
> > Feedback and opinions are very welcome on this topic.
> >
> > Best regards,
> >
> > Quan
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Code coverage for Apache James

[Quan tran hong]

Hi folks,

I think the code coverage report for James would be a helpful factor to
acknowledge how good enough of the tests have been written and will be
written.

Therefore I am experimenting and working on a POC for this adoption. The
first goal is to generate an aggregate report for James modules and then
based on the result we can observe and discuss more about actions to take
e.g: write more tests where are needed and helpful, or even set a code
coverage level for some specific modules.

Jira ticket:
https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3903?filter=allissues
PR: https://github.com/apache/james-project/pull/1522

Feedback and opinions are very welcome on this topic.

Best regards,

Quan

Re: Call for vote: Apache James 3.7.2

[Quan tran hong]

Hi Benoit,

+1 for the release.

Cheers,

Quan

Vào Th 6, 7 thg 10, 2022 vào lúc 14:26 Benoit TELLIER 
đã viết:

> Hi,
>
> I would like to propose a new vote for 3.7.2 release of the Apache James
> server.
>
> You can find:
>
>   - The maven release staged in repository.apache.org as the artifact
> #1066:
> https://repository.apache.org/content/repositories/orgapachejames-1066/
>   - The communication material for release 3.7.2:
> https://github.com/apache/james-project/pull/1234
>
> This release brings various fixes and enhancements. Some security
> related dependency upgrades where done.
>
> Voting rules:
>   - This is a majority approval: this release may not be vetoed.
>   - A quorum of 3 binding votes is required
>   - The vote starts at Friday 7th of October 2022, 2pm30 UTC+7
>   - The vote ends at Friday 14th of October 2022, 2pm30 UTC+7
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> 3 binding votes are expected move forward on this release.
>
> Cheers,
>
> Benoit TELLIER
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Session refactoring discussion

[Quan tran hong]

Hi Benoit,

First of all, thank you for your feedback.

I would prefer a `createUnrestrictedSession`.
> While speaking about names "System types" does not sound relevant
> either. Maybe "component" ?


I agree.

How this change would interact with maibox event serialization and the
> event bus?


 Do you mean the case that a component triggers a mailbox event? Is
Username.of(component class) good enough to you? Otherwise, we would need
to rethink the username/component in maibox event serialization' structure.

Regards,

Quan

Vào Th 5, 4 thg 8, 2022 vào lúc 17:10 Benoit TELLIER 
đã viết:

> +1 for option 1
>
> More details inlined.
>
> How this change would interact with maibox event serialization and the
> event bus?
>
> Regards,
>
> Benoit
>
> On 04/08/2022 10:20, Quan tran hong wrote:
> > Hi folks,
> >
> > Recently when I working with the RSpamD module, I found it is hard to
> > access a message with just a messageId from a system aspect (without a
> > dedicated username context).
> > I have a look again at our MailboxSession code and also the same topic
> > which Rene raised a long time ago (
> > https://lists.apache.org/thread/chmylvglv3mqnmrgwyg49qwmbc5nxfff), I
> > realized that our System session type does not have a really different
> > meaning from normal User session type.
> > I believe that I am not the only one having this issue until now but this
> > is an issue that stayed low in our code base with some disadvantages:
> > - It is hard to implement say curl -XDELETE /msgs/:messageId (without
> > username context).
> > - We are not able to distinguish (eg in logs) between user actions and
> > stuff triggered via a web admin task. Only using users brings in
> > traceability issues...
> >
> > So I decided to raise this topic again to solve this issue one and for
> all.
> > Here is my idea on how to solve this (get inspiration from Rene's old PR:
> > https://github.com/linagora/james-project/pull/3038):
> > - Refactor `createSystemSession` -> `createUserSession` (quickly change
> > system -> user session in code)
> > - Create `createSystemSession` which creates a session with a meaningful
> > System type
> Using the exact same naming is confusing.
>
> Also, if I use `createSystemSession` in an extension, then before that
> change rights are scoped to the user, after that is unrestricted which
> could be a security issue.
>
> I would prefer a `createUnrestrictedSession`.
>
> While speaking about names "System types" does not sound relevant
> either. Maybe "component" ?
> > - Adding tests to make sure System type session have the full right
> access
> > to arbitrary user resources (rename mailbox, delete mailbox, create
> > mailbox, copy message, setFlags, delete message, move message, read
> > message, search mailbox, search message,...).
> >
> > The scope of `TypeSession` refactoring is my concern with the following
> > options:
> > - Option 1: 2 types of session (User/System): easier to archive our
> needs.
> > - Option 2: 4 types of session proposed in the old ML thread
> > (User/ImpersonatedUser/DelegatedUser/System): more comprehensive and good
> > for audit actions (by who) but come with bigger refactoring effort
> > (action's actor information refactoring).
> >
> > I propose we choose option 1 first to fastly archive our need and spend
> > more effort with option 2 later.
> >
> > Here is my proposal on this topic. What do you think? Feedback is welcome
> > :')
> >
> > Best regards,
> >
> > Quan
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Session refactoring discussion

[Quan tran hong]

Hi folks,

Recently when I working with the RSpamD module, I found it is hard to
access a message with just a messageId from a system aspect (without a
dedicated username context).
I have a look again at our MailboxSession code and also the same topic
which Rene raised a long time ago (
https://lists.apache.org/thread/chmylvglv3mqnmrgwyg49qwmbc5nxfff), I
realized that our System session type does not have a really different
meaning from normal User session type.
I believe that I am not the only one having this issue until now but this
is an issue that stayed low in our code base with some disadvantages:
- It is hard to implement say curl -XDELETE /msgs/:messageId (without
username context).
- We are not able to distinguish (eg in logs) between user actions and
stuff triggered via a web admin task. Only using users brings in
traceability issues...

So I decided to raise this topic again to solve this issue one and for all.
Here is my idea on how to solve this (get inspiration from Rene's old PR:
https://github.com/linagora/james-project/pull/3038):
- Refactor `createSystemSession` -> `createUserSession` (quickly change
system -> user session in code)
- Create `createSystemSession` which creates a session with a meaningful
System type
- Adding tests to make sure System type session have the full right access
to arbitrary user resources (rename mailbox, delete mailbox, create
mailbox, copy message, setFlags, delete message, move message, read
message, search mailbox, search message,...).

The scope of `TypeSession` refactoring is my concern with the following
options:
- Option 1: 2 types of session (User/System): easier to archive our needs.
- Option 2: 4 types of session proposed in the old ML thread
(User/ImpersonatedUser/DelegatedUser/System): more comprehensive and good
for audit actions (by who) but come with bigger refactoring effort
(action's actor information refactoring).

I propose we choose option 1 first to fastly archive our need and spend
more effort with option 2 later.

Here is my proposal on this topic. What do you think? Feedback is welcome
:')

Best regards,

Quan

Accomplish remaining James Webadmin CLI command groups - Hacktoberfest 2021

[Quan tran hong]

Hi everyone,

In order to ease the administrator, James servers offer a command-line
interface in order to interact with the server. However, it relies on the
JMX protocol, which is known to be insecure. The mission is to port the
command-line interface in order to rely on the James web admin REST APIs.

There has been some work about this:
https://github.com/apache/james-project/tree/master/server/apps/webadmin-cli.
Some command groups have been finished: domain, mailbox, global quota, user.

But there are some command groups that have not been worked on yet. Please
have a look at: https://james.apache.org/server/manage-webadmin.html.

I created a dedicated ticket for this:
https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3663

We (linagora) decided to take part in Hacktoberfest 2021 and this ticket is
one of our potential topics (of course you can propose topics that you are
interested in). About this ticket, we created 2 issues so you can start
more easily here:
- JAMES-3663 Update James webadmin-cli documentation

- JAMES-3663 Accomplish remaining James Webadmin CLI command groups


We know we join Hacktoberfest this year a little late, so we pick an easy
topic with carefully created issues so you can start quickly. Don't forget
to join and finish 4 Pull Requests so you can receive a T-shirt
from Hacktoberfest. You can more easily accomplish this by starting doing
one CLI command per PR.

If you have any questions, let's discuss them!

Re: About being a more regular member of James community

[Quan tran hong]

Hi Benoit,
It's great to hear from you. I will do as best as I can to contribute to
James.
I hope to get your support in the future.
Quan

Vào Th 2, 18 thg 10, 2021 vào lúc 14:57 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello Quan,
>
> I am personally very happy to have you back, and am looking forward
> seeing your upcoming contributions!
>
> Best regards,
>
> Benoit TELLIER
>
> On 18/10/2021 11:47, Quan tran hong wrote:
> > Hi everyone,
> > My name is Quan. Some of you may already know me through my discussions
> > about the Google Summer of Code topic - Implement Thread support for
> James.
> > Today is my first day as a regular employee at James team - Linagora
> > company. It's a great opportunity for me to contribute more to James
> > project and engage more with the community. I hope to have more
> interesting
> > discussions with you in the near future.
> > Have a good day!
> > Best regards,
> > Quan
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

About being a more regular member of James community

[Quan tran hong]

Hi everyone,
My name is Quan. Some of you may already know me through my discussions
about the Google Summer of Code topic - Implement Thread support for James.
Today is my first day as a regular employee at James team - Linagora
company. It's a great opportunity for me to contribute more to James
project and engage more with the community. I hope to have more interesting
discussions with you in the near future.
Have a good day!
Best regards,
Quan

Re: Implement Thread/changes for JMAP

[Quan tran hong]

Hi Benoit,
Today I did try to think of a more optimized way. I did have some ideas but
it comes with some challenges.

At first, I have a look at EmailChanges to see if I can reuse this for
thread changes. The good thing is EmailChanges be produced after Added and
Expunged which contains MessageMetaData(have threadId). But the first
problem is EmailChanges can be produced also from other events like
FlagsUpdated which do not change thread and the second problem here
a create/update/destroy messageId is not corresponding with a thread event.
So I think we still need a separate ThreadChanges.

My idea is that we need a 1 thread - n messageId table (
ThreadIdToMessageIdTable) . We need to handle thread changes when Added,
Expunged and MailboxDeletion events happen.
*Added*: Before emit Added event, we insert data to this table.
MailboxChangeListener will listen to Added, then get a Set from
this. We need to qualify which threadId is created, which is updated by
doing a query to ThreadIdToMessageIdTable that ask the records number of
that thread:
 - If number = 1 => created
 - If number > 1 => updated
This way should work well with an Added event for each message. I see Added
event emitted when adding a message, but I doubt why in Added structure we
use Map and Collection? If Added event comes with multiple
messages, the above algorithm will not work (in case there are 2 messages
both come in a new thread then that thread could be counted as updated).

*Expunged*: The idea is to query to ThreadIdToMessageIdTable and ask
distinct threadId:
 - If this threadId is not in the result set => destroyed
 - If this threadId is in result set => updated
The problem is we need to delete data in ThreadIdToMessageIdTable before
doing the query. It should happen in DeleteMessageListener, but I think
DeleteMessageListener will not really happen before MailboxChangeListener.
The other choice is to delete that data in MailboxChangeListener, but I am
not sure this is a good idea.

*MailboxDeletion*: Before deletion as above, we need to get all messageId
in that mailbox => get a corresponding Set. Then we do delete
data in ThreadIdToMessageIdTable following MailboxDeletion.
The idea is to query to ThreadIdToMessageIdTable and ask for distinct
threadId:
 - If this threadId is not in the result set => destroyed
 - If this threadId is in result set => updated

In conclusion, this idea may work but comes with some problems.

How do you think about this?

Best regards,
Quan



Vào Th 3, 3 thg 8, 2021 vào lúc 17:17 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello Quan,
>
> Your design proposal would definitly work and I would qualify it of a
> straightforward solution.
>
> With my "efficiency" hat on, I am not really fan of having new events.
>
> I was really surprise to discover that JMAP submissions (draft protocol)
> of a single message did result in the submission of 10 events and had a
> major on overall performance.
>
> My mindset today would more be to 'shrink' that count of events rather
> than increasing them.
>
> Don't we have existing data structures that could be reused as is, or
> slightly modified, to answer the following questions?
>  - A thread is created?
>  - A thread is updated?
>
> Cheers,
>
> Benoit
>
> On 03/08/2021 16:31, Quan tran hong wrote:
> > Hello,
> >
> > I am starting working on the Thread/changes topic, and I think I should
> > discuss how to implement this topic with you.
> >
> > After looking at Thread/changes specs, I think we should add thread
> > event-oriented support to our Event Bus.
> >
> > I propose 3 events: Thread create, thread update and thread destroy.
> These
> > events should store which data is still a topic, but at least they need
> to
> > store changed threadId.
> >
> > A thread create can be easily emitted when we decide a message should
> have
> > a new threadId. But we need to distinguish when thread update and thread
> > destroy event happens.
> I am not sure this distinction is critical: a destroyed thread is just a
> thread with no more messages associated to it.
> >
> > To handle that, I propose a table store 1 threadId to many messageId.
> This
> > table can be used for getThread API, but it’s mainly easy to find changes
> > about a thread for Thread/changes topic which we are discussing.
> >
> > ThreadIdToMessageIdTable? : threadId timeuuid, messageId timeuuid.
> > Partition key: threadId and clustering key: messageId.
> >
> > I suppose some cases how we interact with this table and event bus
> system:
> >
> >-
> >
> >When we decide a message should have a new threadId => store data to
> >ThreadIdToMessageIdTable and emit ThreadCreate event.

Implement Thread/changes for JMAP

[Quan tran hong]

Hello,

I am starting working on the Thread/changes topic, and I think I should
discuss how to implement this topic with you.

After looking at Thread/changes specs, I think we should add thread
event-oriented support to our Event Bus.

I propose 3 events: Thread create, thread update and thread destroy. These
events should store which data is still a topic, but at least they need to
store changed threadId.

A thread create can be easily emitted when we decide a message should have
a new threadId. But we need to distinguish when thread update and thread
destroy event happens.

To handle that, I propose a table store 1 threadId to many messageId. This
table can be used for getThread API, but it’s mainly easy to find changes
about a thread for Thread/changes topic which we are discussing.

ThreadIdToMessageIdTable? : threadId timeuuid, messageId timeuuid.
Partition key: threadId and clustering key: messageId.

I suppose some cases how we interact with this table and event bus system:

   -

   When we decide a message should have a new threadId => store data to
   ThreadIdToMessageIdTable and emit ThreadCreate event.
   -

   When we decide a message should have a old threadId => store data to
   ThreadIdToMessageIdTable and emit ThreadUpdated event.
   -

   When we delete a message (under DeleteMessageListener hood), there will
   be 2 sub-case:
   -

  delete not the last message in that thread: we delete related data in
  ThreadIdToMessageIdTable and emit ThreadUpdate event.
  -

  delete the last message in that thread: we delete related data in
  ThreadIdToMessageIdTable and emit ThreadDestroy event.

So how do we know if it's the last message in that thread or not? First, we
need to be able to lookup threadId by messageId by easily add threadId to
existed ThreadLookupTable. After each message deletion, we will select and
count the number of records by that threadId from ThreadIdToMessageIdTable.
If there still are records of that thread, we know that is an update event.
If there are no records anymore, we know that is a destroyed event.

Now we have to handle these thread events. I propose using
ThreadChangesListener to handle these by calling ThreadChangesRepository/
ThreadChangesRepositoryDAO. Which APIs do we need and what is the
ThreadChangesTable’s structure is still a topic. For now, I can say it can
take inspiration from MailboxChanges.

The above is my outline ideas. Please have a look and do some discussion
together.


Best regards,


Quan

Re: Implement ThreadIdGuessingAlgorithm for the distributed module

[Quan tran hong]

Hi Benoit,

Following the approval of the table's design, I will start to implement
*threadtable *and start with define the table and its DAO layer. I will
come back for the deletion part feature finish the guessing threadid
feature.

Best regards,

Quan

Vào Th 4, 21 thg 7, 2021 vào lúc 18:08 btell...@linagora.com <
btell...@linagora.com> đã viết:

>
> On 21/07/2021 18:06, Quan tran hong wrote:
> > Hi Benoit,
> > Thanks for your great reviews.
> >
> >> Or we can just do several selects, one for each mimeMessageIds.I
> > personaly likely prefer this option.
> >
> > This will make the upper logic layer more complicated. But I agree with
> > your suggestion cause I would prefer balance and performance also.
> >
> >> Remember to do the delete of ThreadTable before threadtable_lookup,
> > otherwise if you delete the pointer before the data you might end up in
> > a case where the actual data is never deleted.
> >
> > Sure. Thanks for your reminder.
> >
> >> When/How do we call it?
> > StoringThreadIdGuessingAlgorithm maybe?
> Not really, this would only be called when appending messages.
>
> Likely this would need to be hooked in DeleteMessageListener.
> >
> > Cheers,
> > Quan
> >
> > Vào Th 4, 21 thg 7, 2021 vào lúc 17:03 btell...@apache.org <
> > btell...@apache.org> đã viết:
> >
> >> Hello quan,
> >>
> >> Nice proposal! I think the Cassandra data model you propose is evolved
> >> enough so that we start implementing it.
> >>
> >> Some comments inlined...
> >>
> >> On 21/07/2021 15:55, Quan tran hong wrote:
> >>> Hi Benoit,
> >>> I did have another try on this. Please have a look.
> >>> CREATE threadtable
> >>>
> >>> CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid,
> username
> >>> text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
> >>> mimemessageid), messageid));
> >>>
> >>> => Partition key: (username, mimemessageid), clustering key: messageid.
> >>>
> >>> [...]
> >>>
> >>> SELECT basesubject, threadId FROM threadtable WHERE username = 'quan'
> >> AND
> >>> mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
> >> This looks way better to me.
> >>
> >> IN usage is PRIMARY KEY is discouraged as it leads to coordination
> >> across partitions.
> >>
> >> Read more for instance in
> >>
> >>
> https://stackoverflow.com/questions/55604857/cassandra-query-performance-using-in-clause-for-one-portion-of-the-composite-pa
> >>
> >> Either we should move "mimeMessageId" to the clustering key (but all the
> >> messages of a user, including their subject would end up in a single
> >> partition, which could be quite large... for instance 1 million messages
> >> x size of the mime message ids and subject could be too much, as
> >> partitions are recommended to be 100MBs at most).
> >>
> >> Or we can just do several selects, one for each mimeMessageIds.I
> >> personaly likely prefer this option.
> >>> [...]
> >>>
> >>> CREATE threadtable_lookup for deletion purpose
> >>>
> >>> Supposed when we delete a message, we would need to delete that
> message’s
> >>> thread-related data in the threadtable also. I guess we just need
> >> messageId
> >>> to delete that message.
> >>>
> >>> We would need to have another table similar to the threadtable but
> looked
> >>> up by messageId to get the needed params for deletion query.
> >>>
> >>> CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
> >>> mimeMessageIds set, PRIMARY KEY(messageid));
> >> The set should be frozen. We will never add or remove data in it, so we
> >> do not need a CRDT (Commutative Replicated Data Type).
> >>
> >> Forbidding adding - removing individual elements avoids lots of issues,
> >> and lead to a more compact storage.
> >>
> >>> To ease testing, I create a table with messageId’s type is int instead.
> >>>
> >>> We will insert the data as same as the original table.
> >>>
> >>> insert into ThreadTable_lookup (messageId, username,
> mimeMessageIds)
> >>> values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2',
> >> 'MimeMessageID3'});
> >>> [...]
> >>>
> >>>
> >

Re: Breaking changes regarding Thread/get implementation?

[Quan tran hong]

Hi Benoit,

I did open a pull request following your suggestion here:
https://github.com/apache/james-project/pull/547

Best regards,
Quan

Re: Implement ThreadIdGuessingAlgorithm for the distributed module

[Quan tran hong]

Hi Benoit,
Thanks for your great reviews.

> Or we can just do several selects, one for each mimeMessageIds.I
personaly likely prefer this option.

This will make the upper logic layer more complicated. But I agree with
your suggestion cause I would prefer balance and performance also.

> Remember to do the delete of ThreadTable before threadtable_lookup,
otherwise if you delete the pointer before the data you might end up in
a case where the actual data is never deleted.

Sure. Thanks for your reminder.

> When/How do we call it?
StoringThreadIdGuessingAlgorithm maybe?

Cheers,
Quan

Vào Th 4, 21 thg 7, 2021 vào lúc 17:03 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello quan,
>
> Nice proposal! I think the Cassandra data model you propose is evolved
> enough so that we start implementing it.
>
> Some comments inlined...
>
> On 21/07/2021 15:55, Quan tran hong wrote:
> > Hi Benoit,
> > I did have another try on this. Please have a look.
> > CREATE threadtable
> >
> > CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
> > text, mimeMessageId text, baseSubject text, PRIMARY KEY((username,
> > mimemessageid), messageid));
> >
> > => Partition key: (username, mimemessageid), clustering key: messageid.
> >
> > [...]
> >
> > SELECT basesubject, threadId FROM threadtable WHERE username = 'quan'
> AND
> > mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3');
> This looks way better to me.
>
> IN usage is PRIMARY KEY is discouraged as it leads to coordination
> across partitions.
>
> Read more for instance in
>
> https://stackoverflow.com/questions/55604857/cassandra-query-performance-using-in-clause-for-one-portion-of-the-composite-pa
>
> Either we should move "mimeMessageId" to the clustering key (but all the
> messages of a user, including their subject would end up in a single
> partition, which could be quite large... for instance 1 million messages
> x size of the mime message ids and subject could be too much, as
> partitions are recommended to be 100MBs at most).
>
> Or we can just do several selects, one for each mimeMessageIds.I
> personaly likely prefer this option.
> > [...]
> >
> > CREATE threadtable_lookup for deletion purpose
> >
> > Supposed when we delete a message, we would need to delete that message’s
> > thread-related data in the threadtable also. I guess we just need
> messageId
> > to delete that message.
> >
> > We would need to have another table similar to the threadtable but looked
> > up by messageId to get the needed params for deletion query.
> >
> > CREATE TABLE ThreadTable_lookup (messageId timeuuid, username text,
> > mimeMessageIds set, PRIMARY KEY(messageid));
> The set should be frozen. We will never add or remove data in it, so we
> do not need a CRDT (Commutative Replicated Data Type).
>
> Forbidding adding - removing individual elements avoids lots of issues,
> and lead to a more compact storage.
>
> > To ease testing, I create a table with messageId’s type is int instead.
> >
> > We will insert the data as same as the original table.
> >
> > insert into ThreadTable_lookup (messageId, username, mimeMessageIds)
> > values (1, 'quan', {'MimeMessageID1', 'MimeMessageID2',
> 'MimeMessageID3'});
> >
> > [...]
> >
> >
> > SELECT * FROM threadtable_lookup;
> >
> >
> > Now we will do a query selection by messageId to get the needed username,
> > mimeMessageIds for original threadtable deletion.
> >
> > Supposed we want to delete message 4.
> >
> > SELECT username, mimemessageids FROM threadtable_lookup WHERE messageid
> = 4;
> >
> >
> > Then we will use these results to do a deletion query on threadtable.
> >
> Remember to do the delete of ThreadTable before threadtable_lookup,
> otherwise if you delete the pointer before the data you might end up in
> a case where the actual data is never deleted.
>
> The algorithm that you propose looks good.
>
> When/How do we call it?
>
>
> Cheers,
>
> Benoit
> > The data of messageId 4 deleted.
> >
> >
> >
> > Best regards,
> >
> > Quan
> >
> >
> > Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
> > btell...@apache.org> đã viết:
> >
> >> Hello Quan,
> >>
> >> On 20/07/2021 17:24, Quan tran hong wrote:
> >>> [...]
> >>>
> >>> SELECT threadId FROM threadtable WHERE username = 'quan' AND
> baseSubject
> >> =
> >>> 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2',

Re: Implement ThreadIdGuessingAlgorithm for the distributed module

[Quan tran hong]

 to do a deletion query on threadtable.


The data of messageId 4 deleted.



Best regards,

Quan


Vào Th 3, 20 thg 7, 2021 vào lúc 18:39 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello Quan,
>
> On 20/07/2021 17:24, Quan tran hong wrote:
> > [...]
> >
> > SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
> =
> > 'baseSubject1' AND mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3')
> > LIMIT 1 ALLOW FILTERING;
> ALLOW FILTERING should not be used as it will result in a full scan and
> is thus a performance disaster.
>
> If you need it, this means you do not have the right table structure and
> likely should rework the CREATE TABLE statement.
> >
> > => This new message should have this threadId.
> > New unrelated message
> >
> > Assume that we do a query for a new unrelated message.
> >
> > SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject
> =
> > 'unrelatedBaseSubject' AND mimeMessageId IN ('MimeMessageID2',
> > 'MimeMessageID3') LIMIT 1 ALLOW FILTERING;
> >
> > => This new message should have a new threadId.
> > Insert new message data
> >
> > After having a threadId, we need to insert new message data into the
> thread
> > table.
> >
> > insert into ThreadTable (messageId, threadId, username, mimeMessageId,
> > baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
> > 'MimeMessageID2', 'baseSubject1');
> >
> > insert into ThreadTable (messageId, threadId, username, mimeMessageId,
> > baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
> > 'MimeMessageID3', 'baseSubject1');
> > Conclusion
> >
> > I think this data model complies with the needed request for the guessing
> > algorithm problem, but it looks like still maybe there is room for
> > improvement.
>
> What Cassandra request do we use to delete the data in there?
>
> >
> >
> > Best Regards,
> >
> > Quan
> >
> >
> >
> >
> >
> > Vào Th 2, 19 thg 7, 2021 vào lúc 18:23 btell...@apache.org <
> > btell...@apache.org> đã viết:
> >
> >> Hello Quan,
> >>
> >> On 19/07/2021 17:59, Quan tran hong wrote:
> >>> Hi,
> >>> I am starting to implement ThreadIdGuessingAlgorithm for the
> distributed
> >>> module. Because this is a breaking change and I am new to Cassandra
> also,
> >>> therefore I want to have some discussion with you about how to do this.
> >> As long as we introduce a new table there is no reason that it creates
> >> breaking change, but getting the format right will ease our life down
> >> the line.
> >>> For the ones who did not catch up with this work, please have a look at
> >>> JMAP Threads specs [1] and my work related to this [2].
> >>>
> >>> So my ideas on how to do this:
> >>> - Add a needed inputs Cassandra Table for guessing threadId algorithm.
> >>> Maybe a table likes:
> >>>  CREATE TABLE ThreadRelatedTable (
> >>> threadId   timeuuid,
> >>> messageId  timeuuid,
> >>> mimeMessageIds SET,
> >>> subject   text,
> >>> PRIMARY KEY (mimeMessageIds, subject)
> >>> );
> >>> - Whenever we guess threadId for a new message, we access this table
> and
> >> do
> >>> the matching query to get related threadId(if there is) or decide new
> >>> message should have a new threadId.
> >>> - Whenever we save a new message, we save the thread-related data to
> this
> >>> table.
> >>>
> >>> This is my first come-up idea. Please express your thoughts about this.
> >> Collections are an advanced data modeling tool, that should be used with
> >> caution. I am not sure using it in a PRIMARY KEY is a good idea. I am
> >> not sure that does what you want (the full primary key should be
> >> specified to know which node hold the data.
> >>
> >> Also, once you found the message related to a thread you want to
> >> validate that the subject matches. This can be done on application side
> >> (James), and avoids complicated data model.
> >>
> >> I encourage you to validate your data model using a Cassandra in docker
> >> and executing CQL commands locally with CQLSH tool to simulate the
> >> queries you whish to do, and learn about your data model before even
> >> starting to implement it. IMO sharing CQL commands for creating the
> >> table, inserting data in it, and retrieving data from it would be a
> >> great follow up to this email.
> >>
> >> How would you populate the data of that table?
> >>
> >> Best regards,
> >>
> >> Benoit
> >>> Best regards,
> >>>
> >>> Quan
> >>>
> >>> [1] https://jmap.io/spec-mail.html#threads
> >>> [2] https://issues.apache.org/jira/browse/JAMES-3516
> >>>
> >> -
> >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> >> For additional commands, e-mail: server-dev-h...@james.apache.org
> >>
> >>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Implement ThreadIdGuessingAlgorithm for the distributed module

[Quan tran hong]

Hi Benoit,
Following your suggestion, I did have some experiments today. Please have a
look.
I represent this with a few outlines so you can read it easily.
Create table

CREATE TABLE ThreadTable (messageId timeuuid, threadId timeuuid, username
text, mimeMessageId text, baseSubject text, PRIMARY KEY(messageId,
mimeMessageId));

=> Partition key: messageId, clustering key: mimeMessageId.
Insert data

I will add:

   -

   2 related message and 1 unrelated message for user ‘quan’
   -

   1 message for user ‘benoit’ (which seem related to some messages of user
   ‘quan’)

// insert message1 data for ‘quan’

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'quan', 'MimeMessageID1',
'baseSubject1');

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'quan', 'MimeMessageID2',
'baseSubject1');

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'quan', 'MimeMessageID3',
'baseSubject1');
// insert message2 data (related to message1) for ‘quan’

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'quan', 'MimeMessageID1',
'baseSubject1');
// insert message3 data (not related to any message) for ‘quan’

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'quan', 'MimeMessageID4',
'baseSubject2');
// insert message4 data (related to message1 but for ‘benoit’)

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'benoit', 'MimeMessageID5',
'baseSubject1');

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), now(), 'benoit', 'MimeMessageID1',
'baseSubject1');
Select all data

The query for guessing new messages' threadIdNew related message

For example, there is a new message coming for user ‘quan’, with some
header fields:

   -

   SET MimeMessageIds (after combine values): {‘MimeMessageID2’,
   ‘MimeMessageID3’}
   -

   Base subject line (after stripping): “baseSubject1”

This message is supposed to be related to 2 other messages of ‘quan’.

We need to query one row related to this new message (if there is).

SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject =
'baseSubject1' AND mimeMessageId IN ('MimeMessageID2', 'MimeMessageID3')
LIMIT 1 ALLOW FILTERING;

=> This new message should have this threadId.
New unrelated message

Assume that we do a query for a new unrelated message.

SELECT threadId FROM threadtable WHERE username = 'quan' AND baseSubject =
'unrelatedBaseSubject' AND mimeMessageId IN ('MimeMessageID2',
'MimeMessageID3') LIMIT 1 ALLOW FILTERING;

=> This new message should have a new threadId.
Insert new message data

After having a threadId, we need to insert new message data into the thread
table.

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
'MimeMessageID2', 'baseSubject1');

insert into ThreadTable (messageId, threadId, username, mimeMessageId,
baseSubject) values (now(), 02294fe1-e941-11eb-a8ee-77de5498f1fa, 'quan',
'MimeMessageID3', 'baseSubject1');
Conclusion

I think this data model complies with the needed request for the guessing
algorithm problem, but it looks like still maybe there is room for
improvement.


Best Regards,

Quan





Vào Th 2, 19 thg 7, 2021 vào lúc 18:23 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello Quan,
>
> On 19/07/2021 17:59, Quan tran hong wrote:
> > Hi,
> > I am starting to implement ThreadIdGuessingAlgorithm for the distributed
> > module. Because this is a breaking change and I am new to Cassandra also,
> > therefore I want to have some discussion with you about how to do this.
> As long as we introduce a new table there is no reason that it creates
> breaking change, but getting the format right will ease our life down
> the line.
> >
> > For the ones who did not catch up with this work, please have a look at
> > JMAP Threads specs [1] and my work related to this [2].
> >
> > So my ideas on how to do this:
> > - Add a needed inputs Cassandra Table for guessing threadId algorithm.
> > Maybe a table likes:
> >  CREATE TABLE ThreadRelatedTable (
> > threadId   timeuuid,
> > messageId  timeuuid,
> > mimeMessageIds SET,
> > subject   text,
> > PRIMARY KEY (mimeMessageIds, subject)
> > );
> > - Whenever we guess threadId for a new message, we access this table and
> do
> > the matching query to get related threadId(if there is) or decide new
> > message should have a new threadId.
> > - Whenever we save a new message, we save the thread-related data to this
> > table.
> 

Implement ThreadIdGuessingAlgorithm for the distributed module

[Quan tran hong]

Hi,
I am starting to implement ThreadIdGuessingAlgorithm for the distributed
module. Because this is a breaking change and I am new to Cassandra also,
therefore I want to have some discussion with you about how to do this.

For the ones who did not catch up with this work, please have a look at
JMAP Threads specs [1] and my work related to this [2].

So my ideas on how to do this:
- Add a needed inputs Cassandra Table for guessing threadId algorithm.
Maybe a table likes:
 CREATE TABLE ThreadRelatedTable (
threadId   timeuuid,
messageId  timeuuid,
mimeMessageIds SET,
subject   text,
PRIMARY KEY (mimeMessageIds, subject)
);
- Whenever we guess threadId for a new message, we access this table and do
the matching query to get related threadId(if there is) or decide new
message should have a new threadId.
- Whenever we save a new message, we save the thread-related data to this
table.

This is my first come-up idea. Please express your thoughts about this.

Best regards,

Quan

[1] https://jmap.io/spec-mail.html#threads
[2] https://issues.apache.org/jira/browse/JAMES-3516

Re: Breaking changes regarding Thread/get implementation?

[Quan tran hong]

Hi Benoit,

Sure. I will do it.

Quan

Vào Th 2, 19 thg 7, 2021 vào lúc 16:32 btell...@apache.org <
btell...@apache.org> đã viết:

> Hello Quan,
>
> I have the impression we missed the upgrade instructions [1] related to
> your mapping changes when adding the threadId to the ElasticSearch index
> [2].
>
> This is a breaking change as after that newly indexed documents will
> reference a field that does not yet exist on previously created indexes,
> we likely need to explicitly modify the mapping.
>
> Could you carry out this change?
>
> Best regards,
>
> Benoit
>
> [1]
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md
> [2] https://github.com/apache/james-project/pull/539
> [3]
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

About sharing my Google Summer of Code 2021 progress

[Quan tran hong]

Hello James developers,

My name is Quan and I am working on the “Implement Thread support for JMAP”
topic for Apache James as my Google Summer of Code 2021 project. My topic
is to implement the logic part of the thread in Apache James: emails that
related to a topic should belong to a thread. As the first evaluation of
Google Summer of Code is coming, I want to share my working progress with
you also.

With the first half of Google Summer of Code past, I almost successfully
implemented Thread/get with the InMemory module using
SearchThreadIdGuessingAlgorithm. My next goal for the rest time of Google
Summer of Code is to enable searching messages by thread using Search
Engine to fasten the search time and then support Thread/get with the
Distributed module.

While Google Summer of Code will enable a basic User experience with
threads on top of JMAP, several enhancements would likely not be covered
due to lack of time. There is some work related to Threads topic that I
will continue to contribute to Apache James after Google Summer of Code:

   -

   Implement Thread/changes
   -

   Push state changes for threads
   -

   Investigate ElasticSearch aggregation for collapse thread
   -

   Implement collapse thread option on top of the message search index
   -

   Implement collapse thread for memory
   -

   Email/query should expose thread options
   -

   Implement Thread support for IMAP maybe?


If you are interested in this topic, please come here (
https://issues.apache.org/jira/browse/JAMES-3516) and have some discussion.
I would be very appreciative.

Wish you a good day!

Best Regards,

Quan Tran Hong

Re: Should we have a checkstyle for Scala?

[Quan tran hong]

+1
I am looking forward to seeing how we select the checkstyle plugin and
configure it for our project.
Quan

Vào Th 6, 25 thg 6, 2021 vào lúc 18:25 btell...@apache.org <
btell...@apache.org> đã viết:

> +1
>
> I would be happy to see contributions on this!
>
> Benoit
>
> On 25/06/2021 13:14, tungtv...@gmail.com wrote:
> > Hello guys,
> >
> > I realize the current checkstyle.xml file doesn't support the Scala
> > convention.
> > Should we have a checkstyle for that?
> >
> > My suggestion:
> > - https://github.com/scalastyle/scalastyle
> >
> > - https://github.com/scalastyle/scalastyle-maven-plugin
> >
> >
> > Best Regards,
> >
> > Tung, Van TRAN
> >
> >
> > -
> > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> > For additional commands, e-mail: server-dev-h...@james.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: Introduction from a new member of Apache James Community

[Quan tran hong]

Hi all,

I am so happy to be a part of Apache James Community!

Quan Tran

Vào Th 2, 5 thg 10, 2020 vào lúc 15:22 David Leangen 
đã viết:

>
> > As Quan did not succeed to send a mail to this list (...) I'm forwarding
> it for him!
>
> > Let me introduce myself first. My name is Tran Hong Quan. I am from
> Vietnam and studying fourth-year at Hanoi university of science and
> technology.
>
> Welcome, Quan!
>
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>