[RESULT] [VOTE] Apache James release 3.7.0

[Tellier Benoit]

|Hi all, I am happy to announce you the vote for the 3.7.0 release did
succeed. The release received 5 positive votes, 3 of them being binding.
Thanks to all contributors, developers and committers who made this
possible! In the coming hours, I will finalize the release process,
namely: - Publish the maven artifacts - Upgrade the download page and
the (old) website - Announce the release Cheers, PMC member name|

Le 07/03/2022 à 15:09, Antoine Duprat a écrit :
> +1
>
> Cheers,
> Antoine
>
> Le mer. 2 mars 2022 à 04:44, Tellier Benoit  a écrit :
>
>> Hi,
>>
>> I would like to propose a new vote for 3.7.0 release of the Apache James
>> server.
>>
>> You can find:
>>
>>  - The maven release staged in repository.apache.org as the
>> artifact #1062:
>> https://repository.apache.org/content/repositories/orgapachejames-1062/
>>  - The changelog for
>> 3.7.0:
>> https://github.com/apache/james-project/blob/master/CHANGELOG.md#unreleased
>>  - Changes to our doc and the website:
>> https://github.com/apache/james-project/pull/900
>>  - Release artifacts are uploaded to ASF dowloads.
>>
>> This release introduces some major bug fixes, stability enhancements and
>> some cool additional components and features!
>>
>> Voting rules:
>>  - This is a majority approval: this release may not be vetoed.
>>  - A quorum of 3 binding votes is required
>>  - The vote starts at Wednesday 2nd of March 2022, 10am UTC+7
>>  - The vote ends at Wednesday 9th of March 2022, 10am UTC+7
>>
>> You can answer to it just with +1 and -1. Down-votes may be motivated.
>>
>> 3 binding votes are expected move forward on this release.
>>
>> Cheers,
>>
>> Benoit TELLIER
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>

Re: Sv: [VOTE] Apache James release 3.7.0

[Tellier Benoit]

Ok.

I created https://issues.apache.org/jira/browse/JAMES-3718 to make sure
this do not get forgotten...

Regards,

Benoit

Le 02/03/2022 à 15:21, Andreas Joseph Krogh a écrit :
> På onsdag 02. mars 2022 kl. 09:10:16, skrev Tellier Benoit
> mailto:btell...@apache.org>>:
>
> Hello Andreas,
>  
>
> Sadly a release is a poor moment to discuss the perimeter.
>
> Running a release process 'just' to come to the vote took me over
> 12 hours, I am personally upset wasting my time running multiple
> releases for nothing. By the way, except for 3.5.0 releasing is a
> burdon that I did carry mostly by my own for several years
> already, for free to the community. I would be glad we better
> share this responsibility.
>
> My mindset (we can discuss this) is that the project should always
> be on a releasable state. Additions according to the will of
> contributors should not compete with the release milestone. That
> is, users and contributors can open tickets and code contribution
> when they encounter the need, independently from releases. When we
> release, what is ready is included, what is not ready / not done
> will go for next release.
>
> In this case, the contributors would have had likely carried the
> task out if we had a JIRA ticket saying "Upgrade the MIME4J code a
> new version to include unreleased changes". The ticket do not
> exist thus it was not carried out. I encourage you to create such
> a ticket. It would then be included in 3.7.1 / 3.8.0.
>  
>
> Finally I sent an email one week ago, seeking consensus toward
> such a release. Raising your voice during consensus seeking phases
> rather than on votes would surely help mitigate such conflicts. CF
> https://www.mail-archive.com/server-dev@james.apache.org/msg71619.html
>
> Regards,
>
> Benoit
>
> Please express your concerns on the email I sent one week before
> explaining I was about to release cf 
>
> No problem, I didn't mean to hold up the release, sorry for the noise.
> It's no big deal for us to include a custom build of mime4j from master.
>
>  
>
> +1 for releasing.
>
>  
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andr...@visena.com <mailto:andr...@visena.com>
> www.visena.com <https://www.visena.com>
> <https://www.visena.com>
>  
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Sv: [VOTE] Apache James release 3.7.0

[Tellier Benoit]

Hello Andreas,

Sadly a release is a poor moment to discuss the perimeter.

Running a release process 'just' to come to the vote took me over 12
hours, I am personally upset wasting my time running multiple releases
for nothing. By the way, except for 3.5.0 releasing is a burdon that I
did carry mostly by my own for several years already, for free to the
community. I would be glad we better share this responsibility.

My mindset (we can discuss this) is that the project should always be on
a releasable state. Additions according to the will of contributors
should not compete with the release milestone. That is, users and
contributors can open tickets and code contribution when they encounter
the need, independently from releases. When we release, what is ready is
included, what is not ready / not done will go for next release.

In this case, the contributors would have had likely carried the task
out if we had a JIRA ticket saying "Upgrade the MIME4J code a new
version to include unreleased changes". The ticket do not exist thus it
was not carried out. I encourage you to create such a ticket. It would
then be included in 3.7.1 / 3.8.0.

Finally I sent an email one week ago, seeking consensus toward such a
release. Raising your voice during consensus seeking phases rather than
on votes would surely help mitigate such conflicts. CF
https://www.mail-archive.com/server-dev@james.apache.org/msg71619.html

Regards,

Benoit

Please express your concerns on the email I sent one week before
explaining I was about to release cf

Le 02/03/2022 à 14:32, Andreas Joseph Krogh a écrit :
> På onsdag 02. mars 2022 kl. 04:44:29, skrev Tellier Benoit
> mailto:btell...@apache.org>>:
>
> Hi,
>
> I would like to propose a new vote for 3.7.0 release of the Apache
> James
> server.
> […]
>
>  
>
> I'd like to see a release of mime4j first then depend on that for
> 3.7.0, as it got some important fixes (for us at least).
>
>  
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andr...@visena.com <mailto:andr...@visena.com>
> www.visena.com <https://www.visena.com>
> <https://www.visena.com>
>  
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [VOTE] Apache James release 3.7.0

[Tellier Benoit]

+1

Le 02/03/2022 à 10:44, Tellier Benoit a écrit :
> Hi,
>
> I would like to propose a new vote for 3.7.0 release of the Apache James
> server.
>
> You can find:
>
>  - The maven release staged in repository.apache.org as the
> artifact #1062:
> https://repository.apache.org/content/repositories/orgapachejames-1062/
>  - The changelog for
> 3.7.0:
> https://github.com/apache/james-project/blob/master/CHANGELOG.md#unreleased
>  - Changes to our doc and the website:
> https://github.com/apache/james-project/pull/900
>  - Release artifacts are uploaded to ASF dowloads.
>
> This release introduces some major bug fixes, stability enhancements and
> some cool additional components and features!
>
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Wednesday 2nd of March 2022, 10am UTC+7
>  - The vote ends at Wednesday 9th of March 2022, 10am UTC+7
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> 3 binding votes are expected move forward on this release.
>
> Cheers,
>
> Benoit TELLIER
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[VOTE] Apache James release 3.7.0

[Tellier Benoit]

Hi,

I would like to propose a new vote for 3.7.0 release of the Apache James
server.

You can find:

 - The maven release staged in repository.apache.org as the
artifact #1062:
https://repository.apache.org/content/repositories/orgapachejames-1062/
 - The changelog for
3.7.0:
https://github.com/apache/james-project/blob/master/CHANGELOG.md#unreleased
 - Changes to our doc and the website:
https://github.com/apache/james-project/pull/900
 - Release artifacts are uploaded to ASF dowloads.

This release introduces some major bug fixes, stability enhancements and
some cool additional components and features!

Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Wednesday 2nd of March 2022, 10am UTC+7
 - The vote ends at Wednesday 9th of March 2022, 10am UTC+7

You can answer to it just with +1 and -1. Down-votes may be motivated.

3 binding votes are expected move forward on this release.

Cheers,

Benoit TELLIER

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: SMTP server configuration issue

[Tellier Benoit]

Hello René,

Thanks for raising the topic here.

Le 05/05/2021 à 16:31, Rene Cordier a écrit :
> Hello guys,
> 
> [...]
> 
> 1. If authRequired is set to false, we should reject verifyIdenty=true,
> as it makes no logical sense. People might need to update their James
> running installation though (but easy)
> 
By far my prefered option.

Yes some people need to reconfigure smtpserver.xml but at least we don't
take implicit decisions.

> 2. If authRequired is set to false, we can silently ignore
> verifyIdentity is set to true.

The option that I like the least... It IMO gives a false sense of safety
(you might believe senders are verified but they actually are not).

> 3. We keep this current behavior, but need to change the documentation
> accordingly and add a warning log as well during James startup.

This look fine to me. Verifying senders implies forcing local users to
authenticate (overwise the work-around is too simple...). However from
an admin perspective it would be harder to diagnose some SMTP
transaction being rejected compared to a server not starting.

> I personally prefer the first one, as this is the way it's documented
> for now and I found it more logical. However, it's completely opened to
> discussion (thus the mail).
> 
> Depending on the feedback, will create the according JIRA fix ticket.

I think we can reuse JAMES-3525 as it is closely related...

Cheers,

Benoit

> 
> Thank you all, have a good day!
> 
> Rene.
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

new committer: Juhan Aasaru

[Tellier Benoit]

The Project Management Committee (PMC) for Apache James
has invited Juhan Aasaru to become a committer and we are pleased
to announce that he has accepted.

In the last few months, Juhan made several Apache James contributions.
Especially regarding JAMES-3564, we believe he is committed to offer a
better support for existing users and as such will have a very positive
impact on the user community.

We believe nominating him as a committer would lead to more frequent
releases, fixing bugs in a timely manner and would give us the means to
maintain a 3.6.x branch.

His membership as part of the Apache Finneract project
shows that he is already following "the Apache way".

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.

Cheers,
The Apache James PMC

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Vote] Call for vote: Apache Mime4J 0.8.4

[Tellier Benoit]

Hello Jean,

I opened https://github.com/apache/james-mime4j/pull/38 on that topic:
 - correct release tracking on the JIRA
 - Detail the changelog for upcoming 0.8.4
 - Merge release notes and the changelog

This should solve your concerns,

Regards,

Benoit

Le 15/04/2021 à 14:12, Jean Helou a écrit :
> Hi benoit,
>
> The changelog has not been updated which is a bit sad (and i know it's
> painful to do but it's s helpful for developers who use the library)
> On a semi-related note : the release notes have not been updated either but
> that last file actually seems to be totally out of sync with the repo : it
> mentions an inexistant 0.9.0 and has not been modified for the last 4 years
> ... maybe delete it ?
>
> jean
>
> On Wed, Apr 14, 2021 at 9:09 AM Tellier Benoit  wrote:
>
>> Hi,
>>
>> Following a user request (https://issues.apache.org/jira/browse/MIME4J-297
>> ),
>>
>> I would like to propose a vote for 0.8.4 release of the MIME4J library.
>>
>> You can find the maven release staged in repository.apache.org as the
>> artifact #1053:
>> https://repository.apache.org/content/repositories/orgapachejames-1053/
>>
>>
>> Voting rules:
>>  - This is a majority approval: this release may not be vetoed.
>>  - A quorum of 3 binding votes is required
>>  - The vote starts at Wedneday 14th of April 2021, 2pm UTC+7
>>  - The vote ends at Wednesday 21th of April 2021, 2pm UTC+7
>>
>> You can answer to it just with +1 and -1. Down-votes may be motivated.
>>
>> 3 binding votes are expected move forward on this release.
>>
>> Cheers,
>>
>> Benoit Tellier
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Tellier Benoit]

Hello Jean,

Answer inlined...

Le 15/04/2021 à 19:01, Jean Helou a écrit :
>> Following the 3.6.0 release I experimented with dockerhub and pushed the
>> following images:
>>
>  THANK YOU ! (yes with caps because that's how grateful I am !)
>
> I also propose to timely edit the existing configuration to:
>>  - No longer use linagora repository for Apache James docker images
>> distribution
>>
>  - Only rely on released docker images (no longer branch master) to
>> better fit in Apache release policies
>>
> I saw the corresponding PR :+1:
>
>
>> To be fairly honest building these images took me too much time. I would
>> highly appreciate automation of docker images construction as part of
>> the maven build. A way to do it is JIB. [1]
>
> That's what I used to build my pure SMTP relay server assembly, it's being
> polished but jib works great :)
>
> demonstrate building a james
>> server image variation for a tierce project so I propose to do something
>> similar, except without requiring a docker daemon. It includes CLI,
>> glowroot, etc... We had been using this JIB generated image on
>> production without problems. I propose we target something similar for
>> James docker image generation. As part of the release process we would
>> then need to load, then push the maven build results. This would lead to
>> the removal of /dockerfiles folder (more or less).
>>
> Since  docker images are not considered release artifacts under apache
> policy (if I understand correctly) would it make sense to try and fully
> automate the image building post release ?
> add a piece to the jenkinsfile that monitors only tags, and builds the jib
> image downloading the artifacts from maven central
This argument make sense.

Maybe we could push automatically RCs to docker hub automatically upon
tagging (before the voting process), then once the vote pass just rename
the images?

That way:
 - It is automatic and convenient
 - We make it clear the artifact is not to be used yet

This would look like what Juhan proposed in
https://issues.apache.org/jira/browse/JAMES-3565 ...

Thoughts?

Cheers,

Benoit

>
>
>> [1]
>>
>> https://github.com/linagora/openpaas-james/tree/master/openpaas-james/apps/memory
>> README and pom.xml are of interest.
>>
>> Cheers,
>>
>> Benoit
>>
>> Le 11/12/2020 à 16:29, Tellier Benoit a écrit :
>>> You're right,
>>>
>>> I did not follow up on this.
>>>
>>> I just created the INFRA ticket.
>>> https://issues.apache.org/jira/browse/INFRA-21180
>>>
>>> As far as I understand, this is not an official release channel but
>>> rather a convenience one.
>>>
>>> Cheers,
>>>
>>> Benoit
>>>
>>> Le 11/12/2020 à 15:18, Jean Helou a écrit :
>>>>> I propose to ask for rights on apache/james repository
>>>> I'm revisiting this thread after seeing
>>>> https://www.docker.com/blog/supporting-open-source-projects-at-docker
>> which
>>>> reminded me of this effort and I was wondering if an INFRA ticket has
>> been
>>>> opened for james (of if there is a trace of the progress somewhere)
>>>> I'm not sur exactly how the program works but if it whitelists prefixes,
>>>> Apache is most likely already in it however in light of the work on CI
>>>> builds on apache infrastructure, I am wondering if/how the credentials
>> for
>>>> pushing form CI can be setup and I was hoping to drop a comment on the
>>>> corresponding ticket.
>>>>
>>>> jean
>>>>
>>> -
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Results] [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Tellier Benoit]

Hello Jean,

Le 15/04/2021 à 19:25, Jean Helou a écrit :
> [...]
> I don't see how 389 blocks the release but I'm ok with the PR anyway :)
I will advertise the release on the announce mailing list. If the first
page of the website is advertising a broken demo, then it is not good
for the overall project, and we miss a good opportunity to promote it.
> Cheers,
> Jean
>
>
>> Otherwise I will proceed, likely on Friday, if not next week when
>> I have time... - I must confess being impatient to be done with it.
>>
>> Cheers,
>>
>> Benoit
>>
>> Le 14/04/2021 à 10:34, Tellier Benoit a écrit :
>>> Hi all,
>>>
>>> I am happy to announce you the vote for the 3.6.0 release did succeed.
>>>
>>> The release received 8 positive votes, 3 of them being binding.
>>>
>>> Thanks to all contributors, developers and committers who made this
>>> possible!
>>>
>>> In the coming hours, I will finalize the release process, namely:
>>>
>>> - Publish the maven artifacts
>>> - Upgrade the download page and the (old) website
>>> - Announce the release
>>> - Apache docker images (convenience downloads, non official release
>>> artifacts CF Apache policy regarding docker images)
>>>
>>> Voting thread:
>>> https://www.mail-archive.com/server-dev@james.apache.org/msg70025.html
>>>
>>> Cheers,
>>>
>>> PMC member name
>>>
>>> -
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Results] [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Tellier Benoit]

Hello there,

I am about to conclude the 3.6.0 release (publish the website & announce)

However I would like the website to advertise the maintained, apache
images rather than the unmaintained ones of an undisclosed third party.

I imagine the changes in
https://github.com/apache/james-project/pull/389 to be consensual.

If the feedback is positive, I can finish the 3.6.0 process as off
today. Otherwise I will proceed, likely on Friday, if not next week when
I have time... - I must confess being impatient to be done with it.

Cheers,

Benoit

Le 14/04/2021 à 10:34, Tellier Benoit a écrit :
> Hi all,
> 
> I am happy to announce you the vote for the 3.6.0 release did succeed.
> 
> The release received 8 positive votes, 3 of them being binding.
> 
> Thanks to all contributors, developers and committers who made this
> possible!
> 
> In the coming hours, I will finalize the release process, namely:
> 
> - Publish the maven artifacts
> - Upgrade the download page and the (old) website
> - Announce the release
> - Apache docker images (convenience downloads, non official release
> artifacts CF Apache policy regarding docker images)
> 
> Voting thread:
> https://www.mail-archive.com/server-dev@james.apache.org/msg70025.html
> 
> Cheers,
> 
> PMC member name
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[Vote] Call for vote: Apache Mime4J 0.8.4

[Tellier Benoit]

Hi,

Following a user request (https://issues.apache.org/jira/browse/MIME4J-297),

I would like to propose a vote for 0.8.4 release of the MIME4J library.

You can find the maven release staged in repository.apache.org as the
artifact #1053:
https://repository.apache.org/content/repositories/orgapachejames-1053/


Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Wedneday 14th of April 2021, 2pm UTC+7
 - The vote ends at Wednesday 21th of April 2021, 2pm UTC+7

You can answer to it just with +1 and -1. Down-votes may be motivated.

3 binding votes are expected move forward on this release.

Cheers,

Benoit Tellier

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Tellier Benoit]

Hello there,

Following the 3.6.0 release I experimented with dockerhub and pushed the
following images:

- apache/james:memory-3.x.x built from
https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/memory
- apache/james:jpa-3.x.x built from
https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/jpa
- apache/james:demo-3.x.x built from
https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/provisioned
- apache/james:cassandra-3.x.x build from
https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra
- apache/james:distributed-3.x.x built from
https://github.com/apache/james-project/tree/master/dockerfiles/run/guice/cassandra-rabbitmq

I however do not have the rights to manage the repository, hence can't
add a description / overview just like apache/tika repo did. I opened
INFRA-21718 regarding this.

I also propose to timely edit the existing configuration to:
 - No longer use linagora repository for Apache James docker images
distribution
 - Only rely on released docker images (no longer branch master) to
better fit in Apache release policies

To be fairly honest building these images took me too much time. I would
highly appreciate automation of docker images construction as part of
the maven build. A way to do it is JIB. [1] demonstrate building a james
server image variation for a tierce project so I propose to do something
similar, except without requiring a docker daemon. It includes CLI,
glowroot, etc... We had been using this JIB generated image on
production without problems. I propose we target something similar for
James docker image generation. As part of the release process we would
then need to load, then push the maven build results. This would lead to
the removal of /dockerfiles folder (more or less).

[1]
https://github.com/linagora/openpaas-james/tree/master/openpaas-james/apps/memory
README and pom.xml are of interest.

Cheers,

Benoit

Le 11/12/2020 à 16:29, Tellier Benoit a écrit :
> You're right,
>
> I did not follow up on this.
>
> I just created the INFRA ticket.
> https://issues.apache.org/jira/browse/INFRA-21180
>
> As far as I understand, this is not an official release channel but
> rather a convenience one.
>
> Cheers,
>
> Benoit
>
> Le 11/12/2020 à 15:18, Jean Helou a écrit :
>>> I propose to ask for rights on apache/james repository
>>
>> I'm revisiting this thread after seeing
>> https://www.docker.com/blog/supporting-open-source-projects-at-docker which
>> reminded me of this effort and I was wondering if an INFRA ticket has been
>> opened for james (of if there is a trace of the progress somewhere)
>> I'm not sur exactly how the program works but if it whitelists prefixes,
>> Apache is most likely already in it however in light of the work on CI
>> builds on apache infrastructure, I am wondering if/how the credentials for
>> pushing form CI can be setup and I was hoping to drop a comment on the
>> corresponding ticket.
>>
>> jean
>>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[Results] [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Tellier Benoit]

Hi all,

I am happy to announce you the vote for the 3.6.0 release did succeed.

The release received 8 positive votes, 3 of them being binding.

Thanks to all contributors, developers and committers who made this
possible!

In the coming hours, I will finalize the release process, namely:

- Publish the maven artifacts
- Upgrade the download page and the (old) website
- Announce the release
- Apache docker images (convenience downloads, non official release
artifacts CF Apache policy regarding docker images)

Voting thread:
https://www.mail-archive.com/server-dev@james.apache.org/msg70025.html

Cheers,

PMC member name

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Leverage the Summer 2021 of Open Source to conduct a proof of concept regarding Pulsar?

[Tellier Benoit]

Hello there,

The Apache James distributed server has the ambition to scale seamlessly
and relies on proven distributed technologies in order to do so
(Cassandra, ElasticSearch, S3).

However, we currently rely on RabbitMQ for the messaging part, and
despite the addition of Quorum queues in 3.8.0 version (not yet
leveraged by James BTW) scalability and distributed set up is still a
concern.

Here are (some) pitfalls of the current RabbitMQ implementation:
 - Manual error management (exponential retries and dead lettering)
 - Custom connection pooling
 - I had issues with connection recovery and had to reboot James
 - Complex logic to re-create exclusive queues upon reboot
 - Scalability of quorum queues
 - Needs of associated tables to 'view the queue content'
 - No support for delays.

Among the alternatives to RabbitMQ, one stands out: Pulsar...
 - Handle 1.000.000 topics seamlessly thus can be used as a backbone for
PUS/SUB
 - Scalable
 - exponential retries and dead-lettering is managed out of the box
 - As a distributed log, we can iterate the content (but we might still
keep track of deleted content)
 - Handles delays
 - And it is another Apache TLP project!

I know some other community members are some Apache Pulsar enthusiasts.

Hence, I wonder if we can not leverage the opportunity of the Summer
2021 of Open Source ( https://summer.iscas.ac.cn/help/en/ ) to conduct a
proof of concept regarding Pulsar integration within the James
eco-system. (all Apache TLPs projects might be eligible)

My views regarding that would be:
 - Start with a basic MailQueue implementation. Could be contributed on
a feature branch / on an unused /server/queue/pulsar mvn project?
 - From there boostrap a custom MTA Guice assembling of James relying on
pulsar. Could it be in /exemple section?
 - Move forward with an EventBus implementation. Again could be
contributed on a feature branch / on an unused /eventbus/pulsar mvn project?
 - Make the custom MTA proof of concept evolve into a MDA (adding
mailbox / jmap bindings).
 - Implement advanced MailQueue features with Pulsar (delays, browse,
purge, flush, ...)
 - Bind the mailqueue management routes in the proof of concept server
 - Of course doing some performance tests for this...

(This proof of concept can likely still rely on RabbitMQ to back the
TaskManager implementation)

Thoughts?

Would there be people interested into mentoring this?

Cheers,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Tellier Benoit]

+1

Le 06/04/2021 à 14:04, Tellier Benoit a écrit :
> I would like to propose a new vote 3.6.0 release of the Apache James server.
> 
> After an initial mistake (sorry), a failed vote (could not achieve a PMC
> quorum), and more issues I did not yet complain about (closing the maven
> artifact #1051 failed for a missing MD5 signature - network issue?),
> here we are with one more vote !!!
> 
> You can find:
> 
> - The maven release staged in repository.apache.org as the artifact #1052:
> https://repository.apache.org/content/repositories/orgapachejames-1052/
> - The changelog for 3.6.0:
>  https://github.com/apache/james-project/blob/master/CHANGELOG.md
> - The compatibility instructions/upgrade recommendation:
>  
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> 
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Tuesday 6th of April 2021, 2pm UTC+7
>  - The vote ends at Tuesday 13th of March 2021, 2pm UTC+7
> 
> You can answer to it just with +1 and -1. Down-votes shall be motivated.
> 
> 3 binding votes are expected move forward on this release.
> 
> Cheers,
> 
> Benoit Tellier
> 
> Le 30/03/2021 à 17:16, Tellier Benoit a écrit :
>> The voting period is over.
>>
>> Only one upvote had been collected this time. We did not achieve the 3
>> PMC quorum, and other contributors did not vote as well.
>>
>> I will re-trigger a vote, this is a good opportunity to integrate
>> https://github.com/apache/james-project/pull/349
>>
>> Cheers,
>>
>> Benoit
>>
>> Le 22/03/2021 à 15:16, Tellier Benoit a écrit :
>>> Hi,
>>>
>>> I would like to propose a new vote 3.6.0 release of the Apache James server.
>>>
>>> You can find:
>>>
>>>  - The maven release staged in repository.apache.org as the
>>> artifact #1050:
>>> https://repository.apache.org/content/repositories/orgapachejames-1050/
>>>  - The changelog for
>>> 3.6.0:
>>> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>>>  - The compatibility instructions/upgrade
>>> recommendation:
>>> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
>>>
>>> Voting rules:
>>>  - This is a majority approval: this release may not be vetoed.
>>>  - A quorum of 3 binding votes is required
>>>  - The vote starts at Monday 22th of March 2021, 4pm UTC+7
>>>  - The vote ends at Monday 29th of March 2021, 4pm UTC+7
>>>
>>> You can answer to it just with +1 and -1. Down-votes may be motivated.
>>>
>>> 3 binding votes are expected move forward on this release.
>>>
>>> Cheers,
>>>
>>> Benoit Tellier
>>>
>>> -
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

[VOTE] Call for vote: Apache James 3.6.0 (round 3)

[Tellier Benoit]

I would like to propose a new vote 3.6.0 release of the Apache James server.

After an initial mistake (sorry), a failed vote (could not achieve a PMC
quorum), and more issues I did not yet complain about (closing the maven
artifact #1051 failed for a missing MD5 signature - network issue?),
here we are with one more vote !!!

You can find:

- The maven release staged in repository.apache.org as the artifact #1052:
https://repository.apache.org/content/repositories/orgapachejames-1052/
- The changelog for 3.6.0:
 https://github.com/apache/james-project/blob/master/CHANGELOG.md
- The compatibility instructions/upgrade recommendation:
 
https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version

Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Tuesday 6th of April 2021, 2pm UTC+7
 - The vote ends at Tuesday 13th of March 2021, 2pm UTC+7

You can answer to it just with +1 and -1. Down-votes shall be motivated.

3 binding votes are expected move forward on this release.

Cheers,

Benoit Tellier

Le 30/03/2021 à 17:16, Tellier Benoit a écrit :
> The voting period is over.
> 
> Only one upvote had been collected this time. We did not achieve the 3
> PMC quorum, and other contributors did not vote as well.
> 
> I will re-trigger a vote, this is a good opportunity to integrate
> https://github.com/apache/james-project/pull/349
> 
> Cheers,
> 
> Benoit
> 
> Le 22/03/2021 à 15:16, Tellier Benoit a écrit :
>> Hi,
>>
>> I would like to propose a new vote 3.6.0 release of the Apache James server.
>>
>> You can find:
>>
>>  - The maven release staged in repository.apache.org as the
>> artifact #1050:
>> https://repository.apache.org/content/repositories/orgapachejames-1050/
>>  - The changelog for
>> 3.6.0:
>> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>>  - The compatibility instructions/upgrade
>> recommendation:
>> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
>>
>> Voting rules:
>>  - This is a majority approval: this release may not be vetoed.
>>  - A quorum of 3 binding votes is required
>>  - The vote starts at Monday 22th of March 2021, 4pm UTC+7
>>  - The vote ends at Monday 29th of March 2021, 4pm UTC+7
>>
>> You can answer to it just with +1 and -1. Down-votes may be motivated.
>>
>> 3 binding votes are expected move forward on this release.
>>
>> Cheers,
>>
>> Benoit Tellier
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [BUILD-UNSTABLE]: Job 'james/ApacheJames-Website/PR-3 [PR-3] [5]'

[Tellier Benoit]

Hi Jean,

Le 02/04/2021 à 02:52, Jean Helou a écrit :
> Hi all,
> 
> Sorry about the spam, I opened a PR to try and improve the build flow for
> james documentation to try and get the staged documentation published
> automatically after each build of james project.

I think we need to update James site asf.yml to report github comments
in the JIRA worklog just like we do for other projects, this will
efficiently reduce the spam. I opened
https://github.com/apache/james-site/pull/6 regqrding this.

> 
> Unfortunately the Jenkinsfile currently in the staging branch of james-site
> is out of date (the JDK name has changed) and It took me these 5 tries to
> realize that none of the changes I was making to the Jenkinsfile were
> accounted for because:
> 
> ‘Jenkinsfile’ has been modified in an untrusted revision
> 
> 
> I'm not sure what the best way to proceed is : this limitation can be
> disabled (would be easiest since I may have further adjustments to make),
> or an official apache committer can push to an alternate PR to get it to
> build with the changes.
> Note that in this change I have also changed the email notifications to
> only notify the list for the live and staging branches and notify the
> requester for other branches.

I opened a branch  from my fork where I will mirror changes on your
branch. This will allow modifications to be trusted.

CF: https://github.com/apache/james-site/pull/5

BTW you have q groovy error to fix ;-)

> 
> best regards
> jean
> 
> On Thu, Apr 1, 2021 at 9:43 PM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
> 
>> BUILD-UNSTABLE: Job 'james/ApacheJames-Website/PR-3 [PR-3] [5]':
>>
>> Check console output at "https://ci-builds.apache.org/job/james/job/ApacheJames-Website/job/PR-3/5/;>james/ApacheJames-Website/PR-3
>> [PR-3] [5]"
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.6.0

[Tellier Benoit]

The voting period is over.

Only one upvote had been collected this time. We did not achieve the 3
PMC quorum, and other contributors did not vote as well.

I will re-trigger a vote, this is a good opportunity to integrate
https://github.com/apache/james-project/pull/349

Cheers,

Benoit

Le 22/03/2021 à 15:16, Tellier Benoit a écrit :
> Hi,
> 
> I would like to propose a new vote 3.6.0 release of the Apache James server.
> 
> You can find:
> 
>  - The maven release staged in repository.apache.org as the
> artifact #1050:
> https://repository.apache.org/content/repositories/orgapachejames-1050/
>  - The changelog for
> 3.6.0:
> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> 
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Monday 22th of March 2021, 4pm UTC+7
>  - The vote ends at Monday 29th of March 2021, 4pm UTC+7
> 
> You can answer to it just with +1 and -1. Down-votes may be motivated.
> 
> 3 binding votes are expected move forward on this release.
> 
> Cheers,
> 
> Benoit Tellier
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.6.0

[Tellier Benoit]

+1

I had been more careful this time. Sorry again for the inconvenience.

Le 22/03/2021 à 15:16, Tellier Benoit a écrit :
> Hi,
> 
> I would like to propose a new vote 3.6.0 release of the Apache James server.
> 
> You can find:
> 
>  - The maven release staged in repository.apache.org as the
> artifact #1050:
> https://repository.apache.org/content/repositories/orgapachejames-1050/
>  - The changelog for
> 3.6.0:
> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> 
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Monday 22th of March 2021, 4pm UTC+7
>  - The vote ends at Monday 29th of March 2021, 4pm UTC+7
> 
> You can answer to it just with +1 and -1. Down-votes may be motivated.
> 
> 3 binding votes are expected move forward on this release.
> 
> Cheers,
> 
> Benoit Tellier
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Call for vote: Apache James 3.6.0

[Tellier Benoit]

Hi,

I would like to propose a new vote 3.6.0 release of the Apache James server.

You can find:

 - The maven release staged in repository.apache.org as the
artifact #1050:
https://repository.apache.org/content/repositories/orgapachejames-1050/
 - The changelog for
3.6.0:
https://github.com/apache/james-project/blob/master/CHANGELOG.md
 - The compatibility instructions/upgrade
recommendation:
https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version

Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Monday 22th of March 2021, 4pm UTC+7
 - The vote ends at Monday 29th of March 2021, 4pm UTC+7

You can answer to it just with +1 and -1. Down-votes may be motivated.

3 binding votes are expected move forward on this release.

Cheers,

Benoit Tellier

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.6.0

[Tellier Benoit]

Sorry I messed up, built the wrong SHA-1.

Let's abort this vote,

Sorry for distrurbance.

Le 18/03/2021 à 20:36, Antoine Duprat a écrit :
> +1
> 
> Le jeu. 18 mars 2021 à 11:54, Tellier Benoit  a écrit :
> 
>> +1
>>
>> Le 18/03/2021 à 17:53, Tellier Benoit a écrit :
>>> Hi,
>>>
>>> I would like to propose a new vote 3.6.0 release of the Apache James
>> server.
>>>
>>>
>>> You can find:
>>>
>>>  - The maven release staged in repository.apache.org as the
>>> artifact #1049:
>>> https://repository.apache.org/content/repositories/orgapachejames-1049/
>>>  - The changelog for
>>> 3.6.0:
>>> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>>>  - The compatibility instructions/upgrade
>>> recommendation:
>>>
>> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
>>>
>>> Voting rules:
>>>  - This is a majority approval: this release may not be vetoed.
>>>  - A quorum of 3 binding votes is required
>>>  - The vote starts at Thursday 18th of March 2021, 6pm UTC+7
>>>  - The vote ends at Thursday 25th of March 2021, 6pm UTC+7
>>>
>>> You can answer to it just with +1 and -1. Down-votes may be motivated.
>>>
>>> 3 binding votes are expected move forward on this release.
>>>
>>> Cheers,
>>>
>>> Benoit Tellier
>>>
>>> -
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.6.0

[Tellier Benoit]

+1

Le 18/03/2021 à 17:53, Tellier Benoit a écrit :
> Hi,
> 
> I would like to propose a new vote 3.6.0 release of the Apache James server.
> 
> 
> You can find:
> 
>  - The maven release staged in repository.apache.org as the
> artifact #1049:
> https://repository.apache.org/content/repositories/orgapachejames-1049/
>  - The changelog for
> 3.6.0:
> https://github.com/apache/james-project/blob/master/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version
> 
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Thursday 18th of March 2021, 6pm UTC+7
>  - The vote ends at Thursday 25th of March 2021, 6pm UTC+7
> 
> You can answer to it just with +1 and -1. Down-votes may be motivated.
> 
> 3 binding votes are expected move forward on this release.
> 
> Cheers,
> 
> Benoit Tellier
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Call for vote: Apache James 3.6.0

[Tellier Benoit]

Hi,

I would like to propose a new vote 3.6.0 release of the Apache James server.


You can find:

 - The maven release staged in repository.apache.org as the
artifact #1049:
https://repository.apache.org/content/repositories/orgapachejames-1049/
 - The changelog for
3.6.0:
https://github.com/apache/james-project/blob/master/CHANGELOG.md
 - The compatibility instructions/upgrade
recommendation:
https://github.com/apache/james-project/blob/master/upgrade-instructions.md#360-version

Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Thursday 18th of March 2021, 6pm UTC+7
 - The vote ends at Thursday 25th of March 2021, 6pm UTC+7

You can answer to it just with +1 and -1. Down-votes may be motivated.

3 binding votes are expected move forward on this release.

Cheers,

Benoit Tellier

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Remove functionality that saves James metrics into Elasticsearch

[Tellier Benoit]

Hello Jean,

I do not know if you can do much about it, but your MUA did incorectly
wrapped the previous lines out its answers, quoting some of you replies
and unquoting some of mine. I found it complicated to read.

Le 15/03/2021 à 15:55, Jean Helou a écrit :
> Hi Benoit,
> 
> 
>> Regarding Prometheus setup, we have implemented an optional HTTP
>> endpoint. I would recommend you to include the following lines in your
>> webadmin.properties:
>>
>> extensions.routes=org.apache.james.webadmin.dropwizard.MetricsRoutes
>>
> 
> Thanks!  I'll have a look.
> 
> 
>> We should then:
>>  - better document this setup
>>  - (provide a docker-compose?)
>>  - Update the metric boards in /grafana-reporting
>>
> 
> Definitely since this is currently very hard to discover without being
> told. Nothing in the name remotely suggests that this is potentially
> compatible with Prometheus pulls and the name MetricsRoute doesn't seem to
> be mentioned anywhere in the documentation.
> 
> With that said my target deployment[1] would have separate instances for :
>  - relay SMTP ( with read access to the DB )
>  - webadmin (administrative VPN )
>  - mailbox and normal SMTP (user VPN, with read access to the user/domain
> db)
> because I like to keep my attack surfaces small and the relay must be
> exposed to the general internet to be of any use so the less code it runs
> the better.
> I will have to see if I can configure my assembly to run a
> restricted webmin with only the metrics routes.
> 
> So my ideal world would have documentation for both a pull and a push :p
> I'll try to contribute the push side if I manage to make it work
> 
> Would you agree with such an approach?

Yes, I will take care of documenting the PUSH.

>>
> 
> Definitely, as I said I don't like ES for metrics either. A better
> alternative is definitely ok.
> 
>> Having the metrics output to logs doesn't help much because of the amount
>>> of processing required to get anything useful out of it (or I failed to
>>> find how to easily ingest it please correct me)
>>
>> Logs for metrics had been introduced by Matthieu to get metrics out in
>> the logs of a performance test platform, without the need to analyses
>> the metrics "live".
>>
> 
> That's interesting information ! Not the part about matthieu (no offense to
> either of you), but the part about it not being intended for production
> monitoring and possibly useful for bench post-mortems. I would definitely
> like to see this mentioned in
> https://james.apache.org/server/metrics.html or maybe in
> https://james.apache.org/server/monitor-logging.html

+1

> We are moving away from JMX for the CLI, I see little reason to
>> encourage its use here too.
>>
>> See https://www.cvedetails.com/cve/CVE-2017-12628/
>>
> 
> I was under the impression that the prometheus jmx_exporter was deployed as
> an agent to be able to access the MBeans directly but it seems to be
> scraping from an exposed JMX tcp endpoint and I concur that this is not a
> good idea.
> 
> Glowroot is an APM, allowing to :
>>  - capture slow traces
>>  - capture DB queries (globally and on slow traces)
>>  - do some flame graphs captures (that might be more or less relevant
>> based on the number of captures performed)
> 
> 
> Yes and also
> * Responsive UI with mobile support
> * MBean attribute capture and **charts**
> * Configurable alerting
> * Historical rollup of all data (1m, 5m, 30m, 4h) with configurable
> retention
> not to mention a centralized collector which is why I said it overlaps
> quite a bit with prometheus

Sure, there's an overlap.

> 
> It provides some insight that regular monitoring tools will not give.
>>
> 
>> I would categorize it more as a developer diagnosis tool (which to me is
>> invaluable for performance issues).
>>
> 
> Do you run it in your production environments in addition to whatever
> metrics reporter you otherwise use or only on dev/bench ? Are there any
> good practices around it, is there any specific support for it in james ?

At Linagora, we did activate glowroot on internal James production
instances in order to learn as much as possible about the actual
behaviour. In addition to the regular metric tools. And it proved being
highly valuable.

I would advise contributors investigating performance issues to use it.

I would consider activating it everywhere overkill. For regular users,
this might not be needed unless some performance issue needs to be
investigated.

This should likely be advertised for a DEV audience only and moved out
of the README.

> Jean
> 
> [1] This is not currently the case in my terraform/helm recipe but it is my
> target.
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Remove functionality that saves James metrics into Elasticsearch

[Tellier Benoit]

Hello Matthieu,

Le 12/03/2021 à 21:14, Matthieu Baechler a écrit :
> Hi,
> 
> My answers below.
> 
> On Thu, 2021-03-11 at 15:17 +0200, Juhan Aasaru wrote:
>> Hi!
>>
>> There is currently ongoing work to upgrade James to Elasticsearch 7.x
>> See https://github.com/apache/james-project/pull/328
>>
>> Current James can be configured to save James metrics to a separate
>> index
>> in Elasticsearch.
>> And then Grafana dashboards can be configured to display the history
>> of
>> these saved metrics.
>> For more detailed documentation refer the configuration parameter
>> "elasticsearch.metrics.reports.enabled" here:
>> https://james.apache.org/server/config-elasticsearch.html
>>
>> This functionality that gathers and publish metrics is provided by an
>> unmaintained library:
>> https://github.com/linagora/elasticsearch-metrics-reporter-java
>> This library is not compatible with Elasticsearch 7.x
> 
> It doesn't support ES7, that's true. However, this support could be
> added.

If that is the way to go we then need to find someone devoting himself
for this maintenance...

> 
>>
>> We are proposing to remove this functionality from James as
>> the industry standard is to use external tools that are purpose built
>> for
>> pulling and storing the metrics.
> 
> I don't think it's the industry standard, it's prometheus standard. The
> debate never settled as far as I know.
> 
>>
>> Users currently relying on this functionality would have to configure
>> some
>> monitoring tool (like Prometheus) to regularly pull and store these
>> metrics
>> if they want to continue displaying history of various James related
>> metrics over time.
> 
> I don't like that idea.
> 
> If we talk about the root of the problem: dropwizard-metrics is quite
> rigit and we decided to include ElasticSearch appender in the product
> to have it working out of the box
> 
> Now, micrometer is doing much better: it works with an SPI much like
> slf4 is doing for logging and allows to choose your implementation by
> dropping the right jar.

Reading the documentation...

...about Prometheus integration we still end up with some custom code,
that really looks like the one we had written on top of dropwizard:
https://micrometer.io/docs/registry/prometheus

...looking at the Elastic integration, we still need to start a metric
registry https://micrometer.io/docs/registry/elastic

I fail at seeing the benefits compared to dropwizard metrics reporters...

A ctrl+f in https://micrometer.io/docs for the term "SPI" yields no
result, what makes you say "we just need to drop a JAR"?

> 
> I suggestion we just port James to micrometer and let people choose how
> they want their metrics exposed.
> 
> WDYT?

More flexibility on the metric side would be IMO nice to have.

I would be happy to see such contributions happens.

(Can you give a precise link explaining how and why you think micrometer
is more flexible than dropwizard?)

Benoit Tellier

> 
> -- Matthieu Baechler
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Remove functionality that saves James metrics into Elasticsearch

[Tellier Benoit]

Hello Jean,

Regarding Prometheus setup, we have implemented an optional HTTP
endpoint. I would recommend you to include the following lines in your
webadmin.properties:

extensions.routes=org.apache.james.webadmin.dropwizard.MetricsRoutes

We should then:
 - better document this setup
 - (provide a docker-compose?)
 - Update the metric boards in /grafana-reporting


Le 12/03/2021 à 21:06, Jean Helou a écrit :
> Hello all,
> 
> I agree that ES is not the best tool for metrics but it's implemented,
> documented and demonstrated
> - https://james.apache.org/server/metrics.html
> - https://james.apache.org/server/config-elasticsearch.html
> -
> https://github.com/apache/james-project/blob/master/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java#L141
> and that's only part of the mentions made throughout the documentation
> 
> So today it's the most discoverable[1] way to collect metrics for james
> especially for people who are not expert "ops". If this is removed, I feel
> documenting at least one recommended way to do it is necessary.

Let's document alternatives to the elasticsearch metric reporter. And
mark the ElasticSearch reporter as legacy.

The cost of maintaining two distinct ElasticSearch  clusters will likely
invite people to do the switch anyway...

Would you agree with such an approach?

> Having the metrics output to logs doesn't help much because of the amount
> of processing required to get anything useful out of it (or I failed to
> find how to easily ingest it please correct me) 

Logs for metrics had been introduced by Matthieu to get metrics out in
the logs of a performance test platform, without the need to analyses
the metrics "live".

> which leaves JMX.
> 
> Since JMX is a JVM standard, people who know the JVM well enough will
> either know or easily deduce that there is a generic way to export JMX
> metrics to whatever tool they want. However a mention of  jmx_exporter[2]
> for prometheus and jmxtrans [3] for other tools in the monitoring/jmx
> sections of the documentation would be kind to non-jvm people who stumble
> on apache james. Ideally having a demo setup/tutorial would be nice [4]

We are moving away from JMX for the CLI, I see little reason to
encourage its use here too.

See https://www.cvedetails.com/cve/CVE-2017-12628/

> 
> One other thing regarding monitoring is that the README mentions Glowroot
> [5] but there is no additional mention in the documentation. I am not
> an ops expert so I can't help but wonder how that fits with the ES exported
> metric ...
> The glowroot documentation mentions a lot of features some of which overlap
> with prometheus. The project doesn't seem very mature:  even if it was
> started 6 years ago, there have been very few contributions in the last
> year, the highest released version is 0.13.6 (james uses 0.13.4) and the
> documentation is quite sparse. I am not sure this is a good recommendation
> to put in the readme of the james project.

Glowroot is an APM, allowing to :
 - capture slow traces
 - capture DB queries (globally and on slow traces)
 - do some flame graphs captures (that might be more or less relevant
based on the number of captures performed)

It provides some insight that regular monitoring tools will not give.

I would categorize it more as a developer diagnosis tool (which to me is
invaluable for performance issues).

It do not replace a proper monitoring tool.

> 
> Jean
> 
> [1]
> https://www.google.com/search?q=monitoring+apache+james+metrics=monitoring+apache+james+metrics
> [2] https://github.com/prometheus/jmx_exporter
> [3] https://github.com/jmxtrans/jmxtrans
> [4] on that note, one reason I have been very quiet lately is that I am
> working on a helm chart and a terraform recipe to deploy james to a k8s
> cluster including all the associated software dependencies. I'm getting
> there and I intend to contribute this back to the project. One of the
> things I don't have yet is ... monitoring ;) Since k8s' monitoring tool of
> choice is prometheus that's what I intended to use but I never used it
> before and I'm only a casual contributor so I progress slowly

That looks cool!

> [5] https://glowroot.org/
> 
> 
> On Thu, Mar 11, 2021 at 2:35 PM Juhan Aasaru  wrote:
> 
>> If the community agrees then I propose the following.
>>
>> * we deprecate this functionality in the next release (3.6.0) with a note
>> that it would be completely removed in the next release (3.7.0)
>> * this functionality (James storing history of its metrics to
>> Elasticsearch) would continue to work with Elasticsearch 6 while rest of
>> the Elasticsearch-related functionality would be upgraded to Elasticsearch
>> 7
>> * anyone still using it would have to run two different Elasticsearch
>> instances (Elasticsearch 6 for history of metrics and Elasticsearch 7 for
>> rest of James)
>>
>> Juhan
>>
>>
>>
>> Kontakt Juhan Aasaru () kirjutas kuupäeval N, 11. märts
>> 2021 kell 15:17:
>>
>>> Hi!

Re: Remove functionality that saves James metrics into Elasticsearch

[Tellier Benoit]

+1 I am fine with both proposals

Maybe the documentation needs to be modified to clearly say it.

Benoit

Le 11/03/2021 à 20:34, Juhan Aasaru a écrit :
> If the community agrees then I propose the following.
> 
> * we deprecate this functionality in the next release (3.6.0) with a note
> that it would be completely removed in the next release (3.7.0)
> * this functionality (James storing history of its metrics to
> Elasticsearch) would continue to work with Elasticsearch 6 while rest of
> the Elasticsearch-related functionality would be upgraded to Elasticsearch 7
> * anyone still using it would have to run two different Elasticsearch
> instances (Elasticsearch 6 for history of metrics and Elasticsearch 7 for
> rest of James)
> 
> Juhan
> 
> 
> 
> Kontakt Juhan Aasaru () kirjutas kuupäeval N, 11. märts
> 2021 kell 15:17:
> 
>> Hi!
>>
>> There is currently ongoing work to upgrade James to Elasticsearch 7.x
>> See https://github.com/apache/james-project/pull/328
>>
>> Current James can be configured to save James metrics to a separate index
>> in Elasticsearch.
>> And then Grafana dashboards can be configured to display the history of
>> these saved metrics.
>> For more detailed documentation refer the configuration parameter
>> "elasticsearch.metrics.reports.enabled" here:
>> https://james.apache.org/server/config-elasticsearch.html
>>
>> This functionality that gathers and publish metrics is provided by an
>> unmaintained library:
>> https://github.com/linagora/elasticsearch-metrics-reporter-java
>> This library is not compatible with Elasticsearch 7.x
>>
>> We are proposing to remove this functionality from James as
>> the industry standard is to use external tools that are purpose built for
>> pulling and storing the metrics.
>>
>> Users currently relying on this functionality would have to configure some
>> monitoring tool (like Prometheus) to regularly pull and store these metrics
>> if they want to continue displaying history of various James related
>> metrics over time.
>>
>> Please respond if you agree or are against removing it.
>>
>> Kind regards
>> Juhan Aasaru
>>
>>
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Time for a 3.6.0 release

[Tellier Benoit]

Hi Matthieu, Hi Juhan,

Le 02/03/2021 à 20:36, Matthieu Baechler a écrit :
> Hi Juhan,
> 
> On Tue, 2021-03-02 at 11:10 +0200, Juhan Aasaru wrote:
> 
> [...]
> 
> We probably won't be able to support both versions in the same product
> for a reasonable cost so I propose to drop ElasticSearch 6 support
> entirely.

+1

> 
> Would it be possible to have a quick recap of the remaining efforts
> needed.
> One place for this could be the Jira task: 
> https://issues.apache.org/jira/browse/JAMES-3492
> 
> If the work required to finish is not too large I could get Andreas
> to come back and work on this starting this Friday. Hopefully this way 
> we have
> a chance to reach the release deadline (or at least have a second
> release
> shortly after the current on)
> 
> Let's define a deadline. That way, rather it's ready on time and
> included or not.

+1

> If you need some help to make it happen, you may find some people
> offering consulting, we are several to be able to do that on the
> mailing-list.
> 
> 
>  > the last release taking me over 3 months!
> 
> Benoit, could you please list the main problems why creating a release
> is time consuming so we could think solutions how some of this could be
> automated.

 - First, previous release had been rejected once. And one time where we
struggle to get the approval on time.
 - Second, the numerous maven project counts makes the upload (mvn
deploy / mvn release prepare / perform) long. I generally plan a day,
and do it on my laptop not to copy my GPG keys around. Which negatively
impact my other tasks.

To be fair, this one should be less of a problem to me this time.

 - Third: changing priorities...

> For example if PGP signing and publishing artifacts to Maven Central is
> time consuming then this could be automated in great deal.
> I created a JIRA issue for this automation initiative: 
> https://issues.apache.org/jira/browse/JAMES-3510
> 
> Thanks, good idea.
> 
> Regarding a release I have planned to raise a new topic that we as a 
> community
> could think about a "long-term-support" release of James. Currently any
> James release is more like
> just a point in time marker but probably many of us have a vision that 
> for a release we could create
> a separate branch and later only merge important security fixes there
> and then we could release patched versions like 3.6.1, 3.6.2 etc
> coming out in parallel with new main releases 3.7.0, 3.8.0 etc
> 
> I'm interested in getting this set up and working on getting the
> patches 
> identified and released
> but for this we would need to dramatically shorten the time and effort 
> it takes to create a (minor/major) release.
> So this is why I would come back to "long-term-support-version" a bit
> later.
> 
> If you want to handle that burden, that's awesome. I think nobody would
> be against having and LTS release for James.
> 
> Cheers,
> 
> -- Matthieu Baechler
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Time for a 3.6.0 release

[Tellier Benoit]

Hi Matthieu,

I think that the cut should be done on the later the 15/03, yes.

Regards,

Benoit

Le 02/03/2021 à 20:36, Matthieu Baechler a écrit :
> Hi Juhan,
> 
> On Tue, 2021-03-02 at 11:10 +0200, Juhan Aasaru wrote:
> 
> [...]
> 
>  > I think (sadly) ElasticSearch V7 migration (and ES V6 backport) will
> not
>  > be ready on time for this release, unless additional efforts are
>  > committed to this issue.
> 
> Since James guice version currently uses Elasticsearch 6.x that has 
> reached end of life
> and our system cannot go live with old Elasticsearch then we would be 
> very interested in getting
> this upgrade into a release (preferably into 3.6.0) and put in work in
> this.
> 
> I know my colleague Andreas has made an effort to add Elasticsearch 7 
> support
> in James and as I understand it currently the problem is to get all the
> tests to
> pass in Elasticsearch 7 related modules. But it is unclear to me what
> is 
> the plan
> to continue supporting Elasticsearch 6 in parallel.
> 
> We probably won't be able to support both versions in the same product
> for a reasonable cost so I propose to drop ElasticSearch 6 support
> entirely.
> 
> 
> 
> Would it be possible to have a quick recap of the remaining efforts
> needed.
> One place for this could be the Jira task: 
> https://issues.apache.org/jira/browse/JAMES-3492
> 
> If the work required to finish is not too large I could get Andreas
> to come back and work on this starting this Friday. Hopefully this way 
> we have
> a chance to reach the release deadline (or at least have a second
> release
> shortly after the current on)
> 
> Let's define a deadline. That way, rather it's ready on time and
> included or not.
> 
> If you need some help to make it happen, you may find some people
> offering consulting, we are several to be able to do that on the
> mailing-list.
> 
> 
>  > the last release taking me over 3 months!
> 
> Benoit, could you please list the main problems why creating a release
> is time consuming so we could think solutions how some of this could be
> automated.
> For example if PGP signing and publishing artifacts to Maven Central is
> time consuming then this could be automated in great deal.
> I created a JIRA issue for this automation initiative: 
> https://issues.apache.org/jira/browse/JAMES-3510
> 
> Thanks, good idea.
> 
> Regarding a release I have planned to raise a new topic that we as a 
> community
> could think about a "long-term-support" release of James. Currently any
> James release is more like
> just a point in time marker but probably many of us have a vision that 
> for a release we could create
> a separate branch and later only merge important security fixes there
> and then we could release patched versions like 3.6.1, 3.6.2 etc
> coming out in parallel with new main releases 3.7.0, 3.8.0 etc
> 
> I'm interested in getting this set up and working on getting the
> patches 
> identified and released
> but for this we would need to dramatically shorten the time and effort 
> it takes to create a (minor/major) release.
> So this is why I would come back to "long-term-support-version" a bit
> later.
> 
> If you want to handle that burden, that's awesome. I think nobody would
> be against having and LTS release for James.
> 
> Cheers,
> 
> -- Matthieu Baechler
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Time for a 3.6.0 release

[Tellier Benoit]

Hey there!

It's been almost a year (!) since we shipped some master code out in a
release (the last release taking me over 3 months!).

As such, I think it would be important, as a project, to make such a
release happen before the next Apache board meeting (mid. April). I
devote myself to perform such a release (let's hope I will be more
efficient this time!).

I would like to propose the following suggestion for this 3.6.0 release:

 - Claim experimental support for JMAP RFC-8621 and RFC-8887 (websocket
transport)
 - As such deprecate JMAP Draft as of 3.6.0, to be removed in 3.7.0. (or
later based on community usage) in order to encourage the migration.
 - Deprecate `server/container/guice/cassandra-guice` product.
Rationals: it do not achieve distribution, this James server cannot be
scaled and maintaining a high project cardinality is hard - at least to
me. The more servers, the harder releasing is.

I would also love to see the following work integrated in it:

 - https://issues.apache.org/jira/browse/JAMES-3506 SMTP stach is slow
and generate high GC when under high traffic (because it is such a nice
enhancement counter-weighting drawbacks of JAMES-3477)
 - https://github.com/apache/james-project/pull/303 JAMES-3499 Use a
module chooser for LDAP users repository
 (as it ease
maintainance efforts)

I think (sadly) ElasticSearch V7 migration (and ES V6 backport) will not
be ready on time for this release, unless additional efforts are
committed to this issue.

Thoughts?

Best regards,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Moving PRs against apache/james-project repository and server-dev signal to noise ratio

[Tellier Benoit]

Hello there,

I want to thank Jean and Matthieu for their awesome work on the Apache
run CI.

We did discuss it with the @linagora team during our daily, and we would
like to handle eventually all of our contributions through
apache/james-project.

Now the last few days experience makes me fear a JIRA message overflow
on this list, eventually burrying all discussions.

I personally find Github and its notifications to be enough to manage
community activities there. As such I would like to propose:

 - To keep JIRA emails on server-dev: I do use this a lot to monitor new
tickets being open.
 - However, github comments should IMO stop triggering JIRA comments
(that then trigger an email) as it is just flooding us.
 - But we should have the very useful JIRA -> Github link, and
automatically if possible.

According to
https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
we just need to turn
https://github.com/apache/james-project/blob/master/.asf.yaml into

  jira_options: link label

The worklog should even allow to have Github events on the ticket
without triggering emails. I would like to experiment with that option.

Thoughts?

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: The case of javax.mail MimeMessage CopyOnWrite optimization

[Tellier Benoit]

Hello Jean,

I do support merging https://github.com/apache/james-project/pull/282
along with https://github.com/apache/james-project/pull/286 as soon as
we get a green build.

I did finally succeed to run an SMTP benchmark against a simple workload
mixing small and big messages, the outcome was good, eventually
answering people wanting Gatling performance tests to complete JMH
benchmarks:
https://github.com/apache/james-project/pull/286#issuecomment-780221455

Cheers,

Benoit

Le 15/02/2021 à 14:58, Jean Helou a écrit :
> Hi everyone,
> 
> In light of benoit's latest comments on the related PRs, I want to report
> that I spent some time setting up an infra to do benchmarks against a full
> server (mx only) I'm not an ops at core, I don't know james all that well
> and spare time is limited and I got sidetracked quite a few times, so I
> still haven't run the benchmarks but I'm getting there.
> 
> I am strongly in favor of integrating at least #282.
> 
> #285 is nice to have for me but I won't have time to investigate the config
> file option rapidly.
> 
> Jean
> 
> Le jeu. 7 janv. 2021 à 18:04, Raphaël Ouazana  a
> écrit :
> 
>> Hi,
>>
>> I completely agree with what Matthieu said.
>>
>> Correctness is needed, so we have to fix this issue.
>>
>> But before merging on master and/or deploying this version I would
>> expect some Gatling runs to show which potential performance decrease we
>> can expect on real use cases.
>>
>> Thanks,
>>
>> Raphaël.
>>
>> Le 04/01/2021 à 14:42, Matthieu Baechler a écrit :
>>> Hi there,
>>>
>>> Thank you for bringing this topic to the mailing-list.
>>>
>>> To me safety and correctness is much more important than raw
>>> performance so I would like the always-copy implementation to replace
>>> the COW version.
>>>
>>> However, keep in mind that the JMH benchmark figures did not told the
>>> full story about the consequences of this change and be ready to
>>> experience slower real-world performances.
>>>
>>> Cheers,
>>>
>>> -- Matthieu Baechler
>>>
>>> On Tue, 2020-12-29 at 12:54 +0700, Tellier Benoit wrote:
>>>> Hello there!
>>>>
>>>> We had been discussing on GitHub recently about an optimization in
>>>> james
>>>> core around the usage of MimeMessage.
>>>>
>>>> Javax.mail MimeMessage is currently used to represent a message of an
>>>> email as part of the mail processing in James. It is part of the Mail
>>>> interface (mailet-api).
>>>>
>>>> As Mail envelope is composed of several recipients, mail related
>>>> operations are performed once for all these recipients (we enqueue
>>>> the
>>>> mail one time, we strip bcc one time etc...). Troubles arise when we
>>>> need different behaviors as part of mail processing across recipients
>>>> (think remote recipients, that needs there mail to be relayed, versus
>>>> local recipients that needs to be locally delivered). The email get's
>>>> duplicated (in MatcherSplitter) and the processing will then be
>>>> distinct
>>>> for both entities. The underlying MimeMessage may - or may not be
>>>> modified.
>>>>
>>>> In order to prevent MimeMessage duplication in the event the
>>>> underlying
>>>> MimeMessage is not modified, a Copy On Write mechanism was introduced
>>>> (I
>>>> guess... Sorry, I was not there yet).
>>>>
>>>> Upon his CI effort, Jean Helou with the help of Matthieu Baechler
>>>> made
>>>> he unpleasant finding that this was not thread safe, that was leading
>>>> to
>>>> build instabilities. The mailet processing happens in Camel, which is
>>>> multi-threaded. Concurrency issues arised between modifications, and
>>>> message disposal, when a same MimeMessage instance was shared. [1]
>>>>
>>>> A first effort was to try to achieve thread-safety, which leaded to a
>>>> brittle double reetrant read-write locks in order to govern data
>>>> access.
>>>> However, another performance enhancement bypassed these lock
>>>> mechanism
>>>> (MimeMessageWrapper allows accessing the data as an InputStream
>>>> instead
>>>> of requiring to copy it). The effort seemed overwhelming, not to
>>>> metion
>>>> possible risks of dead-locks. [2]
>>>>
>>>> We then came up with an always cop

Re: The case of javax.mail MimeMessage CopyOnWrite optimization

[Tellier Benoit]

Of course.

I created https://issues.apache.org/jira/browse/JAMES-3487

Cheers,

Benoit

Le 29/12/2020 à 15:19, Jean Helou a écrit :
>>
>> Please note that above 100KB the MimeMessage would be stored on disk,
>> thus limiting memory impact (see MimeMessageInputStreamSource). Maybe we
>> should make the threshold configurable, via a system property for instance?
>>
> 
> Having the threshold configurable seems like a good idea to me (I know I
> would like it :)) but it can come later since it is already the status quo.
> To me fixing the concurrency issue in such a core component has priority.
> 
> Jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

The case of javax.mail MimeMessage CopyOnWrite optimization

[Tellier Benoit]

Hello there!

We had been discussing on GitHub recently about an optimization in james
core around the usage of MimeMessage.

Javax.mail MimeMessage is currently used to represent a message of an
email as part of the mail processing in James. It is part of the Mail
interface (mailet-api).

As Mail envelope is composed of several recipients, mail related
operations are performed once for all these recipients (we enqueue the
mail one time, we strip bcc one time etc...). Troubles arise when we
need different behaviors as part of mail processing across recipients
(think remote recipients, that needs there mail to be relayed, versus
local recipients that needs to be locally delivered). The email get's
duplicated (in MatcherSplitter) and the processing will then be distinct
for both entities. The underlying MimeMessage may - or may not be modified.

In order to prevent MimeMessage duplication in the event the underlying
MimeMessage is not modified, a Copy On Write mechanism was introduced (I
guess... Sorry, I was not there yet).

Upon his CI effort, Jean Helou with the help of Matthieu Baechler made
he unpleasant finding that this was not thread safe, that was leading to
build instabilities. The mailet processing happens in Camel, which is
multi-threaded. Concurrency issues arised between modifications, and
message disposal, when a same MimeMessage instance was shared. [1]

A first effort was to try to achieve thread-safety, which leaded to a
brittle double reetrant read-write locks in order to govern data access.
However, another performance enhancement bypassed these lock mechanism
(MimeMessageWrapper allows accessing the data as an InputStream instead
of requiring to copy it). The effort seemed overwhelming, not to metion
possible risks of dead-locks. [2]

We then came up with an always copy implementation [3]. Simpler,
safer... The underlying logic is to avoid trying being smarter than
mutability, and leverage immutability to achieve thread safety, which is
a classic functional programming idiom.

JMH benchmarks were conducted. We highlighter little performance
difference for small messages, in the percent realm for both memory
allocation and compute time. Differences are however higher for bigger
messages (~10%) for both metrics.

Please note that above 100KB the MimeMessage would be stored on disk,
thus limiting memory impact (see MimeMessageInputStreamSource). Maybe we
should make the threshold configurable, via a system property for instance?

I just want to further mentioned I encountered that very issue on a
production instance: the underlying email had been corrupted by the
above mentioned COW bug and kept throwing NullPointerExceptions every
time the content was accessed. This resulted (on top of
distributed-james) in a RabbitMQ nack of the message, that ended up in a
dead-letter queue. Replaying its processing required admin intervention
and had been interpreted by the user as an email loss...

To conclude this effort we (Jean an I) would like to merge the "Always
copy" pull request.

Also, would it be beneficial to write an ADR about this topic?

Thoughts?

Cheers,

Benoit

[1] The unfamous COW bug:
https://github.com/apache/james-project/pull/280/commits/09b5554bbcbbb98757910d59bac54f97ee1f8b4f

[2] The double nested reetrant read-write lock fix attempt:
https://github.com/apache/james-project/pull/280
[3] The always copy fix: https://github.com/apache/james-project/pull/282
[4] Benchmarks:
https://github.com/apache/james-project/pull/280#issuecomment-745211736
& https://github.com/apache/james-project/pull/280#issuecomment-745701937
[5] The JIRA ticket: https://issues.apache.org/jira/browse/JAMES-3477


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Jenkins CI setup

[Tellier Benoit]

Hello Jean,

Nice work!

Le 28/12/2020 à 23:21, Jean Helou a écrit :
> Hello again jamers!
> 
> It's time for a new irregular report on the CI effort on apache infra  !
> 
> Let's start with the good news : today I finally reached a successful build
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/45/pipeline
> (the first fully successful build on apache infra)

\o/

> 
> You can see in the pipeline that as discussed before the testing phase is
> split in 2 parts: stable tests vs unstable tests, failure in the first
> phase will fail the build, failures in the unstable phase will not be
> considered a build failure (but should still collect the failed tests in
> the reports, however the recent failures where mostly memory related in
> which case the surefire report is not generated :( )
> 
> Over the last 2 weeks and 45 build attempts, I tagged all failing tests as
> "Unstable", I also increased the heap in the forked surefire to resolve
> some of the OutOfMemoryException failures
> 
> At this stage I would really like to see this merged (if only to be able to
> evaluate dangerous changes such as
> https://github.com/apache/james-project/pull/282)

+1 I think this however deserves a separate thread. I will start it now.

> 
> You can look at https://github.com/apache/james-project/pull/268 to see
> which tests have been marked as Unstable. It was rebased on master this
> morning and I intend to clean up the history tonight.

Will do, maybe tomorrow. On the principles, it is a yes from my side.

As someone operating another CI, I want to play even unstable test on
every runs. Is there some adaptation needed to do this?

> I also removed some invasive logging from the webadmin test code (it used
> to log every single http request made in the tests) the full log is still a
> bit over 30MB...

Nice enhancement, it is likely some long-forgotten debug statements.
> 
> Best regards,
> Jean
> 
> On Fri, Dec 11, 2020 at 12:25 PM Jean Helou  wrote:
> 
>> I conclude that my effort to get CI working is cursed by the gods,
>> remember :
>>
 {"message":"No such image: quay.io/testcontainers/ryuk:0.2.3"}
>>>
>>> which repeats for most tests failures, this seems to be common enough
>>> that there is stack overflow for it
>>>
>>> https://stackoverflow.com/questions/61887363/testcontainers-cant-pull-ryuk-image-quay-io-is-not-reachable
>>> I have attempted to upgrade test containers to 1.15.0 (as it will pull
>>> ryuk from docker hub instead of quay.io since 1.14.3 and we were using
>>> 1.12)
>>> hopefully this will help :)
>>>
>>
>> A docker API change broke most of testcontainers versions, which won't be
>> able to pull the images if they are not already available locally !
>> https://github.com/testcontainers/testcontainers-java/issues/3574
>>> yes, this Docker API change applies to most of Testcontainers versions.
>>
>> They should release  a 1.15.1 to resolve the issue shortly, I have tried
>> explicitly pulling the image in the steps of running the tests but sadly it
>> doesn't seem to have helped :(
>>
>> jean
>>
>>>
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: JMAP vacation mailet

[Tellier Benoit]

Hello Jean,

Sorry for the late answer.

Le 27/12/2020 à 01:29, Jean Helou a écrit :
> Hi,
> 
> While playing around with assemblies, I got an error that vacation mailet
> configuration was missing when I added the JMAP module
> https://github.com/apache/james-project/blob/master/server/container/guice/protocols/jmap/src/main/java/org/apache/james/jmap/draft/JMAPModule.java#L122

This check was added to ensure the mailetcontainer was correctly
configured to answer people upon vacations.

The JMAP module will require this check, as far as I remember even if
the "JMAP protocol" itself is disabled (likely these checks should not
be performed if the configuration disables the JMAP protocol?)

On the subject of those checks, I personally do not like them yet...
It's a constraint but they are easy to cheat, the security they provide
is limited, but it restrict what as an administrator I could do... Like
re-organizing mailetcontainer.xml to perform a single call to
RecipientIsLocal (and thus to the LDAP) which has a performance and
resiliency impact.

> 
> I'm not very familiar with jmap and checked a draft at
> https://tools.ietf.org/html/draft-ietf-jmap-mail-16#section-1.3
> As far as I understand the vacation mailet it is an additional capability
> and could thus be made optional, am I correct ?

You are right.

Now, does it makes sense to expose the JMAP vacation extension in a
separate Guice modules? Is there some cases we want JMAP but not the
vacations?

By the way allowing people to write their own custom JMAP extensions
might become soon a topic, hopefully ;-)

> 
> thanks
> jean
> 

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Jenkins CI setup

[Tellier Benoit]

Le 11/12/2020 à 16:35, Jean Helou a écrit :
 I'm in favor for opening a dedicated ticket and merge a disabled version
 of this test in order to document the problem.

>>>
>>> it"s been busy and I haven't opened the ticket yet, nor have we managed
>> to
>>> fully fix the issue yet
>>>
>>
>> I can devote some of my time to support you on this.
>>
> 
> Thanks, if you can open the jira issue I think I provided the relevant
> information in my previous mail, and if you feel lucky you can take a look
> at the branch I referred to the test demonstrating the issue is still there
> :)
+1

I sadly already did unsuccessful tries with read/write locks instead of
synchronize blocks, it do not seem to help...
> 
> [...]

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Tellier Benoit]

You're right,

I did not follow up on this.

I just created the INFRA ticket.
https://issues.apache.org/jira/browse/INFRA-21180

As far as I understand, this is not an official release channel but
rather a convenience one.

Cheers,

Benoit

Le 11/12/2020 à 15:18, Jean Helou a écrit :
>>
>> I propose to ask for rights on apache/james repository
> 
> 
> I'm revisiting this thread after seeing
> https://www.docker.com/blog/supporting-open-source-projects-at-docker which
> reminded me of this effort and I was wondering if an INFRA ticket has been
> opened for james (of if there is a trace of the progress somewhere)
> I'm not sur exactly how the program works but if it whitelists prefixes,
> Apache is most likely already in it however in light of the work on CI
> builds on apache infrastructure, I am wondering if/how the credentials for
> pushing form CI can be setup and I was hoping to drop a comment on the
> corresponding ticket.
> 
> jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Jenkins CI setup

[Tellier Benoit]

Hello Jean,

Le 11/12/2020 à 15:47, Jean Helou a écrit :
> Hello again jamers!
> 
> It's time for your irregular report on the CI effort on apache infra :)

\o/

>> I'm in favor for opening a dedicated ticket and merge a disabled version
>> of this test in order to document the problem.
>>
> 
> it"s been busy and I haven't opened the ticket yet, nor have we managed to
> fully fix the issue yet
> 

I can devote some of my time to support you on this.

> 
>>> Here is what I would like to do at this stage :
>>> - Isolate the unstable tests under with an unstable tag (akin to "feature
>>> tags")
>>> - exclude these tests from the default surefire execution profile,
>>> - add a parallel pipeline step for these tests where the step failure
>>> doesn't fail the pipeline [2]
>>> - ensure that the build is green
>>> - merge so the project finally has a working public CI
>>>
>>> I intend to start working on this quickly so we can all enjoy a
>> functional
>>> public CI.
>>
> 
> So I added an `Unstable.Tag` and started tagging the known unstable tests,
> it seemed that running the pipeline in parallele led to more issues
> so I reverted the parallel run to a serial run for now. I also changed from
> fail at first error to fail at end to get an idea of the volume of unstable
> tests.

+1

> 
>> I'd advocate a @Disabled tag, referencing both a JIRA ticket specific to
>> the bugfix needed, and the JIRA of the CI build.
>> Having a list of such issues in the JIRA (CI setup) ticket would be
>> valuable. I'd even advise doing subtickets to have a nice checklist.
> 
> 
> Despite Matthieu' remarks I was open to create the tickets and add the
> information to the Unstable.TAG or as a comment next to the tag.
> 
> However the recent CI results make the effort feel overwhelming :
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/7/tests
> : failures 4 new, 33 existing, 27 fixed, total 37
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/6/tests
> : failures 27 new, 27 existing, 0 fixed, total 54
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/5/tests
> : failures 0 new, 8 existing, 6 fixed, total 8
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/4/tests
> (issue with the jenkins file the build did not run)
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/3/tests
> : 4 fixed, 92 existing failures (that uses parallel to run both stable and
> unstable)
> -
> https://builds.apache.org/blue/organizations/jenkins/james%2FApacheJames/detail/PR-268/2/tests
> : 92 existing failures (that uses parallel to run both stable and unstable)
> 
> Now the last 2 runs where triggered after I rebased on master.
> I will trigger the next 3 by modifying random comments in the jenkins file
> to see if the build has a reproductible failure pattern or if all these
> need to be tagged as Unstable.
> Thats a lot of failures some of which don't even make sense to me, in run 7
> the first of the 4 new failures is:
> ```
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryTest
> at
> org.apache.james.user.ldap.ReadOnlyUsersLDAPRepositoryInvalidDnTest.setUp(ReadOnlyUsersLDAPRepositoryInvalidDnTest.java:62)
> ```
> and quite a few of the errors listed are similar NoClassDefFound errors
> which I quite fail to understand ... I would very much welcom feedback if
> one of you encountered this kind of issues before, it feels like I am
> missing something :(

We use a static singleton approach in order for testcontainers docker
containers to be initialised once per surefire fork and not once per
test class. Combined wih a reuseForks=true setting this dramatically
reduce testing time!

The only downside in cryptic NoClassDefound errors if the given docker
container can't start.

To be noted that:
 - some tests reuse existing images
 - some tests like the LDAP one uses a Dockerfile to build their own
image. This is likely the source of some instability.

I also saw Cassandra containers failing initialization in memory
constraint environments (eg on my laptop if less than 5GB are available).

Not sure if it helps.
Not sure if we can hope for changes in the docker environment.

A quick action could be to decrease surefire concurrency in some
projects using expensive-to-start containers (like mailbox/cassandra,
mpt/impl/imap/cassandra, etc...). If we can use en ENV variable for this...

Do give some insight, at linagora we run 1 build only per jenkins slave,
which have a dedicated physical host. I doubt we have such things in the
Apache environment.

Hope it helps.

> 
>> Having a build in the first place, even with the restrictions you
>> describe sounds like a good progress to me.
>>
> 
> As it stands it looks like it's going to take a bit longer :(
> 

Cheers,

Benoit

Re: James requires administrative rights on RabbitMQ (!!!)

[Tellier Benoit]

Le 11/12/2020 à 14:53, Matthieu Baechler a écrit :
> Hi,
> 
> On Fri, 2020-12-11 at 12:42 +0700, Tellier Benoit wrote:
>> Hello James DEVs !!!
>>
>> I want to start a discussion around
>> https://issues.apache.org/jira/browse/JAMES-3475
>>
>> Our issue is that James so far require administrative rights on
>> RabbitMQ
>> server.
>>
>> This of course means that sharing this RabbitMQ with other apps /
>> James
>> servers of other tenant represent a data isolation / security issue,
>> that we leverage by giving James his own dedicated RabbitMQ server,
>> which don't help mutualizing costs.
>>
>> Thus, I would like to leverage Cassandra to keep track of created
>> queues.
>>
>> This is a task that could be quickly tackled by Quan our intern, who
>> wants to learn about NoSQL. This could be a very good sandbox issue
>> for him.
>>
>> Feedback?
> 
> What about using RabbitMQ virtualhost feature instead?
> 
> https://www.rabbitmq.com/vhosts.html

Thanks. Needless to say, we already use that at the AMQP level.

I was unaware VHost restrictions did apply in the management plugin too.

I apologize, my argument is not the right one.

 - Management API access freaks admins out - at least the one I know.
Stopping supporting it will calm them down ;-)

You know, these people have weird checklists, especially external
auditors might not be much open to debate ;-)

 - Actually the management API is *so unstable* that we need a retrier
on it:

0135291 JAMES-2544 Use Feign retry mechanism to retry calls on RabbitMQ
management API

(I agree the commit miss a nice explanation message)

This setup relying on retires (not working without retries!) do not seem
like a "production-ready" thing.

I'd like to enhance this matter of fact!

Cheers,

Benoit

> 
> Cheers,
> 
> -- Matthieu Baechler
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

James requires administrative rights on RabbitMQ (!!!)

[Tellier Benoit]

Hello James DEVs !!!

I want to start a discussion around
https://issues.apache.org/jira/browse/JAMES-3475

Our issue is that James so far require administrative rights on RabbitMQ
server.

This of course means that sharing this RabbitMQ with other apps / James
servers of other tenant represent a data isolation / security issue,
that we leverage by giving James his own dedicated RabbitMQ server,
which don't help mutualizing costs.

Thus, I would like to leverage Cassandra to keep track of created queues.

This is a task that could be quickly tackled by Quan our intern, who
wants to learn about NoSQL. This could be a very good sandbox issue for him.

Feedback?

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: About our usage of LWT in Cassandra related code

[Tellier Benoit]

Le 09/12/2020 à 12:58, Jean Helou a écrit :

>  > Also for ACLs isn't eventual consistency acceptable ?
> 
> 
>> My take is that it is. My first shot was to do just that.
>>
>> Howver the current code enforces a "transaction like" behavior, with an
>> ACLDiff fired on the mailbox event system. Maintaining denormalization
>> was also an issue in the face of reset (requiring to know previously
>> stored data).
>>
>> A full rewritte of the context ACLs are run in would have been needed.
>> And could have been controversial.
>>
>>> using transactions to
>>> avoid non serial writes but accepting stale reads ?
>>
>> Could be too. This could be a quick-win to enhance existing situation,
>> while waiting for stronger decisions and
> 
> What you describe could be a very good short term solution for message
>> flags (I think we will need to challenge CONDSTORE/QRSYNC anyway!)
>>
> 
> Why do you say this would only be a good short term solution ?

On the key table, here is what nodetool tablestat states on one of the
production instances I <3 :

Table: imapuidtable
SSTable count: 3
Space used (live): 472544882
Space used (total): 472544882
SSTable Compression Ratio: 0.40981487390641924
Number of partitions (estimate): 6486085
Local read count: 15793743
Local read latency: 0.233 ms
Local write count: 9171561
Local write latency: 0.037 ms

What you see is that ~40% of the workload on key messages metadata is
writes.

 - IMAP SELECT reset the RECENT flag
 - SEEN flags is updated for most messages
 - People move / delete their messages

That's why I say turning off SERIAL reads on this is a partial solution
that we can activate (via a configuration option to make every body
happy) on the short term.

LWT on writes will stay a problem.

> My understanding is that currently
> - most reads and writes use LWT and that using LWT implies slow writes (to
> something akin to a lock I  suppose) for both read and write operations

Some, those against the table that acts as a source of truth.

> - the amount of reads far outweighs the number of writes (at least for the
> listed use cases of ACLs, users and domains, it sounds like the UID/Flags
> stuff may not be so clear cut)

This is true for acl and mailbox, not for messages hence the different
solution.

> 
> In such a context dropping read consistency while keeping write consistency
> sounds like it would already be a huge gain :) The assertions in a number
> of tests will likely have to be updated to rely on `Awaitility.await()` :)

+1, that would be an enhancements.

> 
> 
> On Wed, Dec 9, 2020 at 2:40 AM Tellier Benoit  wrote:
> 
>> Sorry for repost,
>>
>> I sent that response before but it was lost.
>>
>> Maybe the unfamous text/html format issue.
>>
>> Le 07/12/2020 à 04:33, Jean Helou a écrit :
>>> Hello,
>>>
>>> I'm currently trying to increase overall efficiency of the Distributed
>>>> James server.
>>>>
>>>
>>> I have some concerns but i feel imposterish for posting them as they most
>>> likely come from my own lack of knowledge, i'll still try just in case
>> some
>>> of points are valid :)
>>
>> Thank you very much to dare to do so!
>>
>> You are likely not the only one to lack this knowledge, hopefully
>> discussions will clarify that.
>>
>> I also sometime have problems to express myself clearly, thus
>> explanation are normal.
>>
>>>
>>>  - `users` we rely on LWT for throwing "AlreadyExist" exceptions. LWT
>>>> are likely unnecessary as the webadmin
>>>> presentation layer is offering an idempotent API (and silents the
>>>> AlreadyExist exceptions). Only the CLI
>>>
>>> (soon to be deprecated for Guice products) makes this distinction.
>>>
>>>
>>> So from a user perspective adding a user would always succeed. But would
>> it
>>> succeed by doing nothing (the current behaviour in silencing the
>>> AlreadyExist exception) or would it succeed by effectively overwriting
>> the
>>> user (in a last write wins manner) ?
>>
>> Webadmin so far overwrite the user (and its password) in a last write
>> win manner.
>>
>>> This is a completely different
>>> behaviour which is not necessarily desirable.
>>> this can be further divided into 2 different cases :
>>> - there are concurrent attempts to create the same user (in which case
&g

Re: About our usage of LWT in Cassandra related code

[Tellier Benoit]

I'll answer other threads separately.

I created https://issues.apache.org/jira/browse/JAMES-3468 about this.

I asked Quan, our intern, to take a look.

Regards,

Benoit Tellier

Le 09/12/2020 à 12:58, Jean Helou a écrit :
>>> So from a user perspective adding a user would always succeed. But would
>> it
>>> succeed by doing nothing (the current behaviour in silencing the
>>> AlreadyExist exception) or would it succeed by effectively overwriting
>> the
>>> user (in a last write wins manner) ?
>>
>> Webadmin so far overwrite the user (and its password) in a last write
>> win manner.
>>
> 
> That sounds really scary
> 
> 
>>  - Either we need to distinguish "create" from "update" within the
>> webadmin API
>>
> 
> Well  that would definitely have my vote : as an admin operator I *never*
> want to accidentally overwrite an existing user when trying to create a new
> one (with the possible exception of retrying a create operation that just
> timeouted, in which case my first reflex would be to execute a read to try
> and make sure that the operation that just failed hasn't actually succeeded)

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: About our usage of LWT in Cassandra related code

[Tellier Benoit]

Sorry for repost,

I sent that response before but it was lost.

Maybe the unfamous text/html format issue.

Le 07/12/2020 à 04:33, Jean Helou a écrit :
> Hello,
> 
> I'm currently trying to increase overall efficiency of the Distributed
>> James server.
>>
> 
> I have some concerns but i feel imposterish for posting them as they most
> likely come from my own lack of knowledge, i'll still try just in case some
> of points are valid :)

Thank you very much to dare to do so!

You are likely not the only one to lack this knowledge, hopefully
discussions will clarify that.

I also sometime have problems to express myself clearly, thus
explanation are normal.

> 
>  - `users` we rely on LWT for throwing "AlreadyExist" exceptions. LWT
>> are likely unnecessary as the webadmin
>> presentation layer is offering an idempotent API (and silents the
>> AlreadyExist exceptions). Only the CLI
> 
> (soon to be deprecated for Guice products) makes this distinction.
> 
> 
> So from a user perspective adding a user would always succeed. But would it
> succeed by doing nothing (the current behaviour in silencing the
> AlreadyExist exception) or would it succeed by effectively overwriting the
> user (in a last write wins manner) ? 

Webadmin so far overwrite the user (and its password) in a last write
win manner.

> This is a completely different
> behaviour which is not necessarily desirable.
> this can be further divided into 2 different cases :
> - there are concurrent attempts to create the same user (in which case the
> user data is very likely the same or very close, and has possibly never
> been exposed to a human) in which case the LWW behaviour may be acceptable
> - A user has existed for a long time (definition of long to be defined but
> I would say above a few seconds :) ) in which cas overwriting is most
> likely not acceptable
> 

My take is that we need to make a choice:

 - Either we need to distinguish "create" from "update" within the
webadmin API
 - Or we relax the condition downstream.

I would be in favor of handling conflict as part of the WebAdmin API.

Moreover what you say regarding conflict is very interesting. It suggest
strong consistency might not be needed. A simple "read before your
write" would be enough to see if a long standing user would be updated,
while not dealing with conflict for newly created user.

> 
>>  - `domains` we rely on LWT for throwing "AlreadyExist" exceptions. LWT
>> are likely unnecessary as the webadmin
>> presentation layer is offering an idempotent API (and silents the
>> AlreadyExist exceptions). Only the CLI
>> (soon to be deprecated for Guice products) makes this distinction.
>> Discussions have started on the topic and a proof of
>> concept is available.
>>
> 
> same as above
>


No not really, as domains cary no other information (like user carry a
password).

You don't have the risk to grant access to somebody else account by mistake.

> Why it would be ok to drop LWT for ACL updates only to replace it by
> eventsourcing when you write:
>> LWT are required for `eventSourcing`. As event sourcing usage is limited
> to low-usage use cases, the performance degradations are not an issue.
> Doesn't that mean that ACLs would still rely on LWT but within an
> additional layer ?

Yes. Writes are resolved against event sourcing system using LWT.

Reads are resolved against a projection, free of LWT, maintaines via
subscribers to the event sourcing system.

> Also for ACLs isn't eventual consistency acceptable ? 

My take is that it is. My first shot was to do just that.

Howver the current code enforces a "transaction like" behavior, with an
ACLDiff fired on the mailbox event system. Maintaining denormalization
was also an issue in the face of reset (requiring to know previously
stored data).

A full rewritte of the context ACLs are run in would have been needed.
And could have been controversial.

> using transactions to
> avoid non serial writes but accepting stale reads ?

Could be too. This could be a quick-win to enhance existing situation,
while waiting for stronger decisions and

Here we might need a distinction between reads made as part of the
update process (that are required to be up to date, so need to be
SERIAL!) and regular reads that are acceptable to be stale (and can be
made using QUORUM).

What you describe could be a very good short term solution for message
flags (I think we will need to challenge CONDSTORE/QRSYNC anyway!)

> 
> That's the limit of my understanding : all the flags/UID/IMAP concerns are
> beyond my current knowledge but I'll enjoy reading the comments :)
> 
> jean
> 

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: About our usage of LWT in Cassandra related code

[Tellier Benoit]

Hello Matthieu,

Sadly, I'm unable to see what you did write in the email you sent due to
the absence of quote.

Can you review your email client settings, in order to get a readable
output we can start discussing on?

This time, I made the effort, but I would greatly appreciate a better
display.

Best regards,

Benoit

Le 07/12/2020 à 14:47, Matthieu Baechler a écrit :
> Hi Benoit,
> 
> On Fri, 2020-12-04 at 14:22 +0700, btell...@linagora.com (OpenPaaS) wrote:
> Hi,
> 
> I'm currently trying to increase overall efficiency of the Distributed
> James server.
> 
> As such, I'm pocking around for improvement areas and found a huge
> topic
> around LWT.
> 
> My conclusions so far are that we should keep LWT and SERIAL
> consistency
> level out of the most common use cases.
> 
> I know that this is a massive change in regard of the way the project
> had been working with Cassandra in the past few years. I would
> definitely, in the middle term, would like to reach LWT free reads on
> the Cassandra Mailbox to scale the deployments I am responsible of as
> part of my Linagora job (my long term goal being to decrease the total
> cost of ownership of a "Distributed James" based solution). While I am
> not opposed to diverge from the Apache James project on this point, if
> needed, I do believe an efficient distributed server (with the
> consequences it implies in term of eventual consistency) might be a
> strong asset for the Apache project as well, and would prefer to see
> this work lending on the James project.
> 
> I've been ambitious on the ADR writing, especially in the complementary
> work section. Let's see which consensual ground we find on that! (the
> ML
> version here below serving as a public, immutable reference of my
> thinking!)
> 
> 
> I doubt we can model IMAP without serializability somewhere but let's
> read your proposal as I have LWT as much as you are.

s/have/hate/ ?

> 
> 
> ---
> 
> ## Context
> 
> As any kind of server James needs to provide some level of
> consistencies.
> 
> Strong consistency can be achieved with Cassandra by relying on
> LightWeight transactions. This enables
> optimistic transactions on a single partition key.
> 
> Under the hood, Cassandra relies on the PAXOS algorithm to achieve
> consensus across replica allowing us
> to achieve linearizable consistency at the entry level. To do so,
> Cassandra tracks consensus in a system.paxos
> table. This `system.paxos` table needs to be checked upon reads as well
> in order to ensure the latest state of the ongoing
> consensus is known. This can be achieved by using the SERIAL
> consistency
> level.
> 
> Experiments on a distributed James cluster (4 James nodes, having 4 CPU
> and 8 GB of RAM each, and a 3 node Cassandra
> cluster of 32 GB of RAM, 8 CPUs, and SSD disks) demonstrated that the
> system.paxos table was by far the most read
> and compacted table (ratio 5).
> The table triggering the most reads to the `system.paxos` table was the
> `acl` table. Deactivating LWT on this table alone
> (lightweight transactions & SERIAL consistency level) enabled an
> instant
> 80% throughput, latencies reductions
> as well as softer degradations when load breaking point is exceeded.
> 
> 
> Do you mean that Cassandra is the bottleneck in this setup?
> What is the effect of having more Cassandra nodes?

Yes, it is.

The effect of adding more Cassandra nodes means more costs.

Our ownership cost is so far of 5€/user/year which is around 25 time
more than our competitors. The goal is to lower such costs, in order to
have a viable commercial solution built on top of James.

> 
> ## Decision
> 
> Rely on `event sourcing` to maintain a projection of ACLs that do not
> rely on LWT or SERIAL consistency level.
> 
> Event sourcing is thus responsible of handling concurrency and race
> conditions as well as governing denormalization
> for ACLs. It can be used as a source of truth to re-build ACL
> projections.
> 
> Note that the ACL projection tables can end up being out of
> synchronization from the aggregate but we still have a
> non-questionable source of truth handled via event sourcing.
> 
> ## Consequences
> 
> We expect a better load handling, better response time, and cheaper
> operation costs for Distributed James while not
> compromising the data safety of ACL operations.
> 
> ACL updates being a rare operation, we do not expect significant
> degradation of write performance by relying on
> `eventSourcing`.
> 
> We need to implement a corrective task to fix the ACL denormalization
> projections. Applicative read repairs could be
> implemented as well, offering both diagnostic and on-the-fly
> corrections
> without admin actions (a low probability should
> however be used as loading an event sourcing aggregate is not a cheap
> thing).
> 
> 
> What implementation are you using for Event Sourcing? AFAIK, James on
> Cassandra uses LWT + batchs for Event Store.

I have 

Re: Jenkins CI setup

[Tellier Benoit]

Likely "unstable".

I will have a look at this tomorrow.

Le 26/11/2020 à 17:22, Jean Helou a écrit :
> The good news is that docker does indeed work, the bad news is that the
> tests fail with an issue that's too involved for me :/
> 
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]   
> CassandraMailboxManagerConsistencyTest$FailuresOnDeletion$DeleteOnce.deleteMailboxByPathShouldBeConsistentWhenMailboxPathDaoFails:433
> Multiple Failures (1 failure)
>   
> Expecting:
>   <[]>
> to contain exactly (and in same order):
>   <[#private:user:INBOX]>
> but could not find the following elements:
>   <[#private:user:INBOX]>
> 
> at 
> CassandraMailboxManagerConsistencyTest$FailuresOnDeletion$DeleteOnce.lambda$deleteMailboxByPathShouldBeConsistentWhenMailboxPathDaoFails$8(CassandraMailboxManagerConsistencyTest$FailuresOnDeletion$DeleteOnce.java:440)
> 
> so unless the build for
> 
> * 6fab99364a - JAMES-3448 Rewrite links to
> http://james.apache.org/server/3/ (Mon Nov 23 15:10:36 2020 +0700)
>  N
> 
> is broken which sounds unlikely, I'm going to need help
> 
> jean
> 
> On Thu, Nov 26, 2020 at 10:53 AM Jean Helou  wrote:
> 
>> on a loosely related note : the test suite logs are scary to look at:
>> piles upon piles of stack traces and error logs but the tests actually pass
>> ...
>>
>> On Thu, Nov 26, 2020 at 10:50 AM Jean Helou  wrote:
>>
>>> Thanks benoit,
>>>
>>> Matthieu pointed me to numerous apache projects with jenkinsfiles which
>>> mention docker in
>>> https://github.com/search?q=org%3Aapache++filename%3AJenkinsfile+docker=Code
>>> so I'm trying out things based on that
>>>
>>> the logs seem promising so far :
>>> ```
>>>
>>> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
>>> 0.697 s - in 
>>> org.apache.james.backends.rabbitmq.RabbitMQConnectionFactoryTest
>>>     ℹ︎ Checking the system...
>>> ✔ Docker version should be at least 1.6.0
>>> ✔ Docker environment should have more than 2GB free disk space
>>> [INFO] Running org.apache.james.backends.rabbitmq.RabbitMQTest
>>> ```
>>>
>>>
>>> On Thu, Nov 26, 2020 at 10:40 AM Tellier Benoit 
>>> wrote:
>>>
>>>> Done
>>>>
>>>> Le 26/11/2020 à 16:25, Jean Helou a écrit :
>>>>> hi all,
>>>>>
>>>>> As you know I started a PR to setup jenkins CI, the latest iteration
>>>> sees
>>>>> the compilation of the project complete in 5 minutes ( thanks to T1C)
>>>> but
>>>>> the tests fail to initialize docker containers with the disastrous
>>>>> consequences you can imagine :D
>>>>>
>>>>> I opened https://issues.apache.org/jira/browse/INFRA-21144 to ask if
>>>> it is
>>>>> possible to have the docker service enable don some nodes, since I am
>>>> not
>>>>> official member of the project I think it may be useful if you chimed
>>>> in on
>>>>> the ticket to confirm that this is a legitimate request.
>>>>>
>>>>> Best regards,
>>>>> Jean
>>>>>
>>>>
>>>> -
>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>
>>>>
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Jenkins CI setup

[Tellier Benoit]

Done

Le 26/11/2020 à 16:25, Jean Helou a écrit :
> hi all,
> 
> As you know I started a PR to setup jenkins CI, the latest iteration sees
> the compilation of the project complete in 5 minutes ( thanks to T1C) but
> the tests fail to initialize docker containers with the disastrous
> consequences you can imagine :D
> 
> I opened https://issues.apache.org/jira/browse/INFRA-21144 to ask if it is
> possible to have the docker service enable don some nodes, since I am not
> official member of the project I think it may be useful if you chimed in on
> the ticket to confirm that this is a legitimate request.
> 
> Best regards,
> Jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: patch to fix jpa destination in README.adoc

[Tellier Benoit]

Hello Pablo.

Thanks for the fix.

Would you agree opening a PR on Github for this?

Otherwise, I could extract the patch too...

Cheers,

Benoit

Le 26/11/2020 à 01:17, Pablo Pita Leira a écrit :
> diff --git a/README.adoc b/README.adoc
> index 1c4fec1c26..a7c92ae896 100644
> --- a/README.adoc
> +++ b/README.adoc
> @@ -114,7 +114,7 @@ To retrieve compiled artifacts, one might mount
> these volumes:
> 
>  - --volume $PWD/dockerfiles/run/spring/destination:/spring/destination
> : is the volume used to get the compiled elements for Spring packaging.
>  - --volume
> $PWD/dockerfiles/run/guice/cassandra/destination:/cassandra/destination
> : is the volume used to get the compiled elements for Guice Cassandra
> packaging and Cassandra-LDAP packaging.
> -- --volume
> $PWD/dockerfiles/run/guice/cassandra/destination:/jpa/destination : is
> the volume used to get the compiled elements for Guice JPA packaging.
> +- --volume $PWD/dockerfiles/run/guice/jpa/destination:/jpa/destination
> : is the volume used to get the compiled elements for Guice JPA packaging.
>  - --volume $PWD/swagger:/swagger : is the volume used to get the
> swagger json files for webadmin documentation.
> 
>  Some tests needs a DOCKER_HOST environment variable in order to be
> played, they will be ignored if you don't provide this variable.
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Tellier Benoit]

Le 18/11/2020 à 16:09, Jean Helou a écrit :
> [...]
>> I am not sure where  other apache projects source the machines
>>
>> running the CI but at least https://github.com/apache/pulsar-helm-chart
>>> seems to have access to some serious compute power...
>> INFRA is by default not powerful (likely not enough). A way out is
>> "donating computing power for dedicated project builds".
>>
>> Hence we would need some individual/organization donating computing
>> power to the Apache foundation for Apache James usage.
>>
>> I like the idea, and I would be happy to defend the idea within
>> Linagora, but I need a working build on Apache infra first. Likely it
>> would be a few "dedicated servers", but it would also need internal
>> validation.
>>
>>
> I have asked a friend who may be able to help, I also found
> https://www.scaleway.com/en/about-us/open-source-program/ which seems
> promising. It starts by sending an email to opensource-prog...@scaleway.com
> Scaleway is where I intend to run my SMTP relay (using my own compute
> resources) once it is finished assembling, I started working on a helm
> chart that's deployed on scaleway Kapsule.
> Once it works reasonably well, I intend to contribute it back to
> james/apache but that's gonna take a while
Thanks for sharing!
>
> Jean
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: apache/james on docker hub.

[Tellier Benoit]

Le 18/11/2020 à 15:00, Jean Helou a écrit :
>> I would like us to start as part as the release process to publish
>> convenience binaries under apache name, just like some other Apache
>> projects are doing: https://issues.apache.org/jira/browse/INFRA-19650
> 
> 
> This is a bit of a shameless suggestion but wouldn't it make sense to use
> the opportunity to setup an automated public CI ?

I would love to see this happening but I fail at seeing progress on the
Apache level CI for several months already.

Correlating things together would likely lead to getting nothing done.

I am willing to add publishing docker images to the release process, but
will not find the resources to make this automated public CI happen.

Note that asking dockerhub access is not incompatible with CI work.

> Something that
> automatically publishes release versions from master, and builds PRs would
> be nice. 

Something that automatically builds master branch and peoples PR should
IMO have a higher priority.

BTW given
http://www.apache.org/legal/release-policy.html#approving-a-release a
two stage process would likely be needed.

> I am not sure where  other apache projects source the machines
> running the CI but at least https://github.com/apache/pulsar-helm-chart
> seems to have access to some serious compute power...

INFRA is by default not powerful (likely not enough). A way out is
"donating computing power for dedicated project builds".

Hence we would need some individual/organization donating computing
power to the Apache foundation for Apache James usage.

I like the idea, and I would be happy to defend the idea within
Linagora, but I need a working build on Apache infra first. Likely it
would be a few "dedicated servers", but it would also need internal
validation.

Benoit

> with a bit of luck
> that could also benefit from shared credentials to push apache docker
> images without needing any single user to have specific rights>
> jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

apache/james on docker hub.

[Tellier Benoit]

Hello,

Following this thread:

I would like us to start as part as the release process to publish
convenience binaries under apache name, just like some other Apache
projects are doing: https://issues.apache.org/jira/browse/INFRA-19650

I propose to ask for rights on apache/james repository

I propose to use apache/james:flavor:version as a naming pattern.

Committers, I can include you in the request, so in case you are
interested communicate me your dockerhub id.

Cheers,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: linagora Docker images

[Tellier Benoit]

Le 18/11/2020 à 03:55, Tobias Verbeke a écrit :
> L.S.
>
> When exploring the getting started guide I noticed the Docker image referred 
> to was
>
>   docker run -p "25:25" -p "143:143" linagora/james-jpa-sample:3.5.0 
>
> While I very much appreciate and respect the contributions by the Linagora 
> team, I was wondering whether it would not be cleaner to publish these images 
> under a vendor-neutral Docker hub account e.g. under apache similarly to e.g. 
> hadoop images which are available at
>
> https://hub.docker.com/r/apache/hadoop
+1

Work needs to be done on that. I'll open a INFRA ticket to get the
permission. Commiter conducting the release process would need to build
& tag & push docker images.


On a similar topic we need to work on Guice servers packaging. See
https://issues.apache.org/jira/browse/JAMES-3261,
https://github.com/apache/james-project/pull/221.

I would love to see this happening before the next release.
> Also, as a side note, it might be interesting to add a link pointing to the 
> Git repository / location where the Dockerfile can be found (as a way to 
> further explore the project).
Thanks for the Feedback. This sounds like a good idea.
>
> Best,
> Tobias
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Sorting pom dependencies

[Tellier Benoit]

Heelo Jean!

Le 17/11/2020 à 20:37, Jean Helou a écrit :
> Hello again :D
> 
> [...]
> 
> I suggested creating
> 
>> mvn compile -Psort
> 
> To make running the sort plugin easier and help make a transition to a
> state where the dependencies are all correctly sorted. To which benoit
> answered  :
> 
>> And abort the build if needed? => :+1:
> 
> I looked into this a bit further today, it is easy to make the plugin run
> and fail the build if a pom is incorrectly sorted. However with 150 invalid
> pom files in the project this means the build will be blocked for a while
> unless a massive sorting commit is applied which means probable conflicts
> for everyone else with currently modified pom in their branch.
> 
> While I think this remains the best option (because it converges quicker)

I vote +1 for the best option.

On our side we should be able to rebase our PRs :-)

> here are two alternative ideas
> [..]
> 
> Let me know what you think and if such a change is desirable

That's desirable to me.

Such a shame the intern documenting the sorting module a few years ago
did not think about it ^^

Thanks for bringing this further.

> jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Guice modules organization and naming

[Tellier Benoit]

This sounds legitimate to me.

Across the line, there is however the thought that the main - if not
only - delivery of the Apache James project is a mail server.

Things like mailet-api, mailbox, protocols will end up with the same
naming than some over server components like (say) queue.

I personally agree with such a statement. I wonder if it could bring
further discussions here.

I also agree on the "iterative" approach.

Cheers,

Benoit

Le 13/11/2020 à 17:39, Jean Helou a écrit :
>>  >  Here is why I chose this naming scheme instead of the prevalent one in
>> the
> 
>> guice module:
>>> - I felt the `james-server` prefix which is in most artifact names
>> doesn't
>>> bring much. There are a few other modules which dropped the prefix in
>>> `server/container/guice` , and I prefer the lighter names. I am
>> considering
>>> also dropping the prefix from the 2 other queue modules in
>>> `server/container/guice/queue` if I get positive feedback.
>>
>> Overall +1 but imo we should be consistent.
>>
>> Maybe we should rename them all, or rename none.
>>
> 
> I don't think that's a viable option, at least not in a single PR. It would
> entail a modification of almost every single POM files in the project,
> that in turn has impacts on almost everything (docker images, build
> scripts, documentation, etc which contain various references to artifact
> names)
> 
> As an experiment, I attempted moving the `server/app` module to
> `/server/assemblies/spring-jpa-activemq` because I wanted to separate
> modules which produce actual runnable apps away from modules that only
> define binding modules
> I also renamed the artifact to something a bit closer to what was done for
> the guice based apps.
> This ended up in an evening of tracking dependencies and references all
> over the place and I'm not entirely confident that my end result is
> correct, I dropped that effort.
> 
> A more feasible approach would be to incrementally rename the modules I
> will be touching/needing to build my SMTP only relay and their immediate
> siblings.
> I thus would submit a sequence of PRs to do so, If I identify easy wins
> doing that I´ll include them as well.
> If everyone takes a bit of time to similarly rename the modules they are
> immediately working on this could converge
> 
> I´ll list the modules I intend to rename here in each PR to limit conflicts
> with other's work
> 
> jean
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Guice modules organization and naming

[Tellier Benoit]

Hello Jean!

Nice to see you around ;-)

Many thanks for what looks like a nice future contribution.

Might you need help, do not hesitate.

Cheers,

Ps: It do not seem like the email was duplicated ^^

Benoit

Le 13/11/2020 à 16:34, Jean Helou a écrit :
> Hello everyone,
> 
> I intend to propose a PR which :
> - groups guice queue bindings in server/container/guice/queue
> - groups guice blob bindings in server/container/guice/blob

+1

> 
> I have already done most of the work for the queues but realized I should
> get feedback before spending more time on this before submitting this on
> github.  The changes are visible in
> https://github.com/jeantil/james-project/tree/extract-guice-modules
> 
> The branch currently has5 steps:
> - move activemq bindings to guice/queue
> - move rabbitmq bindings to guice/queue
> - extract memory binding to guice/queue
> - rename activemq binding module to queue-activemq-guice
> - rename rabbitmq binding module to queue-rabbitmq-guice
> 
> Extracting `MemoryMailQueueModule` to its own module ( under
> `server/container/guice/queue/memory` ) led to having to name that new
> module. I ended up naming it `queue-memory-guice` and that in turn led to
> the last 2 steps for consistency in this subtree.
> 
> Here is why I chose this naming scheme instead of the prevalent one in the
> guice module:
> - I felt the `james-server` prefix which is in most artifact names doesn't
> bring much. There are a few other modules which dropped the prefix in
> `server/container/guice` , and I prefer the lighter names. I am considering
> also dropping the prefix from the 2 other queue modules in
> `server/container/guice/queue` if I get positive feedback.

Overall +1 but imo we should be consistent.

Maybe we should rename them all, or rename none.

> -  I found that  the server/container/guice/blob-xxx modules were easier to
> identify/map to useful components than the other modules.
> Looking at the `server/` subdirectories one can easily see different
> components for the server which make perfect sense in the context of email,
> the overall content is inferrable.
> ```
> server
> ├── blob
> ├── data
> ├── dns-service
> ├── mailet
> ├── mailrepository
> ├── protocols
> ├── queue
> ├── task
> ```
> I found that the blob-xxx-guice modules were very natural counterparts for
> server/blob/blob-xxx and chose to follow that naming scheme for the memory
> module.
> 
> Please let me know what you think
> 
> Also I feel this work matches
> https://issues.apache.org/jira/browse/JAMES-1902 but maybe it needs its own
> issue, please advise.
> 
> Best regards,
> Jean
> 
> PS: I sent a first copy of this email before fully completing the
> registration process to the dev list. I'm resending this now because I
> don't see the message in the archive, please accept my apologies if this
> results in a double send to the list.
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Distributed Server: Discussing strong consistency requirements

[Tellier Benoit]

I might start tomorrow relaxing LWT transaction for users & domains.

Let's see where it gets us!

Cheers,

Benoit

Le 20/10/2020 à 09:08, Tellier Benoit a écrit :
> Hello there,
> 
> Cassandra is an eventually consistent datastore, that can be used in a
> consistant fashion. To do so, we rely on a mechanism called "LightWeight
> Transactions (LWT)". Lightweight transactions relies on the PAXOS
> distributed consensus algorithm to enforce a condition upon data
> mutation. A table, system.paxos, is used to track the state of
> transactions. Furthermore, upon writes, several round-trips (two) are
> needed to ensure data integrity accross replica(minimum round trips to
> achieve consensus) and the system.paxos table is read / written to in
> addition to the applicative table.
> 
> All of this causes LWT to be significantly slower than their lower
> consistency counterparts. On some Linagora owned production instances,
> regular reads takes 2ms while reads on tables relying on LWT takes 6ms.
> Similar figures are found for writes. We also noticed some high
> compaction throughtput on the paxos table, leading to many back-ground
> writes.
> 
> Given the massive impact of LWT usage on performance, and given the lack
> of debate upon LWT adoption, I would like to re-challenge their usage...
> 
> Here are the places we rely on LWT for the Distributed Server:
> 
>  - IMAP UID generation (monotic integer) - strong consistency is
> strictly required to not loose data as overwriting a uid means
> overwriting a message.
> 
>  - IMAP ModSeq generation (monotic integer) - strong consistency is
> required, as modseq overwrites can lead to some data not being well
> synchronised.
> 
>  - Domain and users - we rely on LWT to return an error when deleting a
> user that do not exist, or creating an already existing user. It sounds
> unecessary.
> 
>  - Message flags relies on LWT to ensure updates are not overwritten. As
> an often read metadata, the impact is high, for limited criticity for
> the end user. After all, no data is lost, only a user action like
> marking a message as Seen, an action that he can very well perform again
> 
>  - Mailbox path registration, ACL - required to prevent data races
> 
> My proposal would be:
> 
>  - Keep using LWT for UID and modseq generation, as well as Mailbox path
> registration.
>  - Make the use of LWT for message flags update configurable - as an
> admin I can choose to disable it.
>  - I am also fine with completly removing LWT usage for message flags
> update.
>  - No longer use LWT on domain or users. Instead use idempotent create /
> delete. The contract test will thus need to be relaxed.
>  - On the long term, relying on a CRDT to represent ACLs at the
> Cassandra level, instead of serialized JSON, would enable to get rid of
> LWT usage on the ACL table.
> 
> Any feedback?
> 
> Benoit
> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Distributed Server: Discussing strong consistency requirements

[Tellier Benoit]

Hello there,

Cassandra is an eventually consistent datastore, that can be used in a
consistant fashion. To do so, we rely on a mechanism called "LightWeight
Transactions (LWT)". Lightweight transactions relies on the PAXOS
distributed consensus algorithm to enforce a condition upon data
mutation. A table, system.paxos, is used to track the state of
transactions. Furthermore, upon writes, several round-trips (two) are
needed to ensure data integrity accross replica(minimum round trips to
achieve consensus) and the system.paxos table is read / written to in
addition to the applicative table.

All of this causes LWT to be significantly slower than their lower
consistency counterparts. On some Linagora owned production instances,
regular reads takes 2ms while reads on tables relying on LWT takes 6ms.
Similar figures are found for writes. We also noticed some high
compaction throughtput on the paxos table, leading to many back-ground
writes.

Given the massive impact of LWT usage on performance, and given the lack
of debate upon LWT adoption, I would like to re-challenge their usage...

Here are the places we rely on LWT for the Distributed Server:

 - IMAP UID generation (monotic integer) - strong consistency is
strictly required to not loose data as overwriting a uid means
overwriting a message.

 - IMAP ModSeq generation (monotic integer) - strong consistency is
required, as modseq overwrites can lead to some data not being well
synchronised.

 - Domain and users - we rely on LWT to return an error when deleting a
user that do not exist, or creating an already existing user. It sounds
unecessary.

 - Message flags relies on LWT to ensure updates are not overwritten. As
an often read metadata, the impact is high, for limited criticity for
the end user. After all, no data is lost, only a user action like
marking a message as Seen, an action that he can very well perform again

 - Mailbox path registration, ACL - required to prevent data races

My proposal would be:

 - Keep using LWT for UID and modseq generation, as well as Mailbox path
registration.
 - Make the use of LWT for message flags update configurable - as an
admin I can choose to disable it.
 - I am also fine with completly removing LWT usage for message flags
update.
 - No longer use LWT on domain or users. Instead use idempotent create /
delete. The contract test will thus need to be relaxed.
 - On the long term, relying on a CRDT to represent ACLs at the
Cassandra level, instead of serialized JSON, would enable to get rid of
LWT usage on the ACL table.

Any feedback?

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Introduction from a new member of Apache James Community

[Tellier Benoit]

Hello all,

As Quan did not succeed to send a mail to this list (...) I'm forwarding
it for him!

Cheers,

Benoit Tellier

---

Hi,

Let me introduce myself first. My name is Tran Hong Quan. I am from
Vietnam and studying fourth-year at Hanoi university of science and
technology. My major is Computer Engineering. Now I am Java trainee at
Linagora Vietnam company. My company has many open source enthusiasts
and working on many open source projects such as: Apache James,
OpenPaaS, LinShare... About Java team people, they manage and develope a
branch of Apache James (https://github.com/linagora/james-project
).

As a trainee at Java team, my job is writing a new James CLI based on
WebAdmin API to replace the outdated, security-vulnerable JMX
command-line interface. I will work on a wrapper to adapt the old CLI
API. It also aims at providing a more modern and intuitive interface. 


Now I am working on designing the new CLI interface, write documentation
for it. 


I am looking forward to choosing what libraries that I could use for the
new CLI.


I would love to participate in Apache James development, share my work,
contribute to the community.


Quan.


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Le 29/09/2020 à 17:47, Matthieu Baechler a écrit :
> On Tue, 2020-09-29 at 08:59 +0700, Tellier Benoit wrote:
> 
> [...]
> 
>> Spring deprecation could be seen as this big event for most users ?
> 
> You are not very good at public relation, do you? (:
> I don't see feature deprecation as a good opportunity to increment
> major version number.

With Spring deprecation comes Guice applications.

Today they are not even available as a download option, so nobody uses them.

(this should be fixed by the way for the next minor release)

I believe, with all the changes and features that comes with Guice
application, a major release could be justified.

Of course we should not communicate in a "stop spring" way as I did it.

Also I found this link helpfull (and I should try applying it more to my
writings here): https://infra.apache.org/contrib-email-tips.html

" Try not to use the word "you". People get defensive when they think
that comments are directed at them personally. Consider using "one" or
"we" or even "me". "

> 
>>> 4. About defining a vision
>>>
>>> [...]
>>>
>>> What I would really like is to break things! Let's remove all
>>> these anachronic modules, or even better: let's build James 4 by
>>> adopting only well selected modules, ones that are here for a purpose.
>> I do think that such an effort will end up with similar effects for the
>> community than the 3.0 release effort.
> 
> I agree to some extent.
> 
>> What I am looking for is a scalable, modern mail server, we mostly have
>> that already (after 5 years of development which is already a huge
>> commitment).
> 
> I know that it's Linagora intent. However, you are still slow to reach
> that goal partly because of James codebase size. Also, you will keep
> having a clutered product that most people won't see as a "scalable,
> modern mail server".

We agree on the symptoms.

The problem here is that "Linagora" problem about code base size becomes
a community problem.

>> [...]
> 
> How it is different from what we did for years? Did it solve velocity
> issues? Is the project fun now?
> 
> I may start this 3.99 thing on my fork to see how it goes and will keep
> you posted about that.

We deprecated unmaintained outdated components with no active contributors.

Maybe what we should do is discuss what we want to reach as a project.

As a project (as I understood it) we want to answer our end user needs
(self hosting, the distributed server, easy to extend behavior). We
spoke a lot about it lately.

That being said maybe our goal as a project is not to offer a fully
featured email server that we can plug directly into in a collaborative
platform. (our goal at Linagora is to do just that, but our goal as an
Apache project is to *enable* people doing that).

But our goal is more to enable people to do that.

I believe that, going down that road, and removing things, even
maintained, well-written, that ended up in the Apache James code base by
accident (with the classic deprecation / removal procedure)

Third parties, including Linagora, will of course be free to backport them.

I believe it will:
 - enhance the governance of the project
 - simplify the code base
 - force us to have well working extension systems

However (relative) API stability will be needed.

But this could be done "reasonably quickly" with the 3.x code base.

This is what I meant, and as far as I know we never done that.

Sorry for explaining myself elusively.

> 
> Cheers,
> 
> -- Matthieu Baechler

As a conclusion, I'm sure there is a way to arrange the way Linagora
contributes to the Apache James project so that I feel for confident
endorsing both hats at the same time.

It's a long lasting problem, but trying to solve it will in my opinion
solve some of the problems we encounter as a community.

Cheers,

Benoit

> 
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

A big thread, let's jump in.

Le 28/09/2020 à 02:34, Matthieu Baechler a écrit :
> Hi,
>
> I'm not sure which message to answer in the thread so I start a new
> thread to summarize my thoughts on the various topics discussed.
>
> 1. About Roadmap
>
> [...]
>
> 2. About documentation
>
>[...]
>
>TL;DR we have to explain how it already works
+1
>
> 3. About versioning (3.x vs 4.0)
>
>Like roadmaps, major version numbers give people expectations: there
>must be something very new and/or very different because the
>community decided to increment major version number.
>
>Last time the community tried that (3.0) the projects almost died
>because too many things had to ship at the same time and then was
>never ready. It took years to finally release it after Linagora
>started to invest a lot it and by the way, we never finished what
>was supposed to, we just decided that, no software being perfect, we
>had to release this much-better-than-2.x version.
>
>I would like to take the Linux Kernel path: 
>
>* only increment minor version for the time being
>* don't build a backlog or any list of things we want to achieve
>before incrementing
>* release 2 to 4 times a years with what is ready
>* increment major version when what will be ready deserves it or
>when minor number get to big
Spring deprecation could be seen as this big event for most users ?
> 4. About defining a vision
>
> [...]
>
> What I would really like is to break things! Let's remove all
> these anachronic modules, or even better: let's build James 4 by
> adopting only well selected modules, ones that are here for a purpose.
I do think that such an effort will end up with similar effects for the
community than the 3.0 release effort.

What I am looking for is a scalable, modern mail server, we mostly have
that already (after 5 years of development which is already a huge
commitment).

Note that I am also convinced that we need better documentation, and
that we need to simplify the project structure.

With my Linagora hat on now, I see no way to convince my management to
dedicate any effort toward a completely reworked 4.0 with a several
years ETA.

>
> People could either jump to this fresh version of James or keep
> maintaining the 3.x branch. If they lack some modules that were not
> selected for James 4, they could just port these modules to the new
> APIs.
>
> By doing such a move, we could focus to finally solve our longstanding
> problems like: developer experience, newcomers welcoming, having a
> decent and up-to-date documentation, very easy first deployment, etc.
>
> What would you think of such a move ?
I would prefer a more pragmatic alternative.

We as a community could be identifying features / modules that should
not have made it in the 3.x release line. We could decide to deprecate
then remove these modules, hopefully letting time for third party to
backport alternatives.

As such we would end up with a much simpler code base.

The downside is that core API stability would be (to some extent) needed.
>
> Cheers,
>
> -- Matthieu Baechler
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Le 31/08/2020 à 18:28, David Leangen a écrit :
> Hi Benoit,
>
> Ok, cool. I think we are making progress. 
>
>> Would it answer some of your concerns?
> Let me answer this first: yes. Thank you for patiently tolerating my 
> persistence. 
>
>
>> Well from what I recall, we had a nice community meeting answering that
>> very question.
>>
>>  - We will rework product definition, boundaries and branding. Using
>> guice servers we will provide a basic/advanced/distributed server
>>  - We will improve the build experience through Apache CI builds and
>> migrating to graddle.
>>  - Also, some contributors are implementing the final RFC-8621 JMAP release.
>>
>> Maybe we should have a roadmap page somewhere so that people don't have
>> to read the DEV mailing list to find these pieces of information?
> That is indeed my intention, and why I am asking these questions. I am ok 
> with referencing other artifacts, but I think it is important to publish the 
> roadmap on the website. Of course, it can always change, that is fine. It is 
> not a set contract that MUST be undertaken, but it ought to represent the 
> current thoughts of the community.
It turns out there is a roadmap published on our website (
http://james.apache.org/#roadmap ) - I just discovered it, shame on me...

I did open a pull request to add the items of the community call to it.
See https://github.com/apache/james-project/pull/245

Cheers

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Le 31/08/2020 à 09:43, David Leangen a écrit :
>>> From my point of view the most important point is to describe what
>>> James does now. 
>> +1
> Yes, I agree.
>
> Again, I do not disagree that this is “most important”. Let me use a more 
> concrete example to illustrate why it is not the **only** concern.
>
> When I first looked into James, there was a complex grid to choose from 
> between Spring and Guice. Based on the grid, it seemed to me that Spring was 
> better.
>
> When I started asking questions, though, I understood that Spring is not 
> really being maintained anymore, or at least, more energy is being put into 
> Guice.
>
> Well, for me as a user trying to understand what to do, that is pretty darn 
> important information! I **WANT** to know where the project is heading, 
> because I need to be able to make decisions. So not only do I need to know 
> what the project’s vision is (which IMO needs a lot of clarification 
> currently), I need to know more concretely what the plans are for at least 
> the next major release. Else, how can I decide if it’s worth investing or not?

Well from what I recall, we had a nice community meeting answering that
very question.

 - We will rework product definition, boundaries and branding. Using
guice servers we will provide a basic/advanced/distributed server
 - We will improve the build experience through Apache CI builds and
migrating to graddle.
 - Also, some contributors are implementing the final RFC-8621 JMAP release.

Maybe we should have a roadmap page somewhere so that people don't have
to read the DEV mailing list to find these pieces of information?

> On that note, I would propose that for the 4.0 release, we clearly indicate 
> that Spring will **NOT** be available.

+1, and a 4.0 would make the switch very clear. No ambiguity.

I believe the only work remaining toward this is
https://github.com/apache/james-project/pull/221 (and for other guice
servers)

>  It should be deprecated in an upcoming 3.x release, then “completely” 
> eliminated in 4.0. (Perhaps some code could remain if some people really want 
> it, but we need to make it clear in the “user contract" that it is not 
> supported.) That would be precisely what a major release is for, IMO.
We discussed (can't find the thread though) some years ago about the
adoption of time based release for James server.

This ensures we keep delivering improvement and fixing bugs even if we
struggle delivering some major changes.

Here we could maybe:
 - Better communicate about the release calendar (why not having it on
the roadmap page?)
 - Prior the release date, discuss the reach of this release (major vs
minor) and see how we are regarding roadmap items.
 - Also, the release process needs to be run faster...

Would it answer some of your concerns?

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Le 28/08/2020 à 14:48, Raphaël Ouazana-Sustowski a écrit :
> Hi David,
>
> First thank you very much for your involvement into the community, I
> hope you'll be able to continue to do so.
>
> Le 28/08/2020 à 05:11, David Leangen a écrit :
>> Hi Rene,
>>
 I have observed a few different types of approaches to OSS:
   * Haphazard “me” approach.
>>> I think you are correct, I would say that now it is more later
>>> approach as well. Well at least since I started working on the
>>> project (might have been different before).
>> Thank you very much for your comments. Yes, that is very helpful!
>>
>> I fall into this trap often, and I think I am not the only one, but
>> sometimes I explain what I **aspire** a project to be, not really
>> what it is now. I think it is crucial to explain both. We need to be
>> able to give proper information to newcomers so they can make a
>> correct assessment. I think what you mention makes sense and I can
>> work with that.
>>
>> Perhaps we should just wait in case anybody else would also like to
>> add something?
>>
>> So I would describe James as:
>>
>>   * Currently in a state of transition (this is what James is now)
>>   * What we aspire it to be for version xxx (I am proposing v4.0)
>>
>> Just a crazy thought, but technically, I could even create the v4.0
>> of the documentation, and we could even use it almost like a
>> specification to guide the development. Of course, the real work gets
>> done based on JIRA issues, but the issues that get created in the
>> first place could be matched against the v4.0 docs.
>
>
> From my point of view the most important point is to describe what
> James does now. 

+1

> [...]
>
> Raphaël.
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Hello David,

I did actually came back from vacation.

I think I did answer in that thread already. I would love to have other
people feedback as well on the opportunity to market our current
"community roadmap" (documentation, product re-branding, CI) into a
major release.

Le 26/08/2020 à 18:10, David Leangen a écrit :
> Hello,
>
> I am still trying to get a sense of the community’s vision, and it’s thoughts 
> relating to how to treat users. Unfortunately I have not heard too many 
> comments so far on this topic. Trying to read between the lines, I could 
> think of 4 potential conclusions:
>
>  1. People are still on summer holidays and don’t have the time to spend
>  2. I have taken a wrong or confusing approach to the topic, so people don’t 
> know what to answer
>  3. This is already documented somewhere so the assumption is that I already 
> know it
>  4. There is not much interest in discussing the the topic of “users”
>
> To continue with the documentation (at least on the path I have taken so far) 
> I need to better understand the vision. That will allow me to resolve and 
> clarify the topics previously raised regarding the community’s approach to 
> dealing with newbies.
This sentence sounds like the "support/warranty" discussion turned into
a blocker for you.

I think instead of using blockers we should propose consensual solutions.

The approach that emerged from ongoing proposal would be "Through better
branding and documentation, end user would need less support" eventually
making most people happy, without requiring SLAs from community members.

> By the way, this project was much more difficult that I had anticipated. I no 
> longer have much time to dedicate, but I pledge to continue, albeit at a 
> slower pace and in the background, so long as I continue to get engagement 
> from the community members. My main goal is to help document the state of 
> James and the community, and the secondary goal that I discovered in the 
> process is to point out gaps and potential areas of improvement. Thanks again 
> for all the support!!
Thank you again for your involvement!

Cheers,

Benoit
>
> Cheers,
> =David
>
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Which repositories are we using and going to document? Let's retire the ones we don't

[Tellier Benoit]

Hi René

I did a quick review.

As such, given the limited content, I mostly agree with it.

I do think we might want to leverage this opportunity to close
low-traffic unused mailing lists. This includes:

 -site-...@james.apache.org : site discussions happens on server-dev and
not there, it is low traffic, and unused.
 - mailet-...@james.apache.org: It is not active Maybe it is better
to discuss mailet api concerns in server-dev

What would the list think about such an initiative?

Best regards,

Benoit

Le 17/08/2020 à 10:26, Rene Cordier a écrit :
> Hello,
>
>> There are things that need to me documented that are not technical:
>> - what / where are the project resources (website, CI, git repos,
>> mailing list)
>> - how to contribute, engage with the community
>> - etc.
>
> Just to say I started migrating and writing this part of the
> documentation here: https://github.com/apache/james-project/pull/243
>
> Don't hesitate to put comments and give feedbacks.
>
> Thanks!
>
> Rene.
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

Re: [Discussion] Road to 4.0

[Tellier Benoit]

Hi David!

Le 15/08/2020 à 08:20, David Leangen a écrit :
> Hello!
>
> I wanted to tie together a few ongoing threads and make a proposal for the 
> road to a 4.0 release. 

I'm uncomfortable with the 4.0 release switch. It gives the impression
that "much" needs to change whilewhat we need is to re-brand, and better
document what exists.

> My assumption is that there currently is not roadmap to 4.0. If I am 
> mistaken, please do let me know and I will adapt.
>
> [...]
>
> By the way, to coincide with the release, if the objectives are clear, 
> perhaps there is a commercial organization (or individual) who would be 
> willing to provide paid-for professional support starting from 4.0?
Linagora (company for which I am working) already proposes these
services. We already carried out several support / development contracts
for Apache James.
>  I think it would help complete the offering, and hopefully provide a 
> commercial opportunity as well in a way that is beneficial to all, including 
> the James community. We just need to ensure that the “contract” is very 
> clear, and that we avoid any potential conflicts of interest. I think we 
> should include this item in the scope of the discussion as well. 

I'm not sure an Apache project is entitled to broadcast details of
comercial offers. I believe "listing" service provider is enough. It
should be up to the service providers to detail their offer on their own
website (referenced via an hyper-link)

> (Just a thought, but maybe the “Distributed James” could be a commercial 
> offering, rather than a community offering??)
I strongly believe the "distributed server" is a differentiator for
Apache James and helps satisfy some of the community needs.
> To resolve this thread, I will be satisfied once we have a clear statement of 
> objectives regarding the 4.0 release.
I do agree with the already discussed items (documentation & re-branding)
>
> Cheers,
> =David
>
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Problems with sieve

[Tellier Benoit]

Hello David.

I found the issue. org.apache.james.sieve.jpa.model.JPASieveQuota &
org.apache.james.sieve.jpa.model.JPASieveScript are not registered as
persistant class hence the failure.

I opened https://issues.apache.org/jira/browse/JAMES-3348 & a fix for
this here: https://github.com/apache/james-project/pull/242.

Cheers,

Benoit

Le 28/07/2020 à 12:36, David Leangen a écrit :
> Hello,
>
> I am having trouble figuring out the origin of this problem:
>
> 14:30:10.458 [INFO ] o.a.j.s.SendMailHandler - Successfully spooled mail 
> Mail1595914206122-99a764fb-1097-4614-a1c2-022d0b331bee from 
> MaybeSender{mailAddress=Optional[user01@test.local]} on 
> localhost/0:0:0:0:0:0:0:1 for [user01@test.local]
> 14:30:34.797 [INFO ] o.a.j.p.n.BasicChannelUpstreamHandler - Connection 
> closed for 0:0:0:0:0:0:0:1
> 14:31:44.887 [ERROR] o.a.j.t.m.j.d.SieveExecutor - Cannot evaluate Sieve 
> script for user 
> org.apache.openjpa.persistence.ArgumentException: There is no query with the 
> name "findActiveByUsername" defined for any of the known persistent classes: 
> [org.apache.james.mailbox.jpa.quota.model.MaxGlobalMessageCount, 
> org.apache.james.mailbox.jpa.mail.model.JPAUserFlag, 
> org.apache.james.mailbox.jpa.quota.model.MaxUserStorage, 
> org.apache.james.mailbox.jpa.mail.model.openjpa.AbstractJPAMailboxMessage, 
> org.apache.james.mailbox.jpa.quota.model.JpaCurrentQuota, 
> org.apache.james.mailbox.jpa.user.model.JPASubscription, 
> org.apache.james.rrt.jpa.model.JPARecipientRewrite, 
> org.apache.james.mailbox.jpa.mail.model.JPAMailboxAnnotation, 
> org.apache.james.mailbox.jpa.quota.model.MaxDomainMessageCount, 
> org.apache.james.mailbox.jpa.quota.model.MaxGlobalStorage, 
> org.apache.james.mailbox.jpa.mail.model.JPAMailboxAnnotationId, 
> org.apache.james.mailbox.jpa.mail.model.JPAMailbox, 
> org.apache.james.mailrepository.jpa.JPAUrl, 
> org.apache.james.mailbox.jpa.quota.model.MaxDomainStorage, 
> org.apache.james.mailbox.jpa.quota.model.MaxUserMessageCount, 
> org.apache.james.domainlist.jpa.model.JPADomain, 
> org.apache.james.user.jpa.model.JPAUser, 
> org.apache.james.mailbox.jpa.mail.model.JPAProperty, 
> org.apache.james.mailbox.jpa.mail.model.openjpa.JPAMailboxMessage].
> at 
> org.apache.openjpa.meta.MetaDataRepository.getQueryMetaDataInternal(MetaDataRepository.java:1982)
> at 
> org.apache.openjpa.meta.MetaDataRepository.getQueryMetaData(MetaDataRepository.java:1963)
> at 
> org.apache.openjpa.persistence.EntityManagerImpl.createNamedQuery(EntityManagerImpl.java:1200)
> at 
> org.apache.openjpa.persistence.EntityManagerImpl.createNamedQuery(EntityManagerImpl.java:1191)
> at 
> org.apache.james.sieve.jpa.JPASieveRepository.findActiveSieveScript(JPASieveRepository.java:153)
> at 
> org.apache.james.sieve.jpa.JPASieveRepository.lambda$findActiveSieveScript$6(JPASieveRepository.java:147)
> at 
> com.github.fge.lambdas.functions.FunctionChainer.lambda$sneakyThrow$49(FunctionChainer.java:74)
> at 
> org.apache.james.backends.jpa.TransactionRunner.runAndRetrieveResult(TransactionRunner.java:65)
> at 
> org.apache.james.sieve.jpa.JPASieveRepository.findActiveSieveScript(JPASieveRepository.java:146)
> at 
> org.apache.james.sieve.jpa.JPASieveRepository.getActivationDateForActiveScript(JPASieveRepository.java:133)
> at 
> org.apache.james.transport.mailets.jsieve.ResourceLocator.get(ResourceLocator.java:66)
> at 
> org.apache.james.transport.mailets.jsieve.delivery.SieveExecutor.sieveMessage(SieveExecutor.java:127)
> at 
> org.apache.james.transport.mailets.jsieve.delivery.SieveExecutor.execute(SieveExecutor.java:122)
> at org.apache.james.transport.mailets.Sieve.service(Sieve.java:75)
>
>
> Any hints?
>
> Thank you!
> =David
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

[Tellier Benoit]

Le 23/07/2020 à 15:57, Eugen Stan a écrit :
> La 23.07.2020 11:17, David Leangen a scris:
>
> It's great to see progress on this.
> I've also been busy with client/internal work.
>
> I'll make some time to finish up the documentation publishing over the
> weekend probably.
>
> Regarding the javadocs and mailet docs I believe we can make them as
> part of the automated build but not necessarily as part of Antora build.
I think the intent is not exactly the right one here.

I don't want operators to have to go through javadoc to have information
about their mailet / matcher (which was the original intent of my
proposition)
>
> Gradle generates sources and javadocs as part of java builds and publish.
> We can reference the javadoc jar in gradle.
> We can unpack it and push it to some folders as static files.
> Gradle is very powerful and in Jenkinsfile we can call maven as well
> to generate the mailetdocs.
> It's not pretty to use many tools but it should do the job until we
> find better options.
>
> I would stay away from migrating the mailet-docs right now.
> Too much work for too little value IMO.
Me too yes.

I prefer documenting manually each mailet/matcher individually so far,
based on the existing javadoc.
>
> Let's avoid this as much as we can and focus on getting the website up
> so we can feel good about it :).
+1
>
>
> Do you think it's possible to push for a website ready for review by
> 1st august?
I think the "Distributed Server" of the website would be mostly ready.
It could serve as an inspiration while writing other servers
documentation, and reviewing what had been done there first will be
valuable IMO.

I won't have time to work on other parts.

Also, I will be away from keyboard from 1st to 9th August ( vacations :-) )

Regards,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

[Tellier Benoit]

Le 23/07/2020 à 15:44, David Leangen a écrit :
 Cool! I will try to take some time today to comment more thoroughly, 
 likely in the form of a PR.
>>> There is quite a lot of text there and a lot of concepts, so it will take 
>>> me a while to go through it. :-)
>> Maybe more importantly than going through the entire text, the important
>> question for me would be:
>>
>>  - Which concepts do you miss to understand this doc?
> I will need a bit more time.
Take it :-)

It is not a problem if feedback comes a bit late.
>
>>  - Can we address that with a better structure?
> Ok, if I take a very quick top-down view, here are my immediate comments, 
> though I don’t know how useful they are. 
>
> I notice that effectively you mention:
>
>  * Running
>  * Configuring
>  * Operating
>  * Extending
>
> From our User Model, we have:
>
>   • User —> Probably not relevant
>   • Operator —> Relevant
>   • Integrator —> Relevant
>   • Developer —> 
>   • Contributor —> 
>   • Committer —> 
>   • PMC Member —> Probably not relevant
>
> What were your thoughts regarding the target audience of these docs? Was it 
> your intention that:

I'm targeting operators and integrators. This doc is not relevant for
others.

>
>  * Running —> Operator??
Yes. An operator needs to run the software.

As guice servers are not (yet) available for download, an operator have
either so far to compile the software or use docker images.

See https://github.com/apache/james-project/pull/221 for adding a ZIP
distribution of Guice servers that could be advertized on the dowload page.
>  * Configuring —> Operator??
Yes.
>  * Operating —> Operator (this one is clear)
>  * Extending —> Integrator (this one is clear, too)
Here we are more targeting operators that happen to be also extension
developers.

Maybe I need to explain clearly here who the target audience is.
>
> Or did you have some other structure in mind? It’s difficult for me to grasp 
> exactly what you are intending.
>
> I would have thought that “Operating” includes Installing, Configuring, 
> Running. An Operator (as we define in the User Model) would naturally go to 
> the “Operating” section to figure out how to Operate the server.
>
>
> Like I said, those are just my initial gut reactions. I’ll try to dig in more 
> next week.
I believe the "Operating" session is misnamed then. Maybe I mean
"monitoring & administrating". Would it address your concerns?
>> I'm really looking forward into this review ;-)
> Whew, pressure! Hope I don’t disappoint. 
>
>
>> Also note that I did not merge yet the most recent works and
>> improvements for the Distributed Server doc... Do you want me to merge
>> it before your review, to make it easier for you?
> Please, yes! Thank you.
Will do ;-)
>
> Cheers,
> =David
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.5.0

[Tellier Benoit]

Hi all,

I am happy to announce you the vote for the 3.5.0 release did succeed.

The release received 6 positive votes, 4 of them being binding.

Thanks to all contributors, developers and committers who made this
possible!

In the coming hours, I will finalize the release process, namely:

 - Publish the maven artifacts
 - Upgrade the download page and the (old) website
 - Announce the release

Cheers,

Benoit Tellier

Le 16/07/2020 à 13:44, Tellier Benoit a écrit :
> Hi,
>
> I would like to propose a new vote 3.5.0 release of the Apache James server.
>
> I fixes Matthieu remarks since last proposal.
>
> You can see changes proposed to the website at the occasion
> of that release on this GitHub pull
> request: https://github.com/apache/james-project/pull/187
>
> You can find:
>
>  - The maven release staged in repository.apache.org as the
> artifact #1048:
> https://repository.apache.org/content/repositories/orgapachejames-1048/
>  - The changelog for
> 3.5.0:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/upgrade-instructions.md
>
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Thursday 16th of July 2020, 1pm UTC+7
>  - The vote ends at Thursday 23th of July 2020, 2pm UTC+7
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> Cheers,
>
> Benoit Tellier
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

[Tellier Benoit]

Le 23/07/2020 à 14:27, David Leangen a écrit :
>> Cool! I will try to take some time today to comment more thoroughly, likely 
>> in the form of a PR.
> There is quite a lot of text there and a lot of concepts, so it will take me 
> a while to go through it. :-)
Maybe more importantly than going through the entire text, the important
question for me would be:

 - Which concepts do you miss to understand this doc?
 - Can we address that with a better structure?

I think fine grained page-per-page review is important, but is less
critical.

I'm really looking forward into this review ;-)

Also note that I did not merge yet the most recent works and
improvements for the Distributed Server doc... Do you want me to merge
it before your review, to make it easier for you?

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

[Tellier Benoit]

Hello,

I am making good progress on Antora documentation migration for the
Distributed Server, only a couple of pages are left.

I am encountering difficulties to document mailet/matcher an operator
can currently use.

Nowaday we rely on the mailet doc plugin to compile a website page from
the javaDoc of such mailets.

This page currently is in html format and will not be reusable for
Antora. Also the maven plugin will be an obstacle for the graddle
migration. Furthermore, not having to compile James server code when we
build the website seems desirable.

As such I propose to retire the maven-doc plugin, and migrate the
javadoc that exist today as Antora Documentation.

Also, now that I almost finished the Distributed server documentation, I
start thinking of my next step: documenting the other servers. I believe
we should keep documentation writing centralize to ease its maintenance
while keeping each server documentation complete for the reader path to
be straightforward. I believe Antora "include" directives here [1] are
the way to go, and I am thinking of putting that in practice as soon as
I will start documenting another server.

[1] https://docs.antora.org/antora/2.0/asciidoc/include-page/
[2] https://docs.antora.org/antora/2.2/asciidoc/include-partial/

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Which repositories are we using and going to document? Let's retire the ones we don't

[Tellier Benoit]

Le 20/07/2020 à 18:00, Eugen Stan a écrit :
> Hello,
>
> While working on the documentation website I ran into the question of:
> What parts to document?
>
> 1. What are the repositories we still use in James - some like jdkim
> have merged in james-project - others ?
No.

The JDKIM library still lives indepandantly from James.

Only it's mailet integration with the james server had been merged into
james-project.

The usage of JDKIM as a library would still benefit from being documented.
>
> 2. Which ones we don't use?
>
> 3. Which projects we wish to retire - if any? My suggestion: hupa, but
> we need to make a call and see if anyone steps in?
>
>
> Current list of repositories: https://gitbox.apache.org/repos/asf#james
>
> james-hupa.git 
> Apache James Hupa 23 weeks ago
+1 for retirement.
> james-jdkim.git 
> Apache James jdkim 42 weeks ago
This one is used as explained above/
> james-jsieve.git
>  Apache James
> jsieve 42 weeks ago
Used too. Even as
https://www.mail-archive.com/server-dev@james.apache.org/msg65858.html
pointed out we might benefit from a nicer API...
> james-jspf.git 
> Apache James jspf 42 weeks ago 
Used too.
> james-mime4j.git
>  Apache James
> mime4j 3 days ago
Used.
> james-postage.git
>  Apache
> James postage 75 weeks ago
I must confess I never used postage for load testing. I'm a stranger to
this sub-part of the project.

Some alternaves exist as JMeter extensions or Gatling libraries (IMAP,
JMAP & SMTP)

Maybe, given that their is very little contributions on this project we
could consider retiring it?
> james-project.git
>  Apache
> James project 3 days ago
> james-site.git 
> Apache James Website <30 minutes ago
>
>
> == More context
>
> Antora has a nice system - called [component-version] - that allows us
> to keep the documentation next to the sources and have multiple
> version of it.
>
> Right now we have this setup with only James project -
> https://github.com/apache/james-site/blob/live/doc-sites/antora-playbook.yml
>
>
> I do believe we should keep the documentation in every project and
> publish them as components.
+1
>
> We can add edit links back to github for each page.
>
> I'll create a component-version for each project and try to migrate as
> much documentation as I can (help needed).
I've started migrating documentation for the Distributed Server and am
making good progress on it.

I'm expecting the Advanced server to go faster.

I'm also interested into migrating the community part of the james site
to Antora.
>
> The website should be published from james-site which IMO should also
> contain the Jekyll content.
+1 this would be more natural.
>
>
> [component-version] https://docs.antora.org/antora/2.3/component-version/
>
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org

Moving forward on actual Blob deletion

[Tellier Benoit]

Hello all,

The Distributed Server so far is relying on the BlobStore component to
store potentially large binary data. This includes emails header, bodies
and attachments. The mailbox, mailRepository and mailQueue core
components, as well as the DeletedMessageVault extension relies on it.

The available implementations include a memory one (testing), cassandra,
and Object storage (S3). We furthermore added a cache mechanism in order
to fasten recent headers read.

As mail traffic is highly duplicated (a single mail can have several
recipients, a same attachment can be sent several time) we currently
deduplicate blobs stored in the blobStore.

However handling deletions in a deduplicated context is a non trivial
things. The problem is currentlmy covered in an ADR [1] and this work is
currently in (background?) process.

[1]
https://github.com/apache/james-project/blob/master/src/adr/0039-distributed-blob-garbage-collector.md

In order to achieve this work, we decided to distinguish the actual
BlobStore business logic from the Data Access Object layer. See
https://issues.apache.org/jira/browse/JAMES-3028 . This work had been
conducted out for both Memory and Cassandra but is yet to be contributed
for ObjectStorage. While working on this topic, we encoutered issues
with the current JCloud implementation, which is pretty boilerplate.

JCloud was chosen in order to support both S3 and Swift APIs. However,
JCloud don't allow asynchronous requests. This leads to a bad, not
performant threading model. Furthermore recent Swift version also
support S3 APIs. Thus we tried to significantly simplify the code by
dropping swift support and rely directly on a S3 client. [3] is a move
toward this and seems to unlock significant performance enhancements.

[3] https://github.com/linagora/james-project/pull/3430

This is to be noted that deduplication would need a garbage collection
to be run, which brings extra operating complexity. Some user might not
consider deduplication is worth this operational cost. Also we have to
ensure actual blob deletion while waiting a garbage collection solution
to be implemented.

In this context, one of Linagora customer have been deciding to found
the work of providing an alternative to blob deduplication. Most
Linagora contributors are likely going to work on this topic in the
coming weeks/months.

So far, we identified the following strategy:

 - Continue the work on JAMES-3028 and provide a s3 based blobStore DAO
as it is conclusive.

 - Write a DeDuplicatingBlobStore that replaces all current
implementations of the blobStore. It will rely on the BlobStoreDAO
interface (currently DumbBlobStore)

 - Write a PassThroughBlobStore. This BlobStore stores each blob
separately, and don't deduplicate content. It can effectively delete
content right away, without any garbage collection to take place.

 - Expose a configuration option of the DistributedServer for choosing
either the PassThroughBlobStore or the DeDuplicatingBlobStore.

We of course need to ensure configuration management. If I go from
'deduplication.enable=true' to 'deduplication.enable=false' I can end up
deleting some blobs referenced by other entities, making their content
no longer available. In other parts of the code base, event sourcing is
used to handle such concerns.

Given that one cannot "disable deduplication after enabling it" and that
garbage collection is currently not implemented, we should disable
deduplication by default.

 - The current only usage of `BlobStore::delete` is a special case:
DeletedMessageVault::deleteMessage. This operation is intended to
immediatly delete a blob and a garbage collection based algorithm is not
suited for this need. Of course we should avoid a 'feature' poluting the
API of the BlobStore. DeletedMessageVault::deleteMessage could be
handled via a call to the BlobStoreDAO. As a consequence, we will be
able to turn `DeDuplicatingBlobStore::delete` into a noop operation,
while waiting for dereferencing/garbage collection to resume.

 - Once Deduplicated content is no longer aggressively deleted by
BlobStore::delete, we can ensure mailbox, mailqueue and mailRepository
data being effectively deleted when using the PassThroughBlobStore.

Note that in order to simplify this work, I propose to drop the already
deprecated HybridBlobStore.

Do you see some other solutions and workarounds?

Best regards,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Call for vote: Apache James 3.5.0

[Tellier Benoit]

+1

Le 16/07/2020 à 13:44, Tellier Benoit a écrit :
> Hi,
>
> I would like to propose a new vote 3.5.0 release of the Apache James server.
>
> I fixes Matthieu remarks since last proposal.
>
> You can see changes proposed to the website at the occasion
> of that release on this GitHub pull
> request: https://github.com/apache/james-project/pull/187
>
> You can find:
>
>  - The maven release staged in repository.apache.org as the
> artifact #1048:
> https://repository.apache.org/content/repositories/orgapachejames-1048/
>  - The changelog for
> 3.5.0:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/upgrade-instructions.md
>
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Thursday 16th of July 2020, 1pm UTC+7
>  - The vote ends at Thursday 23th of July 2020, 2pm UTC+7
>
> You can answer to it just with +1 and -1. Down-votes may be motivated.
>
> Cheers,
>
> Benoit Tellier
>
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Call for vote: Apache James 3.5.0

[Tellier Benoit]

Hi,

I would like to propose a new vote 3.5.0 release of the Apache James server.

I fixes Matthieu remarks since last proposal.

You can see changes proposed to the website at the occasion
of that release on this GitHub pull
request: https://github.com/apache/james-project/pull/187

You can find:

 - The maven release staged in repository.apache.org as the
artifact #1048:
https://repository.apache.org/content/repositories/orgapachejames-1048/
 - The changelog for
3.5.0:
https://github.com/chibenwa/james-project/blob/website-3.5.0/CHANGELOG.md
 - The compatibility instructions/upgrade
recommendation:
https://github.com/chibenwa/james-project/blob/website-3.5.0/upgrade-instructions.md

Voting rules:
 - This is a majority approval: this release may not be vetoed.
 - A quorum of 3 binding votes is required
 - The vote starts at Thursday 16th of July 2020, 1pm UTC+7
 - The vote ends at Thursday 23th of July 2020, 2pm UTC+7

You can answer to it just with +1 and -1. Down-votes may be motivated.

Cheers,

Benoit Tellier

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Consensus needed: replace james-site master branch with live branh - reset

[Tellier Benoit]

Le 13/07/2020 à 13:01, David Leangen a écrit :
> Hi,
>
>>> If no objections are received in the next 3 days I'll move forward with
>>> the change.
> Thanks for all your efforts. Sorry that I have not moved forward much in the 
> past days. I have been preoccupied with other things.
>
> I am having trouble imagining what you are referring to exactly in your 
> description. It would be very useful for me to see some kind of visual 
> representation of how you are intending to organize the site.
I think Eugen is mostly speeking of where to locate the JenkinsFile with
the instructions to build the site.
>  I was thinking that the current top page is pretty nice, so maybe it could 
> be reworked, then the docs would somehow fit into that.
+1

I propose we keep the same front page (index.html) and migrate the
documentation website whose landing page is currently documentation.html.

This would just mean Changing Antora root index.html into documentation.html

Cheers,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Consensus needed: replace james-site master branch with live branh - reset

[Tellier Benoit]

Le 13/07/2020 à 03:29, Eugen Stan a écrit :
> Hello,
>
> I would like to use james-site#master branch to hold the code to build
> the website automatically.
+1
>
> If no objections are received in the next 3 days I'll move forward with
> the change.
>
>
> Moving forward with the website I would like to replace the existing
> james-site master branch with the current live branch.
>
> The old master is backed up in master-backup.
>
> The old master is generated and is not the current live website.
>
> The current live website is served from asf-site.
>
> Nothing will change with respect to live and staged env, master is
> unsued to my knowledge.
To mine too...
> The goal is to start clean with a fresh site building code.
>
> The code will live in james-site and will use Antora to build the website.
>
> Antora will take the content from the other git repositories.
>
> All this has been tested out and it works, we need to polish it and
> document it. 
+1, I propose to do it in the community page of the antora website.
> I've gone through the Antora docs over the weekend and I'll start
> pushing changes soon.
>
>
> https://github.com/apache/james-site/tree/live
>
> https://github.com/apache/james-site/tree/master
>
> https://github.com/apache/james-site/tree/master-backup
>
> https://github.com/apache/james-site/tree/asf-site 
>
>
> Thanks,
>
Thanks for your investment!

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: Building Antora documentation

[Tellier Benoit]

Le 10/07/2020 à 07:34, David Leangen a écrit :
> Hi Eugen,
>
> Thank you very much for all your efforts. The community is fortunate that you 
> are able to dedicate so much concentrated effort.
>
>> I've given up on building the old site since I failed several times.
> I think that’s ok, though. Honestly, I don’t see a viable path forward in 
> terms of “migrating” the old docs. Yes, there is some good information still 
> in those docs, and some good stuff that could inspire the new docs, however, 
> I think the only practical way forward is to start over, from a top-down 
> perspective. IMO, we need to make a clean break, and at the same time 
> re-evaluate the James mission. (But that’s just my opinion and I’m just a 
> newbie so my voice doesn’t hold as much weight.)
In my opinion the current doc lacks structure.

We miss some concept and "architecture" forwords. We miss a "per server
documentation".

What I mean is that we don't need to write all over again, but in the
process of migrating things we can identify what is missing.
>
> I like the inventory. I can use that as a checklist to make sure the new docs 
> aren’t leaving anything out. Or maybe the old docs could be assembled for the 
> “developer” part of the site, aimed at experienced James developers who 
> already know their way around and just need a reference. But I don’t see how 
> the current docs could be useful for Operators or Integrators who don’t have 
> experience with James.
I prefer dropping it altogether.

We need to simplify things. And the mvn site don't handle versionning.
Even for experienced people I doubt that it can bring value compared to
Antora.
>
> So with that in mind...
>
>> I've create two issues to track the progress of migrating the old site
>> to Antora.
>>
>> * Make an inventory of old site
>> https://issues.apache.org/jira/browse/JAMES-3301
> Cool, thanks! I think this can be useful.
I will have a look, but I don't know if it will be today ;-)
>> * Migrate the content https://issues.apache.org/jira/browse/JAMES-3302
>>
>> Please help out with the inventory and if you have ideas with the content.
>>
>> I'll migrate what I can, we can sort them after.
> I don’t think that’s necessary, but if it’s already done, then great.
IMO:

 - Having the ADRs part of the website is nice
 - For the over contents part of the "migrated" module, we should think
of it as an help for the actual doc writing.
>
>
> Thanks again!
> =David
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: SMTP Relay (Was: Queuing vs. spooling)

[Tellier Benoit]

Le 09/07/2020 à 07:42, David Leangen a écrit :
> Hi Eugen,
>
> Thanks for your support.
>
>> GIven the discussion around the specific topics I think we are getting
>> our documentation.
>> @David: If you can do just that: ask questions and compile the answers
>> it would be a huge win for us.
> Thanks, that is indeed the intent since the beginning. The ability to move 
> forward depends on how much everybody participates in answering my (often 
> stupid) questions.
>
> I am approaching this from a newbie’s perspective, simply because I am one. 
> As I mentioned in the beginning, (although less and less so now) I have the 
> “advantage” of knowing very little about how James works, so it is easier for 
> me to imagine what it’s like for other newbies trying to learn James, and 
> what needs to be done to help make James more usable and thus more popular.
>
> It does require time and patience from everybody though. I hope the efforts 
> will turn out to be worthwhile.
It definitely worth it.

+1 on the conclusion of this thread.

Benoit
>
> Thanks for all the help so far. Please keep it coming!!
>
>
> Cheers,
> =David
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Le 08/07/2020 à 14:40, David Leangen a écrit :
> Sorry, I do have one more question in response to your email…
>
> You wrote:
>
>> In my opinions we should document "How to write hooks with the
>> protocols/smtp library", "How to plug such hooks into a running James
>> server"
>>
>> Then "How to write commands for the protocols/smtp library" (and how to
>> plug them in James)
> This is starting to make better sense to me now. Thanks for your patience.
>
> Moving on… on this page:
>
>   —> https://james.apache.org/server/feature-smtp-hooks.html
>
>
>> The James SMTP Server Component allows to easy write your own code which 
>> will get executed in the SMTP-Transaction. Thats a bit different then using 
>> a Mailet a.k.a Mailet-API.
>>
>> To customize your SMTP Server, you have a few interfaces which helps you to 
>> "hook-in" a specific SMTP Command. That means your class which implements 
>> the given interface(s) will get called after the SMTP-Command was parsed and 
>> depending on your implementation it will handle it.
>>
>> As your code will get executed before the mail was even accepted. This can 
>> help you in many ways, most times its used for rejecting SPAM/Junk within 
>> the SMTP-Dialog. But it can be used for other things too.
>>
>> Its up to you and your use case.
>>
>> But be aware as your code needs to get executed during the SMTP-Transaction 
>> it should not take to long to execute. As it will need to fit in before the 
>> timeout was hit which can be different on every mail server. But as a 
>> general rule as long as your code can get executed within 30 seconds it 
>> should be fine.
>
> There is even a list of hooks on this page:
>
>   —> https://james.apache.org/server/dev-provided-smtp-hooks.html
>
>
> However… Can you provide more examples to help me better understand why I 
> should care about these hooks, and why I would want to consider using a hook 
> instead of a Mailet? Wouldn’t it be simpler just to have a single extension 
> mechanism? What is the value of having two different extension mechanisms?
That is a VERY nice question that needs to be clearly highlighted in the
documentation "given my use case, which extension mechanism should I use".

There's somehow an overlap between hooks and mailets but:

 - Mailet is *asynchronous*. SMTP clients will not be affected by
potentially long processing. Hooks are on the other hand synchronous,
the SMTP client needs to wait for them to happen.
 - Hooks are very cheap. The mail is not stored yet. This makes it
really useful to get rid of spam via IP filtering, MX hosts checks and
so on. Another use case would be DOS prevention.
 - Hook allow configuration of extra additional behaviors at the SMTP
layer like "Sender needs to be valid, etc...".

I believe however that this extension mechanism is less useful than
mailet / matchers.

Does it helps?
>
> Thanks!
>
> =David
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: SMTP Relay (Was: Queuing vs. spooling)

[Tellier Benoit]

Le 08/07/2020 à 15:33, David Leangen a écrit :
> Thanks.
>
> I love this James adventure. It is not boring. Every time I scratch the 
> surface, a new concept pops out. 
>
> Ok, so digging in…
:-)
>> Also as Matthieu said, RemoteDelivery is a side effect of the current 
>> Processing chain.
> I looked at org.apache.james.transport.mailets.RemoteDelivery, with the 
> assumption that it is the correct Mailet.
Correct
> First, the javadoc says:
>> The RemoteDelivery mailet delivers messages to a remote SMTP server able to 
>> deliver or forward messages to their final destination.
> That is consistent with what you wrote. So I looked at the code to try to get 
> a sense of *how* it does this. 
Ouch. Complex and old code relying on javax.mail is waiting you ^^
> What I could gather is that the magic is actually done with 
> org.apache.james.queue.api.MailQueue. All RemoteDelivery does is enqueue the 
> Mail, then it’s done.
It do start DeliveryRunnable that consume the outgoing queue. Relaying
happens there.
> So the queue-api seems now quite important. 
I confirm it is.
> I also gather that it is the same queue that accepts mails from the incoming 
> SMTP Service in the figure. However, by looking at the api code and the 
> sparse javadocs that come with it, I am getting few clues as to what it does 
> or how it works.
>
> I can only guess that the actual heavy lifting is done with an implementation 
> that gets wired with Guice.
It is guice agnostic. Look inside RemoteDelivery::init...
> I will investigate further, but in the meantime, can somebody please point me 
> in the right direction to help speed up my journey?
>
Did I just do that or you need more information?

Cheers,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: SMTP Relay (Was: Queuing vs. spooling)

[Tellier Benoit]

Le 08/07/2020 à 14:39, David Leangen a écrit :
>> The only wrong thing about this picture is the SMTP Service before 
>> "Outgoing".
> Ok, thank you Matthieu.
>
>> As weird as it is, the delivery of messages to a remote server is done by a 
>> Mailet called RemoteDelivery and it's not handled by the SMTP Service.
>> As far as I know, a lot of people are actually forking this Mailet because 
>> they want specific behaviors for delivery so I think this design makes sense.
> Ok sure, but doesn’t that Mailet have to “speak” SMTP? 
No they don't.

They encapsulate what a "Mail" (a Mime message with its transport
context) is and transform it.

They can for instance be used after a JMAP submission servers (as users
can send mails via JMAP too)

Mailet are just a very simple Java interface. There is no hard
dependency to SMTP.

> I.e., doesn’t it effectively become an SMTP client, that opens a session with 
> a remote SMTP server?
>
> Or am I again misunderstanding something fundamental about how SMTP works?
>
>
> Cheers,
> =David
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: SMTP Relay (Was: Queuing vs. spooling)

[Tellier Benoit]

Nice image!

 - there is a queue before SMTP outgoing service (as you don't want to
handle SMTP synchronously anyway)

client -> SMTP (in, as a server) -> queue (spool) -> Processing chain ->
Queue (outgoing) -> RemoteDelivery runnable

Also as Matthieu said, RemoteDelivery is a side effect of the current
Processing chain.

Cheers,

Benoit

Le 08/07/2020 à 14:21, David Leangen a écrit :
>
> Still on the topic of SMTP relay and the original image I posted at
> the beginning of this thread:
>
>  —>  https://james.apache.org/images/james-smtp-relay.png
>
>
> Would the attached (lousy) image be a reasonable representation of the
> general concept of an SMTP Relay?
>

Re: James SMTP Model

[Tellier Benoit]

Le 06/07/2020 à 14:12, David Leangen a écrit :
> Thanks Benoit.
>
> Now, since I already have your attention on this topic, please allow me to 
> really push so I can reach some kind of resolution. The resolution will allow 
> me to figure out how to move forward with several things. Since I have been a 
> bit stuck the past few days (and growing frustrated a little ) I hope you 
> can continue to help me resolve this.
I gonna try my best, given my time constraints :-)
> Let me keep the focus on the SMTP module, because I think we’ll likely be 
> able to extrapolate much of the approach to the other modules as well.
>
> I think this is what we seem to agree on:
>
>  * Separation of API and implementation is generally a good thing (especially 
> if we want clean components)
>  * The current API is a bit messy and could use some cleaning
>  * However, the value of making any changes is questionable at this time
>
>
> Here are also a few things I have discovered on this journey:
>
>  * Actually implementing RFC5321 as a Java API would be VERY challenging
>  - The spec is fairly complex
>  - The spec is very “messy” and not very well or precisely written IMO
>  * As you pointed out, there are some important elements in the James SMTP 
> module, such as commands and hooks
>  - These ought to be somehow highlighted and improved in the API
+1
>  * You have mentioned more than once that it uses a hexagonal architecture
>  - I think this is important, but I still don’t know what to do about it
>
>
> So I guess my question is this: what should I do in the immediate term? As 
> you and others have pointed out (and me also), the documents should show what 
> James *is*, not what James *aspires to be*.
In my opinions we should document "How to write hooks with the
protocols/smtp library", "How to plug such hooks into a running James
server"

Then "How to write commands for the protocols/smtp library" (and how to
plug them in James)

Using protocols-smtp without the server/protocols/protocols-netty super
layer would be so far a non-goal to me, and I would likely not be able
to help you do this.
> So what is the SMTP module, and how should I describe it?
I would state `james-project/protocols/protocols-smtp` is a transport
agnostic implementation of (some of) the SMTP specifications. It is
customizable (hooks) and extensible (additional commands).

`james-project/server/protocols/protocols-smtp` adapts
`james-project/protocols/protocols-smtp` in the James context, providing
hooks interacting with James server components. It also provides a TCP
transport layer, effectively providing a SMTP server.

Does it answers your question?
> In the part of the docs entitled “James model”, if I were to write in all 
> honesty I don’t feel that there is a clear model, or at least it is totally 
> not clear (to me) from the API. James is an implementation of RFC5321, and 
> basically makes no attempt to have a clear java API. Rather, the API is via 
> JMX. (I will have to investigate further to see what the JMX view of the 
> system is.)
Ouch. Please don't.

It's old, not clean, mostly unmaintained, and likely to be removed in
the (not that far) future - for security reasons because JMX. Don't
hesitate if you want details on this statement (maybe a separate thread?)

To 'operate' James we should prefer webadmin. That will mean providing
CLI tool on top of it for the less technical of our users.

What do you try to achieve? (I don't see the link making you go from
SMTP to JMX to be honnest)
> I could mention that there is a “plan” to create cleaner java apis.
>
> That’s about all I can think of right now. Would this approach be acceptable?
There's always a plan in my opinion to consider API enhancement
proposals :-)
>
 Note that "standard SMTP commands" are bundled into a single
 ProtocolHandler, not relying on it means you defines your very own SMTP
 commands.
>>> Ok, these are the types of things that I am very interested in 
>>> understanding better. Are there any docs anywhere to help me out?Not really
>> Not really.
>>
>> http://james.apache.org/server/dev-provided-smtp-hooks.html defines hook
>> you can rely in for the default implementation.
>>
>> https://github.com/apache/james-project/blob/master/src/site/xdoc/server/dev-extend-smtp-hook.xml
>> might be helpful too.
>>
>> Which is not much for "hooks" already. Given the part of the quote you
>> chose, that might not be what you are looking for...
>>
>> Regarding writing your own commands, there is even less.
>>
>> http://james.apache.org/protocols/index.html is too generic
>>
>> http://james.apache.org/protocols/smtp.html is blank
> Would it be useful for me to pursue this in an attempt to document the SMTP 
> module? I could try, but again I’ll most likely need help.
>
> Or should I write something like what I mention above and just be done with 
> it for now?
You proved us writing documentation (and software) needs to be performed
with a usage 

Re: Queuing vs. spooling

[Tellier Benoit]

Le 07/07/2020 à 20:58, David Leangen a écrit :
>> Hope it helps.
> Yes, quite a lot!!
>
> A few clarifications, please. 
>
>> SMTP Service is talking TCP with the client. When it is asked to
>> deliver a message, it simply calls `enqueue` on the MailQueue.
> Can you be more precise about what you mean by “client”?
By SMTP client Matthieu means either a Mail User Agent (eg Thunderbird)
or a remote mail server transferring emails to our server, thus sending
SMTP client commands.
>> As everything happens in the Java process, you don't have a protocol,
>> just a method call.
> By the way, is the message communicated via TCP or via a method call. Sorry, 
> I am a bit confused about the two statements you made above.
>
> Once the mail is received by the mail queue, does all the rest of the process 
> happen via method calls, too?
Via method calls. protocols/protocols-smtp library is agnostic about it,
but server/protocols/protocols-smtp does the link between the smtp
library and other James components (was it your question?)
>> The spooler is the thing taking messages from the queue for processing.
>> The MailQueue allows to decouple the reception from the handling.
>> A spooler usually is able to concurrently process several mails.
>
> Thanks.
>
> Are there any other important “parts” that I should be aware of?
Regarding what?
>
>
> Cheers,
> =David
>
Cheers,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Le 06/07/2020 à 13:12, David Leangen a écrit :
> [...]
>> I believe people are interested by a working SMTP implementation where
>> they can "plug" there behaviors or add the commands they miss.
> I am beginning to think that we may have a fundamentally differing 
> understanding of what an API is. It may be a good idea at some point to have 
> a very clear mutual understanding as to what an “API” is to James.
>
> I say that because I would never have written the sentence you wrote. I would 
> have written “a pluggable component”, but never a “pluggable implementation” 
> because no consumers should ever have to plug anything into an 
> implementation. So how does something get plugged? It’s **necessarily** via 
> the API (while the framework should handle the wiring with the 
> implementation, just like Guide does at system startup). Only the API should 
> ever be exposed to consumers. Consumers should never require the 
> implementation code to compile their own code and work with James. It should 
> always be enough to just have the API jar (consisting almost entirely of 
> interfaces), with NO implementation jar (consisting of implementation 
> classes).
>
> If you need to plug something into the implementation, then by definition the 
> implementation is also an API, but then it means that you are mixing 
> implementation with API, which is not usually a good practice if you want to 
> have a componentized system. Consumers necessarily only interact with an API, 
> by definition.
>
>
> Another purpose of having a solid API is for versioning. Normally an 
> implementation will continue to advance (bug fixes, optimizations etc.), but 
> for consumers of the API, the code they depend on (i.e. the API only) should 
> change as little as possible.
>
> So the API ought to:
>
>  * Communicate the intention VERY well
>  * Not include “extra” stuff that is not absolutely necessary
>  * Definitely not include any implementations
>  * Allow consumers to minimize dependencies (or at least not require 
> additional transitive dependencies)
>  * Allow consumers to swap different implementations (including different 
> implementation versions)
>  * Allow consumers to use more recent implementations without having to 
> change anything in their own system
Globally +1
>
> Here are an article that expresses views about APIs that are similar to mine:
>
>  * 
> https://developer.ibm.com/technologies/java/articles/api-design-practices-for-java/
>
> It only touches on the differences between an API “consumer” and “provider”, 
> but it would probably be a good thing to bring that into the James vocabulary 
> as well so we can have slightly richer exchanges regarding APIs. I think what 
> you are referring to is that we need to consider how consumers use an API, 
> not just providers, and I totally agree.
>> (but again we need to consider the costs - return over investment ratio)
> Always! Completely agree. That is why I mentioned that I was having second 
> thoughts and was starting to think that maybe we could get by WITHOUT having 
> a proper java api (and maybe only a REST api, for instance).
>
> In that scenario, consumers of James would ONLY interact either via a REST 
> API or SMTP / IMAP etc. There would simply be no java API because nobody 
> would program against James.
>
> I think that could work, but we ought to make our approach explicit to avoid 
> confusion.
+1 to be more specific.

We should at least extract the API of what todays works well:
 - Extra commands
- Hooks (additional behaviors)

And extract an API achieving these goals from the protocol-smtp package.

> Note that "standard SMTP commands" are bundled into a single
>> ProtocolHandler, not relying on it means you defines your very own SMTP
>> commands.
> Ok, these are the types of things that I am very interested in understanding 
> better. Are there any docs anywhere to help me out?Not really

Not really.

http://james.apache.org/server/dev-provided-smtp-hooks.html defines hook
you can rely in for the default implementation.

https://github.com/apache/james-project/blob/master/src/site/xdoc/server/dev-extend-smtp-hook.xml
might be helpful too.

Which is not much for "hooks" already. Given the part of the quote you
chose, that might not be what you are looking for...

Regarding writing your own commands, there is even less.

http://james.apache.org/protocols/index.html is too generic

http://james.apache.org/protocols/smtp.html is blank

> Thanks as always for your patience.
>
You are welcome, and I'm happy you raise such an opportunity to
re-qualify or document things!
>
> Cheers,
> =David
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Le 06/07/2020 à 07:13, David Leangen a écrit :
> Hey Benoit,
>
>> Now, that being said, I believe we should always define a "purpose" for
>> an API, and here we mixes things in my opinion.
> Yes, very good point. I agree 100%
>
>
>
>> Are we defining an API for building any arbitrary SMTP server?
> My somewhat vague thought is: I get the impression that in the Java world 
> Apache James has become a kind of reference implementation. It could be that 
> I have just not found anything else, or that nothing else exists, but in any 
> case James is “important” in the intersection of “email” and “JVM”.
I don't think people are interested by an API allowing them to implement
SMTP themselves. (That is quite some work!)

I believe people are interested by a working SMTP implementation where
they can "plug" there behaviors or add the commands they miss.
> I think we should take it upon ourselves to view James and take on the role 
> as the “reference implementation” for the JVM.
>
> This means that we ought to produce java apis that are compliant to the 
> specs, that anybody could implement if they wanted to as technology develops. 
> It also implies that we need to give the API lots of thought as this would 
> become a de-facto industry standard, at least in the JVM world. Heck, if we 
> do a really great job maybe we can even inspire improvements to the email 
> spec itself. We should aim high.
>
>
> Take the fictitious case of a new, awesome “super-nutty” technology that is 
> even better than Netty, but works in a completely different way. I should be 
> able to make my own super-nutty implementation of smtp based on the James 
> reference API.

Globally +1, if it lives as a separate library, we need a more
declarative way off assembling things...

(but again we need to consider the costs - return over investment ratio)

>> Or are we shipping an SMTP implementation with APIs to add new commands
>> support ?
> I am no expert in the spect, but aren’t you referring to ESMTP? Or do you 
> mean something else?
>
> Even if we make a “reference api for Java”, there is nothing from stopping us 
> from making an extension mechanism.
That mechanism already exists: one can register its own "protocol
handlers", be them extra commands or hooks into existing commands.

Note that "standard SMTP commands" are bundled into a single
ProtocolHandler, not relying on it means you defines your very own SMTP
commands.

Cheers,

Benoit


-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: new committer: David Leangen

[Tellier Benoit]

What an awful formatting, let me correct this...

The Project Management Committee (PMC) for Apache James 
has invited David Leangen to become a committer and we 
are pleased to announce that he has accepted. 

David started being active on the James project several 
months ago, and started a huge documentation effort he 
seems committed to.

Furthermore,while doing this documentation work, he asked 
many questions that helps us refining what the Apache James 
project is all about. 

Being a committer enables easier contribution to the project 
since there is no need to go via the patch submission process. 
This should enable better productivity. 

Le 05/07/2020 à 23:27, Tellier Benoit a écrit :
> |The Project Management Committee (PMC) for Apache James has invited
> David Leangen to become a committer and we are pleased to announce that
> he has accepted. ||David started being active on the James project several 
> months ago, and
> started a huge documentation effort he seems committed to. Furthermore,
> while doing this documentation work, he asked many questions that helps
> us refining what the Apache James project is all about. Being a
> committer enables easier contribution to the project since there is no
> need to go via the patch submission process. This should enable better
> productivity. Being a PMC member enables assistance with the management
> and to guide the direction of the project.|
>
>

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

new committer: David Leangen

[Tellier Benoit]

|The Project Management Committee (PMC) for Apache James has invited
David Leangen to become a committer and we are pleased to announce that
he has accepted. ||David started being active on the James project several 
months ago, and
started a huge documentation effort he seems committed to. Furthermore,
while doing this documentation work, he asked many questions that helps
us refining what the Apache James project is all about. Being a
committer enables easier contribution to the project since there is no
need to go via the patch submission process. This should enable better
productivity. Being a PMC member enables assistance with the management
and to guide the direction of the project.|

Re: James SMTP Model

[Tellier Benoit]

Hi Eugen,

You will find my answers inlined.

> 
> I believe you make the assumption that people use and want to use the
> full plethora of protocols we have in James.
> 
> Sort of "All or nothing" approach.
> 
> Why do you think it is so?
> 
> I would argue that we should make little assumptions about how people
> deploy their infrastructure.
> 
> Maybe I have an existing infrastructure that I know and trust for SMTP
> and I just want to use the IMAP server from James.
> 
> Examples:
> 
> 1. I use Amazon Simple Email Service (or Sendgrid or Mailgun) for
> sending emails and I just want to be able to customize the IMAP/JMAP
> experience and control how I store emails.
> 
> 2. I could use James SMTP as a Gateway for my existing IMAP / Webmail setup.
> 
> 3. I could use James SMTP as a Relay for my marketing SaaS platform.

We should of course have an SMTP only server to address these use cases.

https://github.com/apache/james-project/tree/master/server/container/guice/jpa-smtp
achieve it.

> 
> 
> But that is not the main argument for having them separate.
> 
> The main argument IMO is that the protocols are quite complex and having
> the ability to work on just one protocol at a time has significant
> advantages:
> 
> * Lower cognitive effort - I only need to care about this specific
> protocol and it's implementation details, not the other protocols.
> 
>   This is one huge benefit.
> 
> * Faster development feedback cycle - If I work on IMAP or SMTP I should
> get away with running the tests only for that part.
> 
> * Less risk - If I work and break SMTP - the other parts should work.
> With the current project structure - it can be hard to say.
> 
> * Easier to adopt by new people - If people care only about SMTP, LMTP
> or IMAP - they can contribute only to that. The entry barrier will be
> smoother than what we have now.
> 
> * Easier to compose and adapt for new and interesting projects - If we
> package them independently and together in a single product (Advanced
> Server) we have a showcase of how you can do that. It will be clear that
> they are meant to be used in both ways.
> 
> 
> Having them as separate projects may have the downside of integrating
> them together as integrating them requires some form of glue.
> 
> The good side is that we already pay for this integration effort now :).
> 
> We will have to do the effort of splitting them apart while also keeping
> the full version.
> 
> This is doable and while it will take time it's not a hard problem IMO.
> 

Regarding Mail Delivery servers, you end up supporting IMAP/JMAP + SMTP.

I believe we should be doing a better job at making things optional and
packaged separately.

I would love to see for instance POP3, ManagedSieve and many other
things as extensions, bundled in separated JARs, opt-in.

Shipping a small set of servers, with very limited features, each one
with their set of extensions.

I believe it achieves most of what you describes but without requiring
complex integration.

> 
> Our way of using James is not the only way.
> 
> 
>>> [...]
> 
>>> NOTE: Netty should be upgraded to the latest version, I can do that once
>>> we are done with Gradle.
>> +1 <3
>>
>> Likely a huge effort though...
> You questioning my determination ?! (joke)

Never ^^

I don't know which one is easier... Gradle? Or Netty?
> 
> In time I will change the universe :D .

No doubt about it too.

Regards,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Hi David,

As I exposed it earlier, protocols-smtp mixes the protocol
implementation and the interface it defines. We would benefit from
separating the too.

Now, that being said, I believe we should always define a "purpose" for
an API, and here we mixes things in my opinion.

Are we defining an API for building any arbitrary SMTP server?

Or are we shipping an SMTP implementation with APIs to add new commands
support ?

Or are we shipping an SMTP implementation with APIs to add new behavior
on existing commands ?

I believe protocols-smtp addresses the later two, and should be
documented/re-organised if needed with that optic in mind.

Best regards,

Benoit

Le 05/07/2020 à 17:42, David Leangen a écrit :
> 
> Hi there,
> 
> I think that my thoughts about components seem to be quite aligned with those 
> of Eugen, so I won’t really repeat anything here. I agree with pretty much 
> everything he writes about the advantages of having clean components. Perhaps 
> the only thing I would point out is that even with clean APIs, you can still 
> “mix up” the implementations. There is nothing prohibiting using a single 
> “super protocol implement” that implements all of the protocols, even if the 
> APIs are very cleanly separated.
> 
> I will only repeat that to have good componentization, independent (i.e. 
> non-coupled) and very clear APIs are a necessary prerequisite. So to really 
> get the most of Guice, I think there is some work to be done.
> 
> 
> Regarding the hexagonal architecture, I am not quite sure what to say. It 
> sounds good on paper, but IMO any good architecture should help with 
> communication. For the life of me I can’t seem to find my way around even 
> knowing that it is based on this architecture. I don’t find any hints in the 
> code. Can you direct me?
> 
> 
> However….
> 
> 
> Maybe given the current state of James we could just take a completely 
> different approach. Instead of aiming for clean componentization, which would 
> be a huge undertaking, and in any case the specs seem very “dirty” to me, and 
> since James is already “working” maybe we could just use the “black box” 
> approach. It is a closed system, so it doesn’t need APIs for its components.
> 
> The “interface” is simply a clean input and output function for emails.
> 
> Come to think of it, maybe that’s what the hexagonal architecture ends up 
> doing??
> 
> 
> Cheers,
> =David
> 
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Building Antora documentation

[Tellier Benoit]

Hi all,

In an evening effort, I started writing a documentation skeleton for the
Distributed James server with the new Antora Documentation.

In order to juge the results I came up with (and do some corrections as
I am new to this technology) I would like to generate the result of my work.

I don't think I saw explanations of how to generate the James
documentation website, I tried running some commands on the Antora
website but I did not achieve anything conclusive.

Can someone provide me the commands needed to generate the documentation
website? (I successfully installed Antora and http-server)

Regards,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Hello David,

Le 03/07/2020 à 04:30, David Leangen a écrit :
> 
> [...]
>
>> I don't believe the code is super clean but it does work and is quite
>> efficient and fast.
> 
> Yes, that is important, but again, this comment about being “efficient and 
> fast” seems perfectly valid for an implementation, but not really for an API. 
> The API should be more about expressing the model and drawing clear component 
> boundaries. (It should not preclude a good implementation, of course, but 
> should usually not force a particular implementation either.)
> 
>> I agree with the coupling and it is going to take some effort to go
>> through all of the modules and clean them up.
> 
> Since I am pretty much stuck with the documentation efforts (as I have 
> mentioned I need more input from the community if I am to move forward), what 
> I think I’ll try to do next is write an “independent” API for SMTP. My hope 
> is that this will:

>From what I recall, business class exposed by the API are pretty well
defined: it exposes some hooks one could implement.

The big issue here is more that the protocols-smtp project does mix
together the "port" (as in hexagonal architecture) and the
implementation / wiring of it.

I believe, from a documentation perspective much can be achieved by
extracting a 'protocols-smtp-api' exposing the classes I can implement
as a protocols-smtp user, and having the actual smtp protocol
implementation in a separated project.

I believe Gradle will also make this "API extraction" easier.

As part of your documentation effort, I can provide a listing of such
classes; while waiting for the module extraction.

> 
>  * Provide an example as to how to write a “clean” and independent API
>  * Help with documentation (because in a way, the API **is** the 
> documentation because it should reflect the model very well)


Here I have several concerns:

 - We should document what exist, and not a possible desirable future
state that may not be reached.

I believe if some reasonably easy moves could be done to ease the
documentation process, we should give it a go.

(The above proposal tries to achieve this)

>  * Continue to use the “existing” code that “does work and is quite efficient 
> and fast” to implement the new API
> > In the worst case we can throw it away, in the best case it will
provide an example of how to move forward with better componentization
of James.

Here I'm concerned. Not that throwing code and rewritting things is a
bad idea, but it needs to be seen as an investment.

The question I would ask myself is "what do we gain from it?" "what are
the costs?" and "Is it the right moment to do that investment".

Now, protocolHandlers is a widely used API to extend James behavior and
we should pay special care about API stability to prevent our users to
rewritte their extensions.

Regards,

Benoit




-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: James SMTP Model

[Tellier Benoit]

Le 02/07/2020 à 23:03, Eugen Stan a écrit :
> Hello David,
>
> [...]
> 
> Agian, I do think protocols should be independent since they have
> different rules and share only some technical aspects.

+1

Actually IMAP is independent.

LMTP is based on SMTP implementation.

A current limitation of LMTP is that it do not apply the mailet
processing pipeline.

POP3, SMTP and ManageSieve relies on the common protocols-api.

> 
> The 'business' rules should trump over the technical part, but the code
> is quite old on that part and it works so I guess no-one put the work to
> change it much.

+1 SMTP, LMTP and POP3 works globally well.

> 
> I believe each protocol is independent and should build to an
> independent server (SMTP, IMAP, POP3, JMAP) .

That statement is more questionable IMO.

An IMAP server without an SMTP server will not be able to receive mails.

While I agree on differentiating servers on the role they play (Mail
Exchange, Mail Transfer, Mail Delivery, etc...) I believe fully
splitting protocols into independent servers is not desirable.

Today JAMES implementations makes sens to implement a Mail Transfer
agent, and mail processing (achievable with s/jpa-smtp/Mail Processing
server/) and for Mail Delivery (all other flavors). I believe we are not
mature enough for mail exchange.

Maybe the intended role in the mail architecture could be added to the
Antora documentation?

> 
> [...]
> 
> I don't believe the code is super clean but it does work and is quite
> efficient and fast.

Nice explanation!

+1 that's a foundation we may not want to challenge too much.

> 
> NOTE: Netty should be upgraded to the latest version, I can do that once
> we are done with Gradle.

+1 <3

Likely a huge effort though...

> [...]

Regards,

Benoit

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [PROPOSAL] Community call

[Tellier Benoit]

Dear all,

Here are the notes taken during our first community call:

```
Apache James community meeting 25 June 2020

## Turn table & presentations

IEugen Stan
Start with a Google Summer of Code implements Hbase mailbox.
Want to get back into the project because he will use James as a mail
server and to write application for his company
Interested in JMAP. The future looks nice with this new protocol


Raphael
Manage James team at linagora

Rémi
Joins Linagora for one year and a half

Mattieu

Promotor of James at Linagora
Use it too for small scale deployement too.

Gautier
Works at Linagora for 2 years.
Worked principaly on using reactor on James

René

Works for 2 years at Linagora

Duc
Join Linagora for some months

Lan
Join Linagora for some months

David
Joined the Community recently.
Canadian living in Japan.
Interested in email recently.
Have his own email server since 20 years.
Mails are ubiquitous, but are overlooked
His company want to have an extensible mail server to do some processing.
He didn'd find any good tool to process incoming mails except for James
which is a good candidate for him.
Appreciate the support from the Community

Antoine

Used to work on James.
Will work to bring CI to the Community

## Documentation & server offering

David statred a documentation effort on Antora + asciidoc.

Benoit:

- We need versioning for the documentation

- We need per -server documentation

- Documentation focused on use cases

- It would be great if we can have James extensible and provide
functionality as extensions

Matthieu:

- We need a reproducible publish process that we can trust.

- Automated publish on commit process.

- We should focus on what we can do now and not on what we could do.

Eugen:

- I can work on the documentation publishing automation.

- I believe we need to focus on email libraries / components that we can
work colaboratively on.

- Eugen would work on Antora website deployment with gradle.
Configuration should be published online for the community but not
necessarily to the end user.

David:

- I was totally confused by the documentation when I started.

- Be more clear on what James is.

- Be clear on what is  targeting developers and what targets operators.

- It would be nice to have extensions.

## Developer experience

Matthieu:

- opt-in test running

- when we started contributing, things were split but not mature enough,
many cross repository commits. We still have maturity / api-stability
issues.

- We should extract components (mailbox) when they are stable from an
API point of view.

- We have too many submodules in our project / we can leverage gradle to
reduce the number of modules

Benoit:

- We don't deploy the snapshots automatically on Apache SNAPSHOTS ???

- We can simplify things if we re-evaluate what we have and drop things
that we don't use  / maintain

Eugen:
 - Long builds, eventually failing, which can discourage potential
contributors.
 - Migrating to gradle could help, prior attempts
 - Composite builds could help for integrating separate projects
 - Almost ready for the gradle migration

 - We could group submodules based on the underlying technologies
(Cassandra, JPA) because it does not make sense to deploy Cassandra and
JPA in the same deployment.

David:

- Spring cleaning and removing of unused, older modules / components


## Apache builds

Antoine:

- I'm working to add CI build status to Github PR's

- investigation what can be done with
https://gitbox.apache.org/

Gautier:

- existing CI repository is here :
https://github.com/linagora/james-jenkins

Matthieu:

- Linagora CI do not use Jenkinsfile

- Linagora uses smoke tests to test a generated Docker image


### Comunity growth

David:

- What are the goals to grow the community?

Eugen:

- I would like to grow the comunity and awarness around James

- I plan to do some youtube videos regarding Apache James

- I plan to approach Mozilla Thunderbird comunity to collaborate on JMAP
development
```

Notes were taken here
https://cryptpad.fr/pad/#/2/pad/edit/46s0WfDVm6A1cUgWZ5Dk1Bte/

Best regards,

Benoit Tellier


Le 18/06/2020 à 17:02, Tellier Benoit a écrit :
> Hello all,
> 
> Eugen suggested me to be having a community call.
> 
> We have been discussing a lot lately, this could be the occasion to know
> each others better, and find some common ground between community members.
> 
> I know that live meetings tend to disconnect people (if you cannot
> attend you are excluded), thus I think we need to set up some
> organization similar to what is done within the IETF:
> 
>  - Prepare together a clear agenda
>  - Take notes that will be output on this very mailing list
> 
> Regarding the time I propose (given active people time-zones)  Thursday
> 26th June 2020, from 8am UTC to 9am UTC.
> 
> We should of course start with a turn-table so that each person gets to
> present who they are, how did they started cont

Re: Call for vote: Apache James 3.5.0

[Tellier Benoit]

Hi all,

The voting period is over.

The vote failed as Matthieu Baechler downvoted the release, and we did
not reach the binding vote quorum.

I will work on addressing the blocking points regarding this, and soon
re-trigger a vote.

Best regards,

Benoit

Le 19/06/2020 à 10:44, Tellier Benoit a écrit :
> Hi,
> 
> I would like to propose the 3.5.0 release of the Apache James server.
> 
> Here are the changes since the previous proposal:
> 
> ```
> JAMES-3197 Matcher processing should handle NoClassDefFoundError
> f0c6576760 JAMES-3197 Mailet processing should handle NoClassDefFoundError
> 5ad068e2da JAMES-3195 Avoid loosing stacktrace when fails to delete a file
> 0894fab28e JAMES-3192 Upgrade Apache configuration to 2.7
> ba532d0d5b Remove leading line breaks
> 2207d8b3d8 JAMES-3187 Provisionned docker: add a cli helper script
> 3b4828c069 JAMES-3187 Add webadmin enabled by default to jpa-guice
> docker image
> 537af6ed32 JAMES-1541 Fix a typo in MailPriorityHandler in configuration
> 04e8ec7906 JAMES-1541 Remove no longer existing
> SuppressDuplicateRcptHandler from configuration
> 492e458cda JAMES-1541 Remove no longer existing TarpitHandler from
> configuration
> ```
> 
> You can see changes proposed to the website at the occasion
> of that release on this GitHub pull
> request: https://github.com/apache/james-project/pull/187
> 
> You can find:
> 
>  - The maven release staged in repository.apache.org as the
> artifact #1047:
> https://repository.apache.org/content/repositories/orgapachejames-1047/
>  - The changelog for
> 3.5.0:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/CHANGELOG.md
>  - The compatibility instructions/upgrade
> recommendation:
> https://github.com/chibenwa/james-project/blob/website-3.5.0/upgrade-instructions.md
> 
> Voting rules:
>  - This is a majority approval: this release may not be vetoed.
>  - A quorum of 3 binding votes is required
>  - The vote starts at Monday 19th of June 2020, 10am45 UTC
>  - The vote ends at Sunday 26th of June 2020, 10am45 UTC
> 
> You can answer to it just with +1 and -1. Down-votes may be motivated.
> 
> Cheers,
> 
> Benoit Tellier
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [PROPOSAL] Community call

[Tellier Benoit]

Hello,

I am unaware of Eugen speaking French.

Also, some of my Vietnamese coworker are curious and want to join.

I think we will have to go with it in English.

Cheers,

Benoit

Le 25/06/2020 à 11:21, David Leangen a écrit :
> 
> Hi,
> 
> I get the impression that everybody here speaks French.
> 
> Would it be ok to do the call in French instead of English?
> 
> 
> Cheers,
> =David
> 
> 
> 
>> On Jun 24, 2020, at 20:41, Tellier Benoit  wrote:
>>
>> Reminder ;-)
>>
>> Time: Thursday 25th June 2020, from 8am UTC to 9am UTC
>>
>> Location: https://meet.jit.si/apacheJames
>>
>> Agenda:
>>
>> - Turn table & presentation (10 minutes - 1 each)
>> - Documentation effort & James project server offering (15 minutes)
>> - Improve the developer experience (15 minutes)
>> - Builds (15 minutes)
>>
>> Notes will need to be taken.
>>
>> Any potential decision needs to be exposed (and potentially) debated by
>> mail.
>>
>> See you tomorrow ;-)
>>
>> Cheers,
>>
>> Benoit
>>
>> Le 18/06/2020 à 17:06, Tellier Benoit a écrit :
>>> Le 18/06/2020 à 17:02, Tellier Benoit a écrit :
>>>> Hello all,
>>>>
>>>> Eugen suggested me to be having a community call.
>>>>
>>>> We have been discussing a lot lately, this could be the occasion to know
>>>> each others better, and find some common ground between community members.
>>>>
>>>> I know that live meetings tend to disconnect people (if you cannot
>>>> attend you are excluded), thus I think we need to set up some
>>>> organization similar to what is done within the IETF:
>>>>
>>>>  - Prepare together a clear agenda
>>>>  - Take notes that will be output on this very mailing list
>>>>
>>>> Regarding the time I propose (given active people time-zones)  Thursday
>>>> 26th June 2020, from 8am UTC to 9am UTC.
>>>
>>> Errata: Thursday 25th June 2020, from 8am UTC to 9am UTC
>>>
>>> Thanks Gautier
>>>
>>>>
>>>> We should of course start with a turn-table so that each person gets to
>>>> present who they are, how did they started contributing to James, what
>>>> goals they are pursuing by contributing to James, what milestones they
>>>> might get (10-15 minutes).
>>>>
>>>> What other points would you like to cover?
>>>>
>>>> Regards,
>>>>
>>>> Benoit
>>>>
>>>>
>>>> -
>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>
>>>
>>> -
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Re: [PROPOSAL] Community call

[Tellier Benoit]

Reminder ;-)

Time: Thursday 25th June 2020, from 8am UTC to 9am UTC

Location: https://meet.jit.si/apacheJames

Agenda:

 - Turn table & presentation (10 minutes - 1 each)
 - Documentation effort & James project server offering (15 minutes)
 - Improve the developer experience (15 minutes)
 - Builds (15 minutes)

Notes will need to be taken.

Any potential decision needs to be exposed (and potentially) debated by
mail.

See you tomorrow ;-)

Cheers,

Benoit

Le 18/06/2020 à 17:06, Tellier Benoit a écrit :
> Le 18/06/2020 à 17:02, Tellier Benoit a écrit :
>> Hello all,
>>
>> Eugen suggested me to be having a community call.
>>
>> We have been discussing a lot lately, this could be the occasion to know
>> each others better, and find some common ground between community members.
>>
>> I know that live meetings tend to disconnect people (if you cannot
>> attend you are excluded), thus I think we need to set up some
>> organization similar to what is done within the IETF:
>>
>>  - Prepare together a clear agenda
>>  - Take notes that will be output on this very mailing list
>>
>> Regarding the time I propose (given active people time-zones)  Thursday
>> 26th June 2020, from 8am UTC to 9am UTC.
> 
> Errata: Thursday 25th June 2020, from 8am UTC to 9am UTC
> 
> Thanks Gautier
> 
>>
>> We should of course start with a turn-table so that each person gets to
>> present who they are, how did they started contributing to James, what
>> goals they are pursuing by contributing to James, what milestones they
>> might get (10-15 minutes).
>>
>> What other points would you like to cover?
>>
>> Regards,
>>
>> Benoit
>>
>>
>> -
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
> 
> -
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
> 

-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org